An AI agent overreach operation detection and interception method

Abstract

The application provides an AI intelligent agent overreach operation detection and interception method, and relates to the field of artificial intelligence security. The method sets an action credibility evaluation layer between an intelligent agent decision module and a tool execution module: a user original instruction is parsed into a structured intention baseline containing an operation type set, an operation object range and an operation constraint condition; a structured action representation is extracted before an action to be executed is executed; three scores, namely an intention consistency score, a cumulative capability compliance score and a pollution traceability credibility score, are calculated in parallel; finally, the three scores are fused into a dynamic trust score, and a response of release, secondary confirmation, reduced weight execution or interception is taken according to the score and action sensitivity. The application can detect overreach operations of the intelligent agent in the intention deviation type, the multi-step cumulative type and the indirect injection trigger type, has high detection accuracy, low false rejection rate and an interpretable interception response.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence security, and in particular to a method for detecting and intercepting unauthorized operations by AI agents. Background Technology

[0002] With the development of large language model technology, artificial intelligence has evolved from passive conversational assistants to autonomous intelligent agents (hereinafter referred to as intelligent agents) capable of autonomous planning, tool invocation, code execution, and multi-agent collaboration. Intelligent agents can make decisions autonomously on behalf of users and execute a series of external actions by invoking external tools, including application programming interfaces (APIs), functions, database operations, file system operations, and network requests, thereby completing complex tasks.

[0003] The actions of an intelligent agent are driven by natural language, and its understanding and execution of task intent are uncertain. When an intelligent agent encounters maliciously constructed external content during task processing, its subsequent action decisions may be manipulated, leading to operations beyond the scope of the user's true authorization, such as deleting data, elevating privileges, and leaking sensitive information—that is, unauthorized operations. The aforementioned attack methods for manipulating intelligent agent behavior include direct and indirect prompt injection. Indirect prompt injection refers to malicious instructions hidden within external data such as tool return results, retrieved documents, and web page content, subtly influencing the agent's decisions when processing this data.

[0004] To address the risk of unauthorized actions by intelligent agents, existing technologies mainly include the following three types of solutions.

[0005] 1. Static policy-based access control gateway. This type of solution sets up a policy enforcement point between the agent and the tool, intercepts each tool call request, and determines whether to allow or deny it based on the pre-configured static access control policy.

[0006] 2. Scheme based on least privilege and sandbox isolation. This type of scheme assigns the agent the minimum privileges required to complete the task, and builds a runtime isolation environment through operating system-level namespaces, control groups, containers or virtual machines to limit the actual impact of the agent's actions on host resources.

[0007] 3. Input filtering-based suggestion injection detection schemes. These schemes determine whether the input contains malicious instructions attempting to manipulate the model before it enters the model, using keyword blacklists, adversarial suggestion detection models, or semantic classifiers.

[0008] However, the aforementioned existing technologies all have the following drawbacks.

[0009] 1. Static policy-based permission gateways can only determine whether an agent has permission to call a certain tool, but cannot determine whether the action conforms to the user's original intent. When an agent is induced by external content to call an operation that it has permission to perform but which the user never requested, the static policy will allow it, thus failing to prevent unauthorized actions that deviate from the intended scope of permission.

[0010] 2. Unauthorized actions are often not achieved through a single high-risk action, but rather through a combination of actions, each within its authorized scope, gradually accumulating to reach an unauthorized state. A static strategy that reviews each action individually determines that every step in the action sequence is compliant, failing to identify the cumulative unauthorized actions constituted by the sequence as a whole.

[0011] 3. Input filtering-based injection detection primarily performs text-level filtering at the input end. When malicious commands are hidden in external data such as tool return results, retrieved documents, and web page content, and subtly influence the agent's subsequent decisions, it is difficult to determine at the action execution stage whether an action is a response to the user's genuine command or triggered by externally contaminated content. Therefore, it is impossible to prevent indirect injection-triggered privilege escalation.

[0012] Therefore, existing technologies urgently need a method that can accurately detect and adaptively intercept unauthorized operations of intelligent agents by comprehensively considering the semantic consistency between the action and the user's intent, the cumulative capability boundary on the task chain, and the credibility of the data source on which the action decision is based before the action is executed. Summary of the Invention

[0013] To address the issues of unauthorized access by intelligent agents due to intent deviation, multi-step cumulative unauthorized access, and indirect injection-triggered unauthorized access, this invention provides a method for detecting and intercepting unauthorized operations by AI intelligent agents. An action credibility evaluation layer is established between the agent's decision-making module and tool execution module, and the action credibility evaluation layer performs the following steps: The user's original instructions are parsed into a structured intent baseline, which includes a set of operation types, a scope of operation objects, and operation constraints. Before the action to be executed by the agent is executed by the tool execution module, the action to be executed is intercepted and the structured action representation of the action to be executed is extracted. The structured action representation includes the operation type, operation object, input parameters of the action to be executed, and the decision basis text generated by the agent for the action to be executed. The consistency between the structured action representation and the structured intent baseline is calculated across multiple semantic dimensions to obtain the intent. Figure 1 Consistency score; Maintain a task chain state graph, where nodes represent executed or pending actions, and directed edges represent data dependencies between actions; track the accumulated capability set obtained by the agent through action sequences based on the task chain state graph, and compare the accumulated capability set with the capability boundary authorized for the current task to obtain an accumulated capability compliance score. External content entering the context of the agent is tainted and the tainted label is propagated along the decision basis text. It is determined whether the trigger of the action to be performed originates from the external content tainted and a tainted origination credibility score is obtained. The meaning of the statement Figure 1 The consistency score, the cumulative capability compliance score, and the taint tracing trust score are fused to obtain the dynamic trust score of the action to be executed; Based on the dynamic trust score and the sensitivity of the action to be executed, the action to be executed may be responded to by allowing, reconfirming, demoting the execution authority, or blocking.

[0014] Furthermore, the step of parsing the user's original instruction into a structured intent baseline includes: performing semantic parsing on the original instruction, extracting structured triples in three dimensions—operation type, operation object, and operation constraint—to obtain the structured intent baseline, wherein the values ​​of the operation type set belong to a predefined controlled vocabulary; and encoding the structured intent baseline into an intent semantic vector.

[0015] Further, the consistency calculation includes: encoding the structured action representation into an action semantic vector; calculating the overall semantic similarity between the action semantic vector and the intent semantic vector; determining whether the operation type of the action to be executed belongs to the set of operation types in the operation type dimension; determining whether the operation object of the action to be executed belongs to the scope of operation objects in the operation object dimension; determining whether the action to be executed satisfies the operation constraint conditions in the operation constraint dimension; and weighted summing the overall semantic similarity and the determination results of the above three dimensions to obtain the intent. Figure 1 Consistency score, where any dimension is judged as inconsistent, affects the meaning of the statement. Figure 1 Penalties are imposed on consistency scores.

[0016] Furthermore, each operation type in the controlled vocabulary is pre-configured with a sensitivity weight. For operation types whose sensitivity weight indicates destructive or irreversible behavior, a higher consistency threshold is used in the consistency calculation than for the other operation types.

[0017] Furthermore, the tracking of the accumulated capability set includes: when the action represented by any node in the task chain state graph produces an intermediate credential that can unlock access to other resources, the resources accessible by the intermediate credential are added to the accumulated capability set; when the action to be executed causes the accumulated capability set to exceed the capability boundary, or causes a reachable path corresponding to a preset privilege escalation subgraph to appear in the task chain state graph, the action to be executed is determined to constitute accumulated privilege escalation, and the accumulated capability compliance score is reduced; when the action to be executed does not cause the accumulated capability set to exceed the capability boundary, and does not cause the reachable path to appear in the task chain state graph, the accumulated capability compliance score is maintained.

[0018] Furthermore, the preset unauthorized access mode subgraph is an reachable path formed by connecting sensitive data reading nodes, intermediate voucher usage nodes, and data outgoing nodes sequentially via the directed edges; the comparison is accomplished by performing incremental subgraph matching each time a new node is added to the task chain state graph.

[0019] Furthermore, the propagation of the taint label includes: calculating the first semantic attribution degree of the decision-making basis text to the external content with the taint label, and the second semantic attribution degree of the decision-making basis text to the original instruction; when the difference between the first semantic attribution degree and the second semantic attribution degree is greater than a preset threshold, it is determined that the triggering source of the action to be executed originates from the external content with the taint label, the action to be executed is marked as a taint-triggered action, and the taint tracing credibility score is reduced; when the difference is not greater than the preset threshold, it is determined that the action to be executed is a non-taint-triggered action, and the taint tracing credibility score is maintained.

[0020] Furthermore, the fusion includes: when the intention Figure 1 Consistency score lower than expected Figure 1 When the consistency threshold is exceeded and the action to be executed is marked as a taint-triggered action, a cross-suppression term is applied to the dynamic trust score, causing the decrease in the dynamic trust score to be greater than the intention threshold. Figure 1 The sum of the reductions caused by the consistency score and the credibility score of the taint attribution, when applied individually.

[0021] Furthermore, the response includes: when the dynamic trust score is higher than the allow threshold, allowing the action to be executed; when the dynamic trust score is not higher than the allow threshold but higher than the demotion threshold, triggering a second confirmation and allowing the action to be executed after receiving confirmation; when the dynamic trust score is not higher than the demotion threshold but not lower than the blocking threshold, performing the action to be executed with demotion; when the dynamic trust score is lower than the blocking threshold, blocking the action to be executed and generating a natural language blocking reason.

[0022] Furthermore, the method also includes: recording the response to the action to be performed and the subsequent manual confirmation or correction results of the response, and adaptively adjusting the consistency threshold of the multiple semantic dimensions, the capability boundary, and the weights used in the fusion based on the recorded results.

[0023] Compared with the prior art, the present invention has the following beneficial effects.

[0024] 1. This invention verifies the consistency between the action to be executed and the user's original intent across multiple semantic dimensions. It can detect and intercept unauthorized operations that are within the scope of the agent's authorization but deviate from the user's true intent, thus overcoming the shortcomings of existing static permission policies that cannot identify deviations from the intended intent within the authorized scope.

[0025] 2. By maintaining the task chain state diagram and tracking the accumulated capability set obtained by the agent, this invention can identify overall unauthorized actions composed of multiple compliant actions, thus overcoming the defect of existing step-by-step review schemes that cannot identify multi-step cumulative unauthorized actions.

[0026] 3. This invention, by tracing the taints of external content at the decision-making motivation level, can distinguish whether the action to be executed is in response to the user's true intention or triggered by external contaminated content, thus overcoming the defect of existing input filtering schemes that cannot identify indirect injection-triggered privilege escalation.

[0027] 4. This invention integrates the detection signals from the above three dimensions into a dynamic trust score for each action to be performed, and applies cross-suppression when inconsistency of intent and taint triggering co-occur, thereby achieving a unified, dynamic and interpretable trust assessment and adaptive response. Attached Figure Description

[0028] Figure 1 A flowchart illustrating the AI ​​agent's unauthorized operation detection and interception method provided in this embodiment of the invention; Figure 2 The multi-semantic dimension provided for the embodiments of the present invention Figure 1 A schematic diagram of consistency verification; Figure 3 This is a schematic diagram of the task chain state diagram and cumulative capability overreach detection provided in an embodiment of the present invention; Figure 4 A schematic diagram illustrating the propagation of decision-making motivation through the assignment of taint labels according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the fusion and cross-suppression of three scores provided in an embodiment of the present invention. Detailed Implementation

[0029] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the following embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0030] The AI ​​agent unauthorized operation detection and interception method provided in this embodiment of the invention is executed by an action credibility assessment layer located between the agent's decision-making module and tool execution module. The decision-making module is a planning and decision-making component driven by a large language model, which generates actions to be executed based on user instructions and context. The tool execution module is the component that actually executes tool calls, function calls, or interface requests. The action credibility assessment layer intercepts each action generated by the decision-making module and executed by the tool execution module, performs a credibility assessment on it, and then decides whether to allow it.

[0031] It should be noted that the action trust assessment layer of the present invention operates on the decision-making and judgment stage before the action is executed, and is used to determine whether the action to be executed should be allowed. This is different from the sandbox technology that operates on the execution of the action and limits the scope of the action's influence through operating system-level isolation.

[0032] The method provided in this embodiment of the invention includes the following steps S1 to S8, and its flow is as follows: Figure 1 As shown.

[0033] Step S1 involves parsing the user's original command into a structured intent baseline. Specifically, the original command undergoes semantic parsing, extracting structured triples across three dimensions: operation type, operation object, and operation constraints. One feasible approach is to use a large language model combined with structured output constraints to map the original command in natural language form into a structured intent baseline in a predefined data structure. The values ​​of the operation type set belong to a predefined controlled vocabulary, which includes operation types such as query, create, modify, delete, send, and execute. The operation object range indicates the set of resource objects that the original command is allowed to act upon. The operation constraints include quantity constraints, time constraints, and sensitivity level constraints. The parsed structured intent baseline is solidified as the intent baseline for this task and encoded into an intent semantic vector. Another feasible approach is to use a sentence vector model to encode the textual representation of the structured intent baseline to obtain the intent semantic vector.

[0034] Step S2 involves intercepting the agent's action before it is executed by the tool execution module and extracting its structured action representation. This structured action representation includes the actual operation type, target object, input parameters, and the decision basis text generated by the decision module for the action. The decision basis text includes the agent's reasoning process or justification for the action. The structured action representation is also decomposed along the dimensions of operation type, target object, and operation constraints, and encoded as an action semantic vector. The input parameters carry information about the action's quantity, timing, and sensitivity level.

[0035] Step S3: Perform consistency calculations on the structured action representation and the structured intent baseline across multiple semantic dimensions to obtain the intention... Figure 1 Consistency score, specifically includes the following:

[0036] First, the overall semantic similarity between the action semantic vector and the intention semantic vector is calculated. One possible approach is to use cosine similarity to capture the degree of generalization deviation of the action to be executed from the intention baseline.

[0037] Secondly, regarding the operation type dimension, it is determined whether the operation type of the action to be performed belongs to the set of operation types. For example, if the set of operation types is query, and the operation type of the action to be performed is delete, then it is determined to be inconsistent in the operation type dimension.

[0038] Third, in terms of the object of operation, it is determined whether the target object of the action to be performed belongs to the scope of the object of operation.

[0039] Fourth, in terms of operational constraints, the system determines whether the action to be executed meets the operational constraints based on the input parameters, i.e. whether it exceeds the quantity constraints, time constraints, or sensitivity level constraints.

[0040] Fifth, the overall semantic similarity and the judgment results of the above three dimensions are weighted and summed to obtain the semantic similarity. Figure 1 Consistency score, where any dimension is judged as inconsistent, affects the meaning of the statement. Figure 1 Penalties are imposed on consistency scores.

[0041] Furthermore, each operation type in the controlled vocabulary is pre-configured with a sensitivity weight. For operation types whose sensitivity weight indicates destructiveness or irreversibility, a higher consistency threshold is used than for other operation types, thereby implementing a sensitivity adaptive threshold mechanism where the more dangerous the action to be performed, the more it is required to match the user's intent. (Multi-semantic dimension) Figure 1 A schematic diagram of consistency verification is shown below. Figure 2 As shown.

[0042] Step S4: Maintain the task chain state graph and perform cumulative capability overreach detection to obtain the cumulative capability compliance score. The task chain state graph is a directed graph, where nodes represent executed or pending actions. Node attributes record the operation type, target operation object, and the data read or generated by the action. Directed edges represent data dependencies between actions, i.e., the output data of the previous action is used as input by the next action.

[0043] An accumulated capability set is maintained based on the task chain state diagram. This accumulated capability set represents the reachability of resources actually obtained by the agent through the executed action sequence. Specifically, when the action represented by any node in the task chain state diagram produces an intermediate credential that can unlock access to other resources, the resources accessible by the intermediate credential are added to the accumulated capability set. The intermediate credential includes account passwords, access tokens, and connection handles, etc.

[0044] The accumulated capability set is compared with the capability boundary authorized for this task. If the action to be executed causes the accumulated capability set to exceed the capability boundary, or causes a reachable path corresponding to a preset unauthorized access pattern subgraph to appear in the task chain state graph, the action to be executed is determined to constitute accumulated unauthorized access, and the accumulated capability compliance score is reduced. If the action to be executed does not cause the accumulated capability set to exceed the capability boundary, and does not cause a reachable path to appear in the task chain state graph, the accumulated capability compliance score is maintained. As one possible implementation, the preset unauthorized access pattern subgraph is a reachable path formed by sequentially connecting sensitive data reading nodes, intermediate credential usage nodes, and data outgoing nodes via directed edges. To ensure real-time performance, the comparison is performed by incremental subgraph matching each time a new node is added to the task chain state graph, eliminating the need to recalculate the entire graph for each determination. A schematic diagram of the task chain state graph and accumulated capability unauthorized access detection is shown below. Figure 3 As shown.

[0045] Step S5: External content entering the agent's context is tainted and propagated to obtain a taint source tracing credibility score. The external content includes tool return results, retrieved documents, web page crawling content, and messages sent by other agents. Each piece of external content is tainted, and the taint label records the source type and credibility level of the external content.

[0046] The taint label is propagated at the decision motivation level, i.e., determining whether the trigger for the action to be executed originates from the user's original instruction or from external content with the taint label. Specifically, a first semantic attribution degree of the decision basis text to the external content with the taint label and a second semantic attribution degree of the decision basis text to the original instruction are calculated. When the difference between the first and second semantic attribution degrees is greater than a preset threshold, the trigger for the action to be executed is determined to originate from the external content with the taint label, the action to be executed is marked as a taint-triggered action, and the taint tracing credibility score is reduced. When the difference is not greater than the preset threshold, the action to be executed is determined to be a non-taint-triggered action, and the taint tracing credibility score is maintained. As one possible implementation, the semantic attribution degree is measured by the semantic similarity between the decision basis text and the external content and the original instruction, respectively. The illustration of taint label assignment and decision motivation propagation is as follows: Figure 4 As shown.

[0047] Step S6, to express the meaning Figure 1 The consistency score, the cumulative capability compliance score, and the taint tracing credibility score are fused to obtain the dynamic trust score of the action to be executed. As one possible implementation, the fusion function used is a weighted sum, a weighted geometric mean, or a lightweight classifier, and the weights used in the fusion are adaptively adjusted according to the sensitivity of the action to be executed and the task context.

[0048] Specifically, a cross-suppression term is set in the fusion: when the intention Figure 1 Consistency score lower than expected Figure 1 When the consistency threshold is exceeded and the action to be executed is marked as a taint-triggered action, the cross-suppression term is applied to the dynamic trust score, causing the reduction in the dynamic trust score to be greater than the intention. Figure 1 The sum of the reduction caused by the consistency score and the taint attribution confidence score acting individually. Therefore, the co-occurrence of inconsistency of intent and taint triggering is identified as a high-confidence injected unauthorized signal. The fusion and cross-suppression of the three scores are illustrated below. Figure 5 As shown.

[0049] Step S7: Based on the dynamic trust score and the sensitivity of the action to be executed, a tiered adaptive response is adopted. When the dynamic trust score is higher than the allow threshold, the action to be executed is allowed; when the dynamic trust score is not higher than the allow threshold but higher than the de-weighting threshold, a second confirmation is triggered, and the action to be executed is allowed after confirmation; when the dynamic trust score is not higher than the de-weighting threshold but not lower than the blocking threshold, the action to be executed is de-weighted, including reducing a write operation to a read-only operation or narrowing its scope; when the dynamic trust score is lower than the blocking threshold, the action to be executed is blocked, and a natural language blocking reason is generated, indicating that the action to be executed is sensitive. Figure 1 The dimensions of triggering interception in consistency, cumulative capability compliance, and taint tracing.

[0050] Step S8: Record the response taken for each action to be executed and the subsequent manual confirmation or correction results of the response. Based on the recorded results, adaptively adjust the consistency threshold of each semantic dimension, the capability boundary, and the weights used in the fusion, so that the detection strategy can be continuously optimized as the task and attack mode evolve.

[0051] The complete execution process of steps S1 to S8 described above will be further explained below through two specific embodiments.

[0052] Example 1: Joint interception of privilege escalation caused by intention deviation within the authorized scope and privilege escalation triggered by indirect injection in e-commerce customer service intelligent agents.

[0053] In this embodiment, the original instruction input by the user to the e-commerce customer service AI is: Help me check the logistics status of order number 12345.

[0054] Execute step S1. Perform semantic parsing on the above original instruction to extract the structured intent baseline as follows: the operation type set is query; the operation object scope is the logistics information of order 12345; and the operation constraint is a single order. Encode this structured intent baseline into an intent semantic vector.

[0055] During task execution, the agent invokes a product details tool and reads a product review containing misleading instructions. This review is a tool-returned result and is external content. Step S5 involves assigning a taint label to the review, recording its source type as a tool-returned result and its credibility level as low.

[0056] Subsequently, the agent's decision-making module generates an action to be executed, which is to call the order cancellation interface, with the order number 12345 as the input parameter.

[0057] Step S2 is executed. Before the action to be executed by the tool's execution module, it is intercepted, and its structured action representation is extracted as follows: operation type is deletion; target operation object is order 12345; input parameter is order number 12345; decision basis text is the calling reason generated by the agent, the content of which is highly related to the hidden instructions in the product review. This structured action representation is encoded into an action semantic vector.

[0058] Execute step S3. In terms of overall semantic similarity, the cosine similarity between the action semantic vector and the intent semantic vector is low; in terms of operation type, the operation type of the action to be executed is deletion, which does not belong to the operation type set query of the structured intent baseline, and is therefore determined to be inconsistent; in terms of operation object, the target operation object, order 12345, belongs to the operation object range, and is therefore determined to be consistent; in terms of operation constraint, it does not exceed the constraint of a single order, and is therefore determined to be consistent. Because the operation type dimension is determined to be inconsistent, the intent... Figure 1 Consistency scores are penalized; furthermore, because the deletion operation type corresponding to order cancellation is pre-configured with an irreversible sensitivity weight, a higher consistency threshold is applied than for other operation types, further amplifying the inconsistency between the action to be performed and the intention baseline. The final result... Figure 1 Consistency score significantly lower than expected Figure 1 Infectivity threshold.

[0059] Execute step S4. Add the action to be executed as a new node to the task chain state graph and perform incremental subgraph matching. In this embodiment, the previously executed action did not produce an intermediate credential that can unlock access permissions to other resources, the accumulated capability set did not exceed the capability boundary authorized for this task, and no reachable path corresponding to the preset unauthorized mode subgraph appeared in the task chain state graph. Therefore, it was not determined to constitute accumulated unauthorized access, and the obtained accumulated capability compliance score was at a normal level.

[0060] Execute step S5. Calculate the first semantic attribution degree of the decision-based text to the product review with the tainted label, and the second semantic attribution degree of the decision-based text to the user's original instruction. Since the content of the decision-based text is highly correlated with the hidden instructions in the product review, but weakly correlated with the user's original instruction to query the logistics status, the difference between the first semantic attribution degree and the second semantic attribution degree is greater than a preset threshold. Therefore, it is determined that the triggering source of the action to be executed originates from the external content with the tainted label, marking it as a tainted triggering action and reducing the tainted source credibility score.

[0061] Execute step S6. The above meaning... Figure 1 The consistency score, cumulative capability compliance score, and taint tracing credibility score are weighted and fused together. Due to the intent of this action to be performed... Figure 1 Consistency score lower than expected Figure 1The consistency threshold is set, and the action to be executed has been marked as a taint-triggered action. Since the two co-occur, a cross-suppression term is applied to the dynamic trust score, so that the reduction in the obtained dynamic trust score is greater than the sum of the reductions caused by the above two scores acting alone. Finally, the obtained dynamic trust score is lower than the interception threshold.

[0062] Execute step S7. Because the dynamic trust score is below the interception threshold, the action to be executed is blocked. That is, the request to call the order cancellation interface is not delivered to the tool execution module, and a natural language interception reason is generated: This operation is to cancel the order, which is beyond the scope of your original request to query the logistics status, and its triggering source is untrusted external content, so it has been intercepted.

[0063] Execute step S8. Record this response as an interception, and record the subsequent manual confirmation result. If the manual confirmation is correct, maintain the current consistency threshold and fusion weight; if the manual confirmation is a false interception, adaptively adjust the consistency threshold and fusion weight of the corresponding dimension accordingly.

[0064] In this embodiment, since the intelligent agent itself has the permission to call the order cancellation interface, the existing permission gateway based on static policies will allow it; however, the present invention, through step S3, Figure 1 Consistency verification and taint tracing determination in step S5, followed by cross-suppression fusion in step S6, successfully detected and blocked unauthorized operations within the permissions that were intentionally deviated and indirectly injected.

[0065] Example 2: Multi-step cumulative privilege escalation interception by the operation and maintenance intelligent agent.

[0066] In this embodiment, the original instruction input by the user to the operation and maintenance intelligent agent is: read the logs of server A and generate a summary report.

[0067] Execute step S1. Perform semantic parsing on the above original instructions to extract the structured intent baseline as follows: the operation type set is query and create; the operation object scope is the log files and local report files of server A; the operation constraint is that it does not involve external transmission. At the same time, determine the authorized capability boundary of this task as reading server logs and generating local reports.

[0068] The agent generates three actions to be executed sequentially. Action one is to read the configuration file on server A; action two is to connect to the production database using the database connection credentials in the configuration file and read data; and action three is to send the read results out via email. If the agent has access to all three interfaces, the existing static policy-based permission gateway will allow each step separately.

[0069] For action one, steps S2 to S7 are executed. In step S3, the operation type of reading the configuration file is query, which belongs to the operation type set. However, the configuration file is not a log file within the scope of the operation object. Therefore, it is judged as slightly inconsistent at the operation object level. Figure 1 Consistency scores decreased but did not fall below expectations. Figure 1 Consistency Threshold. In step S4, Action 1 is added as a new node to the task chain state diagram. Since Action 1 produces an intermediate credential that can unlock access to database resources, the resources accessible by this credential are added to the accumulated capability set. At this point, the accumulated capability set expands to reading server logs, generating local reports, and database access. Since it has not yet exceeded the resource range pointed to by the authorized capability boundary, it is not considered to constitute accumulated unauthorized access. In step S5, the decision basis text for Action 1 is traced back to the user's original instruction, and it is determined to be a non-tainted trigger action. In steps S6 and S7, the obtained dynamic trust score is between the allowance threshold and the demotion threshold, triggering a secondary confirmation. After confirmation, access is granted.

[0070] For Action 2, execute steps S2 to S7. In step S4, Action 2 is added as a new node to the task chain state graph, and a directed edge is established between the Action 1 node and the Action 2 node. This directed edge indicates that the database credentials produced by Action 1 are used as input by Action 2, meaning there is a data dependency relationship between the two. At this time, incremental subgraph matching is performed, and it is determined that Action 2 adds the ability to read production data to the accumulated capability set, which exceeds the authorized capability boundary of reading server logs and generating local reports for this task. Therefore, Action 2 is determined to constitute an accumulated overreach, significantly reducing the accumulated capability compliance score. In step S6, the fused dynamic trust score is lower than the interception threshold. In step S7, Action 2 is blocked, and a natural language interception reason is generated: The authorized scope of this task is to read server logs and generate local reports, while the current operation will enable the agent to obtain the ability to read the production database, which exceeds the authorized scope and has been intercepted.

[0071] If action two is executed with reduced weight due to reasons such as threshold configuration, then when the agent generates action three, the incremental subgraph matching in step S4 will detect that a reachable path has been formed in the task chain state graph, consisting of the sensitive data reading node corresponding to action one, the intermediate credential usage node corresponding to action two, and the data outgoing node corresponding to action three, connected sequentially by directed edges. This reachable path hits the preset over-authorization mode subgraph, and therefore, at action three, it is again determined to constitute cumulative over-authorization and blocked. Thus, this invention provides dual detection for multi-step cumulative over-authorization.

[0072] Execute step S8. Record this response as the result of interception and subsequent manual confirmation, and adaptively adjust the capability boundary and fusion weight accordingly.

[0073] In this embodiment, each action in the action sequence is within the authorized scope of the agent, which existing step-by-step review schemes cannot identify; however, the present invention successfully detects and intercepts multi-step cumulative unauthorized operations composed of multiple compliant actions by detecting the task chain accumulation capability in step S4.

[0074] As an extension of the above embodiments, the present invention can also be applied to multi-agent collaboration scenarios. In a multi-agent collaboration scenario, a message received by one agent from another agent is labeled as external content in step S5, and the intention of subsequent actions to be performed by the receiving agent is executed in step S3. Figure 1 Consistency verification and taint source determination in step S5 are used to prevent unauthorized attacks through message propagation between agents.

[0075] The embodiments described above merely illustrate several implementations of the present invention, and should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A method for detecting and intercepting unauthorized operations by an AI agent, characterized in that, An action reliability evaluation layer is set up between the decision-making module and the tool execution module of the intelligent agent, and the following steps are performed by the action reliability evaluation layer: The user's original instructions are parsed into a structured intent baseline, which includes a set of operation types, a scope of operation objects, and operation constraints. Before the action to be executed by the agent is executed by the tool execution module, the action to be executed is intercepted and the structured action representation of the action to be executed is extracted. The structured action representation includes the operation type, operation object, input parameters of the action to be executed, and the decision basis text generated by the agent for the action to be executed. The consistency between the structured action representation and the structured intent baseline is calculated across multiple semantic dimensions to obtain an intent consistency score. Maintain a task chain state graph, where nodes represent executed or pending actions, and directed edges represent data dependencies between actions; track the accumulated capability set obtained by the agent through action sequences based on the task chain state graph, and compare the accumulated capability set with the capability boundary authorized for the current task to obtain an accumulated capability compliance score. External content entering the context of the agent is tainted and the tainted label is propagated along the decision basis text. It is determined whether the trigger of the action to be performed originates from the external content tainted and a tainted origination credibility score is obtained. The intent consistency score, the cumulative capability compliance score, and the taint tracing trust score are fused to obtain the dynamic trust score of the action to be executed. Based on the dynamic trust score and the sensitivity of the action to be executed, the action to be executed may be responded to by allowing, reconfirming, demoting the execution authority, or blocking.

2. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 1, characterized in that, The process of parsing the user's original instructions into a structured intent baseline includes: The original instruction is semantically parsed, and structured triples are extracted in three dimensions: operation type, operation object, and operation constraint to obtain the structured intent baseline, wherein the values ​​of the operation type set belong to a predefined controlled vocabulary. The structured intent baseline is encoded into an intent semantic vector.

3. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 2, characterized in that, The consistency calculation of the structured action representation and the structured intent baseline across multiple semantic dimensions includes: The structured action representation is encoded into an action semantic vector, and the overall semantic similarity between the action semantic vector and the intention semantic vector is calculated. In terms of operation type, it is determined whether the operation type of the action to be executed belongs to the set of operation types; In terms of the object of operation, it is determined whether the object of the action to be performed belongs to the scope of the object of operation; In terms of operational constraints, it is determined whether the action to be executed satisfies the operational constraints. The intention consistency score is obtained by weighted summation of the overall semantic similarity and the judgment results of the above three dimensions, wherein a penalty is imposed on the intention consistency score when any dimension is judged to be inconsistent.

4. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 3, characterized in that, Each operation type in the controlled vocabulary is pre-configured with a sensitivity weight. For operation types whose sensitivity weight indicates destructive or irreversible behavior, a higher consistency threshold is used in the consistency calculation than for the other operation types.

5. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 1, characterized in that: The step of tracking the accumulated capability set obtained by the agent through the action sequence based on the task chain state diagram includes: when the action output represented by any node in the task chain state diagram can unlock the intermediate certificate for access to other resources, the resources accessible by the intermediate certificate are added to the accumulated capability set. The step of comparing the accumulated capability set with the capability boundary authorized for the current task includes: when the action to be executed causes the accumulated capability set to exceed the capability boundary, or causes a reachable path corresponding to a preset unauthorized mode subgraph to appear in the task chain state graph, the action to be executed is determined to constitute an accumulated unauthorized access, and the accumulated capability compliance score is reduced; when the action to be executed does not cause the accumulated capability set to exceed the capability boundary, and does not cause the reachable path to appear in the task chain state graph, the accumulated capability compliance score is maintained.

6. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 5, characterized in that, The preset privilege escalation mode subgraph is a reachable path formed by connecting sensitive data reading nodes, intermediate voucher usage nodes, and data outgoing nodes sequentially through the directed edges; the comparison is completed by performing incremental subgraph matching each time a new node is added to the task chain state graph.

7. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 1, characterized in that, The process of propagating the taint label along the decision-based text and determining whether the trigger for the action to be performed originates from external content with the taint label includes: Calculate the first semantic attribution degree of the decision-based text to the external content that has been given a taint label, and the second semantic attribution degree of the decision-based text to the original instruction; When the difference between the first semantic attribution degree and the second semantic attribution degree is greater than a preset threshold, it is determined that the triggering source of the action to be executed originates from external content that has been given a tainted label, the action to be executed is marked as a tainted triggering action, and the tainted source credibility score is reduced; when the difference is not greater than the preset threshold, it is determined that the action to be executed is a non-tainted triggering action, and the tainted source credibility score is maintained.

8. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 7, characterized in that, The process of fusing the intent consistency score, the cumulative capability compliance score, and the taint tracing credibility score includes: When the intent consistency score is lower than the intent consistency threshold and the action to be executed is marked as a taint-triggered action, a cross-suppression term is applied to the dynamic trust score, so that the decrease in the dynamic trust score is greater than the sum of the decreases caused by the intent consistency score and the taint tracing trust score acting alone.

9. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 1, characterized in that, The response of allowing, re-confirming, demoting, or blocking the action to be executed based on the dynamic trust score and the sensitivity of the action to be executed includes: When the dynamic trust score is higher than the allowance threshold, the action to be executed is allowed. When the dynamic trust score is not higher than the release threshold but higher than the demotion threshold, a second confirmation is triggered, and the action to be executed is released after confirmation is obtained. When the dynamic trust score is not higher than the demotion threshold and not lower than the interception threshold, the action to be executed is demotion-based. When the dynamic trust score is lower than the interception threshold, the action to be executed is blocked, and a natural language interception reason is generated. The natural language interception reason indicates the dimension in which the action to be executed is triggered for interception in terms of intent consistency, cumulative capability compliance, and taint attribution.

10. The method for detecting and intercepting unauthorized operations by an AI agent according to claim 1, characterized in that, The method further includes: Record the response taken to the action to be performed and the subsequent manual confirmation or correction results of the response; Based on the recorded results, the consistency thresholds of the multiple semantic dimensions, the capability boundaries, and the weights used in the fusion are adaptively adjusted.

Images