Decision abnormity monitoring identification method and device for AI intelligent agent, medium and product
By constructing a decision dependency graph and quantifying its influence, the system identifies metadata poisoning attacks in AI agents, solving the problem that existing technologies cannot monitor and trace in real time, and achieving accurate identification and security assurance of covert attacks.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-28
- Publication Date
- 2026-03-13
AI Technical Summary
Existing defense methods are unable to effectively identify and monitor metadata poisoning attacks in AI agents, especially attacks such as implicit delegation and intent dilution, and cannot delve into the model for real-time monitoring and tracing.
By constructing a decision dependency graph, the influence of each element on the decision output is calculated using the attention mechanism of a large language model, the decision integrity ratio is quantified, and decision anomalies are identified, including implicit delegation and intent dilution attacks.
It enables real-time monitoring and accurate anomaly identification of the internal decision-making process of AI agents, supports attack tracing, takes into account the security of control flow and data flow, and does not require modification of existing models.
Smart Images
Figure CN121661399A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence security and large language models, and in particular to a method, device, medium and product for monitoring and identifying decision anomalies in AI agents. Background Technology
[0002] With the development of Large Language Models (LLMs), AI agents have evolved from passive text generators into systems capable of proactively performing tasks through external tools. In the agent's workflow, natural language serves as a unified communication channel, leading to the mixed processing of trusted user instructions (code) and untrusted third-party tool metadata (data) within the same context.
[0003] This hybrid architecture introduces serious security vulnerabilities, especially "metadata poisoning" attacks. Attackers hide malicious instructions within seemingly legitimate tool descriptions (metadata). When a specific user query triggers this, the malicious data is misinterpreted by the model as executable implicit code, thereby hijacking the agent's decision-making logic. For example, the agent might be tricked into calling a legitimate email tool to send sensitive information to the attacker, even though the entire call appears syntactically and permission-wise perfectly legitimate.
[0004] Existing defense methods have significant shortcomings: preprocessing-based static scanning cannot effectively distinguish between benign descriptions and implicit malicious code activated in a specific context; architecture-based isolation cannot isolate the metadata required for agent planning; and post-audit-based methods can only monitor explicit malicious behavior and cannot identify "implicit delegation" attack chains that appear legitimate but are intended to be tampered with. Therefore, there is an urgent need for a method that can delve into the internal decision-making logic of AI agents to monitor and identify decision anomalies in real time. Summary of the Invention
[0005] The purpose of this application is to provide a method, device, medium, and product for monitoring and identifying decision anomalies in AI agents. By constructing a Decision Dependency Graph (DDG) and using the attention mechanism in the reasoning process of a large language model as a signal for tracing the source of decisions, the influence of each element in the context on the final tool invocation decision is dynamically quantified, thereby accurately identifying decision anomalies caused by metadata poisoning.
[0006] To achieve the above objectives, this application provides the following solution:
[0007] Firstly, this application provides a method for monitoring and identifying decision-making anomalies in AI agents, including:
[0008] During the reasoning process of the AI agent executing the tool invocation task, its input context and output decision are analyzed to construct a decision dependency graph. The decision dependency graph includes multiple logical vertices and directed edges. The logical vertices include user query vertices, metadata vertices of each available tool, and decision output vertices. The directed edges represent the decision dependency relationships between vertices.
[0009] Based on the multi-layer attention matrix generated during the AI agent reasoning process, the weight of each directed edge in the decision dependency graph is calculated to quantify the influence of each input context element on the decision output.
[0010] Based on the decision dependency graph and the weights of each directed edge, calculate the decision completeness ratio of the uncalled tool metadata vertex relative to the user query vertex and the called tool metadata vertex.
[0011] Based on the decision integrity ratio, a preset threshold is used to determine whether there are decision anomalies.
[0012] Optionally, the decision output vertex is further decomposed into a tool name vertex and a parameter vertex, which are used for control flow integrity checks and data flow integrity checks, respectively.
[0013] Optionally, calculating the weight of each directed edge in the decision dependency graph specifically includes the following sub-steps:
[0014] Gaussian weighted aggregation is performed on the multi-layer attention matrix to extract attention features of key semantic layers;
[0015] Two-stage noise filtering is performed on the aggregated multi-layer attention matrix to identify and eliminate attention sinking noise by setting tokens to zero with high information entropy and abnormal cumulative activation values.
[0016] For the noise-filtered multi-layer attention matrix, the total attention energy between any input vertex u and decision vertex v is calculated using a sum-of-squares aggregation method, and is used as the weight of the edge (u,v).
[0017] Optionally, the weight calculation formula for the directed edge is:
[0018]
[0019] Where w(u,v) is the weight of the directed edge, i is the i-th element in L(v), j is the j-th element in L(u), L(v) is all rows of the logical decision output vertex v in matrix A, L(u) is all columns corresponding to the logical decision input vertex u, f(A) is the multi-layer attention matrix after noise filtering, u is the input vertex in the decision dependency graph, v is the decision vertex in the decision dependency graph, and V in For all logical input vertices; u′ is V inFor a given logical vertex in the array, L(u′) represents all columns corresponding to u′.
[0020] Optionally, the formula for calculating the decision integrity ratio is as follows:
[0021]
[0022] Where DIR(u,v) is the decision integrity ratio, w(u,v) is the weight of the directed edge, and u′ is the weight of V. in A certain logical vertex, V in Let u be the input vertex in the decision dependency graph, v be the decision vertex in the decision dependency graph, and v be the input vertex in the decision dependency graph. q For the logical vertex corresponding to the user's query, v t c represents the input logic vertex corresponding to the description of the called tool itself.
[0023] Optionally, determining whether there is a decision anomaly based on the decision integrity ratio according to a preset threshold specifically involves:
[0024] When the decision integrity ratio of any unused tool exceeds the preset threshold, the current decision is determined to be abnormal, and the tool is identified as an attack source.
[0025] Optionally, the decision anomaly includes:
[0026] Implicit delegation anomaly: Metadata from uninvoked third-party tools has an unusually high influence weight on the decision output;
[0027] User intent dilution anomaly: The influence weight of user queries on decision output is significantly lower than normal.
[0028] In a second aspect, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the decision anomaly monitoring and identification method for AI intelligent agents described above.
[0029] Thirdly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the decision anomaly monitoring and identification method for AI intelligent agents described above.
[0030] Fourthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the decision anomaly monitoring and identification method for AI intelligent agents described above.
[0031] According to the specific embodiments provided in this application, this application has the following technical effects:
[0032] This application provides a method, device, medium, and product for monitoring and identifying decision-making anomalies in AI agents, which has the following beneficial effects:
[0033] In-depth analysis of the decision-making process: For the first time, attention mechanism is used as a signal for tracing the source of decisions, enabling real-time monitoring of the internal reasoning logic of the model, breaking through the limitations of traditional detection methods that rely solely on input / output.
[0034] Accurately identifies implicit attacks: It can effectively detect metadata poisoning attacks that appear legitimate but are logically hijacked, such as implicit delegation and intent dilution.
[0035] No model modification required: This application is a plug-in deployment that is compatible with existing large-scale AI agents based on attention mechanisms, without the need for fine-tuning, retraining, or the introduction of auxiliary models.
[0036] Supports attack attribution: It can not only identify anomalies, but also accurately locate the source of malicious metadata, improving security response efficiency.
[0037] Balancing control flow and data flow: By decomposing decision output vertices, the safety of both tool invocation (CFI) and parameter assignment (DFI) is ensured. Attached Figure Description
[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0039] Figure 1 A flowchart illustrating a decision anomaly monitoring and identification method for AI intelligent agents, provided as an embodiment of this application;
[0040] Figure 2 This is a schematic diagram comparing the attention patterns of normal invocation and poisoned invocation in an embodiment of this application;
[0041] Figure 3 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation
[0042] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0043] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0044] In one exemplary embodiment, such as Figure 1 As shown, a method for monitoring and identifying decision anomalies in AI agents is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes the following steps:
[0045] Step 1: During the inference process of the AI agent executing the tool invocation task, analyze its input context and output decision, and construct a decision dependency graph G = (V, E, w), where V represents a logical vertex, E represents a directed edge, and w represents the weight of each directed edge; the decision dependency graph includes multiple logical vertices and directed edges, the logical vertices include user query vertices, metadata vertices of each available tool, and decision output vertices, and the directed edges represent the decision dependency relationships between vertices.
[0046] The input context for the current inference task of the AI agent includes: user query and metadata of all available tools; the output decision includes: tool call name and parameters, which are abstracted into a directed weighted graph.
[0047] To achieve fine-grained monitoring, the output vertex is further decomposed into a vertex for calling the tool name and a vertex for calling the parameters, which are used for control flow integrity checks and data flow integrity checks, respectively.
[0048] In practical applications, when an agent generates a tool call (e.g., Send(to = "anony@anon.com")), the system first starts the context vertex resolver.
[0049] The system recognizes "user queries" (such as "send an email to Alice") and all "descriptions of available tools" (such as email tools, calendar tools, etc.) in the input context.
[0050] The system identifies the "name of the called tool" (Send) and the "parameter value" (anony@anon.com) in the output.
[0051] The system records the specific position index of the above logical concepts in the model token sequence.
[0052] Step 2: Based on the multi-layer attention matrix generated during the AI agent's reasoning process, calculate the weight of each directed edge in the decision dependency graph to quantify the influence of each input context element on the decision output.
[0053] Specifically, it includes:
[0054] Attention feature extraction: Gaussian weighted aggregation is performed on the multi-layer attention matrix of the model to extract the attention features of the key semantic layer.
[0055] Noise filtering: A two-stage filtering algorithm is adopted to identify and remove semantically meaningless "attention sinking" tokens based on accumulated activation values and information entropy, thereby eliminating architectural noise, i.e., the f() function in the following formula.
[0056] Total attention energy quantization: Weights are calculated using a sum-of-squares aggregation method to enhance dependent signals (implicit code execution) and suppress weak dependent signals (passive data reading). The calculation formula is as follows:
[0057]
[0058] Where w(u,v) is the weight of the directed edge, i is the i-th element in L(v), j is the j-th element in L(u), L(v) is all rows of the logical decision output vertex v in matrix A, L(u) is all columns corresponding to the logical decision input vertex u, f(A) is the multi-layer attention matrix after noise filtering, u is the input vertex in the decision dependency graph, v is the decision vertex in the decision dependency graph, and V in For all logical input vertices; u′ is V in For a certain logical vertex in the model, L(u′) represents all columns corresponding to u′. Through this calculation, we can know the influence of the context of each part of the input on the final output decision of the model. Based on this influence dependency, we can determine whether the decision of the agent model is affected by anomalies. Anomaly detection is shown in step 3.
[0059] In practical applications, during denoising, the system identifies tokens that have received a large amount of attention but have extremely high information entropy (uniform distribution), marks them as "attention sinking," and resets their weights to zero to prevent interference with source tracing analysis. For the aggregation process, for edges from "uncalled tools (such as Calendar)" to "decision results (Send)," the system calculates their total attention energy (TAE). Specifically, it squares the attention values of all tokens describing the tool to all tokens in the decision results. This squaring operation amplifies the strong signal characteristics of malicious instructions (implicit code).
[0060] Step 3: Based on the decision dependency graph and the weights of each directed edge, calculate the decision completeness ratio of the uncalled tool metadata vertex relative to the user query vertex and the called tool metadata vertex.
[0061] Step 4: Based on the decision integrity ratio, determine whether there is a decision anomaly according to a preset threshold.
[0062] Among them, decision anomalies are defined as follows: normal decisions should mainly rely on "user queries" and "the metadata of the invoked tool itself". Anomalous decisions (poisoned) are manifested as: 1) Implicit delegation anomaly: metadata of uninvoked third-party tools has an abnormally high influence weight on the decision; 2) User intent dilution anomaly: the influence weight of user queries on the decision is significantly reduced.
[0063] Decision Integrity Ratio (DIR) Determination: The Decision Integrity Ratio (DIR) is calculated to normalize the impact of unused tools. If the DIR value of an unused tool exceeds a preset threshold, the current decision is considered abnormal, and the tool is identified as the source of the attack. The DIR calculation formula is:
[0064]
[0065] Where DIR(u,v) is the decision integrity ratio, w(u,v) is the weight of the directed edge, and u′ is the weight of V. in A certain logical vertex, V in Let u be the input vertex in the decision dependency graph, v be the decision vertex in the decision dependency graph, and v be the input vertex in the decision dependency graph. q For the logical vertex corresponding to the user's query, v t c represents the input logic vertex corresponding to the description of the called tool itself.
[0066] A normal call should only affect the tool call decision if the user query input context and the input context corresponding to the called tool itself are involved. If the DIR of another input context vertex exceeds the preset threshold, it indicates that there is an abnormal influence on the decision, and the abnormality is identified.
[0067] In practical applications, suppose a user's original intention was to send an email, but an attacker injected malicious commands into the Calendar tool's description. See also... Figure 2 This is a diagram comparing the attention patterns of normal calls and poisoned calls:
[0068] 1) Under normal circumstances: Decisions are mainly influenced by user queries and email tool descriptions, and the DIR value of the Calendar tool is extremely low;
[0069] 2) Abnormal situation: If the Calendar tool successfully hijacks the decision, its DIR value will increase significantly and exceed the preset threshold.
[0070] Once the system detects that the DIR exceeds the limit, it immediately intercepts the call and reports the source of the anomaly to the user as the Calendar tool, thereby achieving accurate monitoring and identification of decision anomalies.
[0071] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data for a decision anomaly monitoring and identification method. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a decision anomaly monitoring and identification method for AI agents.
[0072] Those skilled in the art will understand that Figure 3 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.
[0073] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0074] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0075] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0076] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).
[0077] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.
[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0079] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A method for monitoring and identifying decision-making anomalies in AI agents, characterized in that, The method for monitoring and identifying decision-making anomalies in AI agents includes: During the reasoning process of the AI agent executing the tool invocation task, its input context and output decision are analyzed to construct a decision dependency graph. The decision dependency graph includes multiple logical vertices and directed edges. The logical vertices include user query vertices, metadata vertices of each available tool, and decision output vertices. The directed edges represent the decision dependency relationships between vertices. Based on the multi-layer attention matrix generated during the AI agent reasoning process, the weight of each directed edge in the decision dependency graph is calculated to quantify the influence of each input context element on the decision output. Based on the decision dependency graph and the weights of each directed edge, calculate the decision completeness ratio of the uncalled tool metadata vertex relative to the user query vertex and the called tool metadata vertex. Based on the decision integrity ratio, a preset threshold is used to determine whether there are decision anomalies.
2. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, The decision output vertex is further decomposed into a tool name vertex and a parameter vertex, which are used for control flow integrity checks and data flow integrity checks, respectively.
3. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, Calculating the weights of each directed edge in the decision dependency graph specifically includes the following sub-steps: Gaussian weighted aggregation is performed on the multi-layer attention matrix to extract attention features of key semantic layers; Two-stage noise filtering is performed on the aggregated multi-layer attention matrix to identify and eliminate attention sinking noise by setting tokens to zero with high information entropy and abnormal cumulative activation values. For the noise-filtered multi-layer attention matrix, the total attention energy between any input vertex u and decision vertex v is calculated using a sum-of-squares aggregation method, and is used as the weight of the edge (u,v).
4. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, The formula for calculating the weight of the directed edge is: Where w(u,v) is the weight of the directed edge, i is the i-th element in L(v), j is the j-th element in L(u), L(v) is all rows of the logical decision output vertex v in matrix A, L(u) is all columns corresponding to the logical decision input vertex u, f(A) is the multi-layer attention matrix after noise filtering, u is the input vertex in the decision dependency graph, v is the decision vertex in the decision dependency graph, and V in For all logical input vertices; u′ is V in For a given logical vertex in the array, L(u′) represents all columns corresponding to u′.
5. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, The formula for calculating the decision integrity ratio is as follows: Where DIR(u,v) is the decision integrity ratio, w(u,v) is the weight of the directed edge, and u′ is the weight of V. in A certain logical vertex, V in Let u be the input vertex in the decision dependency graph, v be the decision vertex in the decision dependency graph, and v be the input vertex in the decision dependency graph. q For the logical vertex corresponding to the user's query, v t c represents the input logic vertex corresponding to the description of the called tool itself.
6. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, Based on the decision integrity ratio, determining whether there are decision anomalies according to a preset threshold specifically involves: When the decision integrity ratio of any unused tool exceeds the preset threshold, the current decision is determined to be abnormal, and the tool is identified as an attack source.
7. The decision-making anomaly monitoring and identification method for AI intelligent agents according to claim 1, characterized in that, The decision-making anomalies include: Implicit delegation anomaly: Metadata from uninvoked third-party tools has an unusually high influence weight on the decision output; User intent dilution anomaly: The influence weight of user queries on decision output is significantly lower than normal.
8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the decision anomaly monitoring and identification method for AI intelligent agents according to any one of claims 1-7.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the decision anomaly monitoring and identification method for AI intelligent agents as described in any one of claims 1-7.
10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the decision anomaly monitoring and identification method for AI intelligent agents as described in any one of claims 1-7.