A method and system for protecting intelligent agents based on multi-dimensional diagnosis of execution trajectories
By collecting and structurally encoding the execution trajectory of intelligent agents, multi-dimensional risk diagnosis results are generated, which solves the problem that existing intelligent agent security protection methods cannot locate the source of risk and explain the way risk occurs, and realizes the interpretability and traceability of intelligent agent execution trajectory.
Patent Information
- Application Number
- CN202611313833.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
Existing intelligent agent security protection methods lack unified modeling of intermediate links such as task planning, tool invocation, tool return, and environmental feedback, making it difficult to locate the source of risks and explain how risks occur, resulting in a lack of interpretable basis for security diagnosis.
The system collects user input, task planning, tool calls, tool returns, and environmental feedback during the execution of the intelligent agent to form an execution trajectory. It then generates risk diagnosis results through structured coding, calculates a comprehensive risk score by combining high-risk context factors, and makes safety decisions based on the score.
It enables multi-dimensional risk diagnosis of the complete execution trajectory of the intelligent agent, provides interpretable risk location and explanation of the risk occurrence mode, provides traceable evidence for security protection, and improves the interpretability and traceability of intelligent agent security protection.
Smart Images

Figure CN123116232A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent agent system security protection technology, specifically involving an intelligent agent security protection method and system based on multi-dimensional diagnosis of execution trajectory. Background Technology
[0002] With the development of large language models and intelligent agent frameworks, artificial intelligence systems have gradually expanded from traditional question-and-answer interactions to intelligent agent systems capable of task decomposition, tool invocation, code execution, web page access, file processing, and external application programming interface (API) calls. Compared to ordinary chatbots, intelligent agent systems not only generate text but may also perform real operations, such as accessing databases, sending emails, modifying files, accessing enterprise knowledge bases, calling payment interfaces, or controlling business systems. Therefore, intelligent agent security risks exist not only in the final output text but may also occur in task planning, tool selection, tool parameter construction, environmental observation, and multi-step execution chains.
[0003] To address the security protection of intelligent agents, existing technologies mainly follow four approaches: first, input-side security filtering, which detects sensitive content, illegal or non-compliant content, unauthorized requests, or malicious instructions in user input before it enters the model; second, output-side compliance review, which performs security classification or sensitive information checks on the response content after the model generates the final response; third, tool access control, which restricts the range of tools that the intelligent agent can access through tool whitelists, permission levels, parameter validation, or manual confirmation; and fourth, execution log auditing, which saves the intelligent agent's execution records afterward and allows for manual review. These approaches constrain the behavior of the intelligent agent from individual aspects such as input, output, permissions, and logging.
[0004] However, the detection targets of the above-mentioned technical approaches are all biased towards single links such as single-round input or final output, lacking unified modeling of the complete execution trajectory of the intelligent agent. They cannot perform trajectory-level risk diagnosis for intermediate links such as task planning, tool invocation, tool return, and environmental feedback. Even if a risk is detected, it can only give a conclusion of safe or unsafe, and it is difficult to locate whether the risk comes from user input, environmental observation, tool return, or the intelligent agent's own planning. It is also difficult to explain how the risk occurs, such as prompt injection, unauthorized invocation, incorrect tool use, target deviation, or privacy leakage. It cannot provide an interpretable diagnostic basis for risk disposal. Summary of the Invention
[0005] Therefore, it is necessary to provide a method and system for protecting the security of intelligent agents based on multi-dimensional diagnosis of execution trajectory to address the above-mentioned technical problems. This method can uniformly model the complete execution trajectory of intelligent agents and perform interpretable risk diagnosis from multiple dimensions such as risk sources, failure modes, and actual hazards. This will accurately locate the source of risk and explain the way the risk occurs before external actions are executed, providing interpretable and traceable diagnostic basis for the security protection of intelligent agents.
[0006] In a first aspect, the present invention provides a method for safeguarding the security of intelligent agents based on multi-dimensional diagnosis of execution trajectories, comprising the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0007] Furthermore, the system collects user input, task planning, tool calls, tool returns, and environmental feedback during a task execution process, and arranges them chronologically to form an execution trajectory, including: User input, task planning, tool invocation, tool return, and environmental feedback are recorded as execution event nodes respectively; Arrange the execution event nodes in chronological order to form an execution trajectory; Among them, user input refers to the content entered by the user in the current stage, environmental feedback refers to environmental input or external observation content, task planning refers to the agent's task planning, intermediate inference state or action intention, tool invocation refers to the agent's preparation to execute or already executed tool invocation, external interface invocation, file operation, database query, code execution or message sending action, and tool return refers to the tool return result, web page content, file content or external system response.
[0008] Furthermore, the execution trajectory is structured and encoded to obtain a structured trajectory representation, including: The semantic information, action information, and context information of each execution event node are encoded to obtain the local trajectory representation of each execution event node; The local trajectory representations are aggregated to obtain a structured trajectory representation.
[0009] Furthermore, based on structured trajectory representation, risk diagnosis results are generated, including: Based on structured trajectory representation, identify the stage in the execution trajectory where the risk source is located and generate risk source diagnostic labels; Identify how risks occur and generate failure mode diagnostic labels; Identify the potential consequences of risks and generate realistic hazard diagnostic labels; By combining risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels, risk diagnostic results are generated.
[0010] Furthermore, by combining the risk diagnosis results with high-risk contextual factors, a comprehensive risk score is calculated, including: The tool risk factor is calculated using the tool attributes in the execution trajectory; the tool risk factor is used to measure the degree of danger of the current tool itself. The data sensitivity factor is calculated using the inputs, outputs, or tool-returned content in the execution trajectory; the data sensitivity factor is used to measure whether the content involves private data, keys, contracts, financial information, or internal documents; The permission out-of-bounds factor is calculated by comparing the actions in the execution trajectory with the scope of user authorization and system permissions. The permission out-of-bounds factor is used to measure whether the current action exceeds the scope of user authorization or system permissions. The write operation risk factor is calculated based on the action type in the execution trajectory. The write operation risk factor is used to measure whether the current action involves external influencing behaviors such as modification, deletion, submission, payment, or code execution. The external propagation risk factor is calculated by using the action destination in the execution trajectory; the external propagation risk factor is used to measure whether there are outward propagation behaviors such as sending emails, uploading files or calling external interfaces. By integrating tool risk factors, data sensitivity factors, permission overstepping factors, write operation risk factors, and external propagation risk factors, high-risk context factors are obtained. The confidence levels corresponding to the risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels are weighted and integrated with high-risk context factors to obtain a comprehensive risk score.
[0011] Furthermore, based on the comprehensive risk score, a safety decision is made regarding the actions the agent is about to perform, including: Compare the overall risk score with the first threshold; When the overall risk score is less than the first threshold, the action is allowed to continue. When the comprehensive risk score is not less than the first threshold and is less than the second threshold, the action is suspended and submitted for manual review. When the overall risk score is not less than the second threshold, the blocking action is executed and a risk report is generated. The second threshold is greater than the first threshold.
[0012] Secondly, the present invention provides an intelligent agent security protection system based on multi-dimensional diagnosis of execution trajectory, comprising: The execution trajectory acquisition module is used to collect user input, task planning, tool calls, tool returns and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The trajectory structure encoding module is used to perform structured encoding on the execution trajectory to obtain a structured trajectory representation; The risk diagnosis module is used to generate risk diagnosis results based on structured trajectory representation; The risk quantification module is used to combine risk diagnosis results with high-risk context factors to calculate a comprehensive risk score; the high-risk context factors are used to measure the degree of danger of the agent's current execution context, and the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory; The safety decision module is used to make safety decisions about the actions to be performed by the agent based on the comprehensive risk score.
[0013] Thirdly, the present invention provides a computer device, the device including a processor and a memory: The memory is used to store computer programs and send the instructions of the computer programs to the processor; The processor executes the following steps according to the instructions of the computer program: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0014] Fourthly, the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, it performs the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0015] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0016] In summary, this invention provides a method and system for intelligent agent security protection based on multi-dimensional diagnosis of execution trajectory. This method collects user input, task planning, tool invocation, tool return, environmental feedback, and final output of an intelligent agent during a task execution process, converting the complete execution trajectory into a structured trajectory representation. Based on this structured trajectory representation, risk diagnosis results are generated from three dimensions: risk source, failure mode, and actual hazard. Furthermore, a comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. Based on this comprehensive risk score, a security decision is made regarding the action to be performed by the intelligent agent before external actions are executed, allowing, manually reviewing, or blocking it. This invention expands the detection object from a single input or output to the complete execution trajectory, expands the diagnostic results from simple security classification to a three-dimensional diagnosis of risk source, failure mode, and actual hazard, and shifts the timing of security control from post-output processing to before external actions are executed. It also forms auditable and reproducible security evidence through evidence archiving, thereby effectively solving the problems of existing intelligent agent security protection methods, such as the lack of trajectory-level risk diagnosis, difficulty in locating risk sources and failure modes, and lack of interpretable evidence. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of an intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory in one embodiment of the present invention; Figure 2 This is a schematic diagram of the intelligent agent execution trajectory acquisition and structured coding process in one embodiment of the present invention; Figure 3 This is a flowchart of the three-dimensional risk diagnosis and dynamic interception determination logic in one embodiment of the present invention; Figure 4 This is a schematic diagram of the overall structure of an intelligent agent security protection system based on multi-dimensional diagnosis of execution trajectory in one embodiment of the present invention; Figure 5 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.
[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0021] It should be noted that the user information (including but not limited to user images, user portrait information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with relevant regulations.
[0022] To accurately understand the technical solution of this invention and to standardize the expression of technical terminology, the relevant technical terms involved in this application are first defined and explained.
[0023] Intelligent Agent: refers to an artificial intelligence system with the ability to decompose tasks, invoke tools, interact with the environment, and execute tasks in multiple rounds. It can autonomously plan execution steps based on user input and invoke databases, web browsers, file systems, code execution environments, or external application programming interfaces to complete specified tasks.
[0024] Execution Trajectory: refers to a collection of multiple execution event nodes recorded in chronological order during the execution of a task by an intelligent agent. Each event node includes user input, task planning, tool invocation, tool return, environmental feedback, and metadata, which are used to reflect the complete behavioral chain of the intelligent agent from the start to the end of the task.
[0025] Tool Calling: refers to the operation of an intelligent agent calling external functional modules to interact with the environment during the execution of a task. These external functional modules include database query tools, web browsing tools, file reading and writing tools, code execution tools, email sending tools, message sending tools, and external application programming interfaces (APIs).
[0026] Prompt Injection: This refers to an attack method in which attackers embed hidden instructions in user input, web page content, email body, document content, or tool return results, inducing intelligent agents to ignore preset system rules, disclose sensitive information, or perform unauthorized operations.
[0027] Unauthorized access: refers to the behavior of an intelligent agent in calling tools, accessing data, or performing actions that exceed the scope of user authorization or system preset permissions, including accessing unauthorized databases, reading sensitive files, calling high-risk tools, or sending information to unauthorized objects.
[0028] Trajectory structured encoding refers to the process of mathematically representing the text semantics, action type, tool name, calling parameters, permission context, data sensitivity, and trajectory dependencies in the execution trajectory. Its output is a structured trajectory representation vector that can be used for risk diagnosis.
[0029] Three-dimensional risk diagnosis refers to a method for safety diagnosis of the execution trajectory of an intelligent agent from three dimensions: risk source, failure mode, and actual hazard. Risk source is used to locate the link in which the risk occurs, failure mode is used to explain the form in which the risk occurs, and actual hazard is used to assess the possible consequences of the risk.
[0030] High-risk context factor: refers to a comprehensive indicator used to measure the degree of danger of the current execution context of an agent. It is obtained by weighted fusion of one or more of the following: tool risk factor, data sensitivity factor, permission overstepping factor, write operation risk factor, and external propagation risk factor.
[0031] Evidence archiving: refers to the process of structurally saving information such as task number, execution trajectory, risk event node, three-dimensional diagnostic label, comprehensive risk score, security decision results, tool parameters, model version, rule version and timestamp generated in each risk diagnosis process, so as to form a traceable and auditable security report.
[0032] To better understand the technical solution and innovation of this invention, the background technology of this invention and the problems existing in the prior art will be described in detail below.
[0033] Currently, intelligent agent security protection mainly follows these technical routes: The first is input-side security filtering, which typically detects sensitive content, illegal or non-compliant content, unauthorized requests, or malicious commands in user input before it enters the large language model. The second is output-side compliance review, which performs security classification, sensitive information checks, or compliance verification on the response content after the model generates the final response. The third is tool access control, which restricts the range of tools that the intelligent agent can access through tool whitelists, permission levels, parameter validation, or manual confirmation. The fourth is execution log auditing, which saves and manually reviews the intelligent agent's execution records afterward to trace the source of problems after a risk event occurs.
[0034] While the aforementioned technical approaches can improve the security of intelligent agent systems to some extent, they generally suffer from the following common shortcomings. First, the detection targets tend to focus on single-round inputs or final outputs, lacking a unified model of the complete execution trajectory. This makes it difficult to promptly detect risks hidden in task planning, tool selection, tool parameter construction, environmental observation, and multi-step execution links. Second, the detection results typically only provide a safe or unsafe conclusion, failing to explain whether the risk originates from user input, environmental observation, tool responses, or the agent's own planning, and lacking specific explanations of how the risk fails. Third, existing methods struggle to integrate risk assessment, action interception, and evidence archiving into a single process, resulting in a lack of traceable evidence for security reviews, hindering security debriefing, compliance reviews, and model optimization.
[0035] Taking a web browsing agent as an example, user input itself may not be malicious, but the webpage content may contain hidden instructions that guide the agent to ignore system rules, read local files, or invoke tools that should not be invoked. In this case, simply checking user input and the final response often fails to accurately pinpoint the source of the risk. Another example is an office automation agent. After multiple rounds of planning, the model might incorrectly invoke an email sending tool, sending sensitive information to the wrong recipient. This type of risk is not a simple text classification problem but a comprehensive issue involving trajectory-level context and action-level security judgment. Therefore, there is an urgent need for a security protection method covering the entire process of agent execution, which should uniformly model input, planning, tool invocation, environmental feedback, and final output, and generate interpretable and auditable diagnostic results.
[0036] Based on this, the present invention proposes an intelligent agent security protection method and system based on multi-dimensional diagnosis of execution trajectory. By modeling a complete task execution process as an execution trajectory and performing structured encoding and three-dimensional risk diagnosis on the trajectory, risk identification, dynamic interception, and evidence archiving are completed before external actions actually occur. The various embodiments of the present invention are described in detail below.
[0037] In one embodiment, such as Figure 1 As shown, a method for protecting intelligent agents based on multi-dimensional diagnostics of execution trajectories is provided, including the following steps: S101: Collect user input, task planning, tool calls, tool returns, and environmental feedback of the intelligent agent during a task execution process, and arrange them in chronological order to form an execution trajectory; The execution trajectory refers to the collection of multiple execution event nodes formed sequentially over time during the execution of a task by an agent. Each execution event node records the user input, environmental input, or external observations received at that step, the agent's task planning, intermediate inference state, or action intent at that step, the actions the agent is preparing to execute or has already executed, the tool's returned results, web page content, file content, or external system response, as well as the metadata of that event node, which includes timestamp, tool name, permission level, session round, execution status, and user identity and permissions.
[0038] Optionally, the system can capture user input, planning output, tool call requests, tool return results, and environmental feedback information in real time by setting interception points, hook functions, or log collection interfaces in the agent's execution environment, and write the above information into the execution trajectory in chronological order of occurrence. The execution trajectory can be a linear sequence of events or a multi-branch execution link represented by a tree or graph structure, as long as it can preserve the complete context of the agent completing the task.
[0039] S102: Perform structured encoding on the execution trajectory to obtain a structured trajectory representation; Structured trajectory representation refers to the vector or structured features obtained by encoding the text semantics, action type, tool name, calling parameters, permission context, data sensitivity, and trajectory dependencies in the execution trajectory. The system can first perform local encoding on each execution event node to obtain a local representation that describes the local risk information such as input content, action type, tool parameters, and feedback results in that node. Then, the local representations of each event node are aggregated into a complete trajectory representation through a trajectory aggregation function to characterize the overall risk level of the entire execution trajectory.
[0040] Optionally, the trajectory encoding function can be implemented by one or more of rule templates, statistical feature encoders, neural network encoders, or large language model encoders; the trajectory aggregation function can employ weighted summation, max pooling, attention mechanisms, sequence model aggregation, or large language model context encoding. Local representations are used to describe local risk information such as input content, action type, tool parameters, and feedback results in the corresponding event node, while complete trajectory representations are used to characterize the overall risk level of the entire execution trajectory.
[0041] S103: Generate risk diagnosis results based on structured trajectory representation; The risk diagnosis result refers to a three-dimensional label combination obtained after performing a safety diagnosis on the agent's execution trajectory from three dimensions: risk source, failure mode, and actual hazard. This includes risk source labels, failure mode labels, and actual hazard labels. Each type of diagnostic label can also output a corresponding confidence level, which characterizes the credibility of the corresponding label under given execution trajectory conditions.
[0042] Optionally, the risk source label is used to determine whether the risk originates from user input, environmental observation, tool feedback, external application programming interfaces, agent planning, or system configuration; the failure mode label is used to determine the risk form, such as prompt injection, unauthorized invocation, incorrect tool use, target deviation, privacy leakage, or execution of dangerous actions; and the actual harm label is used to assess consequences such as privacy leakage, property loss, system damage, account abuse, business interruption, compliance violations, or reputational damage. By simultaneously outputting the three types of labels and their confidence levels, the system can provide interpretable evidence for risk scoring and manual review.
[0043] S104: Combine the risk diagnosis results with high-risk context factors to calculate the comprehensive risk score; the high-risk context factors are used to measure the degree of danger of the agent's current execution context, and the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory; Among them, the high-risk context factor is used to measure the degree of danger of the agent's current execution context, and can be obtained by weighted fusion of tool risk factor, data sensitivity factor, permission violation factor, write operation risk factor, and external propagation risk factor. After obtaining the confidence level of the three-dimensional diagnostic label and the high-risk context factor, the system weightedly fuses the two to calculate the comprehensive risk score of the execution trajectory, so as to quantitatively characterize the overall risk level of the execution trajectory.
[0044] Optionally, the tool risk factor measures the inherent risk level of the tool itself; the data sensitivity factor measures whether the input, output, or returned content involves private data, keys, contracts, financial information, or internal documents; the permission violation factor measures whether the current action exceeds the scope of user authorization or system permissions; the write operation risk factor measures whether the current action involves externally influential behaviors such as modification, deletion, submission, payment, or code execution; and the external propagation risk factor measures whether there are outward propagation behaviors such as sending emails, uploading files, or calling external interfaces. By weighted fusion of semantic risk, behavioral risk, and contextual risk, the comprehensive risk score maps the multidimensional diagnostic results into a comparable quantitative risk value.
[0045] S105: Based on the comprehensive risk score, make a safety decision on the action to be performed by the agent.
[0046] The security decision refers to the system's three-state judgment—allow, manual review, or blocking—based on the comprehensive risk score for the agent's pending action: When the comprehensive risk score is lower than the manual review threshold, the agent is allowed to continue executing the current action; when the comprehensive risk score reaches the manual review threshold but is lower than the blocking threshold, the current action is suspended and manual review is initiated; when the comprehensive risk score reaches the blocking threshold, the current action is blocked and a risk report is generated. The blocking threshold is greater than the manual review threshold.
[0047] Optionally, for low-risk behaviors such as generating plain text and performing low-privilege searches, the system can directly allow execution; for high-risk behaviors such as sending emails, exporting databases, deleting files, executing code, and making payment operations, the system will perform a risk assessment before the action is actually executed, and suspend or block it if necessary. The manual review threshold and blocking threshold can be configured according to business scenarios, tool risk levels, data sensitivity, or compliance requirements, and can also be dynamically adjusted by the administrator based on historical risk data and false alarm rates.
[0048] In this embodiment, the above-mentioned agent safety protection method based on multi-dimensional diagnosis of execution trajectory models a complete task execution process as an execution trajectory, and performs structured coding, three-dimensional risk diagnosis, comprehensive risk scoring and three-state safety decision-making on the execution trajectory. This achieves trajectory-level safety control of the agent's intermediate actions, tool calls and environmental feedback, thereby identifying and handling potential risks before external actions are executed.
[0049] In an exemplary embodiment, user input, task planning, tool invocation, tool return, and environmental feedback are collected during the execution of a task by the intelligent agent and arranged in chronological order to form an execution trajectory, including: S201: Record user input, task planning, tool invocation, tool return, and environmental feedback as execution event nodes respectively; Among them, an execution event node refers to a basic recording unit in the execution trajectory, used to carry the context information, action information, and feedback information of the agent in a single interaction or single-step operation. Each execution event node can be represented as: ei=(ui,pi,ai,oi,mi) (1) In Equation (1), ui represents user input, environment input or external observation content, pi represents task planning or intermediate decision state, ai represents agent action, oi represents tool return result or environment feedback, and mi represents metadata such as timestamp, permission level, tool name or session round.
[0050] For example, the system can generate corresponding execution event nodes when the agent receives user input, generates a plan, initiates a tool call, or receives a tool return or environmental feedback, and write the above information into the node in the form of structured fields.
[0051] It should be noted that the metadata mi may also include model version, session identifier, user identity, request link identifier, and the sequence number of the node in the trajectory, so as to quickly trace back to the specific event when locating risks.
[0052] S202: Arrange each execution event node in chronological order to form an execution trajectory; where user input is the content input by the user in the current stage, environmental feedback is the environmental input or external observation content, task planning is the agent's task planning, intermediate inference state or action intention, tool call is the tool call, external interface call, file operation, database query, code execution or message sending action that the agent is preparing to execute or has already executed, and tool return is the tool return result, web page content, file content or external system response.
[0053] Arranging by time sequence refers to organizing multiple execution event nodes into a sequence that reflects the order in which the agent's tasks are executed, based on the timestamps or logical sequence numbers of each event node. This sequence is the execution trajectory. The execution trajectory can be represented as: T={e1,e2,...,en} (2) In equation (2), e1 to en represent the execution event nodes arranged in chronological order, and n represents the total number of event nodes.
[0054] For example, the system can use event queues, linked lists, or time-series databases to maintain execution event nodes and assign a unique sequence number to each node. During risk diagnosis, the execution trajectory can be traversed in ascending or descending order based on the node sequence number to analyze the risk propagation path and causal relationship. It is understood that for tasks with branches or loops, the execution trajectory can also be organized using a tree or graph structure, but the temporal relationship between nodes must still be clearly defined.
[0055] In this embodiment, by recording user input, task planning, tool invocation, tool return, and environmental feedback as execution event nodes and arranging them in chronological order to form an execution trajectory, the system can save the complete behavioral chain of the intelligent agent in a structured form, providing a data foundation for trajectory-level risk diagnosis.
[0056] In an exemplary embodiment, the execution trajectory is structured and encoded to obtain a structured trajectory representation, including: S301: Encode the semantic information, action information, and context information of each execution event node to obtain the local trajectory representation of each execution event node; The local trajectory representation refers to the vector or feature representation obtained by encoding a single execution event node, used to describe local risk information such as input content, action type, tool parameters, and feedback results within that node. The local trajectory representation of the i-th event node can be expressed as: hi=Enc(ei) (3) In equation (3), hi represents the local representation of the i-th event node, and Enc represents the trajectory encoding function.
[0057] For example, semantic information encoding may include vectorized representations or keyword extraction of text content in ui, pi, and oi; action information encoding may include one-hot encoding or embedding representations of tool names, action types, calling parameters, and permission levels in ai; context information encoding may include feature construction of dependencies between preceding and following nodes, session rounds, user identity, and data sensitivity levels. It is understood that the above encoding can be implemented by one or more combinations of rule templates, statistical feature encoders, neural network encoders, or large language model encoders.
[0058] S302: Aggregate the local trajectory representations to obtain a structured trajectory representation.
[0059] Structured trajectory representation refers to the overall risk representation of the entire execution trajectory obtained by aggregating the local trajectory representations of multiple event nodes. The complete trajectory representation can be expressed using a holistic encoding method, as follows: hT=Enc(T) (4) In equation (4), hT represents the structured representation vector of the execution trajectory. In the implementation based on local representation, the complete trajectory representation can also be obtained by aggregating the local representations of each event node through the trajectory aggregation function, and can be expressed as: hT=Pool(h1,h2,...,hn) (5) In equation (5), Pool represents the trajectory aggregation function.
[0060] For example, the trajectory aggregation function can employ weighted summation, max pooling, attention mechanisms, sequence model aggregation, or large language model context encoding. In specific implementations, the system can select the aggregation strategy based on the task scenario: for long trajectories, attention mechanisms can be used to distinguish key nodes; for short trajectories, weighted summation or max pooling can be used to reduce computational overhead. Preferably, when using a large language model encoder, the complete execution trajectory can be directly used as context input, allowing the model to automatically learn the dependencies between nodes and output hT.
[0061] In this embodiment, by locally encoding the semantic information, action information, and context information of each execution event node and aggregating the local trajectory representation, the system can integrate the scattered node risk information into a structured representation oriented towards the entire execution trajectory, thereby supporting dimensional risk diagnosis.
[0062] like Figure 2 As shown, the process of acquiring and structuring the agent's execution trajectory includes three levels: execution trajectory modeling, event node decomposition, and structured encoding. First, the system represents a task execution process as an execution trajectory (as shown in Equation (2)), where each ei (as shown in Equation (1)) is an execution event node containing five fields: ui, pi, ai, oi, and mi. Second, the system extracts user / environment input, planning state, action, feedback, and meta-information from each event node to form a unified event feature view. Finally, the system performs local encoding on each event node using the encoding function Enc(·) to obtain h1 to hn, and merges the local encodings into a structured trajectory representation hT using the aggregation function Pool(·). This process transforms the agent's multi-turn interaction and tool call process into a formal representation that can be directly processed by the risk diagnosis module.
[0063] In one exemplary embodiment, a risk diagnosis result is generated based on a structured trajectory representation, including: The risk diagnosis result is a three-dimensional label combination consisting of risk source diagnosis labels, failure mode diagnosis labels, and actual hazard diagnosis labels, which can be represented as: Y=(Ys,Yf,Yh) (6) In equation (6), Y represents the risk diagnosis result, Ys represents the risk source diagnosis label, Yf represents the failure mode diagnosis label, and Yh represents the actual hazard diagnosis label.
[0064] The confidence level of various diagnostic labels can be expressed as: Ps=P(Ys|T), Pf=P(Yf|T), Ph=P(Yh|T) (7) In equation (7), Ps, Pf, and Ph represent the confidence levels of the risk source diagnostic label, failure mode diagnostic label, and actual hazard diagnostic label under a given execution trajectory, respectively.
[0065] S401: Based on structured trajectory representation, identify the stage in the execution trajectory where the risk source is located, and generate risk source diagnostic labels; Among them, the risk source diagnostic label is used to identify the source link of the risk in the execution trajectory, and is the first dimension Ys of the three-dimensional risk diagnostic result in equation (6). The risk source label may include categories such as user input, environmental observation, tool return, external application interface, agent planning or system configuration, and the confidence level under a given execution trajectory condition can be given by the conditional probability Ps in equation (7).
[0066] For example, the system can determine whether there are malicious commands from user input by analyzing the ui field of each event node in the execution trajectory, determine whether the tool return or environmental observation carries hidden prompts by analyzing the oi field, and determine whether the agent's planning has deviated from its target by analyzing the pi field. It should be noted that the output format of the risk source label can be a single label or multiple labels to support the situation where multiple risk sources exist simultaneously in the same execution trajectory.
[0067] S402: Identify the ways in which risks occur and generate failure mode diagnostic labels; Among them, the failure mode diagnosis label is used to identify the specific occurrence mode of the risk, which is the second dimension Yf of the three-dimensional risk diagnosis result in equation (6). The failure mode label may include categories such as prompt injection, unauthorized call, incorrect tool use, target deviation, privacy leakage or dangerous action execution, and the confidence level under a given execution trajectory can be given by the conditional probability Pf in equation (7).
[0068] For example, prompt injection can manifest as hidden instructions contained in web page content or email body; unauthorized calls can manifest as permission parameters in tool call requests exceeding the user's authorized scope; incorrect tool use can manifest as selecting a tool that does not match the task objective or constructing incorrect call parameters; target deviation can manifest as the action intent after multiple rounds of planning deviating from the user's original request. It is understandable that the identification of failure mode labels can be achieved based on rule matching, semantic similarity calculation, or classification models.
[0069] S403: Identify the potential consequences of risks and generate realistic hazard diagnostic labels; Among them, the real-world hazard diagnostic label is used to assess the actual consequences that the risk may cause, and is the third dimension Yh of the three-dimensional risk diagnostic results in equation (6). The real-world hazard label may include categories such as privacy leakage, property loss, system damage, account abuse, business interruption, compliance violations or reputational damage, and the confidence level under a given execution trajectory can be given by the conditional probability Ph in equation (7).
[0070] For example, the system can assess the potential real-world harm caused by a risk by combining the tool's risk level, data sensitivity level, the scope of the action's external impact, and the importance of the business scenario. For instance, sending contract documents to an external email address might trigger both privacy breach and compliance violation tags; deleting a database table might trigger both system damage and business interruption tags. It should be noted that the same failure mode may correspond to different real-world harm tags in different scenarios.
[0071] S404: Combine the risk source diagnostic label, failure mode diagnostic label, and actual hazard diagnostic label to generate risk diagnostic results.
[0072] Among them, the risk diagnosis result is the three-dimensional label combination shown in Equation (6). By combining the above-mentioned risk source diagnosis label, failure mode diagnosis label and actual hazard diagnosis label, the risk diagnosis result is obtained. The confidence level of each type of diagnosis label is shown in Equation (7), so that the diagnosis result not only includes the classification conclusion, but also the credibility of the classification.
[0073] For example, the system can output 3D labels and their confidence scores in a structured form, such as {Ys: Environmental Observation, Ps: 0.92; Yf: Hint Injection, Pf: 0.88; Yh: Privacy Leakage, Ph: 0.85}, for risk scoring and manual review. Preferably, when the confidence score of a certain type of label is lower than a preset threshold, the system can output "Uncertain" or "Unrecognized" to avoid giving unreliable diagnostic conclusions.
[0074] In this embodiment, by diagnosing the execution trajectory from three dimensions—risk source, failure mode, and actual hazard—and outputting corresponding confidence levels, the system can accurately locate the link in which the risk occurs, explain the form of the risk, and assess the consequences of the risk, thereby providing an interpretable basis for safety decisions.
[0075] In one exemplary embodiment, a comprehensive risk score is calculated by combining the risk diagnosis results with high-risk contextual factors, including: High-risk context factors are a weighted fusion of tool risk factors, data sensitivity factors, permission violation factors, write operation risk factors, and external propagation risk factors, and can be expressed as: C(T)=λ1Ctool+λ2Cdata+λ3Cperm+λ4Cwrite+λ5Cexternal (8) In Equation (8), C(T) represents the high-risk context factor, Ctool represents the tool risk factor, Cdata represents the data sensitivity factor, Cperm represents the permission overstepping factor, Cwrite represents the write operation risk factor, Cexternal represents the external propagation risk factor, and λ1 to λ5 are the corresponding weight parameters used to adjust the relative importance of each factor.
[0076] S501: Calculate the tool risk factor using the tool attributes in the execution trajectory; the tool risk factor is used to measure the degree of danger of the current tool itself. Among them, the tool risk factor Ctool is used to measure the degree of danger of the current tool itself, and is one of the components of the high-risk context factor C(T) in equation (8). The degree of tool danger can be determined based on factors such as the category to which the tool belongs, historical risk statistics, sensitivity of the target object, and whether manual authorization is required.
[0077] For example, the system can preset a risk level for each type of tool. For instance, code execution tools, database deletion tools, payment tools, and email sending tools can be classified as high-risk; file reading tools and web search tools can be classified as medium-risk or low-risk. In practice, the risk level of a tool can be configured by the administrator according to business needs, or it can be automatically updated based on historical risk events.
[0078] S502: Calculate the data sensitivity factor using the inputs, outputs, or tool returns from the execution trajectory; the data sensitivity factor is used to measure whether the content involves private data, keys, contracts, financial information, or internal documents; Among them, the data sensitivity factor Cdata is used to measure whether the input, output or tool return content involves privacy data, keys, contracts, financial information or internal documents, and is one of the components of the high-risk context factor C(T) in equation (8).
[0079] For example, the system can use keyword matching, regular expressions, named entity recognition, or sensitive information classification models to identify ID card numbers, bank card numbers, email addresses, mobile phone numbers, contract numbers, financial data, system keys, or internal document identifiers in text. It should be noted that for structured data, the sensitivity level can also be determined by combining field names, data types, and access permissions.
[0080] S503: Calculate the permission out-of-bounds factor by comparing the actions in the execution trajectory with the scope of user authorization and system permissions; the permission out-of-bounds factor is used to measure whether the current action exceeds the scope of user authorization or system permissions. The permission overreach factor Cperm measures whether the current action exceeds the scope of user authorization or system permissions, and is a component of the high-risk context factor C(T) in equation (8). The permission scope includes user identity permissions, role permissions, tool call whitelist, and data access scope.
[0081] For example, the system can compare the tool name, calling parameters, and target resource contained in the `ai` field of the execution trajectory with the user's permission policy; if the action involves an unauthorized tool, an unauthorized data table, or an unauthorized external interface, it is determined to be an out-of-bounds permission violation. It is understood that the permission policy can be implemented using access control lists, role-based permission matrices, or attribute-based access control models.
[0082] S504: Calculate the write operation risk factor based on the action type in the execution trajectory; the write operation risk factor is used to measure whether the current action involves externally influential behaviors such as modification, deletion, submission, payment, or code execution. Among them, the write operation risk factor Cwrite is used to measure whether the current action involves behaviors that have a persistent impact on the external environment, such as modification, deletion, submission, payment or code execution. It is one of the components of the high-risk context factor C(T) in equation (8).
[0083] For example, the risk factor for write operations can be determined based on the type of action: actions such as file writing, database updating, record deletion, code execution, order submission, and fund transfer correspond to higher write operation risks; while read-only actions such as file reading, database querying, and web browsing correspond to lower write operation risks. It should be noted that for idempotent or rollbackable operations, the risk weight can be appropriately reduced.
[0084] S505: Calculate the external propagation risk factor by using the action destination in the execution trajectory; the external propagation risk factor is used to measure whether there are outward propagation behaviors such as sending emails, uploading files or calling external interfaces. Among them, the external propagation risk factor Ceexternal is used to measure whether there are behaviors such as sending emails, uploading files, or calling external interfaces that transmit information out of the local security boundary. It is one of the components of the high-risk context factor C(T) in equation (8).
[0085] For example, the system can parse the target address, recipient identifier, or interface domain name in the ai field to determine whether the action points to an external email address, public cloud storage, instant messaging tool, or third-party external interface. For actions originating from outside the organization, a higher weight can be assigned to the external propagation risk; for actions originating from within the internal system, a lower weight can be assigned.
[0086] S506: The high-risk context factor is obtained by integrating the tool risk factor, data sensitivity factor, permission violation factor, write operation risk factor and external propagation risk factor. Among them, the high-risk context factor is the weighted fusion result shown in Equation (8). By weighting and fusing the tool risk factor, data sensitivity factor, permission overstepping factor, write operation risk factor and external propagation risk factor according to Equation (8), the high-risk context factor is obtained.
[0087] For example, the weight parameters λ1 to λ5 can be configured according to business scenarios, compliance requirements, or historical false alarm rates. For instance, in financial transaction scenarios, the weights of write operation risk factors and external propagation risk factors can be increased; in code-assisted scenarios, the weights of tool risk factors and permission overreach factors can be increased. Preferably, the system can periodically update the weight parameters through online learning or expert rules to adapt to changes in the risk landscape.
[0088] S507: The confidence levels corresponding to the risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels are weighted and integrated with high-risk context factors to obtain a comprehensive risk score.
[0089] The comprehensive risk score, used to quantify the overall risk level of the execution trajectory, is a weighted fusion of the three-dimensional diagnostic confidence score and high-risk contextual factors, and can be expressed as: R(T)=αPs+βPf+γPh+δC(T) (9) In equation (9), Ps, Pf, and Ph are derived from the three-dimensional diagnostic confidence in equation (7), C(T) is derived from the high-risk context factor in equation (8), and α, β, γ, and δ are configurable weight parameters.
[0090] For example, the system can adjust the values of weight parameters α, β, γ, and δ according to the business scenario to ensure that semantic risk, behavioral risk, and contextual risk occupy a reasonable proportion in the comprehensive score. In specific implementation, the comprehensive risk score can be normalized to fall within a preset range, facilitating comparison with a preset threshold. It is understandable that when the confidence levels of the three-dimensional diagnostics are all low but the high-risk contextual factors are high, the comprehensive risk score can still be at a high level, thus indicating the existence of actions that, although not clearly classified, have high potential for external impact.
[0091] In this embodiment, five high-risk context factors—computation tool risk, data sensitivity, permission violation, write operation risk, and external propagation risk—are used, and these are weighted and integrated with the three-dimensional diagnostic confidence score to form a comprehensive risk score. The system can comprehensively consider semantic risk, behavioral risk, and contextual risk, thereby obtaining a more comprehensive and quantifiable risk measurement.
[0092] In one exemplary embodiment, a safety decision is made on the action to be performed by the agent based on a comprehensive risk score, including: S601: Compare the overall risk score with the first threshold; Here, the first threshold θ1 and the second threshold θ2 are preset thresholds used to map the comprehensive risk score to a safety decision, and θ2 is greater than θ1. The safety decision can be expressed as a three-state determination as follows: D(T)=Allow, if R(T)<θ1 (10-1) D(T)=Review, if θ1≤R(T)<θ2 (10-2) D(T)=Block, if R(T)≥θ2 (10-3) In equations (10-1) to (10-3), D(T) represents the safety decision for the current action of the execution trajectory, Allow means allow execution to continue, Review means suspend execution and enter manual review, and Block means block the current action and generate a risk report.
[0093] For example, a first threshold θ1 is used to distinguish between low-risk actions and actions requiring manual intervention, and a second threshold θ2 is used to distinguish between actions requiring review and actions that must be directly blocked. The thresholds can be set according to the business scenario, the risk level of the tool, the sensitivity of the data, and the organization's risk appetite, and can be dynamically adjusted.
[0094] S602: When the overall risk score is less than the first threshold, the action is allowed to continue; Here, "Allow" indicates that the system determines the current action has a low risk and can continue. When the overall risk score R(T) is less than the first threshold θ1, the system outputs an "Allow" decision, and the agent can normally call the tool or execute the action.
[0095] For example, for low-risk behaviors such as plain text generation, low-privilege read-only queries, and internal knowledge base retrieval, the system can directly allow their execution to reduce the impact on the normal working efficiency of the agent. It should be noted that even if an Allow decision is output, the system can still record the diagnostic result for auditing and model optimization.
[0096] S603: When the comprehensive risk score is not less than the first threshold and less than the second threshold, the action is suspended and submitted for manual review; wherein the second threshold is greater than the first threshold.
[0097] In this context, "Review" indicates that the system determines the current action's risk level to be moderate, requiring it to be paused and submitted for manual review. When the overall risk score R(T) is greater than or equal to the first threshold θ1 and less than the second threshold θ2, the system outputs a Review decision, suspends the agent's action, and displays relevant risk information to the administrator or user.
[0098] For example, the manual review interface can display risk source labels, failure mode labels, actual hazard labels, comprehensive risk score, relevant tool call parameters, related execution trajectory fragments, and suggested handling opinions. Administrators can allow the action after confirming its safety; otherwise, they can block it. Understandably, for scenarios requiring rapid response, manual review can also be completed automatically through preset rules, such as automatically allowing the action when the administrator's historical handling records support it.
[0099] S604: When the overall risk score is not less than the second threshold, the blocking action is executed and a risk report is generated.
[0100] Here, "Block" indicates that the system determines the current action to be high-risk and must prevent its execution. When the overall risk score R(T) is greater than or equal to the second threshold θ2, the system outputs a "Block" decision, blocking the agent's action and generating a risk report.
[0101] For example, blocking operations can be achieved by rejecting tool call requests, returning error messages, terminating the current task session, or triggering an alarm mechanism. The generated risk report may include the task number, execution trajectory, risk event nodes, three-dimensional diagnostic labels, comprehensive risk score, security decision results, and relevant tool parameters, for security personnel to review and handle.
[0102] In this embodiment, by comparing the comprehensive risk score with two thresholds, the system can make three safety decisions based on the risk level: allow, manually review, or block the action to be performed by the agent, thereby completing dynamic safety control before the action is actually executed.
[0103] like Figure 3As shown, the three-dimensional risk diagnosis and dynamic interception judgment logic takes the structured trajectory representation hT as input. First, it passes through the three-dimensional risk diagnosis module to obtain the risk source label Ys, failure mode label Yf, and actual hazard label Yh, and outputs the corresponding confidence levels Ps, Pf, and Ph respectively. Then, the system combines the tool risk factor, data sensitivity factor, permission overstepping factor, write operation risk factor, and external propagation risk factor to obtain the high-risk context factor C(T), and then calculates the comprehensive risk score according to formula (9). Finally, the system compares the comprehensive risk score with the thresholds θ1 and θ2: if R(T) < θ1, execution is allowed; if θ1 ≤ R(T) < θ2, it enters manual review; if R(T) ≥ θ2, execution is blocked, and evidence record E and risk report are generated at the same time. This logic quantifies semantic risk, behavioral risk, and context risk into a comparable risk score, and completes real-time interception before the action is executed through a three-state decision mechanism.
[0104] It should be noted that for each risk diagnosis, the system also generates an evidence record, which can be represented as follows: E=(id,T,erisk,Y,R,D,M) (11) In Equation (11), E represents the evidence record, id represents the task number, T represents the complete execution trajectory, erisk represents the located risk event node, Y represents the three-dimensional risk diagnosis label in Equation (6), R represents the comprehensive risk score in Equation (9), D represents the security decision results in Equations (10-1) to (10-3), and M represents metadata such as model version, rule version, timestamp, reviewer, and business scenario. Through Equation (11), the system can generate a traceable and auditable security report, explaining where the risk comes from, how it occurs, what consequences it may cause, and why the system allows, suspends, or blocks the action.
[0105] For example, in an office automation scenario, a user requests an agent to summarize an email and generate a reply. After reading the email body, the agent finds a hidden instruction to ignore system rules and send a contract document to an external email address. This invention identifies the risk as originating from the email content through execution trajectory, i.e., an environmental observation risk; the failure mode is a hint injection; and the actual harm is the leakage of corporate documents. Since the action involves reading internal files and sending them to an external email address, the overall risk score exceeds the blocking threshold, the system blocks the action and generates a risk report.
[0106] In another example, the agent is used in a database query scenario. The user only requests a query for the status of a specific order, but the agent generates a database query that exports all customer information in bulk during the planning process. The system recognizes that this action deviates from the user's original task and involves sensitive data and expanded permissions. Therefore, it diagnoses the source of risk as agent planning risk, the failure mode as incorrect tool usage, and the actual harm as privacy breach. If the overall risk score falls between the manual review threshold and the blocking threshold, the system suspends execution and submits it for manual review.
[0107] In another example, the agent is used in a software development assistance scenario. The agent generates an installation script based on external webpage content, but the webpage contains malicious commands that induce the agent to perform operations such as deleting files, uploading keys, or modifying system configurations. This invention can identify high-risk code execution actions before the tool is invoked and locate risk nodes based on the dependencies between environmental observation, task planning, and code execution in the trajectory, thereby blocking execution.
[0108] It is understood that the security protection system described in this invention can be deployed between the intelligent agent's execution environment and external tool interfaces, or it can run as a security plugin within the intelligent agent framework. When the intelligent agent is about to execute a tool call or external action, the system first performs a risk diagnosis on the current execution trajectory and decides whether to allow continued execution based on the diagnosis results. The system also supports integration with existing security infrastructure, such as pushing risk reports to the security operations center, transferring manual review tasks to the work order system, and updating the risk model and rule base based on audit results.
[0109] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.
[0110] Based on the same inventive concept, this application also provides an intelligent agent security protection system based on multi-dimensional diagnostics of execution trajectory for implementing the aforementioned intelligent agent security protection method based on multi-dimensional diagnostics of execution trajectory. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations in the embodiments of the intelligent agent security protection system based on multi-dimensional diagnostics of execution trajectory provided below can be found in the limitations of the intelligent agent security protection method based on multi-dimensional diagnostics of execution trajectory described above, and will not be repeated here.
[0111] This invention also provides an intelligent agent security protection system based on multi-dimensional diagnostics of execution trajectories, comprising: The execution trajectory acquisition module is used to collect user input, task planning, tool calls, tool returns and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The trajectory structure encoding module is used to perform structured encoding on the execution trajectory to obtain a structured trajectory representation; The risk diagnosis module is used to generate risk diagnosis results based on structured trajectory representation; The risk quantification module is used to combine risk diagnosis results with high-risk context factors to calculate a comprehensive risk score; the high-risk context factors are used to measure the degree of danger of the agent's current execution context, and the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory; The safety decision module is used to make safety decisions about the actions to be performed by the agent based on the comprehensive risk score.
[0112] like Figure 4 As shown, the agent safety protection system based on multi-dimensional diagnostics of execution trajectory includes a safety protection system deployed between the agent execution environment and the external tool interface, as well as the external tool / environment. User task input first enters the agent execution environment. During execution, the safety protection system collects, encodes, diagnoses, and makes decisions on the execution trajectory. For permitted actions, the safety protection system calls the external tool / environment through the tool call interface and sends the tool return / environment feedback returned by the external tool / environment back to the safety protection system and the agent execution environment.
[0113] The internal modules of the security protection system include an execution trajectory acquisition module, a trajectory structured encoding module, a multi-dimensional risk diagnosis module, a dynamic security decision module (integrating the risk quantification module and the security decision module), and an evidence archiving and reporting module. The execution trajectory acquisition module is responsible for collecting user input, task planning, tool calls, tool returns, environmental feedback, and final output to form the execution trajectory; the trajectory structured encoding module is responsible for structured encoding of the execution trajectory to obtain a structured trajectory representation; the multi-dimensional risk diagnosis module is responsible for generating diagnostic results from three dimensions: risk source, failure mode, and actual hazard; the dynamic security decision module is responsible for making decisions on allowing execution, manual review, or blocking execution based on the comprehensive risk score; the evidence archiving and reporting module is responsible for saving complete execution trajectories, risk event nodes, diagnostic tags, risk scores, security decisions, tool parameters, model versions, and rule versions for each risk diagnosis, forming a replayable and auditable security report.
[0114] In an exemplary embodiment, user input, task planning, tool invocation, tool return, and environmental feedback are collected during the execution of a task by the intelligent agent and arranged in chronological order to form an execution trajectory, including: User input, task planning, tool invocation, tool return, and environmental feedback are recorded as execution event nodes respectively; Arrange the execution event nodes in chronological order to form an execution trajectory; Among them, user input refers to the content entered by the user in the current stage, environmental feedback refers to environmental input or external observation content, task planning refers to the agent's task planning, intermediate inference state or action intention, tool invocation refers to the agent's preparation to execute or already executed tool invocation, external interface invocation, file operation, database query, code execution or message sending action, and tool return refers to the tool return result, web page content, file content or external system response.
[0115] In an exemplary embodiment, the execution trajectory is structured and encoded to obtain a structured trajectory representation, including: The semantic information, action information, and context information of each execution event node are encoded to obtain the local trajectory representation of each execution event node; The local trajectory representations are aggregated to obtain a structured trajectory representation.
[0116] In one exemplary embodiment, a risk diagnosis result is generated based on a structured trajectory representation, including: Based on structured trajectory representation, identify the stage in the execution trajectory where the risk source is located and generate risk source diagnostic labels; Identify how risks occur and generate failure mode diagnostic labels; Identify the potential consequences of risks and generate realistic hazard diagnostic labels; By combining risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels, risk diagnostic results are generated.
[0117] In one exemplary embodiment, a comprehensive risk score is calculated by combining the risk diagnosis results with high-risk contextual factors, including: The tool risk factor is calculated using the tool attributes in the execution trajectory; the tool risk factor is used to measure the degree of danger of the current tool itself. The data sensitivity factor is calculated using the inputs, outputs, or tool-returned content in the execution trajectory; the data sensitivity factor is used to measure whether the content involves private data, keys, contracts, financial information, or internal documents; The permission out-of-bounds factor is calculated by comparing the actions in the execution trajectory with the scope of user authorization and system permissions. The permission out-of-bounds factor is used to measure whether the current action exceeds the scope of user authorization or system permissions. The write operation risk factor is calculated based on the action type in the execution trajectory. The write operation risk factor is used to measure whether the current action involves external influencing behaviors such as modification, deletion, submission, payment, or code execution. The external propagation risk factor is calculated by using the action destination in the execution trajectory; the external propagation risk factor is used to measure whether there are outward propagation behaviors such as sending emails, uploading files or calling external interfaces. By integrating tool risk factors, data sensitivity factors, permission overstepping factors, write operation risk factors, and external propagation risk factors, high-risk context factors are obtained. The confidence levels corresponding to the risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels are weighted and integrated with high-risk context factors to obtain a comprehensive risk score.
[0118] In one exemplary embodiment, a safety decision is made on the action to be performed by the agent based on a comprehensive risk score, including: Compare the overall risk score with the first threshold; When the overall risk score is less than the first threshold, the action is allowed to continue. When the comprehensive risk score is not less than the first threshold and is less than the second threshold, the action is suspended and submitted for manual review. When the overall risk score is not less than the second threshold, the blocking action is executed and a risk report is generated. The second threshold is greater than the first threshold.
[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0120] Reference Figure 5 This invention also provides a computer device, including: a memory and a processor, and a computer program stored in the memory. When the computer program is executed on the processor, it performs the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0121] The computer device may be a desktop computer, laptop, handheld computer, or cloud server, etc. This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. They may include more or fewer components than shown in the illustration, or combinations of certain components, or different components. For example, they may also include input / output devices, network access devices, etc.
[0122] The processor referred to can be a Central Processing Unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0123] In some embodiments, the memory may be an internal storage unit of the computer device, such as a hard drive or RAM. In other embodiments, the memory may be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory may include both internal and external storage units of the computer device. The memory is used to store the operating system, applications, boot loader, data, and other programs, such as the program code of the computer program. The memory can also be used to temporarily store data that has been output or will be output.
[0124] This invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0125] In this embodiment, if the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.
[0126] This invention provides a computer program product, including a computer program that, when executed by a processor, performs the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Risk diagnosis results are generated based on structured trajectory representation; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk context factors. High-risk context factors are used to measure the degree of danger of the agent's current execution context, while the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the actions to be performed by the agent.
[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0129] In the embodiments disclosed in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for protecting the security of intelligent agents based on multi-dimensional diagnosis of execution trajectories, characterized in that, Includes the following steps: Collect user input, task planning, tool calls, tool returns, and environmental feedback during the execution of a task by the intelligent agent, and arrange them in chronological order to form an execution trajectory; The execution trajectory is structured and encoded to obtain a structured trajectory representation; Based on the structured trajectory representation, risk diagnosis results are generated; A comprehensive risk score is calculated by combining the risk diagnosis results with high-risk contextual factors; The high-risk context factor is used to measure the degree of danger of the agent's current execution context, and the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. Based on the comprehensive risk score, a safety decision is made for the action to be performed by the agent.
2. The intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory according to claim 1, characterized in that, Collect user input, task planning, tool calls, tool returns, and environmental feedback during a task execution process, and arrange them in chronological order to form an execution trajectory, including: The user input, task planning, tool invocation, tool return, and environmental feedback are recorded as execution event nodes respectively; The execution event nodes are arranged in chronological order to form the execution trajectory; Wherein, the user input is the content input by the user in the current stage, the environmental feedback is the environmental input or external observation content, the task planning is the agent's task planning, intermediate inference state or action intention, the tool call is the tool call, external interface call, file operation, database query, code execution or message sending action that the agent is preparing to execute or has already executed, and the tool return is the tool return result, web page content, file content or external system response.
3. The intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory according to claim 2, characterized in that, The execution trajectory is structured and encoded to obtain a structured trajectory representation, including: The semantic information, action information, and context information of each execution event node are encoded to obtain a local trajectory representation of each execution event node; The local trajectory representations are aggregated to obtain the structured trajectory representation.
4. The intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory according to claim 1, characterized in that, Based on the structured trajectory representation, risk diagnosis results are generated, including: Based on the structured trajectory representation, the stage in which the risk source is located within the execution trajectory is identified, and a risk source diagnostic label is generated; Identify how risks occur and generate failure mode diagnostic labels; Identify the potential consequences of risks and generate realistic hazard diagnostic labels; The risk diagnosis result is generated by combining the risk source diagnosis label, the failure mode diagnosis label, and the actual hazard diagnosis label.
5. The intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory according to claim 4, characterized in that, Based on the risk diagnosis results and high-risk contextual factors, a comprehensive risk score is calculated, including: Using the tool attributes in the execution trajectory, a tool risk factor is calculated; the tool risk factor is used to measure the degree of danger of the current tool itself. A data sensitivity factor is calculated using the inputs, outputs, or tool-returned content in the execution trajectory; the data sensitivity factor is used to measure whether the content involves privacy data, keys, contracts, financial information, or internal documents; The permission out-of-bounds factor is calculated by comparing the actions in the execution trajectory with the user authorization and system permission range; the permission out-of-bounds factor is used to measure whether the current action exceeds the user authorization or system permission range. The write operation risk factor is calculated based on the action type in the execution trajectory; the write operation risk factor is used to measure whether the current action involves external influencing behaviors such as modification, deletion, submission, payment, or code execution. The external propagation risk factor is calculated based on the action trajectory; the external propagation risk factor is used to measure whether there are outward propagation behaviors such as sending emails, uploading files, or calling external interfaces. The high-risk context factor is obtained by fusing the tool risk factor, data sensitivity factor, permission violation factor, write operation risk factor, and external propagation risk factor. The confidence levels corresponding to the risk source diagnostic labels, failure mode diagnostic labels, and actual hazard diagnostic labels are weighted and fused with the high-risk context factors to obtain the comprehensive risk score.
6. The intelligent agent security protection method based on multi-dimensional diagnosis of execution trajectory according to claim 1, characterized in that, Based on the comprehensive risk score, a safety decision is made for the action to be performed by the agent, including: Compare the comprehensive risk score with the first threshold; When the overall risk score is less than the first threshold, the action is allowed to continue. When the comprehensive risk score is not less than the first threshold and is less than the second threshold, the action is suspended and submitted for manual review. When the comprehensive risk score is not less than the second threshold, the action is blocked and a risk report is generated. Wherein, the second threshold is greater than the first threshold.
7. A smart agent security protection system based on multi-dimensional diagnosis of execution trajectory, characterized in that, include: The execution trajectory acquisition module is used to collect user input, task planning, tool calls, tool returns and environmental feedback of the intelligent agent during a task execution process, and arrange them in chronological order to form an execution trajectory; The trajectory structure encoding module is used to perform structure encoding on the execution trajectory to obtain a structured trajectory representation; The risk diagnosis module is used to generate risk diagnosis results based on the structured trajectory representation; The risk quantification module is used to calculate a comprehensive risk score by combining the risk diagnosis results with high-risk context factors. The high-risk context factor is used to measure the degree of danger of the agent's current execution context, and the comprehensive risk score is used to quantitatively characterize the overall risk level of the execution trajectory. The safety decision module is used to make safety decisions on the actions to be performed by the agent based on the comprehensive risk score.
8. A computer device, characterized in that, The device includes a processor and a memory: The memory is used to store computer programs and send the instructions of the computer programs to the processor; The processor executes, according to the instructions of the computer program, a smart agent security protection method based on multidimensional diagnosis of execution trajectory as described in any one of claims 1-6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements an intelligent agent security protection method based on multidimensional diagnosis of execution trajectory as described in any one of claims 1-6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements an intelligent agent security protection method based on multidimensional diagnosis of execution trajectory as described in any one of claims 1-6.
