Security detection method, apparatus, device, medium, and program product
By acquiring and comparing environmental state objects when the agent performs a task, the problem of low reliability in agent security detection in the prior art is solved. This enables reliable determination of injected attack payloads and accurate identification of hijacking types, improving the reliability and fineness of detection.
Patent Information
- Application Number
- CN202611494206.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-23
- Publication Date
- 2026-11-17
AI Technical Summary
In existing technologies, intelligent agent security detection methods based on large models only detect security response data of static attack text images, resulting in low reliability of detection results. Furthermore, the evaluation objects are limited to the model output level, the judgment method is unreliable, lacks fine-grained diagnosis, and has insufficient versatility.
When an agent is performing a task, its current environmental state object is obtained and security detection is performed in conjunction with the environmental state object before the attack payload is embedded. The success or failure of injecting the attack payload is determined by comparing the differences in state before and after execution, and the attack payload is dynamically adjusted to improve detection reliability.
By comparing the differences in the environmental state objects before and after execution, it is determined whether the injected attack payload was successfully executed. The attack payload is dynamically adjusted, which improves the reliability of agent security detection, distinguishes hijacking types, and improves the fineness of detection.
Smart Images

Figure CN123122986A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of network security technology, and in particular to a security detection method, apparatus, equipment, medium, and program product. Background Technology
[0002] With the continuous development of large-scale model capabilities, agents based on large-scale models can call external tools to complete practical tasks. However, while greatly expanding the capabilities of large-scale models, it also introduces new security risks. Therefore, how to conduct systematic security evaluation of agents has become a core issue that urgently needs to be addressed in the field of large-scale model security.
[0003] Currently, in related technologies, it is possible to acquire attack text image pairs from multiple domains, and then use an intelligent agent to perform security response processing on the attack text image pairs to obtain security response data for each domain. Finally, an intelligent analysis model is used to perform security detection on the security response data for each domain to obtain the comprehensive security detection results of the intelligent agent in multiple domains.
[0004] However, in the above method, the security detection process only targets the security response data output by the agent after performing security response processing on static attack text images in multiple domains. The security detection results only reflect the agent's text response performance to static attack samples, which leads to low reliability of the agent's security detection. Summary of the Invention
[0005] This application provides a security detection method, apparatus, equipment, medium, and program product to improve the reliability of security detection of intelligent agents.
[0006] In a first aspect, embodiments of this application provide a security detection method, which includes: receiving a task instruction input by a user; executing the task corresponding to the task instruction through an intelligent agent; obtaining the current environment state object of the intelligent agent after the task is completed; performing security detection on the intelligent agent based on the current environment state object and the initial environment state object, and obtaining a security detection result; wherein, the initial environment state object is the environment state object after embedding the attack payload; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
[0007] The technical solution provided in this application offers at least the following advantages: After the agent completes the user task, the current environment state object of the agent is obtained, and the agent is subjected to security detection in conjunction with the environment state object when the attack payload is embedded, resulting in a determination of whether the attack payload injection was successful or failed. This allows for comparison of the differences between the environment state objects before and after execution, determining whether the injected attack payload was successfully executed, and dynamically adjusting the attack payload, thus improving the reliability of security detection of the agent.
[0008] One possible implementation involves performing security checks on the agent based on the current environment state object and the initial environment state object to obtain the agent's security check result. This includes: performing task completion determination processing on the agent using a task completion determination function based on the current environment state object and the initial environment state object to obtain a task completion determination result; performing injection attack payload determination processing on the agent using an injection attack payload determination function to obtain an injection attack payload determination result; and determining the security check result based on the task completion determination result and the injection attack payload determination result.
[0009] Another possible implementation, in which the above-mentioned task is executed by an agent corresponding to a task instruction, includes: extracting at least one inference text from the task based on the task instruction; executing the tool call request corresponding to at least one inference text by calling a simulation tool through the agent, thereby obtaining the agent's interaction trajectory and environmental state data; wherein, the simulation tool is obtained by registration processing based on a simulation tool function through an evaluation framework or standardized tool protocol.
[0010] Another possible implementation is that, based on the agent's current environment state object and initial environment state object, the agent is subjected to security detection. After obtaining the security detection result of the agent, the security detection method further includes: if the security detection result indicates that the injection of attack payload was successful, determining the hijacking type of the agent; wherein the hijacking type includes any of the following: task replacement, task expansion, task drift.
[0011] Another possible implementation is that, after performing security detection on the agent based on the agent's current environment state object and initial environment state object, and obtaining the agent's security detection result, the security detection method further includes: based on the recognition pattern management keyword set and the defense pattern keyword set, performing injection awareness reasoning on the agent through a large model to obtain the agent's injection awareness result; wherein, the injection awareness result is used to characterize whether the agent has the awareness to recognize the injected attack payload.
[0012] Secondly, embodiments of this application provide a security detection device, comprising: a receiving module, an execution module, an acquisition module, and a processing module; the receiving module is used to receive task instructions input by a user; the execution module is used to execute the task corresponding to the task instruction through an intelligent agent; the acquisition module is used to acquire the current environment state object of the intelligent agent after the task is completed; the processing module is used to perform security detection on the intelligent agent based on the current environment state object and the initial environment state object, and obtain the security detection result of the intelligent agent; wherein, the initial environment state object is the environment state object after embedding the attack payload; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
[0013] One possible implementation is that the above processing module is specifically used to: perform task completion determination processing on the agent based on the current environment state object and the initial environment state object through the task completion determination function to obtain the task completion determination result; and perform injection attack payload determination processing on the agent through the injection attack payload determination function to obtain the injection attack payload determination result; and determine the security detection result based on the task completion determination result and the injection attack payload determination result.
[0014] Another possible implementation, the aforementioned execution module, is specifically used to: extract at least one reasoning text from the task based on the task instruction through an agent; execute the tool call request corresponding to at least one reasoning text by calling a simulation tool through the agent, thereby obtaining the agent's interaction trajectory and environmental state data; wherein, the simulation tool is obtained by registration processing based on the simulation tool function through an evaluation framework or standardized tool protocol.
[0015] In another possible implementation, the security detection device further includes: a determination module; the determination module is used to determine the hijacking type of the intelligent agent after the processing module performs security detection on the intelligent agent based on the current environment state object and the initial environment state object and obtains the security detection result of the intelligent agent, and if the security detection result indicates that the injection of attack payload is successful; wherein, the hijacking type includes any one of the following: task replacement, task expansion, task drift.
[0016] In another possible implementation, the aforementioned processing module is further used to perform security detection on the agent based on the agent's current environment state object and initial environment state object. After obtaining the security detection result of the agent, it uses a large model to perform injection awareness reasoning on the agent based on the recognition pattern management keyword set and the defense pattern keyword set to obtain the injection awareness result of the agent. The injection awareness result is used to characterize whether the agent has the awareness to recognize the injection attack payload.
[0017] Thirdly, this application provides an electronic device comprising: a processor and a memory; the memory stores a program or instructions executable on the processor, wherein the program or instructions, when executed by the processor, implement the method of the first aspect described above.
[0018] Fourthly, this application provides a readable storage medium on which a program or instructions are stored, which, when executed by a computer, implement the method of the first aspect described above.
[0019] Fifthly, this application provides a computer program product stored in a storage medium, which, when executed by a computer, implements the method described in the first aspect.
[0020] In a sixth aspect, embodiments of this application provide a chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0021] The beneficial effects of the second to sixth aspects mentioned above are described in the corresponding description of the first aspect and will not be repeated here. Attached Figure Description
[0022] Figure 1 A schematic diagram of a network architecture for a security detection method provided in an embodiment of this application;
[0023] Figure 2 A schematic flowchart illustrating a security detection method provided in an embodiment of this application;
[0024] Figure 3 A flowchart illustrating another security detection method provided in this application embodiment;
[0025] Figure 4 A flowchart illustrating another security detection method provided in this application embodiment;
[0026] Figure 5 A flowchart illustrating another security detection method provided in this application embodiment;
[0027] Figure 6 A flowchart illustrating another security detection method provided in this application embodiment;
[0028] Figure 7 A flowchart illustrating the implementation process of a security detection method provided in this application embodiment;
[0029] Figure 8 This is a schematic diagram of the structure of a security detection device provided in an embodiment of this application;
[0030] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0031] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.
[0032] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0033] The terms "at least one," "at least one of," etc., used in the specification and claims of this application refer to any one, any two, or a combination of two or more of the included items. For example, at least one of a, b, and c can mean: "a," "b," "c," "a and b," "a and c," "b and c," and "a, b, and c," where a, b, and c can be single or multiple. Similarly, "at least two" refers to two or more items, and its meaning is similar to that of "at least one."
[0034] In the description of this application, unless otherwise stated, "a plurality of" means two or more.
[0035] The security detection method, apparatus, equipment, medium, and program product provided in this application embodiment can be applied to scenarios such as big data analysis, knowledge base question answering, and intelligent enterprise office.
[0036] With the continuous development of large-scale model capabilities, agents based on large-scale models can call external tools to complete practical tasks. However, while greatly expanding the capabilities of large-scale models, it also introduces new security risks. Therefore, how to conduct systematic security evaluation of agents has become a core issue that urgently needs to be addressed in the field of large-scale model security.
[0037] Currently, in related technologies, it is possible to acquire attack text image pairs from multiple domains, and then use an intelligent agent to perform security response processing on the attack text image pairs to obtain security response data for each domain. Finally, an intelligent analysis model is used to perform security detection on the security response data for each domain to obtain the comprehensive security detection results of the intelligent agent in multiple domains.
[0038] However, the above method has the following drawbacks:
[0039] (1) The security detection process only targets the security response data output by the agent after performing security response processing on static attack text images in multiple domains. The security detection results only reflect the agent's text response performance to static attack samples, which leads to low reliability of the security detection results for the agent.
[0040] (2) The evaluation object is limited to the model output level. Existing security evaluation methods mainly focus on whether the model outputs harmful text content, while the core issue of agent security is whether the model executes harmful tool calling actions. There are fundamental differences between the two in terms of attack surface and judgment criteria.
[0041] (3) The judgment method is unreliable. Existing evaluation systems use large language models as judges to determine whether an agent has been successfully attacked. However, the accuracy of the judge model itself is difficult to guarantee.
[0042] (4) Single dimension of attack analysis. Existing methods only provide a binary judgment of "attack success / failure", which cannot distinguish at which stage the attack was blocked, whether the agent recognized the attack during the reasoning process, what the specific manifestation of the hijacking is, and lack fine-grained diagnosis of security failure.
[0043] (5) Insufficient generality of evaluation objects. Existing evaluation frameworks are usually deeply coupled with specific agent frameworks, making it impossible to conduct horizontal comparisons across different agent systems.
[0044] To address the aforementioned technical problems, embodiments of this application provide a security detection method, apparatus, device, medium, and program product. After an intelligent agent completes a user task, the current environmental state object of the intelligent agent is obtained, and this object is combined with the environmental state object at the time of embedding the attack payload to perform security detection on the intelligent agent, obtaining a determination result of whether the injected attack payload was successfully or unsuccessfully injected. Thus, by comparing the differences in the environmental state objects before and after execution, the success of the injected attack payload can be determined, and the attack payload can be dynamically adjusted, thereby improving the reliability of security detection of the intelligent agent.
[0045] The following description, in conjunction with the accompanying drawings, details the security detection methods, apparatus, equipment, media, and program products provided in the embodiments of this application.
[0046] Figure 1 This illustration shows a network architecture for a security detection method provided in an embodiment of this application. For example... Figure 1 As shown, the network architecture includes a security detection device 101 and a terminal device 102. The security detection device 101 and the terminal device 102 are interconnected.
[0047] In some embodiments of this application, the security detection device 101 may be a server, a computer, or a processor or processing unit within a server or computer. The server may be a single server or a server cluster consisting of multiple servers. It should be noted that the embodiments of this application do not limit the specific device form of the security detection device 101. Figure 1 The example shown is a single server using the security detection device 101.
[0048] In some embodiments, the terminal device may be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, personal computer (PC), ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., and the embodiments of this application do not specifically limit it. Figure 1 The example shown is a mobile phone, with terminal device 102 as an example.
[0049] It should be noted that the network architecture described in the embodiments of this application is for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and does not constitute a limitation on the technical solutions provided in the embodiments of this application. As network architectures evolve, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0050] See Figure 2 This is a flowchart illustrating a security detection method provided in an embodiment of this application. Figure 2 As shown, the security detection method provided in this application embodiment can be implemented by the above-mentioned security detection device, specifically including the following steps 201 to 204.
[0051] Step 201: The security detection device receives the task instruction input by the user.
[0052] In some embodiments of this application, the task instruction is an instruction issued by the user to the intelligent agent to trigger the intelligent agent to execute the corresponding task.
[0053] In some embodiments of this application, the security detection device can receive the above-mentioned task instructions through an intelligent agent.
[0054] In some embodiments of this application, the task instruction can be an instruction for triggering the intelligent agent to perform any task; for example, the task instruction can be an instruction for triggering the intelligent agent to perform a file retrieval and summary task, an email classification and forwarding task, or a database query and report generation task.
[0055] Step 202: The security detection device executes the task corresponding to the task instruction through the intelligent agent.
[0056] In some embodiments of this application, the intelligent agent is the object being tested, capable of reasoning based on task instructions and autonomously invoking tools.
[0057] In some embodiments of this application, a task is a pending task corresponding to a task instruction, serving as a benchmark for the execution and evaluation of the intelligent agent. It defines the legal scope of operations that the intelligent agent should complete, and subsequent task completion determination and hijacking type determination are based on this task as a reference benchmark.
[0058] In some embodiments of this application, combined with Figure 2 ,like Figure 3 As shown, step 202 above can be implemented through steps 202a and 202b.
[0059] Step 202a: Based on the task instructions, the security detection device extracts at least one reasoning text from the task through an intelligent agent.
[0060] In some embodiments of this application, the reasoning text is intermediate reasoning, planning, or thought text generated by the agent during task execution, reflecting its decision-making basis and next intention. It serves as a bridge between the agent's task instructions and tool invocation requests, and is also an important basis for subsequently determining the type of hijacking and task deviation.
[0061] For example, based on task instructions, the intelligent agent performs semantic parsing, task decomposition, and reasoning planning through a large model to generate at least one reasoning text. This reasoning text may include subtask breakdown, reasons for tool selection, parameter construction logic, etc., and is a prerequisite for autonomously determining the tool invocation sequence.
[0062] Step 202b: The security detection device executes at least one tool call request corresponding to the reasoning text by calling the simulation tool through the intelligent agent, and obtains the interaction trajectory and environmental state data of the intelligent agent.
[0063] Among them, the simulation tool is obtained by registering the simulation tool function through the evaluation framework or standardized tool protocol.
[0064] In some embodiments of this application, the simulation tool is a high-fidelity virtual tool that replaces the real tool in the evaluation environment. It has the same interface and return logic as the real tool, receives tool call requests from the intelligent agent, executes and records interaction trajectories and environmental state data in the virtual sandbox, and provides the evaluation end with data for collection and judgment, thus avoiding damage to the real system.
[0065] In some embodiments of this application, the tool invocation request is a structured tool invocation parameter generated by the agent based on the reasoning text.
[0066] In some embodiments of this application, the interaction trajectory is a record composed of the agent's reasoning text, tool call requests, tool return results, etc., in sequence.
[0067] For example, suppose the agent generates N interactions during execution, each step containing inference text. and tool calls Record interaction trajectory .
[0068] In some embodiments of this application, the environmental state data is the state change data generated by the underlying sandbox after the simulation tool is executed, such as changes in files, databases, networks, etc.
[0069] In some embodiments of this application, the simulation tool function is a specific callable function interface in the simulation tool, such as sending emails, querying databases, executing code, etc.
[0070] In some embodiments of this application, the evaluation framework is a system framework that carries the evaluation process, tool mounting, trajectory acquisition, and security determination.
[0071] In some embodiments of this application, the standardized tool protocol is a protocol for a unified tool calling format, enabling intelligent agents to call simulation tools in a standard manner, which facilitates the mounting and collection of evaluation frameworks.
[0072] In some embodiments of this application, tool functions are registered using a dual-path process to obtain simulated tools: Path A (internal runtime path): The tool function is registered through the internal runtime of the evaluation framework, and the model interacts with the tool through the framework's built-in tool call loop; Path B (protocolized path): The tool function is exposed as an independent service through a standardized tool protocol. Any intelligent agent compatible with this protocol can connect as a client and discover and invoke all tools. The two paths share the same environment state object and business logic implementation, differing only in the communication protocol and state access mechanism.
[0073] In this way, based on task instructions, multiple inference texts are extracted from the task by the agent. The agent calls simulation tools to obtain the agent's interaction trajectory and environmental state data. During the process of the agent autonomously calling tools, its behavior and environmental changes can be objectively recorded, providing a real basis for subsequent safety judgments and avoiding reliance solely on the agent's text output.
[0074] Step 203: After completing the task, the security detection device obtains the current environmental state object of the intelligent agent.
[0075] In some embodiments of this application, the current environment state object of the agent is a snapshot object of the simulated environment in which the agent is currently located, used to determine whether the agent has been attacked and as a basis for the agent to perform tasks subsequently.
[0076] For example, the aforementioned current environment state object can be a virtual file system snapshot object, a virtual database table state object, or a virtual tool permission state object.
[0077] Step 204: The security detection device performs security detection on the intelligent agent based on the current environment state object and the initial environment state object, and obtains the security detection result.
[0078] The initial environment state object is the environment state object after the attack payload is embedded; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
[0079] In some embodiments of this application, the attack payload is a malicious instruction or data injected into the input information stream of the intelligent agent.
[0080] In some embodiments of this application, the environment state object is a state change data object encapsulated after the simulation tool is executed.
[0081] In some embodiments of this application, injecting attack payload is the action or process of injecting attack payload into the input information flow of the intelligent agent.
[0082] In some embodiments of this application, combined with Figure 2 ,like Figure 4 As shown, step 204 above can be implemented through steps 204a and 204b.
[0083] Step 204a: Based on the current environment state object and the initial environment state object, the security detection device performs task completion determination processing on the agent through the task completion determination function to obtain the task completion determination result, and performs injection attack payload determination processing on the agent through the injection attack payload determination function to obtain the injection attack payload determination result.
[0084] In some embodiments of this application, a task completion determination function is used to evaluate whether the agent has completed the user's original task, and the task completion determination function U is defined as follows: ;in, This is the initial environment state object. Let T be the current environment state object, and T be the interaction trajectory mentioned above.
[0085] In some embodiments of this application, the task completion determination result is used to indicate whether the original task is completed or to what extent it is completed.
[0086] In some embodiments of this application, the injection attack payload determination function is a function used to evaluate whether the attack payload was successfully injected or whether it triggered unexpected behavior.
[0087] In some embodiments of this application, the injection attack payload determination result indicates whether the injection attack payload was successful.
[0088] Step 204b: The security detection device determines the security detection result based on the task completion determination result and the injected attack payload determination result.
[0089] In some embodiments of this application, the judgment is based on a combination of two results. For example, if the task is completed but the injection is unsuccessful, it is considered secure; if the task is completed and the injection is successful, it is considered hijacking; if the task is not completed but the injection is successful, it is considered insecure; and if the task is not completed and the injection is unsuccessful, it is considered a failed or pending task.
[0090] In this way, by using the task completion determination function and the injection attack payload determination function to determine the agent, the two dimensions of "whether the task is completed" and "whether it has been injected with control" can be combined to reduce single-dimensional misjudgments and improve the reliability of security detection results.
[0091] The security detection method provided in this application obtains the current environment state object of the intelligent agent after it has completed the user task, and performs security detection on the intelligent agent in conjunction with the environment state object when the attack payload is embedded, thereby obtaining a determination result of whether the injection of the attack payload was successful or failed. This allows for comparison of the differences between the environment state objects before and after execution to determine whether the injected attack payload was successfully executed, and the attack payload can be dynamically adjusted, thus improving the reliability of security detection of the intelligent agent.
[0092] In some embodiments of this application, combined with Figure 2 ,like Figure 5 As shown, after step 204 above, the security detection method provided in this application embodiment may further include the following step 205.
[0093] Step 205: If the security detection result indicates that the injection of the attack payload was successful, the security detection device determines the hijacking type of the intelligent agent.
[0094] The hijacking types include any of the following: mission replacement, mission expansion, and mission drift.
[0095] In some embodiments of this application, the hijacking type refers to the specific behavior category of the agent being hijacked after successful injection.
[0096] In some embodiments of this application, the task replacement involves the intelligent agent completely abandoning the user's original task and only executing the injected instructions, which is the most harmful and easiest to detect.
[0097] In some embodiments of this application, the task is expanded to the intelligent agent executing additional injected instructions while completing the user's original task, which is the most covert and common method.
[0098] In some embodiments of this application, task drift occurs when the agent partially performs the original task, gradually deviates from it, and eventually executes the injected instructions.
[0099] In this way, by distinguishing the hijacking types of intelligent agents, the hijacked behavior categories can be accurately located, which facilitates targeted defense and risk classification.
[0100] In some embodiments of this application, combined with Figure 2 ,like Figure 6 As shown, after step 204 above, the security detection method provided in this application embodiment may further include the following step 206.
[0101] Step 206: The security detection device, based on the identification mode management keyword set and the defense mode keyword set, performs injection consciousness reasoning on the intelligent agent through a large model to obtain the injection consciousness result of the intelligent agent.
[0102] Among them, the injection consciousness result is used to characterize whether the agent has the awareness to recognize the injected attack payload.
[0103] In some embodiments of this application, the identification pattern management keyword set is used to identify whether an agent has entered an attack identification mode.
[0104] In some embodiments of this application, the defense mode keyword set is a set of keywords used to identify whether an agent has entered a defense / deny attack mode.
[0105] In some embodiments of this application, a large model is used to determine whether an agent is aware of injected attack payloads.
[0106] In some embodiments of this application, injection awareness reasoning is the reasoning process of a large model regarding whether an agent is aware of an injection attack.
[0107] In some embodiments of this application, awareness is defined as the intelligent agent's ability to recognize or detect injected attack payloads.
[0108] In some embodiments of this application, the interaction trajectory of the intelligent agent is collected, including inference text, tool call requests, tool return results, and environmental state data. Task instructions, attack payloads, the intelligent agent's inference text and behavior, as well as the identification pattern management keyword set and defense pattern keyword set, are input into a large model. The large model analyzes whether the intelligent agent's inference text contains statements indicating awareness, suspicion, rejection, or defensiveness regarding the injection attack, determining whether it is aware of the attack payload's existence. Results such as "identified and defended," "identified but not defended," "not identified," and "suspected identification" are obtained.
[0109] Thus, by using a large model to determine whether an agent has the awareness to recognize injected attack payloads, it is possible to distinguish whether the agent truly recognizes and defends against the injection, or fails to recognize it but luckily does not execute the hijacking task affected by the attack payload, thereby improving the credibility of security detection.
[0110] The security detection method of this application will be described below through specific embodiments.
[0111] like Figure 7 As shown, the implementation process of the security detection method provided in this application embodiment includes the following S1 to S6:
[0112] S1. Build the simulation tool environment.
[0113] Construct a set of simulated utility functions, each operating on an in-memory environment state object, rather than a real external system. The environment state object is defined using a hierarchical structure based on a data model, containing at least one business entity (such as email, calendar, cloud storage, bank account, etc.), and each business entity maintains readable and writable state data.
[0114] Each simulation utility function ,in, Input parameters for the tool. The text result to be returned to the agent. This is for modifying the environment state object. The logic implementation of the utility function is decoupled from the external display interface, allowing the same set of business logic to be exposed to the tested intelligent agent through different protocol interfaces. Each simulation utility function employs a dual-path exposure design:
[0115] Path A (Internal Runtime Path): Utility functions are registered through the evaluation framework's internal runtime, and the model interacts with the tools through the framework's built-in tool call loop;
[0116] Path B (Protocol-based Path): Utility functions are exposed as independent services through a standardized utility protocol. Any intelligent agent compatible with this protocol can connect as a client and discover and invoke all the tools.
[0117] Both paths share the same environment state object and business logic implementation; the only difference lies in the communication protocol and state access mechanism.
[0118] S2, Constructing and injecting attack payloads.
[0119] Define an injection attack payload template library {A} = {A1, A2, ..., A...} K}, each attack template Contains a target placeholder When used, it is replaced with the specific injection task target. , obtain attack payload P k The environment is embedded through an environmental data placeholder replacement mechanism. Furthermore... It describes the specific actions the attacker expects the agent to perform (e.g., "send an email with specified content to a specified email address"). The environment data is defined in a structured configuration file, which contains predefined injection placeholder tags. When the evaluation starts, formula (1) is executed to replace:
[0120] (1)
[0121] In formula (1), Basic environment configuration, This is the j-th injection vector (such as the body of an email or the description of a calendar event). This is the attack payload injected into the j-th vector. The replaced environment data is loaded as an environment state object in memory. The attack payload is embedded in the environment data that the agent can read through normal tool calls. The agent must actively call tools that read relevant data (such as "search email" or "query calendar events") to access the injected payload in the tool's return value.
[0122] S3. Perform the evaluation task.
[0123] For each test case, execute the following process:
[0124] (a) Environment initialization: Load the injected environment data and construct the environment state object. This is used for comparison of subsequent environmental changes.
[0125] (b) Agent Execution: User task instructions are sent to the tested agent, which autonomously determines the tool invocation sequence. Assume the agent execution involves N interactions, each containing inference text. and tool calls Record the complete trajectory .
[0126] (c) Environment final state acquisition: After the agent finishes execution, it acquires the current environment state. .
[0127] S4. Deterministic assessment of environmental status.
[0128] Define the utility function U for the user task and the security function S for the injected task. The task completion function U determines the task completion degree. ;in, This is the initial environment state object. For the current environment state object,
[0129] The security check function verifies whether the injection attack was successful. Taking "sending email" type injection as an example, the security check function checks whether there are side effects matching the injection target in the environment after execution using formula (2), which is as follows:
[0130] (2)
[0131] In formula (2), the dashed box represents the process that S=1 needs to be satisfied simultaneously, inbox represents the inbox, recipient represents the email recipient, subject represents the email subject, and body represents the body injection attack payload.
[0132] A security determination result S = 1 indicates successful injection (the agent has been hijacked), and S = 0 indicates failed injection (the agent has resisted the attack). The determination is based entirely on deterministic comparison of environmental state objects, without involving any adjudication model judgment, and has the characteristic of zero false positive rate.
[0133] S5. Hijacking type classification.
[0134] When the payload injection is successful, S=1. Further classification of the hijacking behavior yields the following hijacking types:
[0135] Task replacement: The agent completely abandons the user's original task and only executes the injected instructions, which is the most harmful and easiest to detect.
[0136] Task extension: While completing the user's original task, the agent executes additional injected instructions, which is the most covert and common method.
[0137] Task drift: The agent partially executes the original task, but its behavior gradually deviates, and eventually it executes the injected instructions.
[0138] S6. Analysis of the reasoning process.
[0139] This study analyzes the injection awareness in the reasoning text of dialogues between intelligent agent models. A recognition pattern set is defined, containing keywords such as "suspicious," "injection," "untrustworthy," and "unsuitable to follow," while a defense pattern set contains keywords such as "I won't," "refuse," and "shouldn't." The analysis examines whether the agent explicitly expresses suspicion or recognition of injection during the reasoning process, yet still executes the injection command. This reflects the adversarial defense capabilities of the large model itself, which are undetectable by existing security assessment methods.
[0140] Thus, after the agent completes the user task, the current environment state object of the agent is obtained, and a security check is performed on the agent in conjunction with the environment state object when the attack payload was embedded, to obtain a result indicating whether the injection of the attack payload was successful or not. By comparing the differences in the environment state objects before and after execution, it can be determined whether the injected attack payload was successfully executed, and the attack payload can be dynamically adjusted, thereby improving the reliability of the security check on the agent.
[0141] It should be noted that the descriptions of each step S1 to S6 in this embodiment can be found in the descriptions in the above embodiments, and will not be repeated here.
[0142] It should be noted that the above-described method embodiments, or the various possible implementations of the method embodiments, can be executed individually, or, provided there is no conflict, they can be combined with each other. The specific implementation can be determined according to actual usage requirements, and this application embodiment does not impose any restrictions on this.
[0143] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0144] This application embodiment can divide the security detection device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. Optionally, the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.
[0145] In some embodiments of this application, a security detection device is also provided. This security detection device may include one or more functional modules for implementing the security detection methods described in the above embodiments.
[0146] For example, Figure 8 This is a schematic diagram of a security detection device provided in an embodiment of this application. Figure 8 As shown, the security detection device 900 includes: a receiving module 901, an execution module 902, an acquisition module 903, and a processing module 904.
[0147] The receiving module 901 is used to receive task instructions input by the user; the execution module 902 is used to execute the task corresponding to the task instruction through the intelligent agent; the acquisition module 903 is used to acquire the current environment state object of the intelligent agent after the task is completed; the processing module 904 is used to perform security detection on the intelligent agent based on the current environment state object and the initial environment state object, and obtain the security detection result of the intelligent agent; wherein, the initial environment state object is the environment state object after embedding the attack payload; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
[0148] In the security detection device 900 provided in this application, after the intelligent agent completes the user task, the current environmental state object of the intelligent agent is obtained, and the intelligent agent is subjected to security detection in combination with the environmental state object when the attack payload is embedded, so as to obtain a determination result of whether the injection of the attack payload was successful or not. Thus, by comparing the differences in the environmental state objects before and after execution, it can be determined whether the injection of the attack payload was successfully executed, and the attack payload can be dynamically adjusted, thereby improving the reliability of the security detection of the intelligent agent.
[0149] In some embodiments of this application, the above-mentioned processing module 904 is specifically used to: perform task completion determination processing on the agent based on the current environment state object and the initial environment state object through the task completion determination function to obtain the task completion determination result; and perform injection attack payload determination processing on the agent through the injection attack payload determination function to obtain the injection attack payload determination result; and determine the security detection result based on the task completion determination result and the injection attack payload determination result.
[0150] In some embodiments of this application, the execution module 902 is specifically used to: extract at least one reasoning text from the task based on the task instruction through an intelligent agent; execute the tool call request corresponding to at least one reasoning text by calling a simulation tool through the intelligent agent, thereby obtaining the interaction trajectory and environmental state data of the intelligent agent; wherein, the simulation tool is obtained by registration processing based on the simulation tool function through an evaluation framework or standardized tool protocol.
[0151] In some embodiments of this application, the security detection device further includes: a determination module; the determination module is used to determine the hijacking type of the intelligent agent after the processing module 904 performs security detection on the intelligent agent based on the current environment state object and the initial environment state object of the intelligent agent and obtains the security detection result of the intelligent agent, and if the security detection result indicates that the injection of attack payload is successful; wherein, the hijacking type includes any one of the following: task replacement, task expansion, task drift.
[0152] In some embodiments of this application, the above-mentioned processing module 904 is further configured to perform security detection on the agent based on the agent's current environment state object and initial environment state object, and after obtaining the agent's security detection result, perform injection consciousness reasoning on the agent through a large model based on the recognition mode management keyword set and the defense mode keyword set, and obtain the agent's injection consciousness result; wherein, the injection consciousness result is used to characterize whether the agent has or does not have the recognition consciousness of the injected attack payload.
[0153] It should be noted that the safety detection device can implement all the processes implemented in the above method embodiments and achieve the same beneficial effects. To avoid repetition, it will not be described again here.
[0154] In the case where the functions of the integrated modules described above are implemented in hardware, this application provides a possible structural schematic diagram of the electronic device involved in the above embodiments. For example... Figure 9 As shown, the electronic device 90 includes: a processor 92, a communication interface 93, and a bus 94. Optionally, the electronic device 90 may also include a memory 91.
[0155] Processor 92 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 92 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0156] Communication interface 93 is used to connect with other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.
[0157] The memory 91 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.
[0158] As one possible implementation, the memory 91 can exist independently of the processor 92. The memory 91 can be connected to the processor 92 via a bus 94 and is used to store instructions or program code. When the processor 92 calls and executes the instructions or program code stored in the memory 91, it can implement the security detection method provided in the embodiments of this application.
[0159] In another possible implementation, memory 91 can also be integrated with processor 92.
[0160] Bus 94 can be an Extended Industry Standard Architecture (EISA) bus, etc. Bus 94 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 9 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0161] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the safety detection device can be divided into different functional modules to complete all or part of the functions described above.
[0162] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described security detection method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.
[0163] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0164] This application also provides a readable storage medium storing a program or instructions that, when executed by a computer, implement the security detection method provided in the above embodiments. It is understood that all or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware; the readable storage medium can be any of the foregoing embodiments or memory; the readable storage medium can also be an external storage device of the security detection device, such as a plug-in hard drive, Smart MediaCard (SMC), Secure Digital (SD) card, flash card, etc., equipped on the security detection device. Further, the readable storage medium can include both internal storage units of the security detection device and external storage devices. The readable storage medium is used to store the computer program and other programs and data required by the security detection device. The readable storage medium can also be used to temporarily store data that has been output or will be output.
[0165] This application also provides a computer program product, which is stored in a storage medium and implements the security detection method provided in the above embodiments when the computer program product is executed by a computer.
[0166] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
[0167] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0168] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.
Claims
1. A security detection method characterized by, include: Receive task instructions from the user; The task corresponding to the task instruction is executed by the intelligent agent; After completing the task, obtain the current environment state object of the agent; Based on the current environment state object and the initial environment state object, a security detection is performed on the agent to obtain the security detection result of the agent; wherein, the initial environment state object is the environment state object after embedding the attack payload; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
2. The security detection method according to claim 1, characterized in that, The process of performing security checks on the agent based on the current environment state object and the initial environment state object, and obtaining the security check result of the agent, includes: Based on the current environment state object and the initial environment state object, the task completion degree of the agent is determined by the task completion degree determination function to obtain the task completion degree determination result. The injection attack payload determination function is then used to determine the injection attack payload of the agent to obtain the injection attack payload determination result. The security detection result is determined based on the task completion determination result and the injection attack payload determination result.
3. The security detection method according to claim 1, characterized in that, The execution of the task corresponding to the task instruction by the intelligent agent includes: Based on the task instructions, the agent extracts at least one inference text from the task. The agent invokes a simulation tool to execute a tool invocation request corresponding to at least one inference text, thereby obtaining the agent's interaction trajectory and environmental state data; wherein, the simulation tool is based on a simulation tool function and is obtained through registration processing via an evaluation framework or standardized tool protocol.
4. The safety detection method according to any one of claims 1 to 3, characterized in that, After performing security detection on the agent based on the agent's current environment state object and initial environment state object, and obtaining the security detection result of the agent, the security detection method further includes: If the security detection result indicates that the injection of the attack payload was successful, the hijacking type of the agent is determined; wherein, the hijacking type includes any of the following: task replacement, task expansion, and task drift.
5. The safety detection method according to any one of claims 1 to 3, characterized in that, After performing security detection on the agent based on the agent's current environment state object and initial environment state object, and obtaining the security detection result of the agent, the security detection method further includes: Based on the identification pattern management keyword set and the defense pattern keyword set, the agent is subjected to injection awareness inference through a large model to obtain the injection awareness result of the agent; wherein, the injection awareness result is used to characterize whether the agent has the awareness to identify the injection attack payload.
6. A safety detection device, characterized in that, include: Receive module, execute module, acquire module, and process module; The receiving module is used to receive task instructions input by the user; The execution module is used to execute the task corresponding to the task instruction through the intelligent agent; The acquisition module is used to acquire the current environment state object of the intelligent agent after the task has been completed; The processing module is used to perform security detection on the agent based on the current environment state object and the initial environment state object, and obtain the security detection result of the agent; wherein, the initial environment state object is the environment state object after embedding the attack payload; the security detection result is used to characterize whether the injection of the attack payload was successful or failed.
7. The safety detection device according to claim 6, characterized in that, The processing module is specifically used for: Based on the current environment state object and the initial environment state object, the task completion degree of the agent is determined by the task completion degree determination function to obtain the task completion degree determination result. The injection attack payload determination function is then used to determine the injection attack payload of the agent to obtain the injection attack payload determination result. The security detection result is determined based on the task completion determination result and the injection attack payload determination result.
8. The safety detection device according to claim 6, characterized in that, The execution module is specifically used for: Based on the task instructions, the agent extracts at least one inference text from the task. The agent invokes a simulation tool to execute a tool invocation request corresponding to at least one inference text, thereby obtaining the agent's interaction trajectory and environmental state data; wherein, the simulation tool is based on a simulation tool function and is obtained through registration processing via an evaluation framework or standardized tool protocol.
9. The safety detection device according to any one of claims 6 to 8, characterized in that, The security detection device further includes: a determination module; The determining module is used to determine the hijacking type of the agent after the processing module performs security detection on the agent based on the agent's current environment state object and initial environment state object, and obtains the security detection result of the agent. If the security detection result indicates that the injection of attack payload was successful, the hijacking type includes any one of the following: task replacement, task expansion, and task drift.
10. The safety detection device according to any one of claims 6 to 8, characterized in that, The processing module is further configured to perform security detection on the agent based on the agent's current environment state object and initial environment state object, and after obtaining the security detection result of the agent, perform injection awareness reasoning on the agent based on the recognition mode management keyword set and the defense mode keyword set through a large model to obtain the injection awareness result of the agent; wherein, the injection awareness result is used to characterize whether the agent has the awareness to recognize the injection attack payload.
11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the security detection method as described in any one of claims 1 to 5.
12. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a computer, implement the security detection method as described in any one of claims 1 to 5.
13. A computer program product, characterized in that, The computer program product is stored in a storage medium, and when executed by a computer, the computer program product implements the security detection method as described in any one of claims 1 to 5.
