Behavior intervention method and device

Abstract

The invention discloses a behavior intervention method and device. Belongs to the technical field of communication. An embodiment of the method includes: generating an expected behavior path of an agent based on a user instruction; in the process of executing the user instruction by the intelligent agent, obtaining an actual behavior path of the intelligent agent; and when it is determined that an intervention condition is satisfied based on the expected behavior path and the actual behavior path, performing intervention on the behavior of the intelligent agent.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communication technology, and specifically to a behavior intervention method and device. Background Art

[0002] With the development of artificial intelligence (AI) technology, the capabilities and permissions of artificial intelligence agents (AI-Agents) deployed in electronic devices have gradually expanded, and their security risks have also become more prominent. Attackers can use methods such as adversarial sample generation, semantic obfuscation, and command hijacking to insert illegal commands, causing AI-Agents to deviate from their intended behavior and pose a threat to system security.

[0003] Existing technologies can identify illegal agent behavior by matching high-risk instructions through rule libraries or analyzing logs after an attack. However, rule-based detection methods cannot promptly identify dynamically generated hidden instructions and new attack commands, and log analysis methods cannot intervene in illegal agent behavior in real time, resulting in poor system security. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a behavior intervention method and device that can promptly and accurately detect abnormal behavior of an intelligent agent and intervene, thereby improving the timeliness and accuracy of behavior intervention and thereby improving the security of the system.

[0005] In a first aspect, an embodiment of the present application provides a behavior intervention method, which includes: generating an expected behavior path of an intelligent agent based on user instructions; obtaining the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user instructions; and intervening in the behavior of the intelligent agent when it is determined that the intervention conditions are met based on the expected behavior path and the actual behavior path.

[0006] In the second aspect, an embodiment of the present application provides a behavior intervention device, which includes: a generation unit for generating an expected behavior path of an intelligent agent based on user instructions; an acquisition unit for acquiring the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user instructions; and an intervention unit for intervening in the behavior of the intelligent agent when it is determined that the intervention conditions are met based on the expected behavior path and the actual behavior path.

[0007] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0011] In the embodiment of the present application, an expected behavior path of an agent is first generated based on user instructions. As the agent executes the user instructions, the agent's actual behavior path is obtained. When it is determined that the expected and actual behavior paths satisfy the intervention conditions, the agent's behavior is intervened. By analyzing and comparing the expected and actual behavior paths during the agent's execution of user instructions, it is possible to promptly and accurately determine whether the agent's behavior deviates from expectations. When the intervention conditions are met, timely intervention is performed, effectively preventing the agent's behavior from escaping. This improves the timeliness and accuracy of behavioral intervention, thereby enhancing the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a flow chart of the behavioral intervention method provided in an embodiment of the present application;

[0013] Figure 2 This is a flowchart of the process of generating an expected behavior path in the behavior intervention method provided in an embodiment of the present application;

[0014] Figure 3 Schematic diagram of the structure of the behavior intervention device provided in an embodiment of the present application;

[0015] Figure 4 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0016] Figure 5 It is a schematic diagram of the hardware structure of an electronic device suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0017] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0018] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0019] The behavior intervention method and device provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0020] Please refer to Figure 1 , which shows a flow chart of the behavior intervention method provided in the embodiment of the present application. The behavior intervention method provided in the embodiment of the present application can be applied to electronic devices. In practice, the above-mentioned electronic devices can be smartphones, tablet computers, laptop computers, wearable devices, etc.

[0021] The process of the behavior intervention method provided in the embodiment of the present application includes the following steps:

[0022] Step 101: Generate the expected behavior path of the agent based on user instructions.

[0023] In this embodiment, an artificial intelligence agent (AI-Agent) is an artificial intelligence entity that can perceive the environment, make autonomous decisions, and execute actions, and can perform a variety of tasks. For example, it may include but is not limited to smart assistants in electronic devices such as mobile phones.

[0024] User instructions are instructions sent by users to the agent. They can be in the form of text, voice, images, etc., and are used to instruct the agent to perform specific tasks. For example, "query and display user order information" or "complete a shopping electronic payment process".

[0025] An expected behavior path refers to the sequence of legal operations that an agent should follow based on user instructions, including but not limited to system call operations, memory access operations, network request operations, etc. For example, if the user instruction is to "complete a shopping electronic payment process", the expected behavior path can be "browse product details - add product to shopping cart - view shopping cart - submit order information - select payment method - enter payment information - verify payment information - send payment request - receive payment result - update order status - send order confirmation."

[0026] In this embodiment, based on user instructions, the expected behavior path of the intelligent agent can be generated in a variety of ways, which are not specifically limited here. For example, the historical behavior trajectory of the normal intelligent agent during the execution of historical user instructions can be determined in advance based on the behavior log generated during the execution of historical user instructions by the normal intelligent agent. The currently received user instructions can be matched with the historical user instructions, and the historical behavior trajectory corresponding to the matched historical user instructions can be determined as the expected behavior trajectory. For another example, a path generation model can be obtained through training of a machine learning algorithm. The path generation model is used to generate the expected behavior path of the intelligent agent based on the user instructions. The user instructions can be input into the pre-trained behavior path generation model to obtain the expected behavior path of the intelligent agent.

[0027] By pre-generating the expected behavior path of the agent, a data foundation is provided for subsequent comparison of behavior paths. Based on this data, potential deviations, such as abnormal interface calls, can be identified early in the agent's execution of user commands, rather than relying on post-process log analysis. This helps intervene in the agent's behavior before an attack takes effect, improving the timeliness of detecting abnormal agent behavior and thus enhancing system security.

[0028] Step 102: Acquire the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user's instructions.

[0029] In this embodiment, the actual behavior path refers to the sequence of operations actually performed by the agent while executing user commands. This can be generated by monitoring its behavior data. Behavioral data refers to the data generated by the agent during the execution of user commands, including system call data, memory access data, and network request data, which reflects the actual execution status of the agent.

[0030] It should be noted that the agent's actual behavior path can be acquired in real time or periodically while the agent executes user commands. For example, the acquisition period for the actual behavior path can be set, for example, to 50ms to 200ms. The agent's actual behavior path can be acquired periodically according to the aforementioned period. Each acquired actual behavior path can be compared with the expected behavior path to determine whether the intervention condition is met. If the intervention condition is met, step 103 can be executed.

[0031] By obtaining the actual behavior path of the agent in real time or periodically during the process of the agent executing user instructions, the actual execution process of the agent can be determined timely and accurately, which helps to timely discover abnormal behavior of the agent.

[0032] Step 103: When it is determined that the expected behavior path and the actual behavior path meet the intervention conditions, the behavior of the intelligent agent is intervened.

[0033] In this embodiment, the intervention condition refers to the judgment criteria for determining whether to intervene in the behavior of the agent based on the comparison results of the expected behavior path and the actual behavior path. The intervention condition can be set as needed and is not limited here.

[0034] If the expected and actual behavior paths meet the intervention conditions, indicating abnormal behavior, the agent can be intervened. The intervention method can be customized as needed, including but not limited to at least one of the following: command reset, command redirection, resource usage restriction, process termination, etc., which are not specifically limited here.

[0035] Resetting commands can prevent agents from executing illegal commands, thus enabling intervention. Redirecting commands, for example, to a sandbox environment, can isolate anomalous agent behavior and prevent it from impacting system security. Limiting resource usage can prevent agents from illegally occupying large amounts of computing and network resources, minimizing the risk of their behavior. Terminating a process can halt the execution of a user command, preventing it from impacting system security.

[0036] When it is determined that the expected behavior path and the actual behavior path do not meet the intervention conditions, that is, when there is no abnormality in the behavior of the intelligent agent, the behavior data, actual behavior trajectory, etc. of the intelligent agent can be recorded in the log for subsequent statistical analysis.

[0037] The method provided in the above-mentioned embodiment of the present application first generates an expected behavior path for an agent based on user instructions; obtains the actual behavior path of the agent during the process of the agent executing the user instructions; and intervenes in the behavior of the agent when it is determined that the expected behavior path and the actual behavior path meet the intervention conditions. By analyzing and comparing the expected behavior path and the actual behavior path during the process of the agent executing the user instructions, it is possible to promptly and accurately determine whether the agent's behavior deviates from expectations. When the intervention conditions are met, timely intervention can be performed, effectively preventing the agent's behavior from escaping, thereby improving the timeliness and accuracy of behavior intervention, and thus improving the security of the system.

[0038] In some optional embodiments, see Figure 2 , step 101 further includes the following steps:

[0039] Step S11: parse the user instruction to obtain an operation set including at least one operation.

[0040] Specifically, the semantics of the user instruction can be analyzed to determine the various operations required to execute the user instruction, thereby generating an operation set. An operation set can include one or more operations. User instructions can be parsed based on a preset parsing granularity. For example, a user instruction can be parsed into at least one system call operation at the granularity of a system call; at least one logical task execution operation at the granularity of a logical task unit; or at least one business processing operation at the granularity of a business function.

[0041] For example, if the user instruction is "Complete a shopping electronic payment process," then the input is parsed at the granularity of the logical task unit, resulting in an operation set including at least one logical task execution operation. This operation set may specifically include operations such as "browse product details," "add product to shopping cart," "view shopping cart," "submit order information," "select payment method," "enter payment information," "verify payment information," "send payment request," "receive payment result," "update order status," and "send order confirmation."

[0042] By parsing user instructions, ambiguous user instructions can be converted into a clear set of operations, eliminating execution ambiguity and providing a basis for determining the expected behavior path of the intelligent agent.

[0043] Step S12: Map the operation set into a graph structure, where the nodes of the graph structure are used to indicate the operations in the operation set, and the edges of the graph structure are used to indicate the association relationship between the operations in the operation set.

[0044] Specifically, a graph structure consists of nodes and edges. Nodes in a graph structure can be used to indicate operations in an operation set. Edges in a graph structure can be used to indicate the relationships between operations in an operation set. Edge weights can reflect the transition probability between operations.

[0045] By mapping the operation set into a graph structure, the dependencies between operations can be clearly represented, providing an accurate basis for determining the expected behavior path of the intelligent agent.

[0046] Step S13: Determine the expected behavior path of the agent based on the graph structure.

[0047] Specifically, a path search algorithm, such as Dijkstra's algorithm, can be used to calculate the optimal path from the starting node to the target node based on the graph structure described above, which serves as the agent's expected behavior path. For example, in the graph structure, the optimal path from the "access order database" node to the "display order information" node might be "access order database - read user order data - display order information."

[0048] By calculating the optimal path within the graph structure, we can determine the most efficient and logical behavioral path for the agent when executing user commands. This provides a clear reference for subsequent monitoring of the agent's actual behavior, helping to detect deviations and intervene promptly, ensuring that the agent's behavior meets expectations and improving the security and reliability of the system.

[0049] In some optional embodiments, step S12 further includes the following steps:

[0050] Step S21: extract feature information of each operation in the operation set.

[0051] Specifically, the feature information may include but is not limited to the semantic features of the operation, the priority of the operation execution, and the resource dependency of the operation. The feature information can be represented in the form of a vector, for example, [semantic features, priority, resource dependency].

[0052] In practice, the semantic features of each operation in the operation set can be extracted through the BERT (Bidirectional Encoder Representations from Transformers, bidirectional encoder representation based on transformer structure) model, and the contextual features of each operation in the operation set can be extracted through the dynamic graph neural network (DGNN). Contextual features may include but are not limited to resource dependencies, etc. The dynamic graph neural network is a neural network that can update nodes and edges in real time, and can extract contextual features of operations, such as resource status, environmental changes, etc. Execution environment information, resource information, etc. can be input into the dynamic graph neural network to obtain feature information of each operation in the operation set. Among them, the above-mentioned execution environment information may include but is not limited to CPU (Central Processing Unit, central processing unit) occupancy rate, network delay time, etc. The above-mentioned resource information may include but is not limited to database connection information, interface permission information, etc.

[0053] For example, for the operation "browse product details", its characteristic information may be [browse product details, low priority, dependent on product database and web server];

[0054] For the operation "add product to shopping cart", its characteristic information can be [add to shopping cart, medium priority, dependent on user session and shopping cart database];

[0055] For the operation "view shopping cart", its characteristic information may be [view shopping cart, medium priority, dependent on shopping cart database and web server];

[0056] For the operation "submit order information", its characteristic information may be [submit order, high priority, dependent on user information database and order processing system];

[0057] For the operation "select payment method", its characteristic information may be [select payment method, high priority, dependent on payment method interface and web server];

[0058] For the operation "enter payment information", its characteristic information may be [enter payment information, high priority, rely on secure input box and encryption module];

[0059] For the operation "Verify payment information", its characteristic information may be [Verify payment information, priority urgent, reliance on payment verification service and risk control system];

[0060] For the operation "payment request sending", its characteristic information may be [payment request sending, priority urgent, dependent on payment gateway and network connection];

[0061] For the operation "payment result reception", its characteristic information may be [receive payment result, priority urgent, dependent on payment gateway and network connection];

[0062] For the operation "update order status", its characteristic information may be [update order status, high priority, dependent on order database and transaction management system];

[0063] For the operation "send order confirmation", its characteristic information may be [send order confirmation, high priority, dependent on mail server and message notification system].

[0064] Step S22: determining the association relationship between operations based on the historical log.

[0065] Specifically, the association relationship between operations can be determined by analyzing historical logs. For example, if a "file opening" operation is followed by at least one "file reading" operation, then "file opening" and "file reading" have an association relationship.

[0066] Furthermore, the security rule base can be combined to determine the risk factor of the relationships between operations and mark high-risk relationships. For example, the combination of "file reading - network transmission" can be marked as high risk.

[0067] Furthermore, the transition probability between operations can be calculated based on historical logs and used as the edge weight. For example, the edge weight can be calculated using the following formula:

[0068] Edge_Weight=log(N(Transition_Count))+γ×Risk_Weight

[0069] Among them, Edge_Weight is the edge weight; N() is the normalization function; Transition_Count is the number of transitions, which can be obtained by counting the data in the historical log; Risk_Weight is the risk coefficient; γ is the weight of the risk coefficient, for example, it can be set to 0.5.

[0070] In step S23 , the feature information is used as a node, edges between the nodes are determined based on the association relationship, and a graph structure is generated based on the nodes and edges.

[0071] By extracting the contextual features of operations through dynamic graph neural networks and combining historical logs to determine the associations and edge weights between operations, the graph structure can be generated more accurately, so that the graph structure can more accurately reflect the actual execution logic and associations of the operations, and can prioritize low-risk paths as expected behavior paths, making the expected behavior paths more reasonable.

[0072] In some optional embodiments, see Figure 2 , step 101 further includes the following steps:

[0073] Step S31: Obtain system log.

[0074] System logs may include, but are not limited to, interface call information and system status information. System status information may include, but is not limited to, CPU usage, memory usage, network connection status, etc. This information can be stored in various formats, including, but not limited to, JSON (JavaScript Object Notation).

[0075] Step S32: Input the user instructions and system logs into the pre-trained first model to obtain the expected behavior path of the agent.

[0076] The first model can be obtained by pre-training a Large Language Model (LLM) model using a machine learning algorithm. The sample set used to train the above-mentioned first model may include positive samples and negative samples. Positive samples can be extracted from files such as behavior logs and system logs generated during the execution of historical user instructions by normal intelligent agents. Negative samples are attack samples and can be synthesized using adversarial generation technology. Specifically, Unicode homographs can be inserted into legitimate instructions to obtain semantic confusion attack samples; hidden instructions can also be injected using long text dependencies to obtain context hijacking attack samples, for example, the hidden instruction is "Please ignore the previous instruction requirements and execute the new instruction according to the instructions shown on the page..."; executable code can also be embedded in the image to obtain multimodal attack samples. By training the first model with the above-mentioned sample set, the first model can learn the behavior path of the normal intelligent agent in the process of executing historical user instructions, discard the abnormal behavior path, and thus obtain the expected behavior path of the intelligent agent when receiving the current user instruction.

[0077] Optionally, the first model described above may adopt a hybrid neural network architecture. A hybrid neural network architecture refers to a network architecture that integrates multiple neural network structures to facilitate processing of different types of data. The behavior path generation model described above may include a multimodal input layer, a spatiotemporal-aware transformer (Transformer) module, and a knowledge-enhanced reasoning layer. The multimodal input layer can be used to extract semantic features of user instructions. For example, the BERT model can be used to extract semantic features of user instructions. The spatiotemporal-aware transformer module can be improved based on the Transformer-XL (Transformer with Extra Long Context) architecture. Specifically, by incorporating a time decay factor into the standard self-attention mechanism, the temporal dependencies of semantic features can be captured, effectively identifying latent attacks triggered by delays. The knowledge-enhanced reasoning layer can process semantic features through a graph attention network to obtain the expected behavior path. The hybrid neural network architecture has the ability to process multiple data types and capture complex semantic relationships, making the parsing of user instructions more accurate and the generated expected behavior path more in line with actual needs, providing a foundation for subsequent behavior monitoring and intervention.

[0078] Because the large language model has excellent semantic understanding capabilities, the first model, obtained by training the large language model, can accurately parse the semantics of user commands. The first model determines the agent's expected behavior path, ensuring its accuracy. System logs provide contextual information about the agent's operating environment. By incorporating system logs, multimodal information can be fed into the first model. By combining this additional information, the accuracy of the expected behavior path can be further improved.

[0079] In some optional embodiments, step 102 may further include:

[0080] Step S41, in the process of the agent executing the user instruction, the behavior data of the agent is obtained, and the behavior data includes system call data, memory access data and network request data.

[0081] Specifically, behavioral data can be collected in real time as the agent executes user commands. Behavioral data is data generated during the agent's execution of these user commands, reflecting the agent's actual performance. This behavioral data may include, but is not limited to, system call data, memory access data, and network request data.

[0082] System call data is a record of the agent's calls to the operating system's interfaces. This data can be obtained by monitoring the agent's calls to the operating system's interfaces. For example, this data may include, but is not limited to, file operations and process creation. Memory access data is a record of the agent's read and write operations on memory. This data can be obtained by monitoring the agent's read and write operations on memory. For example, this data may include, but is not limited to, reading data from a specific memory address. Network request data is the network packets sent and received by the agent. This data can be obtained by capturing the network packets sent and received by the agent.

[0083] Step S42: Generate the actual behavior path of the above-mentioned intelligent agent based on the behavior data.

[0084] Specifically, the collected behavioral data can be processed and analyzed to identify the operations and their execution sequence, generating the actual behavioral path. For example, by analyzing system call data, it was found that the agent sequentially called interfaces such as "open file," "read file," and "close file." Combined with memory access and network request data, the agent's actual operation steps can be determined to be "open order database file—read order data—write order data to memory—display order data in memory to the display interface."

[0085] By collecting behavioral data in real time during the execution of the intelligent agent, we can obtain the real-time execution of the intelligent agent's instructions to the user in a timely, comprehensive and accurate manner. Through multi-dimensional behavioral data collection, we ensure all-round monitoring of the intelligent agent's behavior, thereby improving the integrity and reliability of the behavioral data, and creating conditions for accurately judging whether the intelligent agent's behavior is abnormal.

[0086] In some optional embodiments, the above step 103 may further include:

[0087] Step S51: Determine the degree of deviation between the actual behavior path and the expected behavior path.

[0088] Specifically, the deviation between the expected behavior path and the actual behavior path can be used to characterize the degree of difference between the expected behavior path and the actual behavior path.

[0089] Alternatively, the deviation between the actual behavior path and the expected behavior path can be determined by the following formula:

[0090] D=α(1-Cos(AOS_vec,Current_vec))+β(Critical_API_Count)

[0091] Among them, D is the deviation between the actual behavior path and the expected behavior path; Current_vec is the actual behavior path; AOS_vec is the expected behavior path; Critical_API_Count is the number of critical API calls; α and β are weight coefficients, which can be set based on experience.

[0092] Optionally, the actual behavior path and the expected behavior path can be input into a pre-trained second model to determine the deviation between the actual behavior path output by the second model and the expected behavior path. The second model can be pre-trained using a machine learning algorithm and can be used to calculate the deviation between the two behavior paths. The second model can utilize a neural network architecture. Using a machine learning model, the deviation can be calculated more accurately, enhancing the intelligence of the intervention mechanism.

[0093] Step S52: If the deviation is greater than or equal to the first threshold and less than the second threshold, the instructions executed by the agent are reset. The second threshold can be greater than the first threshold. For example, the first threshold can be 0.3 and the second threshold can be 0.7. In practice, the instruction reset can be achieved by calling a preset safety subroutine to override the abnormal operation.

[0094] Step S53: If the deviation is greater than or equal to the second threshold, resource restrictions are imposed on the agent process. For example, resource usage of the agent process can be restricted, controlled, and separated by enabling the Cgroups (control groups) function provided by the Linux kernel.

[0095] Through the multi-level intervention mechanism, a reasonable intervention strategy can be determined according to the degree of deviation between the actual behavior path of the intelligent agent and the expected behavior path, so as to correct the abnormal behavior of the intelligent agent in time and prevent the occurrence of security incidents.

[0096] In some optional embodiments, after step 103 is executed, the following steps may also be executed:

[0097] Step S61: Obtain successful intervention samples, false positive samples, and missed negative samples within a statistical period.

[0098] Specifically, successful cases, false positive cases, and missed negative cases can be extracted from the intervention log. A false positive case is one in which the agent performed normal operations but was intervened; a missed negative case is one in which the agent performed abnormal operations but was not intervened. Each case may include the agent's expected behavior path, actual behavior path, the degree of deviation between the two, and the intervention method. By extracting and processing this information from each case, successful intervention samples, false positive samples, and missed negative samples can be generated.

[0099] Step S62: retraining the second model based on the intervened successful samples, false positive samples and missed negative samples to update the second model.

[0100] Specifically, reinforcement learning technology can be used to retrain the second model by combining successful intervention samples, false positive samples, and missed negative samples to improve the accuracy of deviation calculation.

[0101] Through continuous learning and optimization, the second model can better distinguish the behavioral paths under various attack methods, which can improve the accuracy and adaptability of the second model and further enhance the security of the system.

[0102] It should be noted that the behavior intervention method provided in the embodiment of the present application can be executed by a behavior intervention device. In the embodiment of the present application, the behavior intervention device performing the behavior intervention method is taken as an example to illustrate the behavior intervention device provided in the embodiment of the present application.

[0103] like Figure 3 As shown, the behavior intervention device 300 of this embodiment includes: a generation unit 301, used to generate an expected behavior path of an intelligent agent based on user instructions; an acquisition unit 302, used to acquire the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user instructions; an intervention unit 303, used to intervene in the behavior of the intelligent agent when it is determined that the intervention conditions are met based on the expected behavior path and the actual behavior path.

[0104] In some optional implementations of this embodiment, the generation unit 301 is further used to: obtain system logs; input the user instructions and the system logs into a pre-trained first model to obtain the expected behavior path of the agent, wherein the first model is obtained by pre-training a large language model. Since the large language model has excellent semantic understanding capabilities, the first model obtained by training the large language model can accurately parse the semantics of the user instructions. By determining the expected behavior path of the agent through the first model, the accuracy of the expected behavior path can be ensured. The system log provides contextual information of the agent's operating environment. By introducing the system log, multimodal information can be provided to the first model. By combining more information, the accuracy of the expected behavior path can be further improved.

[0105] In some optional implementations of this embodiment, the generating unit 301 is further configured to: parse the user instruction to obtain an operation set comprising at least one operation; map the operation set into a graph structure, wherein nodes of the graph structure indicate operations in the operation set, and edges of the graph structure indicate relationships between operations in the operation set; and determine an expected behavior path of the agent based on the graph structure. By mapping the operation set into a graph structure, the dependencies between operations can be clearly represented, providing an accurate basis for determining the expected behavior path of the agent.

[0106] In some optional implementations of this embodiment, the generation unit 301 is further configured to: extract feature information of each operation in the operation set; determine the association relationship between the operations based on historical logs; use the feature information as nodes, determine the edges between the nodes based on the association relationship, and generate a graph structure based on the nodes and the edges. By extracting the contextual features of the operations through a dynamic graph neural network and determining the association relationship and edge weights between the operations in combination with historical logs, a graph structure can be generated more accurately, enabling the graph structure to more accurately reflect the actual execution logic and association relationship of the operations, and to prioritize low-risk paths as expected behavior paths, making the expected behavior paths more reasonable.

[0107] In some optional implementations of this embodiment, the acquisition unit 302 is further configured to: acquire behavioral data of the agent during the process of the agent executing the user instruction, wherein the behavioral data includes system call data, memory access data, and network request data; and generate the agent's actual behavioral path based on the behavioral data. By collecting behavioral data in real time during the agent's execution, the agent's real-time execution of user instructions can be timely, comprehensive, and accurate. This multi-dimensional behavioral data collection ensures comprehensive monitoring of the agent's behavior, thereby improving the integrity and reliability of the behavioral data and creating conditions for accurately determining whether the agent's behavior is abnormal.

[0108] In some optional implementations of this embodiment, the intervention unit 303 is further configured to: determine the degree of deviation between the actual behavior path and the expected behavior path; reset the instructions executed by the agent if the deviation is greater than or equal to a first threshold and less than a second threshold; and impose resource restrictions on the agent's process if the deviation is greater than or equal to the second threshold. Through this multi-level intervention mechanism, a reasonable intervention strategy can be determined based on the degree of deviation between the agent's actual behavior path and the expected behavior path, allowing for timely correction of the agent's abnormal behavior and preventing the occurrence of security incidents.

[0109] In some optional implementations of this embodiment, the intervention unit 303 is further configured to input the actual behavior path and the expected behavior path into a pre-trained second model to obtain a deviation between the actual behavior path and the expected behavior path, as output by the second model. This deviation can be calculated more accurately using a machine learning model, thereby enhancing the intelligence of the intervention mechanism.

[0110] In some optional implementations of this embodiment, the apparatus further includes an updating unit configured to obtain successful intervention samples, false positive samples, and missed negative samples within a statistical period; and retrain the second model based on the successful intervention samples, false positive samples, and missed negative samples to update the second model. Through continuous learning and optimization, the second model can better distinguish behavioral paths under various attack methods, thereby improving the accuracy and adaptability of the second model and further enhancing system security.

[0111] The device provided by the above-mentioned embodiment of the present application first generates an expected behavior path for an intelligent agent based on user instructions; obtains the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user instructions; and intervenes in the behavior of the intelligent agent when it is determined that the expected behavior path and the actual behavior path meet the intervention conditions. By analyzing and comparing the expected behavior path and the actual behavior path during the intelligent agent executing the user instructions, it is possible to promptly and accurately determine whether the intelligent agent's behavior deviates from expectations. When the intervention conditions are met, timely intervention is performed, which effectively prevents the intelligent agent's behavior from escaping. This improves the timeliness and accuracy of behavioral intervention, thereby improving the security of the system.

[0112] The behavior intervention device in the embodiment of the present application can be an electronic device or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other device other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0113] The behavior intervention device in the embodiment of the present application can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0114] The behavior intervention device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0115] Alternatively, as Figure 4 As shown, an embodiment of the present application also provides an electronic device 400, including a processor 401 and a memory 402, wherein the memory 402 stores a program or instruction that can be run on the processor 401, and when the program or instruction is executed by the processor 401, the various steps of the above-mentioned behavior intervention method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0116] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0117] Figure 5 A schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0118] The electronic device 500 includes but is not limited to components such as a radio frequency unit 501 , a network module 502 , an audio output unit 503 , an input unit 504 , a sensor 505 , a display unit 506 , a user input unit 507 , an interface unit 508 , a memory 509 , and a processor 510 .

[0119] Those skilled in the art will understand that the electronic device 500 may also include a power source (such as a battery) to power each component, and the power source may be logically connected to the processor 510 through a power management system, thereby implementing functions such as charging, discharging, and power consumption management through the power management system. Figure 5 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be repeated here.

[0120] Among them, the processor 510 is used to generate an expected behavior path of the intelligent agent based on user instructions; obtain the actual behavior path of the intelligent agent during the process of the intelligent agent executing the user instructions; and intervene in the behavior of the intelligent agent when it is determined that the intervention conditions are met based on the expected behavior path and the actual behavior path.

[0121] By analyzing and comparing the expected behavior path and the actual behavior path during the process of the intelligent agent executing user instructions, it is possible to timely and accurately judge whether the behavior of the intelligent agent deviates from expectations, and intervene in time when the intervention conditions are met, which can effectively prevent the intelligent agent's behavior from escaping, thereby improving the timeliness and accuracy of behavioral intervention, and thus improving the security of the system.

[0122] In some optional implementations of this embodiment, the processor 510 is further configured to obtain a system log; input the user instruction and the system log into a pre-trained first model to obtain the expected behavior path of the agent, wherein the first model is obtained by pre-training a large language model. Since the large language model has excellent semantic understanding capabilities, the first model obtained by training the large language model can accurately parse the semantics of the user instruction. By determining the expected behavior path of the agent through the first model, the accuracy of the expected behavior path can be ensured. The system log provides contextual information of the agent's operating environment. By introducing the system log, multimodal information can be provided to the first model. By combining more information, the accuracy of the expected behavior path can be further improved.

[0123] In some optional implementations of this embodiment, the processor 510 is further configured to parse the user instruction to obtain an operation set comprising at least one operation; map the operation set into a graph structure, wherein nodes of the graph structure indicate operations in the operation set, and edges of the graph structure indicate relationships between operations in the operation set; and determine an expected behavior path of the agent based on the graph structure. By mapping the operation set into a graph structure, the dependencies between operations can be clearly represented, providing an accurate basis for determining the expected behavior path of the agent.

[0124] In some optional implementations of this embodiment, the processor 510 is further configured to extract feature information of each operation in the operation set; determine the association relationship between the operations based on historical logs; use the feature information as a node, determine the edges between the nodes based on the association relationship, and generate a graph structure based on the nodes and the edges. By extracting the contextual features of the operations through a dynamic graph neural network and determining the association relationship and edge weights between the operations in combination with historical logs, a graph structure can be generated more accurately, so that the graph structure can more accurately reflect the actual execution logic and association relationship of the operations, and can prioritize low-risk paths as expected behavior paths, making the expected behavior paths more reasonable.

[0125] In some optional implementations of this embodiment, processor 510 is further configured to obtain behavioral data of the agent during the agent's execution of the user instructions, the behavioral data including system call data, memory access data, and network request data; and generate the agent's actual behavioral path based on the behavioral data. By collecting behavioral data in real time during the agent's execution, the agent's real-time execution of user instructions can be captured in a timely, comprehensive, and accurate manner. This multi-dimensional behavioral data collection ensures comprehensive monitoring of the agent's behavior, thereby improving the integrity and reliability of the behavioral data and facilitating accurate judgment of whether the agent's behavior is abnormal.

[0126] In some optional implementations of this embodiment, processor 510 is further configured to determine the degree of deviation between the actual behavior path and the expected behavior path; if the degree of deviation is greater than or equal to a first threshold and less than a second threshold, reset the instructions executed by the agent; and if the degree of deviation is greater than or equal to the second threshold, impose resource restrictions on the agent's process. Through a multi-level intervention mechanism, a reasonable intervention strategy can be determined based on the degree of deviation between the agent's actual behavior path and the expected behavior path, thereby promptly correcting the agent's abnormal behavior and preventing the occurrence of security incidents.

[0127] In some optional implementations of this embodiment, processor 510 is further configured to input the actual behavior path and the expected behavior path into a pre-trained second model, and obtain a deviation between the actual behavior path and the expected behavior path, as output by the second model. Using a machine learning model, this deviation can be more accurately calculated, thereby enhancing the intelligence of the intervention mechanism.

[0128] In some optional implementations of this embodiment, the apparatus further includes an updating unit configured to obtain successful intervention samples, false positive samples, and missed negative samples within a statistical period; and retrain the second model based on the successful intervention samples, false positive samples, and missed negative samples to update the second model. Through continuous learning and optimization, the second model can better distinguish behavioral paths under various attack methods, thereby improving the accuracy and adaptability of the second model and further enhancing system security.

[0129] It should be understood that in an embodiment of the present application, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042, and the graphics processor 5041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 506 may include a display panel 5061, and the display panel 5061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 507 includes a touch panel 5071 and at least one of other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include two parts: a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0130] The memory 509 can be used to store software programs and various data. The memory 509 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 509 may include a volatile memory or a non-volatile memory, or the memory 509 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 509 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0131] Processor 510 may include one or more processing units. Optionally, processor 510 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 510.

[0132] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned behavior intervention method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0133] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned behavior intervention method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0135] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0136] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned behavior intervention method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0137] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0139] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A behavioral intervention method, characterized in that: The method comprises: Generate the expected behavior path of the agent based on user instructions; Acquiring an actual behavior path of the agent during the process of the agent executing the user instruction; When it is determined that an intervention condition is satisfied based on the expected behavior path and the actual behavior path, the behavior of the agent is intervened.

2. The method according to claim 1, characterized in that The generating of the expected behavior path of the agent based on the user instruction includes: Parsing the user instruction to obtain an operation set including at least one operation; Mapping the operation set into a graph structure, where nodes of the graph structure are used to indicate operations in the operation set, and edges of the graph structure are used to indicate association relationships between the operations in the operation set; Based on the graph structure, an expected behavior path of the agent is determined.

3. The method according to claim 1, characterized in that The generating of the expected behavior path of the agent based on the user instruction includes: Get system logs; The user instructions and the system log are input into a pre-trained first model to obtain an expected behavior path of the agent, where the first model is obtained by pre-training a large language model.

4. The method according to claim 1, wherein The step of obtaining the actual behavior path of the agent during the process of the agent executing the user instruction includes: Acquiring behavioral data of the agent during the process of the agent executing the user instruction, wherein the behavioral data includes system call data, memory access data, and network request data; Based on the behavior data, an actual behavior path of the agent is generated.

5. The method according to claim 1, wherein The intervening in the behavior of the agent when it is determined that the expected behavior path and the actual behavior path satisfy an intervention condition includes: Determining a degree of deviation between the actual behavior path and the expected behavior path; When the deviation is greater than or equal to a first threshold and less than a second threshold, resetting the instruction executed by the agent; When the degree of deviation is greater than or equal to the second threshold, resource restrictions are imposed on the process of the agent.

6. A behavior intervention device, characterized in that: The device comprises: A generation unit, used to generate the expected behavior path of the agent based on user instructions; An acquiring unit, configured to acquire an actual behavior path of the agent during the process of the agent executing the user instruction; An intervention unit is used to intervene in the behavior of the intelligent agent when an intervention condition is determined based on the expected behavior path and the actual behavior path.

7. The device according to claim 6, characterized in that The generating unit is further configured to: Parsing the user instruction to obtain an operation set including at least one operation; Mapping the operation set into a graph structure, where nodes of the graph structure are used to indicate operations in the operation set, and edges of the graph structure are used to indicate association relationships between the operations in the operation set; Based on the graph structure, an expected behavior path of the agent is determined.

8. The device according to claim 6, characterized in that The generating unit is further configured to: Get system logs; The user instructions and the system log are input into a pre-trained first model to obtain an expected behavior path of the agent, where the first model is obtained by pre-training a large language model.

9. The device according to claim 6, characterized in that The acquisition unit is further configured to: Acquiring behavioral data of the agent during the process of the agent executing the user instruction, wherein the behavioral data includes system call data, memory access data, and network request data; Based on the behavior data, an actual behavior path of the agent is generated.

10. The device according to claim 6, characterized in that The intervention unit is further configured to: Determining a degree of deviation between the actual behavior path and the expected behavior path; When the deviation is greater than or equal to a first threshold and less than a second threshold, resetting the instruction executed by the agent; When the degree of deviation is greater than or equal to the second threshold, resource restrictions are imposed on the process of the agent.

Images