Method and system for risk management and control of long-term autonomous agent based on behavior evolution

Abstract

The application discloses a long-term autonomous agent risk management and control method and system based on behavior evolution. The method collects long-term autonomous agent operation data to construct behavior state, extracts behavior driving factors affecting the change of behavior state, and calculates the behavior state at the next moment according to a preset behavior evolution rule; the behavior states at continuous multiple moments are organized into behavior trajectories and subjected to behavior dynamics analysis, a risk state is calculated based on the behavior state, the behavior trajectory and the dynamics index, and the tool call restriction, the permission level and the execution constraint of the agent are adjusted accordingly. The system comprises a collection module, an extraction module, an analysis module, a calculation module and an adjustment module. The application improves the safety analysis object from discrete events to continuous behavior evolution, can identify long-term evolution risks such as time aggregation attacks, intention drift and gradual change of permissions, and realizes computable and predictable dynamic security control.

Description

Technical Field

[0001] This application relates to the field of information technology, specifically to a method and system for risk management of long-term autonomous intelligent agents based on behavioral evolution. Background Technology

[0002] With the development of large language models and intelligent agent technology, artificial intelligence systems are evolving from question-and-answer models into long-term autonomous execution systems with long-term memory, autonomous planning, continuous tool invocation, dynamic goal modification, and the accumulation of trust and permissions. Most existing security technologies use discrete events such as single inputs, outputs, invocations, or sessions as analysis units, relying on rule filtering, anomaly detection, risk scoring, or policy engines for judgment. These methods struggle to depict the continuous evolution of intelligent agent behavior over time and cannot describe the relationships and changes between behavioral dimensions such as intent, memory, permissions, trust, identity, and data scope.

[0003] However, existing intelligent agent security solutions, including injection detection, output content filtering, tool call whitelists, policy engines, execution graph analysis, state machine control, risk scoring, and manual approval, still focus on inputs, outputs, tool calls, or process nodes. For long-term autonomous intelligent agents, the real risk is not a single action, but the gradual change in behavioral state over time: for example, individual accesses may be legitimate, but over time, the data scope expands; targets gradually drift during multiple interactions; permissions gradually increase with accumulated trust; and multiple agents may act safely individually but collectively create a risk chain. Because existing technologies lack a unified behavioral evolution model, they cannot identify long-term evolutionary risks such as time-aggregation attacks, delayed-triggered attacks, intent drift, gradual permission changes, multi-agent collaborative attacks, and slow data outflows, making it difficult to provide calculable, predictable, and controllable security guarantees. Summary of the Invention

[0004] This specification describes a method and system for risk management of long-term autonomous intelligent agents based on behavioral evolution through several embodiments.

[0005] Firstly, the embodiments of this specification provide a long-term autonomous intelligent agent risk management method based on behavioral evolution, including the following steps:

[0006] Collect operational data during the long-term operation of an autonomous intelligent agent, and construct a behavioral state S(t) that represents the comprehensive behavioral characteristics of the agent at time t;

[0007] Extract the set of behavioral driving factors D(t) that affect the change of the behavioral state S(t), and calculate the next behavioral state S(t+Δt) based on the current behavioral state S(t) and the set of driving factors D(t) according to the preset set of behavioral evolution rules R, using the state transition function Φ.

[0008] The behavioral states at multiple consecutive moments are organized in chronological order into behavioral trajectories BT, and behavioral dynamics analysis is performed on the behavioral trajectories BT to obtain at least one behavioral dynamics index.

[0009] Based on the risk mapping function Ψ, the risk state Q(t) is calculated based on the behavioral state S(t), the behavioral trajectory BT, and the dynamic index.

[0010] Based on the risk state Q(t), adjust the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous intelligent agent.

[0011] Secondly, embodiments of this specification provide a long-term autonomous intelligent agent risk management system based on behavioral evolution, including:

[0012] The data acquisition module collects operational data during the long-term operation of the autonomous intelligent agent and constructs a behavioral state S(t) that represents the comprehensive behavioral characteristics of the agent at time t.

[0013] The extraction module extracts the set of behavioral driving factors D(t) that affect the change of the behavioral state S(t), and calculates the behavioral state S(t+Δt) at the next moment based on the current behavioral state S(t) and the set of driving factors D(t) according to the preset behavioral evolution rule set R and the state transition function Φ.

[0014] The analysis module organizes the behavioral states at multiple consecutive moments into behavioral trajectories BT in chronological order, and performs behavioral dynamics analysis on the behavioral trajectories BT to obtain at least one behavioral dynamics index.

[0015] The calculation module calculates the risk state Q(t) based on the risk mapping function Ψ, the behavioral state S(t), the behavioral trajectory BT, and the dynamic index.

[0016] The adjustment module adjusts the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous intelligent agent based on the risk state Q(t).

[0017] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following:

[0018] In several embodiments of this specification, the present invention constructs a unified model system integrating behavioral state space, behavioral driving space, behavioral evolution rules, behavioral trajectory, behavioral dynamics, and risk evolution. This elevates the object of security analysis from discrete events to continuous behavioral evolution, uniformly characterizing the continuous changes in behavioral states such as intent, memory, permissions, trust, identity, and data scope from a time dimension. This enables the identification of latent risks such as time-aggregation attacks (where a single action is safe but only becomes apparent after long-term accumulation), slow data outflows, and gradual permission changes; delayed-triggered attacks (early implantation, later triggering); and intent drift risks caused by gradual deviation of task objectives. Furthermore, it can identify collaborative attacks that are legitimate individually but dangerous overall through multi-agent trajectory fusion and drive propagation. Simultaneously, the present invention quantitatively predicts behavioral deviation trends based on behavioral trajectories, behavioral dynamics indicators, and risk evolution stages, and implements tiered interventions for different evolution stages. It provides early warnings during the pre-deviation stage, restricts permissions and adds manual confirmation during the accumulation or critical stage, and terminates the behavioral chain and triggers audits during the outbreak stage. This achieves calculable, predictable, and controllable predictive security management, with the control results feeding back into subsequent behavioral states to form a closed loop. Furthermore, behavioral snapshots, behavioral trajectories, and risk evolution records can completely preserve evidence of the agent's behavioral evolution, forming an auditable, replayable, and accountable chain of evidence. Moreover, this invention is applicable to various long-lifecycle autonomous execution scenarios such as large-scale model agents, enterprise offices, software development, browsers, automated operation and maintenance, and multi-agent collaboration, and has good scalability and versatility.

[0019] Other features and advantages of various embodiments of this specification will be further revealed in the following detailed description and accompanying drawings. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a long-term autonomous intelligent agent risk management method based on behavioral evolution;

[0022] Figure 2 Generate a schematic diagram for the behavior snapshot (tk);

[0023] Figure 3 This is a schematic diagram of the behavior-driven space.

[0024] Figure 4 A diagram illustrating the rules of behavioral evolution;

[0025] Figure 5 This is a schematic diagram of the behavioral trajectory;

[0026] Figure 6 This is a schematic diagram of behavioral dynamics analysis;

[0027] Figure 7 This is a diagram illustrating the stages of risk evolution.

[0028] Figure 8 A schematic diagram illustrating the behavioral evolution of intelligent agents in enterprise offices;

[0029] Figure 9 A schematic diagram illustrating the behavioral evolution of intelligent agents in software development;

[0030] Figure 10 A schematic diagram illustrating the evolution of browser agent behavior;

[0031] Figure 11 A schematic diagram of trajectory fusion in a multi-agent collaborative system;

[0032] Figure 12 A schematic diagram illustrating the behavioral evolution of an automated operation and maintenance intelligent agent;

[0033] Figure 13 This is an architecture diagram of a long-term autonomous intelligent agent risk management system based on behavioral evolution. Detailed Implementation

[0034] The technical solutions of the embodiments of this specification will be explained and described below with reference to the accompanying drawings. However, the following embodiments are only preferred embodiments of this specification and not all of them. Other embodiments obtained by those skilled in the art based on the embodiments in the implementation methods without creative effort are all within the protection scope of this specification.

[0035] The terms "first," "second," "third," etc., in the description, claims, and accompanying drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0036] In the following description, terms such as “inner,” “outer,” “upper,” “lower,” “left,” and “right” are used only to facilitate the description of the embodiments and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this specification.

[0037] All data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data comply with the relevant laws, regulations, and standards of the relevant countries and regions. In this specification, a long-term autonomous intelligent agent refers to an artificial intelligence execution system that possesses long-term memory and autonomous planning capabilities, can continuously invoke tools to execute tasks, and whose permissions and trust status can accumulate and change during operation. Examples include intelligent agents driven by large language models, enterprise office intelligent agents, software development intelligent agents, browser intelligent agents, automated operation and maintenance intelligent agents, and intelligent agents in multi-agent collaborative systems.

[0038] Example 1

[0039] This embodiment provides a long-term autonomous intelligent agent risk management method based on behavioral evolution, the process of which is as follows: Figure 1 As shown, it includes the following steps:

[0040] Step S101) Collect operational data during the long-term operation of the autonomous intelligent agent and construct the behavioral state S(t).

[0041] Specifically, operational data can originate from the agent's input / output logs, tool call logs, long-term memory write and read records, task planning records, permission change records, identity switching records, external communication records, manual confirmation records, system feedback records, and model inference result summaries, etc. This step collects the aforementioned multi-source data and then constructs the behavioral state S(t) representing the agent's comprehensive behavioral characteristics at time t through rule extraction, model classification, embedding vector encoding, or multi-source fusion.

[0042] In one implementation, the behavioral state S(t) is represented in vector form as S(t) = [I(t), M(t), P(t), T(t), U(t), G(t), C(t), E(t), L(t), X(t)]. Here, I(t) represents the intention state, M(t) represents the memory state, P(t) represents the permission state, T(t) represents the trust state, U(t) represents the identity state, G(t) represents the data scope state, C(t) represents the planning state, E(t) represents the external interaction state, L(t) represents the tool capability state, and X(t) represents the reasoning mode state. In other implementations, the number of dimensions can be increased or decreased, and may include dimensions such as environmental perception, resource consumption, task dependence, multi-agent association, context stability, and feedback response.

[0043] Furthermore, to facilitate recording and comparison, this step also generates a behavioral snapshot Snapshot(tk)={S(tk),Ctx(tk),Env(tk),Meta(tk)}, such as Figure 2As shown in the diagram. Here, S(tk) represents the behavioral state at time tk, Ctx(tk) represents context information, Env(tk) represents environment information, and Meta(tk) represents metadata, including session identifier, task identifier, source identifier, and version identifier. Behavioral snapshots are used to save key states during behavioral evolution, construct behavioral trajectories, compare behavioral states at different times, calculate behavioral offsets and evolution rates, and perform cross-session, cross-task, and cross-agent correlation analysis.

[0044] Step S102) Extract the set of behavioral driving factors D(t) that affect the change of behavioral state S(t), and construct the behavioral driving space BDS.

[0045] Specifically, behavioral driving factors are the factors that cause changes in the behavioral state of an agent, including one or more of the following: goal-driven factors, memory-driven factors, environment-driven factors, policy-driven factors, permission-driven factors, trust-driven factors, identity-driven factors, tool-driven factors, data-driven factors, and collaboration-driven factors. Among these: goal-driven factors include changes in task objectives, goal refinement, goal conflict, and goal replacement; memory-driven factors include adding memories, retrieving memories, memory reinforcement, and memory forgetting; environment-driven factors include external events, context changes, user feedback, and system interruptions; policy-driven factors include policy constraints, policy relaxation, policy updates, and policy version switching; permission-driven factors include permission escalation, permission contraction, permission inheritance, and permission isolation; trust-driven factors include trust accumulation, trust decay, trust reset, and trust confirmation; identity-driven factors include role switching, role inheritance, role impersonation, and role authorization; tool-driven factors include tool addition, tool failure, tool switching, and tool retries; data-driven factors include expanded data access scope, changes in access paths, and changes in data types; and collaboration-driven factors include multi-agent dependencies, changes in task division, result handover, and state synchronization.

[0046] Further, this step constructs the behavior-driven space BDS={D(t),Ω,Γ}, as follows: Figure 3 As shown in the diagram. Here, Ω represents the relationship structure between driving factors, and Γ represents the weight, strength, or influence parameter of the driving factor. In one implementation, the behavior-driven space is represented by a directed graph: directed edges represent the causal influence relationship between driving factors, edge weights represent driving strength, node attributes represent driving categories, and subgraphs represent local evolutionary links; in another implementation, hypergraphs, tensors, weighted networks, or probabilistic graphical models can also be used to represent the dependencies, conflicts, collaborations, and propagation relationships between multiple factors.

[0047] Step S103) Calculate the next behavioral state S(t+Δt) using the state transition function Φ based on the preset set of behavioral evolution rules R.

[0048] Specifically, the set of behavioral evolution rules R defines how driving factors influence behavioral states, including at least one of the following: propagation rules, inheritance rules, decay rules, superposition rules, conflict rules, branching rules, convergence rules, reset rules, freezing rules, and upgrade rules. Figure 4 As shown in the diagram. Here, Δt is the preset state update interval, which can be determined by a fixed sampling period, or, in another implementation, by a key event trigger. The propagation rule describes the influence of driving factors on the behavioral state propagated along the behavioral link; the inheritance rule describes the inheritance of previous behavioral states by subtasks or subsequent sessions; the decay rule describes the decay of historical states and the influence of driving factors over time; the superposition rule describes the superposition of the effects of multiple driving factors; the conflict rule describes the competition and resolution between contradictory driving factors; the branching rule describes the generation of multiple evolution paths under the driving force of behavior; the convergence rule describes the convergence of multiple paths to the same state; the reset rule describes the reset of the behavioral state under specific conditions; the freeze rule describes the unchanged behavioral state under specific conditions; and the upgrade rule describes the transition upgrade of the behavioral state when conditions are met.

[0049] In one implementation, the state transition function Φ is implemented using rule-based reasoning: taking the current behavioral state S(t) and the set of driving factors D(t) as input, it matches the rules in the behavioral evolution rule set R one by one, and outputs the behavioral state S(t+Δt) at the next moment. In another implementation, the state transition function Φ is implemented using matrix operations, graph propagation, statistical models, machine learning models, or deep learning models. When there are multiple possible evolution paths for the behavior, this step can output a set of candidate states for the next moment, which can be used for subsequent trajectory construction and risk analysis. Further, the actual behavioral state constructed at time t+Δt is compared with the behavioral state S(t+Δt) at the next moment, and the deviation between the two is used as one of the inputs for calculating the behavioral offset and the risk state.

[0050] Step S104) Organize the behavioral states of multiple consecutive time points into behavioral trajectories BT in chronological order.

[0051] Specifically, the behavior trajectory BT can be constructed using one of the following methods: a time-series method that connects behavior snapshots after sorting by timestamps; an event-triggered method that generates trajectory nodes when a critical event occurs; a session-based method that concatenates behavior snapshots from multiple sessions according to their relationships; a graph-based method that represents multiple behavior branches, merging, and jump relationships as a graph structure; or a cross-agent fusion method that merges local trajectories of different agents into a global trajectory, such as... Figure 5 As shown.

[0052] Furthermore, when constructing the behavior trajectory BT, state consistency constraints are imposed on the behavior state. These constraints include: time consistency constraints, requiring that the change in behavior state at adjacent times does not exceed a preset threshold; logical consistency constraints, requiring that the matching degree between the change in behavior state and the task objective and context logic is higher than a preset reference value; permission consistency constraints, requiring that permission changes conform to a preset permission policy; memory consistency constraints, requiring that the correlation between subsequent behaviors and valid memories exceeds a preset correlation threshold; identity consistency constraints, requiring that identity switching conforms to preset authorization conditions; trust consistency constraints, requiring that trust upgrades match historical trust upgrade records; and collaboration consistency constraints, requiring that the state handover between multiple agents has traceable links.

[0053] Step S105) Perform behavioral dynamics analysis on the behavioral trajectory BT to obtain at least one behavioral dynamics index.

[0054] Specifically, behavioral dynamics indicators include at least one of the following: direction of change of behavioral state, rate of change, acceleration of change, stability of change, curvature of change, bifurcation of change, and convergence of change, such as... Figure 6 As shown in the diagram. The direction of change is used to determine which state the behavior is evolving towards; the rate of change is used to determine how fast the behavior state evolves; the acceleration of change is used to determine whether the evolution trend is accelerating or decelerating; the stability of change is used to determine the severity of the behavior state fluctuations; the curvature of change is used to determine the degree of curvature of the behavior trajectory; the bifurcation of change is used to determine whether the behavior exhibits multi-path differentiation; and the convergence of change is used to determine whether multiple behavior paths ultimately point to the same state. This is suitable for identifying long-term attacks that involve multiple steps, multiple stages, and cross sessions.

[0055] Furthermore, this step also calculates the behavioral offset ΔB = Dist(BT_cur, BT_norm). Here, BT_norm is the preset baseline trajectory for normal behavior, BT_cur is the current behavioral trajectory, and Dist is one of the following: distance function, similarity function, graph edit distance, embedding distance, or probability bias function. The behavioral offset can be calculated separately for at least one behavioral type, such as intent, memory, permissions, trust, data scope, tools, identity, external interaction, planning, and reasoning, and different thresholds can be set for different dimensions. When the offset of a certain dimension exceeds the threshold, a warning, review, demotion, or freeze mechanism is triggered. In addition, the offset also considers long-term cumulative bias: even if each offset is small, as long as the cumulative offset continues to grow, it constitutes a high-risk behavioral evolution. The set consisting of at least one of the above behavioral dynamics indicators is denoted as BD.

[0056] Step S106) Calculate the risk state Q(t) based on the risk mapping function Ψ, the behavioral state S(t), the behavioral trajectory BT, and the dynamic index.

[0057] Specifically, the risk mapping function Ψ maps changes in behavioral state to changes in risk state, i.e., Q(t) = Ψ(S(t), BT, BD, BDS, R), as shown below. Figure 7 As shown, BD is the set of behavioral dynamics indicators obtained in step S105. The risk state Q(t) characterizes the evolution stage of risk, including at least two of the following: stable stage, pre-deviation stage, accumulation stage, critical stage, outbreak stage, and regression stage. Specifically: the stable stage indicates that behavior is within a normal range; the pre-deviation stage indicates that behavior begins to deviate slightly; the accumulation stage indicates that the deviation continues to accumulate and gradually approaches the risk boundary; the critical stage indicates that risk factors overlap to form a critical change; the outbreak stage indicates that high-risk behavior appears significantly; and the regression stage indicates that behavior returns to a safe state after being controlled.

[0058] Furthermore, this step maps the behavioral evolution process into typical risk types, including: single access is secure, but the cumulative access scope gradually expands over a long period of time, forming a high-risk time aggregation attack; delayed triggering attacks where malicious conditions are written early and later triggered by driving rules; intent drift where the task objective gradually evolves from a low-risk direction to a high-risk direction; gradual expansion of permission boundaries eventually leading to a gradual change in permissions that make high-risk capabilities available; multi-agent collaborative attacks where each subsystem is secure when observed individually, but the overall combination forms a risky link; and slow data outflows where each transmission is small, but the cumulative long-term data leaks a complete data breach.

[0059] Step S107) Based on the risk state Q(t), adjust the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous agent to implement dynamic security control.

[0060] Specifically, dynamic security control measures include at least one of the following: restricting tool calls, lowering permission levels, freezing long-term memory, trimming context information, adding manual confirmation, isolating external communication, restricting task paths, terminating the current behavior chain, resetting trust state, and switching to a low-risk execution mode.

[0061] Furthermore, dynamic security control operates in stages based on the risk evolution phase: when a risk is in the pre-offset phase, mild intervention measures such as early warnings are used; when a risk enters the accumulation or critical phase, measures such as restricted access, increased manual confirmation, and isolation of external communication are employed; when a risk enters the outbreak phase, the current behavioral chain is immediately terminated and auditing is conducted. When a risk is in the rollback phase, tool access capabilities and access levels are gradually restored while auditing is maintained; when a risk is in the stable phase, no intervention is applied.

[0062] Furthermore, the dynamic control results act inversely on subsequent behavioral states, causing the agent to form new behavioral trajectories under control, thus constituting closed-loop control. Steps S101 to S107 are executed cyclically during the agent's continuous operation, and model parameters such as behavioral baselines and risk thresholds are continuously updated based on the operational results.

[0063] Example 2

[0064] This embodiment provides the application of the present invention in an enterprise office intelligent agent scenario, such as... Figure 8 As shown. Enterprise office intelligent agents are used to handle tasks such as emails, spreadsheets, documents, meeting minutes, internal knowledge base queries, approval processes, and external collaborations. They have a long lifespan and typically retain user preferences, historical tasks, and organizational structure information, making them particularly suitable for this invention.

[0065] In this scenario, the behavioral state S(t) can include the following dimensions: current task type, current email access permissions, current document access scope, currently stored organizational information, whether it has external sending capabilities, whether it is requesting high-level approval, whether it is currently performing batch operations, and whether it is currently collaborating across departments.

[0066] For example, an office intelligence agent is initially only authorized to compile public meeting minutes, and its behavior is in a low-risk zone. Subsequently, due to repeated authorizations from the user, its permissions gradually increase, allowing access to more internal documents. If the system continues to expand its tool capabilities based on historical trust, its behavior trajectory BT may gradually evolve from compiling public materials to reading internal materials, summarizing sensitive content, batch exporting, and even sending externally. Each of these individual actions may conform to the current permissions, but from the perspective of behavior trajectory, its behavior has clearly deviated. This embodiment identifies the cumulative changes in permissions and trust through behavior-driven spatial recognition, discovers such progressive risks through behavior trajectory and behavior dynamics analysis, and performs dynamic control before permission expansion reaches a critical threshold. Possible control strategies include: adding manual confirmation for batch export actions, adding fine-grained auditing for reading highly sensitive documents, adding manual approval for sending external emails, implementing memory isolation for cross-departmental information summaries, and rolling back abnormal permission increases.

[0067] Example 3

[0068] This embodiment provides an application of the present invention in the scenario of intelligent agents in software development, such as... Figure 9 As shown, software development intelligent agents can be used for code generation, test writing, repository retrieval, dependency analysis, patch submission, CI / CD configuration, and troubleshooting. They typically need to access source code, configurations, keys, build systems, and multiple development tools, and their behavioral risks exhibit a clear characteristic of long-term accumulation.

[0069] In this scenario, the behavioral state S(t) can include the following dimensions: the scope of the currently accessed code repository, whether the key or configuration file is currently accessed, whether the private dependency is currently accessed, whether the commit permission is currently granted, whether the deployment permission is currently granted, whether the current plan includes destructive operations, whether the current stage is integration testing, testing or release, whether the current project calls historical memory across projects, and whether the current code snippets are actively sent to external entities.

[0070] For example, a research and development agent might initially be used only to fix defects in a single function. Through prolonged interaction, it gradually gains more repository context, module access permissions, and automated execution capabilities, eventually enabling it to submit code and trigger build and release processes independently. If an attacker implants a delayed task early on with harmless hints, the agent might begin automatically extracting code, compressing dependencies, and distributing it several days later. This embodiment identifies delayed drivers in memory through behavioral evolution rules, recognizes the gradual expansion of permissions through trajectory analysis, and identifies the increasing rate of external interactions through dynamic analysis, thereby blocking the behavioral chain before data actually leaks out. Possible control strategies include: implementing tiered permissions for code repository access, setting double confirmation for commit and release actions, implementing sensitive access isolation for key files, setting trajectory audits for cross-repository searches, and re-verifying behaviors that remain in the same high-privilege state for extended periods.

[0071] Example 4

[0072] This embodiment provides an application of the present invention in a browser-based intelligent agent scenario, such as... Figure 10 As shown, browser agents are commonly used to automatically search web pages, log in to platforms, fill out forms, retrieve information, and execute online processes. They easily access external websites, third-party content, and dynamic web pages, and their behavior exhibits strong environmental dependence and external driving characteristics.

[0073] In this scenario, the behavioral state S(t) can include the following dimensions: the current access domain, the current login session state, the current download and upload capabilities, whether autofill is currently allowed, whether cookie reading capabilities are currently available, whether cross-site redirection is currently in progress, whether external misleading text exists, and whether the user is currently on a high-risk page.

[0074] Under normal circumstances, a browser agent might only search publicly available information. However, after encountering injection prompts on external pages, it may be gradually lured to access emails, documents, cloud storage, or third-party services, integrating these external services into its long-term behavioral trajectory. If the system only detects based on single webpage content, it is difficult to detect this cross-page, cross-site, and cross-session behavioral evolution. This embodiment identifies external page inducements through driver factor analysis, identifies cross-site behavior continuation through behavioral trajectory analysis, and determines whether the agent is gradually moving towards an external leakage path through risk evolution analysis. Possible control strategies include: adding access restrictions to unknown domains, adding confirmation for cross-site logins and autofill, isolating instruction text in external pages using driver factors, adding auditing for file downloads and uploads, and interrupting abnormal redirect links.

[0075] Example 5

[0076] This embodiment provides an application of the present invention in a multi-agent cooperative system scenario, such as... Figure 11 As shown, in multi-agent scenarios, different agents are responsible for tasks such as searching, summarizing, executing, reviewing, and transmitting. The behavior of a single agent may be completely legal, but the combination of multiple agents may form new high-risk links.

[0077] In this scenario, each sub-agent possesses its own behavioral state, and the system also maintains a global collaborative behavioral state, including: the current responsibility of each sub-agent, dependencies between sub-agents, shared memory state, shared task state, state synchronization frequency, handover content, permission allocation method, and result aggregation method. In this embodiment, the behavioral driving factor also includes a collaborative driving factor, used to represent cross-agent dependencies, changes in task division, result handover, and state synchronization.

[0078] For example, agent A is responsible for reading data, agent B for organizing content, and agent C for sending emails. Individually, each agent appears to be operating without violating rules; however, from a global behavioral perspective, the trajectories of multiple agents converge into a path for sensitive data leakage. This embodiment merges multiple local trajectories into a global trajectory and uses drive propagation to identify how changes in the behavior of one agent affect another, thereby identifying collaborative attacks and cross-agent risks. Possible control strategies include: auditing cross-agent data exchanges, implementing write isolation for shared memory, adding confirmation to actions sent at the end of the link, escalating risks at the convergence points of multiple agent trajectories, and manually reviewing key nodes in collaborative links.

[0079] Example 6

[0080] This embodiment provides the application of the present invention in the scenario of automated operation and maintenance and cloud control intelligent agent, such as... Figure 12As shown, automated operations and maintenance agents can perform tasks such as log analysis, resource scaling, container operations, configuration updates, service restarts, fault location, and alarm handling. They typically have high privileges, and if their behavior evolves out of control, it may cause widespread impact.

[0081] In this scenario, the behavioral state S(t) can include the following dimensions: the currently controlled cluster scope, whether there are permissions to restart, delete, expand, or rollback, whether the current operation involves the production environment, whether batch modifications are being performed, whether the system is in fault recovery mode, whether there are abnormal command combinations, and whether cross-service operations are occurring. This embodiment uses permission trajectories, behavioral dynamics, and risk evolution to determine whether the operations and maintenance agent is evolving from local repair to high-risk destructive operations, and can implement tiered approval, isolation execution, or delayed execution for dangerous operations.

[0082] It should be noted that the present invention is not limited to the above embodiments. Without affecting the core idea of ​​the present invention, it can also be implemented in a variety of alternative ways, including: the behavioral state S(t) can be represented by vector, matrix, probability distribution, embedding representation, graph structure or hybrid structure, and the number of dimensions can be increased or decreased according to the application scenario; the set of behavioral driving factors D(t) can be represented by rule set, causal graph, attention weight, control variable, event flow or neural network feature representation; the behavioral trajectory BT can be represented by linear sequence, branch graph, time window sequence, hierarchical trajectory or multi-agent fusion trajectory; the calculation of risk state Q(t) can be implemented by rule scoring, graph propagation, statistical model, machine learning model, deep learning model or hybrid model; dynamic security control can be executed through API gateway, proxy layer, sandbox, permission manager, memory manager, task orchestrator or manual approval system.

[0083] Example 7

[0084] This embodiment provides a long-term autonomous intelligent agent risk management system based on behavioral evolution, the overall structure of which is as follows: Figure 13 As shown. The system includes a data acquisition module 100, an extraction module 200, an analysis module 300, a calculation module 400, and an adjustment module 500. The modules are connected in sequence, and the output of the adjustment module 500 acts on the long-term autonomous intelligent agent. The operating data of the intelligent agent is fed back to the data acquisition module 100, forming a closed-loop control.

[0085] Specifically:

[0086] The acquisition module 100 is used to collect operational data during the long-term operation of the autonomous intelligent agent and construct the behavioral state S(t) of the agent at time t, which represents the comprehensive behavioral characteristics of the agent. In one embodiment, the acquisition module 100 includes a log acquisition unit and a state construction unit: the log acquisition unit acquires input data, output data, tool call data, memory read / write data, permission change data, identity switching data, and external interaction data through the log interface, message bus, or interception agent of the agent's operating environment; the state construction unit performs normalization, feature extraction, and encoding on the operational data to generate the behavioral state S(t), which can be represented in the vector form S(t)=[I(t),M(t),P(t),T(t),U(t),G(t),C(t),E(t),L(t),X(t)], and can also generate such as Figure 2 The behavior snapshot shown is Snapshot(tk).

[0087] The extraction module 200 is used to extract the set of behavioral driving factors D(t) that influence the change of behavioral state S(t), and calculate the next behavioral state S(t+Δt) based on the current behavioral state S(t) and the set of driving factors D(t) using the state transition function Φ, according to the preset set of behavioral evolution rules R. In one embodiment, the extraction module 200 includes a driving factor identification unit, a driving space construction unit, and a state transition unit: the driving factor identification unit identifies target driving factors, memory driving factors, environmental driving factors, strategy driving factors, permission driving factors, trust driving factors, identity driving factors, tool driving factors, data driving factors, and collaboration driving factors based on rule templates, classification models, or embedding matching; the driving space construction unit constructs a behavioral driving space BDS={D(t),Ω,Γ}, and represents the relationship structure and influence strength between driving factors using directed graphs, hypergraphs, tensors, weighted networks, or probabilistic graphical models, such as... Figure 3 As shown; the state transition unit implements the state transition function Φ according to the propagation, inheritance, decay, superposition, conflict, branching, convergence, reset, freeze, and upgrade rules in the behavior evolution rule set R, and outputs the behavior state S(t+Δt) at the next time step, as follows. Figure 4 As shown.

[0088] The analysis module 300 is used to organize the behavioral states of multiple consecutive time points into behavioral trajectories BT in chronological order, and to perform behavioral dynamics analysis on the behavioral trajectories BT to obtain at least one behavioral dynamics index. The analysis module 300 includes a trajectory construction unit and a dynamics analysis unit: the trajectory construction unit constructs the behavioral trajectories BT using one of the following methods: timestamp sorting, key event triggering, session splicing, graph association, or cross-agent fusion. Figure 5As shown, constraints are imposed on behavioral states based on temporal consistency, logical consistency, permission consistency, memory consistency, identity consistency, trust consistency, and collaborative consistency. The dynamic analysis unit calculates indicators such as the direction, speed, acceleration, stability, curvature, bifurcation, and convergence of behavioral state changes. Figure 6 As shown, the behavior offset ΔB=Dist(BT_cur,BT_norm) is calculated.

[0089] The calculation module 400 is used to calculate the risk state Q(t) based on the risk mapping function Ψ, the behavioral state S(t), the behavioral trajectory BT, and dynamic indicators. The calculation module 400 has a built-in risk mapping unit that implements the risk mapping function Ψ using rule scoring, graph propagation, statistical models, machine learning models, deep learning models, or hybrid models. It outputs the risk state Q(t) and its evolution stages, which include a stable stage, a pre-offset stage, an accumulation stage, a critical stage, an outbreak stage, and a regression stage, such as... Figure 7 As shown, the behavioral evolution process can be mapped into risk types such as time aggregation attack, delayed triggering attack, intent drift, permission change, multi-agent cooperative attack, and slow data outflow.

[0090] The adjustment module 500 is used to adjust the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous agent according to the risk state Q(t). The adjustment module 500 includes a strategy decision-making unit and a control execution unit: the strategy decision-making unit selects a hierarchical control strategy based on the evolutionary stage of the risk state Q(t), such as... Figure 7 As shown, for example, early warning prompts are used in the pre-offset phase, permission contraction, manual confirmation and isolation of external communication are used in the accumulation or critical phase, and the current behavior chain is terminated and audit is triggered in the outbreak phase; the control execution unit performs at least one of the following control actions through API gateway, proxy layer, sandbox, permission manager, memory manager, task orchestrator or manual approval system: restricting tool calls, reducing permission level, freezing long-term memory, trimming context information, adding manual confirmation, isolating external communication, restricting task path, terminating current behavior chain, resetting trust state and switching to low-risk execution mode.

[0091] During system operation, the various modules work collaboratively: the acquisition module 100 acquires the agent's operational data and constructs the behavioral state S(t); the extraction module 200 extracts the set of behavioral driving factors D(t) and calculates the behavioral state at the next moment based on the set of behavioral evolution rules R; the analysis module 300 organizes the behavioral states at multiple consecutive moments into behavioral trajectories BT and completes behavioral dynamics analysis; the calculation module 400 calculates the risk state Q(t) based on the risk mapping function Ψ; and the adjustment module 500 implements dynamic safety control based on the risk state Q(t). After the control results are applied to the agent, its subsequent behaviors are again acquired by the acquisition module 100, forming a closed-loop control system.

[0092] In another implementation, the system also includes an audit module and a model update module (not shown in the figures), wherein the audit module saves evidence chains such as behavior snapshots, behavior trajectories and risk evolution records; and the model update module updates the behavior baseline, risk threshold and behavior evolution rule parameters according to the running results.

Claims

1. A long-term autonomous intelligent agent risk management method based on behavioral evolution, characterized in that, Including the following steps: Collect operational data during the long-term operation of the autonomous intelligent agent, and construct a behavioral state S(t) that represents the comprehensive behavioral characteristics of the agent at time t; Extract the set of behavioral driving factors D(t) that affect the change of the behavioral state S(t), and calculate the next behavioral state S(t+Δt) based on the current behavioral state S(t) and the set of driving factors D(t) according to the preset set of behavioral evolution rules R, using the state transition function Φ. The behavioral states at multiple consecutive moments are organized in chronological order into behavioral trajectories BT, and behavioral dynamics analysis is performed on the behavioral trajectories BT to obtain at least one behavioral dynamics index. Based on the risk mapping function Ψ, the risk state Q(t) is calculated based on the behavioral state S(t), the behavioral trajectory BT, and the dynamic index. Based on the risk state Q(t), adjust the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous intelligent agent.

2. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 1, characterized in that, The operational data includes at least one of the following: input data, output data, tool call data, memory read / write data, permission change data, identity switching data, and external interaction data. The behavioral state S(t) includes the intent dimension, memory dimension, permission dimension, trust dimension, identity dimension, and data scope dimension.

3. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 1, characterized in that, The behavioral state S(t) is represented in vector form as S(t) = [I(t), M(t), P(t), T(t), U(t), G(t), C(t), E(t), L(t), X(t)], where I(t) represents the intention state, M(t) represents the memory state, P(t) represents the permission state, T(t) represents the trust state, U(t) represents the identity state, G(t) represents the data scope state, C(t) represents the planning state, E(t) represents the external interaction state, L(t) represents the tool capability state, and X(t) represents the reasoning mode state. The set of behavioral driving factors D(t) includes one or more of the following: goal-driven factors, memory-driven factors, environment-driven factors, strategy-driven factors, permission-driven factors, trust-driven factors, identity-driven factors, tool-driven factors, data-driven factors, and collaboration-driven factors. The set of behavioral evolution rules R includes at least one of the following: propagation rules, inheritance rules, decay rules, superposition rules, conflict rules, branching rules, convergence rules, reset rules, freezing rules, and upgrade rules. The behavioral dynamics indicators include at least one of the following: direction of change of behavioral state, rate of change, acceleration of change, stability of change, curvature of change, bifurcation of change, and convergence of change. The risk state Q(t) characterizes the evolution stage of the risk, which includes at least two of the following: stable stage, pre-offset stage, accumulation stage, critical stage, outbreak stage, and retreat stage.

4. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 1, characterized in that, It also includes constructing the Behavior-Driven Space (BDS), specifically including the following steps: Construct a BDS = {D(t), Ω, Γ}, where Ω represents the relationship structure between driving factors, and Γ represents the weight, strength, or influence parameter of the driving factors. The behavior-driven space is represented by directed graphs, hypergraphs, tensors, weighted networks, or probabilistic graphical models.

5. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 1, characterized in that, The risks include: A single access may be secure, but long-term cumulative access can gradually expand the scope of access, creating a high-risk time-aggregation attack. Delayed-triggered attacks where malicious conditions are written early and later triggered by driver rules. The intention to shift from a low-risk objective to a high-risk objective is a form of intentional drift. The gradual expansion of permission boundaries eventually leads to a gradual change in the availability of high-risk capabilities. Individual subsystems may appear safe when viewed separately, but when combined as a whole, they form a risky chain of multi-agent collaborative attacks. Each transmission is small, but over a long period of time, they accumulate to form a slow, complete data leak.

6. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 5, characterized in that, The methods for adjusting the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous intelligent agent include: Restrict tool calls, lower permission levels, freeze long-term memory, trim context information, add manual confirmation, isolate external communication, restrict task paths, terminate the current behavior chain, reset trust status, and switch to a low-risk execution mode.

7. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 1, characterized in that, The behavioral trajectory BT is constructed through one of the following methods: The behavior trajectory BT is constructed by connecting the time series of behavior snapshots after sorting by timestamps. When a critical event occurs, the event trigger that generates the trajectory node constructs the behavior trajectory BT; The behavior trajectory BT is constructed by stitching together the behavior snapshots from multiple sessions according to their relationships. The behavior trajectory BT is constructed by representing multiple behavioral branches, merging, and jumping relationships as graph associations in a graph structure. The behavioral trajectory BT is constructed by cross-agent fusion, which merges the local trajectories of different agents into a global trajectory.

8. The long-term autonomous intelligent agent risk management method based on behavioral evolution according to claim 7, characterized in that, Constructing the behavior trajectory BT further includes imposing state consistency constraints on the behavior states, wherein the state consistency constraints adopt one or more of the following constraint methods: A time consistency constraint is adopted, ensuring that the change in behavior state between adjacent time points does not exceed a preset threshold; Logical consistency constraint is applied: the degree of matching between behavioral state changes and task objectives and contextual logic is higher than the preset reference value; Adopt permission consistency constraints: permission changes conform to the preset permission policy; A memory consistency constraint is applied: the correlation between subsequent behavior and effective memory exceeds a preset correlation threshold. Identity consistency constraint is adopted: identity switching must meet preset authorization conditions; Adopt a trust consistency constraint: trust improvement is matched with historical trust improvement records; Cooperative consistency constraints are adopted: the state handover between multiple agents is traceable.

9. The risk management method for long-term autonomous intelligent agents based on behavioral evolution according to claim 7, characterized in that, Methods for calculating behavior offset ΔB include: Calculate ΔB=Dist(BT_cur,BT_norm), where BT_norm is the preset normal behavior baseline trajectory, BT_cur is the current behavior trajectory, and Dist is one of the following: distance function, similarity function, graph editing distance, embedding distance, or probability deviation function. The behavior offset ΔB is calculated on at least one behavior type.

10. A long-term autonomous intelligent agent risk management system based on behavioral evolution, characterized in that, include: The data acquisition module collects operational data during the long-term operation of the autonomous intelligent agent and constructs a behavioral state S(t) that represents the comprehensive behavioral characteristics of the agent at time t. The extraction module extracts the set of behavioral driving factors D(t) that affect the change of the behavioral state S(t), and calculates the behavioral state S(t+Δt) at the next moment based on the current behavioral state S(t) and the set of driving factors D(t) according to the preset behavioral evolution rule set R and the state transition function Φ. The analysis module organizes the behavioral states at multiple consecutive moments into behavioral trajectories BT in chronological order, and performs behavioral dynamics analysis on the behavioral trajectories BT to obtain at least one behavioral dynamics index. The calculation module calculates the risk state Q(t) based on the risk mapping function Ψ, the behavioral state S(t), the behavioral trajectory BT, and the dynamic index. The adjustment module adjusts the tool invocation restrictions, permission levels, and execution constraints of the long-term autonomous intelligent agent based on the risk state Q(t).

Images