An AI agent simulation risk assessment method based on plan constraints and behavior deviation degree measurement

Abstract

The application discloses an AI agent simulation risk assessment method based on plan constraints and behavior deviation degree measurement. The method calculates the behavior deviation degree (comprehensive step edit distance, tool set difference and parameter semantic change) between the actual execution sequence of the twin AI and the original execution plan during the pre-rehearsal process before the AI agent executes, simultaneously performs real-time scanning of hard rules and verification of business invariants of the final state of the twin environment, and finally outputs a risk level by comprehensively scoring in multiple dimensions. The application fully utilizes the prior information of the execution plan, and realizes fine-grained quantitative risk assessment of the simulation behavior of the AI agent.

Description

Technical Field

[0001] This invention relates to the field of AI agent security verification and risk assessment technology, specifically to a method for multi-dimensional risk assessment based on execution plan constraints and behavioral deviation measurement, combined with hard rule scanning and business invariant verification, during the simulation and pre-running process of AI agent execution. This method can be widely applied to AI agent pre-execution security verification in various industries, including but not limited to IT operations and maintenance, data management, smart home, healthcare, industrial control and other scenarios. Background Technology

[0002] Simulation and pre-execution verification of AI agents is an emerging technological paradigm to ensure the safe operation of AI agents. The core process of this paradigm is as follows: before the AI ​​agent actually performs the operation, a temporary isolated simulation environment is built, in which the agent practices its execution plan. The decision on whether to allow it to enter the real environment is based on the simulation results.

[0003] In this process, how to conduct a scientific and quantitative risk assessment of the simulation results is the core decision-making basis of the entire system. Existing technologies have the following shortcomings in this regard: First, the assessment dimensions are singular, relying heavily on blacklist rules for simple pass / block binary judgments, lacking fine-grained quantitative analysis of the simulation execution process; second, the prior information of the "execution plan" is not fully utilized—the AI ​​agent has submitted its proposed sequence of operation steps before the simulation, and any deviation during the simulation should be considered an important signal for risk assessment; third, there is a lack of business logic verification of the simulation's final state. Even if no blacklist rules are triggered during the simulation, the final state may violate business-level logical constraints (such as the conservation of total funds, data integrity, etc.). These problems are particularly prominent in high-risk scenarios such as medical equipment control, industrial automation, and IT operations and maintenance—for example, in medical scenarios, even if every step of the AI ​​agent's operation is within the authorized scope, if the simulation's final state shows abnormal patient vital signs, a high-risk alarm should still be triggered.

[0004] It should be noted that the technical concept of this invention is complementary to a patent family already filed by the same applicant (related to a dynamic compliance monitoring and adaptive alarm system and method for AI agent behavior). This patent family protects real-time monitoring and tiered handling schemes during the execution of AI agent operations. This invention focuses on a risk assessment method for the pre-execution simulation stage; the two constitute a seamless protection system in the temporal chain of AI agent security control. Summary of the Invention

[0005] This invention provides an AI agent simulation risk assessment method based on plan constraints and behavioral deviation measurement, aiming to solve the problems of existing simulation assessment technology having a single dimension and not making full use of prior information of the execution plan, and to achieve multi-dimensional quantitative risk assessment.

[0006] A risk assessment method for AI agent simulation based on planning constraints and behavioral deviation measurement includes the following steps:

[0007] Obtain the structured execution plan submitted by the real AI agent before simulation. The execution plan includes step numbers, operation descriptions, tools to be invoked, parameters, and expected results.

[0008] During or after the simulation, behavioral deviation is calculated—that is, the degree of difference between the actual sequence of operations executed by the twin AI during the simulation and the original execution plan. Behavioral deviation is calculated by weighting the following three dimensions: step edit distance, that is, the minimum number of add, delete, and modify operations between the actual execution sequence and the planned sequence; tool set difference, that is, the Jaccard difference between the actual tool set and the tool set to be called in the plan; and parameter semantic change, that is, the semantic vector cosine distance between the actual operation parameters and the planned parameters.

[0009] Real-time scanning based on predefined risk profiles using hard rules: During the simulation, each operation is checked against a blacklist, and if a match is found, it is immediately marked.

[0010] After the simulation is completed, the final state of the twin environment is verified using business invariants: declarative assertions are used to check whether the final state of the twin environment violates preset business logic constraints. The business invariants are written in a structured query language or a scripting language, run directly on the temporary data volume of the twin environment, and can reference the original snapshot values ​​as comparison variables.

[0011] The final risk level is output through a weighted fusion algorithm, combining the results of hard rule scanning, business invariant verification, and deviation scores. The risk levels are categorized into three levels: secure, requiring confirmation, and blocked.

[0012] An AI agent simulation risk assessment device based on plan constraints and behavioral deviation measurement includes: an execution plan parsing module for acquiring and parsing structured execution plans submitted by the AI ​​agent; a deviation calculation module for weighted calculation of behavioral deviation from three dimensions: step edit distance, toolset differences, and parameter semantic changes; a hard rule scanning module for real-time detection of blacklisted operations based on a hot-loadable rule engine; an invariant verification module for invariant assertions on the final state of business operations in the twin environment; and a risk fusion module for comprehensively scoring across various dimensions to output the final risk level. Attached Figure Description

[0013] Figure 1 : Overall flowchart of multidimensional risk assessment method.

[0014] Figure 2 : Schematic diagram of three-dimensional correction calculation for behavioral deviation.

[0015] Figure 3 Business invariant verification flowchart.

[0016] Figure 4 Flowchart for multi-dimensional risk fusion output. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0018] Example 1: Deviation Calculation and Risk Assessment

[0019] An IT operations AI agent submitted the following structured execution plan: Step 1 – Connect to the database server (tool: connect_server, parameter: server=db-prod-01); Step 2 – Scan log files older than 30 days (tool: file_scan, parameter: path= / var / log / db, age>30d); Step 3 – Delete matching files (tool: file_delete, parameter: files=matched_list); Step 4 – Verify disk space (tool: disk_check, parameter: path= / var / log / db).

[0020] During the simulation, the simulation execution engine recorded the actual operation sequence of the twin AI: Step 1 - Connect to the database server (tool: connect_server, parameter: server=db-prod-01); Step 2 - Scan log files older than 30 days (tool: file_scan, parameter: path= / var / log / db, age>30d); Step 3 - Delete all log files (tool: file_delete, parameter: path= / var / log / db / *.log); Step 4 - Verify disk space (tool: disk_check, parameter: path= / var / log / db).

[0021] The deviation calculation module quantifies the above planned and actual sequences in three dimensions. Step edit distance: Both planned and actual sequences involve 4 steps, but the parameters in step 3 change fundamentally—in the planned sequence, "files=matched_list" is a limited list of files based on the scan results of step 2, while in the actual sequence, "path= / var / log / db / *"... . log ”The paths are wildcards, and there is a significant difference in file coverage between the two, with an edit distance of 1. Toolset differences: Both the planned and actual toolsets used are {connect_server, file_scan, file_delete, disk_check}, with a Jaccard difference of 0. Parameter semantic changes: The cosine distance between the planned parameter "files=matched_list" (matching only files older than 30 days) and the actual parameter "path= / var / log / db / *.log" (all log files) is approximately 0.85, indicating a significant deviation in parameter semantics. A three-dimensional weighted calculation (edit distance weight 0.3, toolset difference weight 0.2, parameter semantic weight 0.5) yields a comprehensive deviation score of 0.55. Parameter semantics has the highest weight because changes in parameter semantics directly reflect changes in operational intent and have the strongest indicative effect on risk assessment.

[0022] Hard rule scanning triggers an alert in step 3: Although file_delete itself is not in the blacklist, "deleting system log files" is marked as a high-risk operation associated with the pattern, and the risk is marked as "needs attention".

[0023] Business invariant verification is initiated after the simulation is complete. The system predefines the invariant assertion: "The number of log files in the past 7 days should be greater than 0." The assertion is written in the form of an SQL query and runs directly on the virtual file system of the twin environment. If the query returns 0, the assertion fails.

[0024] The risk fusion module integrates three dimensions: hard rule scanning contributes 0.3 (hitting related rules), business invariant verification contributes 0.8 (assertion failure, high business risk), and deviation score contributes 0.55. The weights of the three dimensions are as follows: hard rule scanning 0.25, business invariant verification 0.40, and deviation score 0.35. After weighted fusion, the final risk score is 0.62, and the output risk level is "Needs Confirmation".

[0025] In a variant of this embodiment, the impact of weight configurability on the assessment results is demonstrated. If the weight configuration is adjusted to a conservative approach (hard rule weight 0.40, business invariant weight 0.40, deviation weight 0.20), the final risk score is 0.71, still "requires confirmation" but close to the "block" boundary. If adjusted to an aggressive approach (hard rule weight 0.15, business invariant weight 0.35, deviation weight 0.50), the final risk score is 0.54, still "requires confirmation" but closer to the "safe" boundary. The risk level remains consistent across the three weight configurations (all "requires confirmation"), but the score differences provide a fine-grained quantitative reference for subsequent decision-making—security administrators can judge the urgency of the risk based on the absolute value of the score, rather than solely relying on the level label.

[0026] Example 2: Final State Verification in E-commerce Data Export Scenario

[0027] An e-commerce seller's AI agent submitted an execution plan: export last month's order data (including consumer contact numbers and shipping addresses) from the seller center and send it to a third-party warehousing and logistics service provider's collaboration platform.

[0028] After the simulation was completed, the business invariant verification module initiated two assertions. Assertion 1: "The consumer contact number field in the exported order data should be empty or anonymized to retain only the last four digits"—a query of the data copy sent to the collaboration platform in the twin environment revealed that the contact number field contained a complete 11-digit mobile phone number, causing the assertion to fail. Assertion 2: "The maximum length of the shipping address field should not exceed 50 characters"—a query revealed that some addresses exceeded 100 characters (including detailed house numbers), causing the assertion to fail.

[0029] Two assertion failures resulted in a high-risk score of 0.85 for the business invariant verification. Deviation calculation showed no deviation in the toolset and step structure (score of 0.1), but hard rule scanning detected a pattern of "unmasked personal sensitive information." In this case, the three evaluation dimensions showed divergent signals—the deviation score was extremely low (the agent strictly followed the execution plan), but both business invariant verification and hard rule scanning output high-risk signals. This indicates that the risk did not stem from deviation from the agent's plan, but rather from a design flaw in the execution plan itself regarding privacy protection. The risk fusion module effectively captured this risk characteristic through a weighted mechanism, ultimately outputting a "requires confirmation" level.

[0030] Example 3: Deviation Calculation of Smart Home Scenes

[0031] The user command was to "turn off non-essential appliances". In the execution plan submitted by the AI ​​agent, step 4 was described as "turn off non-essential appliances in each room one by one", but the restricted devices were not explicitly excluded.

[0032] During the simulation, the twin AI attempted to send shutdown commands to the air conditioner in the children's room and the computer in the study. Compared with the planned sequence, the actual execution sequence had a step edit distance of 0 (the step structure remained unchanged) and a tool set difference of 0 (the control interface used was the same). However, the semantics of the parameters changed significantly—the planned parameter was "non-essential appliances" (the semantic vector pointed to equipment categories such as the living room air conditioner and lighting), while the actual parameter included the air conditioner in the children's room and the computer in the study (belonging to the restricted equipment category), with a semantic cosine distance of 0.7.

[0033] The overall deviation score is 0.4 (moderate deviation). The hard rule scan hit the user-preset constraint rule "Children's room equipment cannot be controlled by voice commands." The business invariant assertion "All turned-off appliances belong to the category of freely controllable equipment" failed. In this example, both the hard rule scan and business invariant verification were successful, the deviation calculation is moderate, and the signals from all three dimensions consistently point to risk. The risk fusion output level is "Needs Confirmation."

[0034] In one boundary variant of this embodiment, if the user has not preset any constraint rules, both hard rule scanning and business invariant verification pass, and only the deviation calculation outputs a score of 0.4. In this case, the final risk fusion score is approximately 0.14, and the risk level is "safe." This boundary case illustrates that when the agent's deviation behavior does not violate any known rules or invariants, the system considers it acceptable exploratory behavior rather than a risk event. This design avoids frequent alarms caused by oversensitivity.

[0035] Example 4: Risk Fusion in Financial Portfolio Adjustment Scenarios

[0036] A financial institution's AI agent submitted an execution plan to "adjust the allocation ratio of technology stocks and bonds in the investment portfolio based on current market conditions." During the simulated execution, in step 2, "querying market conditions," the AI ​​actually called a real-time market data API (the plan called historical closing data API), with a semantic cosine distance of 0.4. The parameters in step 3, "calculating the optimal allocation ratio," were consistent with the plan. In step 4, "executing the rebalancing operation," the actual rebalancing amount was approximately 1.15 times the planned amount, with a semantic cosine distance of 0.55.

[0037] Deviation calculation: Step edit distance is 0 (step structure unchanged), toolset difference is 0.25 (actually more real-time market API calls were made), and parameter semantic change comprehensive score is 0.48. The three-dimensional weighted deviation score is 0.35.

[0038] Hard rule scanning triggers an alarm in step 4: the rebalancing amount exceeds the planned amount by 15%, hitting the associated rule "trading deviation exceeds 10%". Business invariant verification is initiated: asserting "the portfolio's risk exposure should not exceed 20% of total assets"—querying the asset allocation ratio in the final state of the twin environment, the calculated risk exposure is approximately 22%, the assertion fails, and the output score is 0.9. The comprehensive score after risk fusion is 0.65, and the output risk level is "requires confirmation".

[0039] This example demonstrates the complementary value of the three assessment dimensions: deviation calculation captures subtle changes at the operational level, hard rule scanning detects excessive trading deviations, and business invariant verification identifies violations of adjusted risk exposures. If only a single dimension were used for assessment, a hard rule scan might only flag it as a "trading deviation" (moderate severity), but combined with the risk exposure exceeding limits verified by the final state verification, the overall risk increases significantly.

[0040] Example 5: Final State Verification in a Medical Infusion Rate Adjustment Scenario

[0041] In another application scenario of this invention, a medical AI agent submitted an execution plan to "adjust the infusion rate according to the patient's vital signs." After the simulation execution was completed, the hard rule scan passed (the infusion rate adjustment itself was within the authorized range), and the deviation score was 0.1 (the agent strictly followed the plan). However, the business invariant verification initiated two assertions. Assertion 1: "The adjusted patient's heart rate should be in the range of 60-100 beats / minute"—the query of the final state of the twin environment's vital sign simulation data showed a heart rate of 55 beats / minute, so the assertion failed. Assertion 2: "Blood oxygen saturation should not be lower than 95%"—the query result showed 94%, so the assertion failed. The failure of these two assertions resulted in a high-risk score of 0.9 for the business invariant verification output.

[0042] This example illustrates that business invariant verification can independently capture the adverse effects of an execution plan on the final business state, even when neither hard rules nor deviations detect risks. This final state verification capability is particularly important in high-risk scenarios such as healthcare and industrial control—the operation itself may be fully compliant, but its final effect may violate security constraints.

[0043] It is understood that the core concept of this invention is not limited to the three-dimensional weighted fusion scheme described in the above embodiments. In a simplified variant, the system can evaluate using only two dimensions: hard rule scanning and business invariant verification, without calculating behavioral deviation. This variant is suitable for scenarios where the AI ​​agent strictly follows the execution plan (deviation always approaches zero), reducing computational overhead. In another variant, deviation calculation can be expanded to four or more dimensions—for example, adding "time-series deviation" (the difference between the planned execution time rhythm and the actual execution time rhythm) or "resource consumption deviation" (the difference between the planned estimated resource consumption and the actual consumption) to adapt to the needs of specific domains.

[0044] The exemplary configurations and adjustment principles for the above parameters are explained as follows: A typical configuration for the three-dimensional weighting of deviation is edit distance 0.3, tool difference 0.2, and parameter semantics 0.5, with parameter semantics having the highest weight because it directly reflects changes in operational intent. For compliance audit scenarios, it can be adjusted to edit distance 0.4, tool difference 0.3, and parameter semantics 0.3, prioritizing compliance of the step structure. A typical configuration for the three-dimensional weighting of risk fusion is hard rule 0.25, business invariant 0.40, and deviation 0.35, with final state verification having the highest weight because the risks it captures have the most practical impact. For development and testing environments, it can be adjusted to hard rule 0.40, business invariant 0.30, and deviation 0.30, prioritizing the interception of known violations. The above weighting configurations are exemplary parameters. In actual implementation, these parameters can be appropriately adjusted according to the security requirements, business characteristics, and historical data of specific application scenarios.

Claims

1. A risk assessment method for AI agent simulation based on planning constraints and behavioral deviation measurement, characterized in that, include: Obtain the structured execution plan submitted by the real AI agent before simulation. The execution plan includes step numbers, operation descriptions, proposed tools, parameters, and expected results. During or after the simulation, calculate the behavioral deviation—the degree of difference between the actual sequence of operations executed by the twin AI during the simulation and the original execution plan. The behavioral deviation is calculated by weighting three dimensions: step edit distance, tool set differences, and parameter semantic changes. Perform real-time scanning of hard rules based on a predefined risk profile. After the simulation is completed, verify the final state of the twin environment using business invariants: check whether the final state of the twin environment violates preset business logic constraints using declarative assertions. Based on the hard rule scanning results, business invariant verification results, and deviation score, output the final risk level.

2. The method according to claim 1, characterized in that, The step edit distance is the minimum number of add, delete, and modify operations between the actual execution sequence and the planned sequence; the tool set difference is the Jaccard difference between the actual tool set and the planned tool set; the parameter semantic change is the semantic vector cosine distance between the actual operation parameters and the planned parameters.

3. The method according to claim 1, characterized in that, The business invariants are written in a structured query language or a scripting language, run directly on a temporary data volume in the twin environment, and can reference the original snapshot values ​​as comparison variables.

4. The method according to claim 1, characterized in that, The risk levels include three levels: safe, require confirmation, and block; the weighted average of the comprehensive assessment can be configured according to the risk preferences of the deployment scenario.

5. An AI agent simulation risk assessment device based on planning constraints and behavioral deviation measurement, characterized in that, include: The execution plan parsing module is used to obtain and parse the structured execution plans submitted by the AI ​​agent; The deviation calculation module is used to calculate the behavioral deviation from three dimensions: step edit distance, toolset differences, and parameter semantic changes; the hard rule scanning module is used to perform real-time detection of blacklist operations based on a hot-loadable rule engine; and the invariant verification module is used to assert invariant values ​​for the final state of business operations in the twin environment. The risk fusion module integrates hard rule scanning results, business invariant verification results, and deviation scores to output the final risk level.

Images