Risk-controlled operation and maintenance handling method and system based on multi-agent task graph orchestration

Abstract

This invention discloses a risk-controlled operation and maintenance (O&M) method and system based on multi-agent task graph orchestration, belonging to the field of intelligent and automated O&M technology. Addressing the problems of uncontrollable change risks, unverifiable execution, lack of rollback guarantees, and insufficient audit evidence in existing automated O&M, this invention achieves full-process autonomy from fault perception to change intent understanding, task graph generation, risk assessment, approval access control, idempotent canary execution, effect verification, automatic rollback upon failure, and audit accounting, under the constraints of dependency topology, configuration baseline, and policy guardrails. This invention can achieve minute-level self-healing and traceable O&M while ensuring business continuity, and is applicable to microservice system O&M scenarios in enterprise private cloud, public cloud, and hybrid cloud environments.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance and automated operation and maintenance technology, and in particular to a risk-controlled operation and maintenance handling method and system based on multi-agent task graph orchestration. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Intelligent Operations and Maintenance (AIOps) refers to a technical system that monitors, analyzes, and manages the operational status of information systems through observable data collection, anomaly identification, root cause analysis, automated orchestration, and change execution. As enterprise production systems evolve towards cloudification, microservices, and multi-cluster architectures, the dependencies between services are becoming increasingly complex. The impact of a single failure or change often spans multiple call chains and technology stacks, significantly increasing the demands for precision and controllability in operations and maintenance.

[0004] While existing AIOps solutions have introduced anomaly detection and root cause analysis capabilities, their focus is primarily on the problem discovery phase, with limited involvement in how to safely and controllably translate diagnostic conclusions into actionable change plans while ensuring business continuity. Traditional SOAR or Runbook solutions, relying on pre-programmed fixed scripts and processes, lack dynamic adaptability in the face of complex topology dependencies and long-tail fault scenarios, making it difficult to meet the comprehensive security and compliance requirements of automated operations and maintenance in production environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a risk-controlled operation and maintenance (O&M) management method and system based on multi-agent task graph orchestration. It unifies the collection of multi-source O&M data and constructs standardized event objects. By planning intelligent agents to generate O&M task graphs with idempotent keys and rollback strategies, and having execution agents execute tasks in batches according to a gray-scale ratio, and verification agents verify target achievement based on service level indicators, it solves the technical problems of uncontrollable change risks, unverifiable execution, lack of rollback guarantees, and insufficient audit evidence in automated O&M processes through timely submission and automatic rollback at rollback points in case of failure. This achieves a self-governing O&M closed loop with quantitative risk assessment, verifiable effects, rollback capability in case of failure, and full-process auditability throughout the entire change process.

[0006] On the one hand, a risk-controlled operation and maintenance management method based on multi-agent task graph orchestration is provided, including: Metrics, logs, tracing, alarm events, work order information, and change records are collected from the observability platform and operation and maintenance system. After time alignment and entity parsing, event objects are constructed by combining CMDB, dependency topology, and configuration baseline. The diagnostic agent outputs root cause candidates and evidence chains based on the event objects; The planning agent generates an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects; The simulation evaluation agent performs a pre-simulation of the operation and maintenance task graph and calculates a risk score based on the topological impact surface and historical baseline. The guardrail intelligent agent determines the approval level and grayscale ratio based on the risk score, and performs compliance verification and approval access control. The execution agent executes the operation and maintenance task graph in batches according to the grayscale ratio. After each batch is executed, the verification agent is called to verify the execution results of the current batch based on the target achievement of service level indicators. When the verification is successful, the execution agent submits the current batch and enters the next batch. When the verification fails, the execution agent immediately stops the subsequent batches and triggers automatic rollback. The verification agent writes the evidence of the entire process into the audit ledger in the form of a hash chain.

[0007] Furthermore, the event object includes at least: event identifier, time window, affected services and instances, dependent topology slices, configuration version fingerprint, evidence index, target state, constraint set, and rollback point; wherein, the evidence index uses a reference method to point to the original telemetry data, query statement, time window, and external system query results; the target state is used to describe the verifiable result that is expected to be achieved in this operation and maintenance action; the constraint set includes change window, permission range, upper limit of grayscale ratio, and rollback requirements; the rollback point corresponds to the configuration version, image version, status file version, or resource snapshot.

[0008] Furthermore, the diagnostic agent outputs root cause candidates and evidence chains based on the event object, specifically as follows: The diagnostic agent determines the anomaly propagation path based on dependent topology slicing, narrowing the suspicious range to candidate nodes on the anomaly propagation path; it identifies the main delay contributing factors by combining critical path tracing analysis; it identifies the distribution of anomaly log templates through log template clustering; it aligns the above analysis results with the most recent change records for correlation, calculates the confidence score of each candidate root cause, and outputs one or more root cause candidates in descending order of confidence; each root cause candidate is attached with an evidence index, which is used to point to the original telemetry data, query statements, time windows, and external system query results.

[0009] Furthermore, the planning agent generates an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects, specifically as follows: The planning agent matches the recovery actions corresponding to the root cause candidates with the target state, filters the candidate actions according to the constraint set, and excludes actions that do not meet the requirements of change window, grayscale upper limit, or rollback; the actions that meet the constraint set are organized into an operation and maintenance task graph according to the dependency relationship. The operation and maintenance task graph is a directed acyclic graph or a state machine graph with rollback edges. Its nodes represent operation and maintenance actions or verification actions, and the edges represent dependencies and execution order. Each operation and maintenance action node is configured with an idempotent key, preconditions, execution interface, timeout threshold, number of retries, and rollback interface, and the corresponding rollback strategy is bound according to the rollback point in the event object.

[0010] Furthermore, the simulation evaluation agent performs a pre-analysis of the operation and maintenance task graph, and calculates a risk score based on the topological impact surface and historical baseline, specifically as follows: The simulation evaluation agent performs a pre-play of the operation and maintenance task graph in a digital twin environment, shadow environment, or read-only simulation environment, estimates the impact surface based on the dependent topology slice, and outputs conflict detection results, resource capacity prediction results, and suggested rollback points. The risk score is calculated based on at least the following factors: the business importance weight of the affected service, the size of the topology impact, the risk level of the change type, the success rate of similar historical changes, and the deviation of the current load and error rate baseline. The risk score is then mapped to the approval level and grayscale ratio, with a higher score resulting in a higher approval level and a lower grayscale ratio.

[0011] Furthermore, the smart fence agent determines the approval level and grayscale ratio based on the risk score, and performs compliance verification and approval access control, specifically as follows: The guardrail agent loads a pre-built policy rule base using a policy-as-code approach. This policy rule base includes at least minimum permission constraints, whitelist and blacklist constraints, change window constraints, compliance rule constraints, and business continuity constraints. The guardrail agent performs policy and permission checks on the operation and maintenance task graph. When all constraints are met, it outputs a compliance approval conclusion; when any constraint is not met, it outputs a blocking conclusion and converts the operation and maintenance task graph into a read-only suggestion. After policy and permission checks are passed, the guardrail agent determines the approval level based on the risk score, triggers the approval process corresponding to the approval level, and only after approval can the operation and maintenance task graph be released to the execution agent.

[0012] Furthermore, the execution agent executes the maintenance task graph in batches according to the aforementioned grayscale ratio. After each batch is executed, the verification agent is invoked to verify the achievement of the target based on service level indicators in the execution results of the current batch. Specifically: The execution agent compiles the operation and maintenance task graph into a state machine, with node states including at least pending execution, execution in progress, execution successful, execution failed, rollback in progress, and rollback completed. It determines the instance range for each batch of execution according to the grayscale ratio and executes operation and maintenance actions batch by batch. After each batch is completed, it calls the verification agent to verify the execution results of the current batch. The verification agent collects service level indicators within the observation window and determines whether they meet the corresponding service level target. It also combines synthetic probing transactions, critical business regression test cases, and the rate of change trend relative to the baseline window to output a pass or fail verification conclusion. If verification passes, the execution agent submits the current batch and proceeds to the next batch. If verification fails, the execution agent immediately stops subsequent batches and performs automatic rollback according to the rollback point bound to the operation and maintenance task graph. The automatic rollback includes configuration snapshot recovery, image version rollback, execution of rollback scripts, or infrastructure-as-code state rollback.

[0013] On the other hand, a risk-controlled operation and maintenance management system based on multi-agent task graph orchestration is also provided, including: The unified telemetry access and event objectification module is configured to: collect metrics, logs, tracing links, alarm events, work order information and change records from the observability platform and operation and maintenance system, and after time alignment and entity parsing, construct event objects in combination with CMDB, dependency topology and configuration baseline; The diagnostic agent module is configured to output root cause candidates and evidence chains based on the event object; The planning intelligent agent module is configured to generate an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects; The simulation evaluation agent module is configured to: perform a pre-simulation of the operation and maintenance task graph and calculate a risk score based on the topological impact surface and historical baseline; The guardrail intelligent agent module is configured to: determine the approval level and grayscale ratio based on the risk score, and perform compliance verification and approval access control; The execution intelligent agent module is configured to execute the operation and maintenance task map in batches according to the gray scale ratio. After each batch is executed, the verification intelligent agent is called to verify the target achievement based on the service level index of the current batch. When the verification is passed, the current batch is submitted and the next batch is entered. When the verification fails, the subsequent batches are stopped immediately and the execution is automatically rolled back. The verification agent module is configured to: verify the achievement of the target based on service level indicators for the execution results of the current batch, and write the evidence of the entire process into the audit ledger in the form of a hash chain.

[0014] In another aspect, a computer device is also provided, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein when the processor executes the program, it performs the method described in the first aspect.

[0015] In another aspect, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, performs the method described in the first aspect.

[0016] The above technical solution has the following advantages or beneficial effects: By simulating and evaluating the combination of features of the intelligent agent calculating risk scores based on the topological influence surface and historical baseline, and the guardrail intelligent agent performing compliance verification and approval access control, this invention completes multi-dimensional risk quantification assessment and pre-compliance verification before the change is executed. This effectively prevents non-compliant or high-risk operations from entering the execution stage and significantly reduces the probability of production accidents caused by configuration changes or fault repair operations. Compared with the existing automated operation and maintenance solutions that lack risk assessment and approval access control, this invention achieves pre-controllable change risks.

[0017] By combining features such as planning an operation and maintenance task graph that generates nodes and binds rollback strategies to intelligent agents, executing intelligent agents in batches according to a gray-scale ratio, and verifying intelligent agents to verify the achievement of goals based on service level indicators, the system submits the verification when it passes and automatically executes the rollback at the rollback point when it fails. This achieves real-time gray-scale verification and rollback capability in case of failure. Compared to traditional solutions that rely on manual judgment of execution effects and manual rollback, the minute-level closed-loop self-healing capability of this invention can significantly shorten fault recovery time. At the same time, by executing in batches in a gray-scale manner, the potential impact of changes is controlled to a minimum, effectively ensuring business continuity during the change process.

[0018] The diagnostic agent outputs root cause candidates with accompanying evidence indexes, while the verification agent writes the entire process evidence, including plan summary, execution parameters, and verification conclusions, into the audit ledger in the form of a hash chain. Compared to existing technologies where execution process records are scattered and lack anti-tampering mechanisms, this invention achieves end-to-end traceability and immutability from fault diagnosis to change execution and effect verification, significantly improving the compliance auditing capabilities of operation and maintenance operations and enabling post-event auditability of the operation and maintenance process.

[0019] Successful operation and maintenance task diagrams, risk scores, and verification results are compiled into reusable strategy templates and case libraries. As the number of operation and maintenance operations increases, the efficiency and accuracy of plan generation in subsequent similar scenarios continue to improve, the level of intelligence of automated operation and maintenance is constantly improved, and the operation and maintenance costs are gradually reduced. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart illustrating the autonomous operation and maintenance method described in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the multi-agent collaboration and barrier release mechanism described in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of the audit entry and failure rollback closed loop process described in Embodiment 1 of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. Those skilled in the art should understand that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] Example 1 This embodiment provides a risk-controlled operation and maintenance method based on multi-agent task graph orchestration, aiming to solve problems such as uncontrollable change risks, lack of rollback guarantees, unverifiable execution effects, and insufficient audit evidence in existing automated operation and maintenance. This invention can be deployed in enterprise private cloud, public cloud, or hybrid cloud environments and can be integrated with existing observability platforms, ITSM, CMDB, and CI / CD pipelines. For ease of understanding, the following explanation uses the fault self-healing and high-risk change control of a microservice system as an example.

[0025] Figure 1 This is an overall flowchart of the method in Embodiment 1 of the present invention, as follows: Figure 1 As shown, the method includes the following steps: S1: Collect metrics, logs, tracing data, alarm events, work order information, and change records from the observability platform and operation and maintenance system. After time alignment and entity parsing, construct event objects by combining CMDB, dependency topology, and configuration baseline. S2: The diagnostic agent outputs root cause candidates and evidence chains based on event objects; S3: The planning agent generates an operation and maintenance task graph based on the target state and constraint set in the root cause candidates and event objects; S4: The simulation evaluation agent performs a pre-drill of the operation and maintenance task graph and calculates the risk score based on the topological impact surface and historical baseline; S5: The guardrail intelligent agent determines the approval level and grayscale ratio based on the risk score, and performs compliance verification and approval access control. S6: The execution agent executes the operation and maintenance task graph in batches according to the gray scale ratio. After each batch is executed, the verification agent is called to verify the execution results of the current batch based on the service level indicators to achieve the target. When the verification is successful, the execution agent submits the current batch and enters the next batch. When the verification fails, the execution agent immediately stops the subsequent batches and triggers the automatic rollback of the execution. The verification agent writes the evidence of the whole process into the audit ledger in the form of a hash chain.

[0026] like Figure 2 As shown, this method achieves full-process autonomy from fault perception to diagnosis, planning, evaluation, approval, execution, verification, rollback, and auditing through a collaborative closed loop of six intelligent agents: a diagnostic agent, a planning agent, a simulation evaluation agent, a guardrail agent, an execution agent, and a verification agent. Using a unified event object as the semantic carrier, it realizes this autonomy. Specifically, the diagnostic agent generates root cause candidates based on evidence citations and provides interpretable evidence chains; the planning agent expresses the disposal plan as an executable task graph under the target state and constraint set; the simulation evaluation agent estimates the impact surface and checks for conflicts on the task graph based on the topology and configuration baseline, and provides risk scores and gray-scale suggestions; the guardrail agent brings compliance, permissions, and approval access control to the planning stage, transforming execution access conditions into computable policy judgments; the execution agent implements execution under idempotency, gray-scale, retry, and rollback mechanisms; and the verification agent collects service level indicators, determines whether the corresponding service level target has been achieved, and verifies the execution effect by combining synthetic probing transactions and key business regression test cases, providing a decision to submit or rollback.

[0027] In this embodiment, the specific implementation of each step is as follows.

[0028] Step S1 is the data foundation and semantic starting point of the entire autonomous operation and maintenance closed loop. Its purpose is to transform the raw telemetry data collected from different operation and maintenance tools and observability platforms into standardized event objects that can be recognized and invoked by subsequent intelligent agents.

[0029] Specifically, the process begins with continuous access to metrics, logs, tracing data, event data, ticket information, and change records from the observability platform and operations system. The observability platform can include a metric collection system, a log retrieval system, and a tracing system, while the operations system can include a continuous integration / continuous delivery system, a ticket system, and a configuration management database. Data from different sources differs in timestamp format and service naming conventions. To address this issue, time alignment and entity resolution operations are performed on the collected data during the access phase: time alignment unifies data reported by different systems within the same time window to the same time coordinate system, while entity resolution maps the identity identifiers (such as service name, instance identifier, Pod name, node information, and version number) of the same service across different systems, ensuring stable identification of the same service across different data sources.

[0030] Based on the above mapping, the processed data is combined with asset information in the CMDB, inter-service dependency topology relationships, and preset configuration baselines for correlation and aggregation. This is used to form affected services, dependency topology slices, and configuration version fingerprints, and to construct standardized structured event objects. These event objects serve as a unified semantic carrier for subsequent intelligent agent identification and invocation, unifying diagnostic conclusions, change intentions, execution plans, verification evidence, and rollback points within the same data structure, thereby resolving the semantic fragmentation problem between different operation and maintenance tools.

[0031] In the aforementioned aggregation process, abnormal signals within the same time window are aggregated to form reproducible evidence references. These evidence references do not involve copying the original data into the system; instead, they are abstracted into reproducible queries and pointers, such as log query statements and indexes, indicator sequence names and tag sets, trace ID sets, change order or pipeline numbers, and configuration snapshots or image summaries. This design allows auditors to retrieve and review any conclusions retrospectively using the same references, avoiding the storage overhead of copying large amounts of data and supporting audit traceability.

[0032] An event object includes at least an event identifier, time window, affected services and instances, dependent topology slices, configuration version fingerprint, evidence index, target state, constraint set, and rollback point. The evidence index references the original telemetry data, query statements, time windows, and external system query results. The target state describes the verifiable result expected from this autonomous operation, such as the error rate falling below a threshold and running stably for a specified duration (e.g., 5-10 minutes). The constraint set includes change windows, permission ranges, upper limits for grayscale ratios, and rollback requirements. The constraint set describes production governance requirements, such as allowing execution only within the maintenance window, prohibiting direct deletion of core resources, and ensuring the grayscale ratio does not exceed a preset upper limit. The rollback point corresponds to the configuration version, image version, status file version, or resource snapshot. The rollback point describes the system's recovery method when verification fails, such as reverting to the previous stable version's image digest or restoring to a specified configuration snapshot identifier. The event object also includes abnormal symptoms, including error rate, latency, alarms, abnormal logs, and slow tracing.

[0033] By uniformly writing the above elements into the event object, subsequent planning and execution no longer rely on the implicit experience of operations and maintenance personnel, but are instead made verifiable decisions by the machine within clearly defined constraints.

[0034] To enhance the traceability and rollback capability of event objects, a change association alignment operation can also be performed when constructing event objects: CI / CD release records, work order changes, and configuration drifts are associated according to time windows and service topologies to generate a recently changed field; at the same time, a configuration version fingerprint is generated, including configuration file hash value, image digest, or infrastructure i.e. code module version, for subsequent rollback operations and version comparison.

[0035] Furthermore, the evidence index can be layered according to evidence type: read-only evidence corresponds to query operations, lightweight evidence corresponds to sampled logs or trace fragments, and heavy evidence may include host snapshots, container images, or configuration snapshots. The guardrail agent can only allow the collection of read-only evidence before approval, thereby reducing the risk of intrusion into the production system.

[0036] Through the processing in step S1 above, the raw telemetry data is unified into standardized structured event objects. These event objects carry abnormal symptoms, evidence indexes, and engineering execution constraints, providing a unified semantic foundation and data input for subsequent root cause analysis of diagnostic agents, task graph generation of planning agents, risk scoring of simulation evaluation agents, compliance verification of guardrail agents, and execution and verification processes.

[0037] In step S2, the diagnostic agent analyzes the abnormal propagation path, call chain, log pattern and recent changes based on the abnormal symptoms, dependency topology slices, evidence index and configuration fingerprint contained in the event object constructed in step S1, and outputs one or more root cause candidates and corresponding evidence chains.

[0038] Specifically, the diagnostic agent first determines the anomaly propagation path based on dependent topology slicing and narrows the suspicious scope to candidate nodes on the anomaly propagation path. In one implementation, when the downstream service error rate increases, it retains its upstream call path and nodes that exhibit anomalies within the same time window, excluding services that are not on the propagation path and do not exhibit anomaly signals. Subsequently, it combines critical path tracing analysis to determine the main delay contributing factors, such as identifying which call chain (span) contributes the P95 delay, and clustering high-frequency error templates into clusters through log template clustering to identify the distribution pattern of abnormal logs, thereby compressing candidate root causes to the actionable component level.

[0039] After completing the above analysis, the diagnostic agent aligns the analysis results with the most recent change logs to calculate change correlations, such as whether the probability of both error templates and delays occurring simultaneously increases significantly after the release of the same component. Based on this, the diagnostic agent evaluates each candidate root cause using a scoring mechanism combining evidence coverage and consistency: coverage measures how much of the anomalous signals a candidate root cause can explain, including anomalous indicators, error logs, and slow tracking records; consistency measures whether the candidate root cause matches the most recent change logs and the propagation direction of the dependency topology.

[0040] After summarizing the above scores, the diagnostic agent outputs one or more root cause candidates in descending order of confidence. Each candidate must be accompanied by an evidence index. The evidence index is used to point to the original telemetry data, query statements, time windows, and external system query results, enabling the root cause conclusions to be retrieved, reproduced, and verified.

[0041] Through the processing in step S2 above, the diagnostic agent transforms the original symptom information in the event object into structured root cause candidates and verifiable evidence chains, providing a clear basis for fault location and handling direction for the task graph generation of the planning agent in the next stage.

[0042] In step S3, the planning agent transforms the root cause candidates obtained in step S2 and the target state and constraint set in the event object constructed in step S1 into an executable, rollbackable and acceptable operation and maintenance task graph.

[0043] Specifically, the planning agent first matches the recovery actions corresponding to the root cause candidates with the target state. For example, when the target is to restore the service level target and the constraint set is that the grayscale ratio does not exceed the preset upper limit and must be rollbackable, the planning agent will prioritize low-intrusion, reversible actions (such as capacity expansion, rate limiting, adjusting timeout or retry parameters) as candidate actions. Subsequently, the planning agent filters the candidate actions according to the constraint set in the event object, excluding actions that do not meet the requirements of change window, grayscale upper limit, or rollback. On this basis, the actions that meet the constraint set are organized into an operation and maintenance task graph according to the dependency relationship. This operation and maintenance task graph is a directed acyclic graph or a state machine graph with rollback edges. Its nodes represent operation and maintenance actions or verification actions, and the edges represent dependencies and execution order. Each operation and maintenance action node is configured with an idempotent key, preconditions, execution interface, timeout threshold, number of retries, and rollback interface, and the corresponding rollback strategy is bound according to the rollback point in the event object. When low-intrusion actions cannot meet the objectives, the planning agent will generate more intrusive actions (such as rolling back versions or migrating traffic) and explicitly insert approval nodes and shadow rehearsal nodes into the task graph.

[0044] Optionally, the planning agent can adopt a hierarchical task network (HTN) or a template-based task decomposition method: first, decompose the high-level goal (such as the service level restoration goal) into policy layer goals (rate limiting, scaling up, or rollback), and then further decompose it into specific actions (calling the application programming interface of the container orchestration platform (such as the Kubernetes API), modifying the gateway configuration, or rolling back the release order). Each action is bound to preconditions, execution commands or interfaces, rollback commands or interfaces, verification probes, and timeout thresholds.

[0045] Through the processing in step S3 above, the planning agent transforms root cause candidates and recovery targets into a structured operation and maintenance task graph. This operation and maintenance task graph contains the complete execution order, dependencies, and execution parameters and rollback strategies for each node.

[0046] In step S4, the simulation evaluation agent takes the operation and maintenance task graph generated in step S3 and the dependent topology slices, configuration baselines and current load status in the event objects constructed in step S1 as inputs, and performs a pre-simulation of the operation and maintenance task graph in a digital twin environment, shadow environment or read-only simulation environment. It outputs conflict detection results, resource capacity prediction results and suggested rollback points, and calculates risk scores based on topology impact surface, historical similar change records and current operating baseline.

[0047] The simulation evaluation agent is used to quantify the potential impact and risk level of a change operation in a production environment. In one implementation, the simulation evaluation agent uses dependency topology slicing to estimate the impact surface: the impact surface includes the number of service nodes affected by the change, the number of critical business links affected, and the fan-out and fan-in status of dependencies; it also combines the configuration baseline deviation (e.g., the magnitude of the parameter change), the success rate of similar historical changes, the current system load status (e.g., whether it is during peak business hours), and the error rate baseline deviation for a comprehensive evaluation.

[0048] Based on this, a simulation evaluation of the intelligent agent's computational comprehensive risk score is conducted. This score is based on at least the following factors: the business importance weight of the affected service, the size of the topology impact surface, the risk level of the change type, the success rate of similar historical changes, the deviation of the current load and error rate baseline, and whether high-risk resource categories are involved. The risk score is not limited to providing a quantitative score, but is directly used for decision-making. The score is mapped to the corresponding approval level and grayscale ratio.

[0049] Specifically, the simulation evaluation agent determines the approval level and grayscale ratio based on the risk score: when the risk is low, automatic execution is allowed and the first grayscale ratio is used; when the risk is medium, on-duty personnel are required to confirm and a second grayscale ratio lower than the first grayscale ratio is used; when the risk is high, shadow simulation and dual approval are required and only the suggestion mode is allowed; the first grayscale ratio and the second grayscale ratio are pre-configured according to business importance and governance strategy.

[0050] Through the processing in step S4 above, the simulation evaluation agent integrates and analyzes the operation and maintenance task diagram generated in step S3 with the topology, load and historical data of the current production environment, outputs risk scores, suggested gray scale ratios and approval levels, and fills the suggested rollback points back into the task diagram. This provides clear risk quantification basis and decision parameters for the subsequent compliance verification and approval access control of the guardrail agent and the batch gray scale execution of the execution agent.

[0051] In step S5, the guardrail agent uses the risk score output in step S4 and the operation and maintenance task diagram reinforced by the risk assessment as input to perform compliance verification, permission verification, and approval access control on the operation and maintenance task diagram, ensuring that only plans that meet the preset governance requirements can enter the execution stage. The core task of the guardrail agent is to pre-define compliance and production governance requirements as machine-readable policy rules, blocking unacceptable operations at the source from the plan generation stage.

[0052] Specifically, the guardrail agent loads a pre-defined policy rule base using a policy-as-code approach. This policy rule base includes at least the least privilege constraint, whitelist and blacklist constraints, change window constraints, compliance rule constraints, and business continuity constraints. It can express "prohibited actions" (e.g., prohibiting the direct deletion of core namespace resources), "necessary conditions" (e.g., database parameter changes must have a rollback point and be in a maintenance window), and "permission constraints" (e.g., only tokens with the least privilege can be obtained through temporary credential proxies).

[0053] Based on this, the guardrail intelligent agent performs policy and permission verification on the operation and maintenance task diagram, checking item by item whether the task diagram meets each policy rule. For example, whether changes are executed within the allowed maintenance window, whether the operation object is in the whitelist, whether the operation has a complete rollback plan, and whether the permission scope of temporary credentials matches the operation requirements, etc.

[0054] Based on the verification results, the guardrail agent outputs corresponding verification conclusions: when all constraints are met, a compliance pass conclusion is output; when any constraint is not met, a blocking conclusion is output and the operation and maintenance task graph is converted into a read-only suggestion, preventing it from entering the execution phase. Simultaneously, the guardrail agent writes the triggered policy rule clauses and verification context into audit evidence for subsequent audit traceability.

[0055] After compliance verification is passed, the guardrail agent determines the approval level based on the risk score and triggers the corresponding approval process. The approval access control adopts a tiered strategy: automatic execution when the risk score is below the first threshold; confirmation from on-duty personnel is required when the risk score is between the first and second thresholds; dual approval and successful pre-rehearsal in a shadow environment are required when the risk score is not lower than the second threshold; for actions involving core databases or permission changes, approval is required regardless of the risk score, and execution must be performed through a sandbox. The first and second thresholds are pre-configured by the policy rule base; only after approval can the maintenance task graph be released to the executing agent.

[0056] During the aforementioned verification process, the guardrail agent can also enable a sandbox isolation environment and temporary credentials for high-risk actions such as database changes, permission changes, and cross-regional migrations. It can also directly block or postpone the execution of plans that are not within the maintenance window, thereby putting access control and compliance governance into practice.

[0057] Through the processing in step S5 above, the guardrail agent completes compliance verification, permission verification, and approval access control verification of the operation and maintenance task diagram, ensuring that only operation and maintenance task diagrams that meet the preset governance requirements can enter the execution phase, providing a prerequisite guarantee for the safe execution of subsequent execution agents.

[0058] like Figure 3As shown, step S6 forms a closed loop of audit entry and failure rollback. In step S6, the execution agent takes the operation and maintenance task map and its corresponding grayscale ratio, which were approved by the guardrail agent in step S5, as input, and executes the operation and maintenance task map in batches according to the grayscale ratio. After each batch is executed, the verification agent is called to verify the execution results of the current batch based on the target achievement of service level indicators, and decides to submit the current batch or trigger a rollback based on the verification conclusion. At the same time, the verification agent writes the evidence of the entire process into the audit ledger in the form of a hash chain.

[0059] Specifically, the execution orchestration includes: the execution agent first compiles the operation and maintenance task graph into a state machine, with node states including at least pending execution, executing, successful execution, failed execution, rolling back, and rolling back complete. Each operation and maintenance action node is configured with an idempotent key, a timeout threshold, and a number of retries. The idempotent key is used to determine whether repeated execution will produce side effects. The execution agent determines the scope of instances to be executed in each batch according to the grayscale ratio determined by the guardrail agent, and executes the operation and maintenance actions batch by batch. For example, the first batch may only select a certain percentage of instances as the total number of instances for operation.

[0060] In practice, the canary execution method can be flexibly adapted to different underlying infrastructures: for Kubernetes scenarios, canary execution can be achieved through batch updates and traffic splitting of Rollout or Deployment; for Infrastructure as Code (IaC) scenarios, execution can be achieved through a closed-loop link of planning, application and verification, and the state file version or resource snapshot is used as the rollback point.

[0061] After each batch of execution is completed, the execution agent calls the verification agent to verify the execution results of the current batch. Specifically, the verification agent collects service level indicators within the observation window, determines whether the service level indicators have reached the corresponding service level target, and outputs a verification conclusion of pass or failure based on synthetic probing transactions, key business regression test cases, and the rate of change trend relative to the baseline window.

[0062] When the verification result is successful, the execution agent submits the current batch and proceeds to the next batch; when the verification result is unsuccessful, the execution agent immediately stops subsequent batches and performs automatic rollback according to the rollback point bound to the operation and maintenance task diagram. Automatic rollback includes restoration based on configuration snapshots, image version rollback, execution of preset rollback scripts, or infrastructure-as-code state rollback. After the rollback is completed, the verification agent performs verification again to confirm that the system has been restored to a safe state and writes the rollback evidence into the audit ledger in the form of a hash chain.

[0063] Regardless of whether the verification passes or fails, the verification agent writes all the evidence in the form of a hash chain into the audit ledger. Each block in the hash chain contains the hash value of the previous block, the timestamp of the current block, and the content hash. It records at least the plan summary, approval conclusion, execution parameters, evidence index, verification conclusion, and rollback conclusion to support post-event accountability and compliance auditing.

[0064] The aforementioned "execute-verify-commit or rollback" loop mechanism ensures the controllability and recoverability of each change. During canary deployments, a batch-verify-commit loop pattern can be adopted: execution is divided into batches based on the instance set. After each batch is executed, an observation window is opened, and the verification agent calculates the rate of change and confidence interval of the metrics within the window. If successful, it proceeds to the next batch; otherwise, a rollback is triggered, and failed samples are written to the case library for subsequent learning. The rollback strategy can employ a multi-layered rollback mechanism: first, rollback the configuration to achieve the fastest recovery speed; second, rollback the version (image or installation package); and finally, rollback the infrastructure (infrastructure i.e., code state). Each rollback layer has independent verification actions to avoid the risk of secondary accidents where the system remains in an abnormal state after rollback.

[0065] The following two specific application examples further illustrate the execution effect of step S6 in real-world scenarios.

[0066] In a typical application example, suppose the observability platform detects a simultaneous increase in both the error rate and latency of a payment service. The system automatically constructs event objects and slices the dependency topology. The diagnostic agent locates a retry storm caused by jitter in a downstream cache cluster, outputting root cause candidates and evidence chains. Based on this, the planning agent generates a task graph including actions such as rate limiting, cache expansion, and abnormal instance reclamation. Simulation evaluation shows that the computational impact of the agent is small and the risk is controllable. The guardrail agent is allowed to execute automatically and has a set grayscale ratio. The execution agent executes in batches according to the grayscale ratio. After each batch of execution, the verification agent verifies whether the service level indicators have reached the corresponding service level target. After all batches pass verification, changes are submitted, and relevant evidence is written to the audit ledger.

[0067] In another typical application example, suppose database connection pool parameters need to be adjusted in a production environment. The planning agent generates a task graph containing parameter adjustment actions and binds a rollback point, i.e., the previous version configuration snapshot. The simulation evaluation agent calculates a high risk score, and based on this, the guardrail agent requests manual approval and sets a second grayscale ratio. The execution agent performs the parameter adjustment according to the second grayscale ratio. After the verification agent executes the adjustment, it verifies that latency degradation has occurred. The system immediately and automatically rolls back to the configuration snapshot according to the rollback point and verifies the recovery. The audit ledger records the approval conclusion, execution parameters, evidence of degradation, and rollback actions, enabling accountability and review.

[0068] The execution agent can also achieve secure isolation and access control of the execution environment through script sandboxes and credential brokers. The script sandbox is used to isolate the execution environment and restrict file system access, network communication, and system call permissions to prevent operation and maintenance scripts from accessing production resources or performing unexpected operations during execution. The credential broker is used to provide temporary credentials to the execution agent. These credentials have a minimum permission range and a preset validity period, and automatically expire after execution. At the same time, the credential broker supports periodic credential rotation and operation auditing to ensure that each execution operation can be traced back to the specific credential and call chain.

[0069] Through the processing in step S6 above, the execution agent and the verification agent collaborate to complete a full closed loop from canary execution and effect verification to submission or rollback. The execution agent achieves the engineering goals of controllable execution, interruptibility, resumability, and rollback. The verification agent decides to submit or rollback based on whether the service level indicators have met the corresponding service level objectives, and writes the entire process into the audit ledger in the form of a hash chain to ensure that the entire process is traceable and tamper-proof, providing a reliable data foundation for subsequent compliance audits and continuous optimization.

[0070] Through the processing of steps S1 to S6 above, Embodiment 1 of the present invention realizes a complete autonomous operation and maintenance closed loop from the acquisition of original telemetry data to the verification of change effects and the auditing of accounting. In this closed loop, the diagnostic agent completes the anomaly propagation path analysis and root cause localization based on the dependency topology slices and evidence indexes in the event objects, and outputs root cause candidates with verifiable evidence; the planning agent transforms the target state and constraint set into an operation and maintenance task graph with node binding idempotent keys, timeout thresholds, retry counts and rollback strategies; the simulation evaluation agent estimates the topology impact surface through a digital twin environment, shadow environment or read-only simulation environment, calculates the risk score by comprehensively considering multiple factors and maps it to the approval level and grayscale ratio; the guardrail agent executes the task graph in a policy-as-code manner. Policy verification, permission verification, and approval access control ensure that only plans that meet governance requirements are released. The execution agent executes the task map in batches according to a gray-scale ratio. After each batch is executed, the verification agent is called to verify the achievement of the target based on service level indicators. If it passes, it is submitted; if it fails, it automatically executes the reverse rollback according to the rollback point. The verification agent writes the plan summary, approval conclusion, execution parameters, evidence index, verification conclusion, and rollback conclusion into the audit ledger in the form of a hash chain to ensure that the whole process is traceable and tamper-proof, thereby achieving controllable risks before the event, verifiable gray-scale during the event, rollback for failures, and auditability after the event.

[0071] Therefore, this embodiment realizes the automated generation of task graphs for operation and maintenance intentions, the quantitative assessment and differentiated control of change risks, the rollbackable execution of key actions, the pre-verification of policy compliance, and the non-repudiation audit recording based on hash chains, providing a complete technical solution for autonomous operation and maintenance in the production environment.

[0072] Example 2 This embodiment provides a risk-controlled operation and maintenance management system based on multi-agent task graph orchestration, including: The unified telemetry access and event objectification module is configured to: collect metrics, logs, tracing links, alarm events, work order information and change records from the observability platform and operation and maintenance system, and after time alignment and entity parsing, construct event objects in combination with CMDB, dependency topology and configuration baseline; The diagnostic agent module is configured to output root cause candidates and evidence chains based on event objects; The planning intelligent agent module is configured to generate an operation and maintenance task graph based on the target state and constraints in root cause candidates and event objects. The simulation evaluation agent module is configured to: perform a pre-simulation of the operation and maintenance task graph and calculate risk scores based on the topological impact surface and historical baseline; The guardrail intelligent agent module is configured to: determine the approval level and grayscale ratio based on risk scores, and perform compliance verification and approval access control. The execution agent module is configured to execute the operation and maintenance task map in batches according to the gray scale ratio. After each batch is executed, the verification agent is called to verify the execution results of the current batch based on the service level indicators to achieve the target. When the verification is successful, the current batch is submitted and the next batch is entered. When the verification fails, the subsequent batches are stopped immediately and the execution is automatically rolled back. The verification agent module is configured to: verify the achievement of the target based on service level indicators for the execution results of the current batch, and write the evidence of the entire process into the audit ledger in the form of a hash chain.

[0073] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation process is the same, so it will not be repeated here.

[0074] Example 3 This embodiment also provides a computer device, including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor. When the processor executes the program, it completes the method described in Embodiment 1.

[0075] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0076] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0077] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0078] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0079] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0080] Example 4 This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0081] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A risk-controlled operation and maintenance method based on multi-agent task graph orchestration, characterized in that, include: Metrics, logs, tracing, alarm events, work order information, and change records are collected from the observability platform and operation and maintenance system. After time alignment and entity parsing, event objects are constructed by combining CMDB, dependency topology, and configuration baseline. The diagnostic agent outputs root cause candidates and evidence chains based on the event objects; The planning agent generates an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects; The simulation evaluation agent performs a pre-simulation of the operation and maintenance task graph and calculates a risk score based on the topological impact surface and historical baseline. The guardrail intelligent agent determines the approval level and grayscale ratio based on the risk score, and performs compliance verification and approval access control. The execution agent executes the operation and maintenance task graph in batches according to the grayscale ratio. After each batch is executed, the verification agent is called to verify the execution results of the current batch based on the target achievement of service level indicators. When the verification is successful, the execution agent submits the current batch and enters the next batch. When the verification fails, the execution agent immediately stops the subsequent batches and triggers automatic rollback. The verification agent writes the evidence of the entire process into the audit ledger in the form of a hash chain.

2. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The event object includes at least: event identifier, time window, affected services and instances, dependent topology slices, configuration version fingerprint, evidence index, target state, constraint set, and rollback point; wherein, the evidence index uses a reference method to point to the original telemetry data, query statement, time window, and external system query results; the target state is used to describe the verifiable result that this operation and maintenance action is expected to achieve; the constraint set includes change window, permission range, grayscale ratio upper limit, and rollback requirements; the rollback point corresponds to the configuration version, image version, status file version, or resource snapshot.

3. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The diagnostic agent outputs root cause candidates and evidence chains based on the event object, specifically: The diagnostic agent determines the anomaly propagation path based on dependent topology slicing, narrowing the suspicious range to candidate nodes on the anomaly propagation path; it identifies the main delay contributing factors by combining critical path tracing analysis; it identifies the distribution of anomaly log templates through log template clustering; it aligns the above analysis results with the most recent change records for correlation, calculates the confidence score of each candidate root cause, and outputs one or more root cause candidates in descending order of confidence; each root cause candidate is attached with an evidence index, which is used to point to the original telemetry data, query statements, time windows, and external system query results.

4. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The planning agent generates an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects, specifically as follows: The planning agent matches the recovery actions corresponding to the root cause candidates with the target state, filters the candidate actions according to the constraint set, and excludes actions that do not meet the requirements of change window, grayscale upper limit, or rollback; the actions that meet the constraint set are organized into an operation and maintenance task graph according to the dependency relationship. The operation and maintenance task graph is a directed acyclic graph or a state machine graph with rollback edges. Its nodes represent operation and maintenance actions or verification actions, and the edges represent dependencies and execution order. Each operation and maintenance action node is configured with an idempotent key, preconditions, execution interface, timeout threshold, number of retries, and rollback interface, and the corresponding rollback strategy is bound according to the rollback point in the event object.

5. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The simulation evaluation agent performs a pre-analysis of the operation and maintenance task graph, and calculates a risk score based on the topological impact surface and historical baseline, specifically as follows: The simulation evaluation agent performs a pre-play of the operation and maintenance task graph in a digital twin environment, shadow environment, or read-only simulation environment, estimates the impact surface based on the dependent topology slice, and outputs conflict detection results, resource capacity prediction results, and suggested rollback points. The risk score is calculated based on at least the following factors: the business importance weight of the affected service, the size of the topology impact, the risk level of the change type, the success rate of similar historical changes, and the deviation of the current load and error rate baseline. The risk score is then mapped to the approval level and grayscale ratio, with a higher score resulting in a higher approval level and a lower grayscale ratio.

6. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The intelligent fence agent determines the approval level and grayscale ratio based on the risk score, and performs compliance verification and approval access control, specifically as follows: The guardrail agent loads a pre-built policy rule base using a policy-as-code approach. This policy rule base includes at least minimum permission constraints, whitelist and blacklist constraints, change window constraints, compliance rule constraints, and business continuity constraints. The guardrail agent performs policy and permission checks on the operation and maintenance task graph. When all constraints are met, it outputs a compliance approval conclusion; when any constraint is not met, it outputs a blocking conclusion and converts the operation and maintenance task graph into a read-only suggestion. After policy and permission checks are passed, the guardrail agent determines the approval level based on the risk score, triggers the approval process corresponding to the approval level, and only after approval can the operation and maintenance task graph be released to the execution agent.

7. The risk-controlled operation and maintenance method based on multi-agent task graph orchestration according to claim 1, characterized in that, The execution agent executes the maintenance task map in batches according to the stated grayscale ratio. After each batch is executed, the verification agent is invoked to verify the achievement of the target based on service level indicators. Specifically: The execution agent compiles the operation and maintenance task graph into a state machine, with node states including at least pending execution, execution in progress, execution successful, execution failed, rollback in progress, and rollback completed. It determines the instance range for each batch of execution according to the grayscale ratio and executes operation and maintenance actions batch by batch. After each batch is completed, it calls the verification agent to verify the execution results of the current batch. The verification agent collects service level indicators within the observation window and determines whether they meet the corresponding service level target. It also combines synthetic probing transactions, critical business regression test cases, and the rate of change trend relative to the baseline window to output a pass or fail verification conclusion. If verification passes, the execution agent submits the current batch and proceeds to the next batch. If verification fails, the execution agent immediately stops subsequent batches and performs automatic rollback according to the rollback point bound to the operation and maintenance task graph. The automatic rollback includes configuration snapshot recovery, image version rollback, execution of rollback scripts, or infrastructure-as-code state rollback.

8. A risk-controlled operation and maintenance system based on multi-agent task graph orchestration, characterized in that, include: The unified telemetry access and event objectification module is configured to: collect metrics, logs, tracing links, alarm events, work order information and change records from the observability platform and operation and maintenance system, and after time alignment and entity parsing, construct event objects in combination with CMDB, dependency topology and configuration baseline; The diagnostic agent module is configured to output root cause candidates and evidence chains based on the event object; The planning intelligent agent module is configured to generate an operation and maintenance task graph based on the root cause candidates and the target states and constraints in the event objects; The simulation evaluation agent module is configured to: perform a pre-simulation of the operation and maintenance task graph and calculate a risk score based on the topological impact surface and historical baseline; The guardrail intelligent agent module is configured to: determine the approval level and grayscale ratio based on the risk score, and perform compliance verification and approval access control; The execution intelligent agent module is configured to execute the operation and maintenance task map in batches according to the gray scale ratio. After each batch is executed, the verification intelligent agent is called to verify the target achievement based on the service level index of the current batch. When the verification is passed, the current batch is submitted and the next batch is entered. When the verification fails, the subsequent batches are stopped immediately and the execution is automatically rolled back. The verification agent module is configured to: verify the achievement of the target based on service level indicators for the execution results of the current batch, and write the evidence of the entire process into the audit ledger in the form of a hash chain.

9. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the risk-controlled operation and maintenance handling method based on multi-agent task graph orchestration as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the risk-controlled operation and maintenance handling method based on multi-agent task graph orchestration as described in any one of claims 1-7.

Images