Automatic operation and maintenance scheme and system based on heterogeneous multi-Agent and sandbox inspection

Abstract

The invention relates to the technical field of automatic operation and maintenance, and provides an automatic operation and maintenance scheme and system based on heterogeneous multi-Agent and sandbox inspection, and the automatic operation and maintenance scheme comprises the following steps: constructing a three-in-one dynamic knowledge base; performing root cause identification and complex task disassembly of the multi-source abnormal data based on the trinity dynamic knowledge base by utilizing the BosAgent; calculating the confidence coefficient of root cause identification, comparing the confidence coefficient with a preset threshold value, when the confidence coefficient exceeds the preset threshold value, activating a plurality of ExpertAgents, and enabling the ExpertAgents to cooperatively generate a repair scheme through an MCP protocol; when the confidence coefficient does not exceed a preset threshold value, activating the SeniorAgent to process the high-complexity abnormity, and generating a repair scheme by the SeniorAgent; deploying the repair scheme into a double-iteration low-delay simulation sandbox to perform safety and applicability inspection, if an inspection result accords with expectation, sending the repair scheme to a target operating system to perform repair, and generating a repair process report; and if the test result does not conform to the expectation, the abnormal information is returned to the BosAgent to regenerate the decision.

Description

Technical Field

[0001] This invention relates to the field of automated operation and maintenance technology, and in particular to an automated operation and maintenance scheme and system based on heterogeneous multi-agent and sandbox inspection. Background Technology

[0002] Current technologies primarily rely on traditional log analysis and single-agent diagnostic models for operating system maintenance. Traditional log diagnostics analyzes log fragments in isolation, lacking multi-source data correlation of KPI indicators, system topology relationships, and trace call stacks. This results in low accuracy in root cause tracing, frequent false positives and false negatives, and long fault localization cycles. While some solutions introduce multi-agent technology, they mostly use a single model to handle anomalies across all scenarios, lacking dynamic knowledge base support. This makes it difficult to accurately break down complex tasks and to coordinate tasks, often leading to incomplete or inefficient diagnosis due to mismatched agent capabilities. In the maintenance and repair phase, existing basic sandbox verification tools generally have a runtime latency exceeding 12 seconds and lack a closed-loop design of "verification-execution-feedback." Repair scripts are deployed directly to the production environment without sufficient testing, easily causing secondary failures due to compatibility issues. Repeated trial and error significantly increases maintenance costs and system stability risks. These shortcomings make it difficult to meet the needs of complex systems (such as KylinOS) for efficient, accurate, and secure automated maintenance. Summary of the Invention

[0003] To address the aforementioned shortcomings, the present invention aims to propose an automated operation and maintenance scheme and system based on heterogeneous multi-agent and sandbox verification. It seeks to support confidence-driven heterogeneous multi-agent collaborative diagnosis by constructing a three-in-one dynamic knowledge base of "KPI tree-topology graph-Trace call stack," and to develop a dual-iteration low-latency simulated sandbox to achieve secure verification and closed-loop feedback of repair solutions. This enables seamless integration of accurate root cause localization of operating system anomalies, efficient generation of repair solutions, and secure execution, significantly improving the accuracy and stability of automated operation and maintenance.

[0004] To achieve this objective, the present invention adopts the following technical solution: An automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification includes the following steps: S1: Construct a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; S2: Utilize BossAgent to perform root cause identification and complex task decomposition of multi-source abnormal data based on the aforementioned three-in-one dynamic knowledge base; S3: Calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, activate several ExpertAgents and use the MCP protocol to enable the ExpertAgents to collaboratively generate a remediation plan. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. S4: Deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, send the repair solution to the target operating system for repair and generate a repair process report. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

[0005] Preferably, in step S1, the construction of the three-in-one dynamic knowledge base includes: using KPI trees to vertically correlate system performance indicators to locate anomalies and deduce causal chains, combining topology graphs to horizontally map system components and their dependencies to analyze anomaly propagation paths, and simultaneously using Trace call stacks to dynamically capture system kernel call sequences to provide underlying execution evidence, forming a full-dimensional anomaly localization system covering vertical indicator chains, horizontal component chains, and deep causal chains, thus obtaining the three-in-one dynamic knowledge base.

[0006] Preferably, step S2 includes: Based on the aforementioned three-in-one dynamic knowledge base, KPI nodes associated with anomalies are anchored through BossAgent, and the dependencies between components are traced according to the topology graph to construct the anomaly propagation path. At the same time, the complete execution link process is restored through Trace call stack. Root cause identification is achieved through comprehensive analysis of the KPI nodes, anomaly propagation path, and complete link process, and the operation and maintenance tasks are decomposed according to the identified root causes.

[0007] Preferably, calculating the confidence level of the root cause identification and comparing the confidence level with a preset threshold includes: When the confidence level exceeds the preset threshold: Synchronize the KPI nodes, topology relationships, and Trace call stack information and collaboration targets associated with the anomaly, assign processing roles to each Agent, unify the data interaction format to achieve cross-Agent data interoperability, form a consensus evidence chain through multiple rounds of evidence exchange and judgment correction, and distribute tasks and generate repair solutions based on the consensus evidence chain. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair solution.

[0008] Preferably, the ExpertAgents include at least one of the following types: DiskAgent, KernelAgent, SystemAgent, ProcessAgent, CPUAgent, MemAgent, and NetAgent; The DiskAgent is used to handle solutions for disk space exhaustion and IO latency anomalies; the KernelAgent is used to handle solutions for kernel deadlock and parameter exhaustion anomalies; the SystemAgent is used to handle solutions for system resource exhaustion and service anomalies; the ProcessAgent is used to handle solutions for process scheduling and resource leakage anomalies; the CPUAgent is used to handle solutions for processor overload and interrupt anomalies; the MemAgent is used to handle solutions for memory exhaustion and swapping anomalies; and the NetAgent is used to handle solutions for network connection and transmission anomalies. ExpertAgent invokes the appropriate tools to generate repair scripts based on the assigned exception type.

[0009] Preferably, step S4 includes: The repair solution is deployed to a dual-iteration low-latency simulation sandbox using a deployment tool. The dual-iteration simulation capability of the sandbox is used to simulate the operating environment to verify the safety and applicability of the repair solution. At the same time, the resource scheduling logic of the sandbox is optimized to reduce runtime latency. Based on the test results of the sandbox, the process is divided: when the test results meet expectations, the repair plan is sent to the target operating system to perform the repair operation, and the execution results are synchronized to ReportAgent to generate a total repair process report; When the test results do not meet expectations, a feedback mechanism is triggered to return the abnormal information to the BossAgent to regenerate the decision.

[0010] Preferably, the verification process of the dual-iteration low-latency simulation sandbox includes: The remediation scheme is subjected to a security scan, including static syntax risk scanning, high-risk command behavior interception, unauthorized command execution, and illegal access to sensitive paths detection; Configure environment synchronization using deployment tools, complete environment consistency verification, and then start the sandbox environment. Initial runtime data is collected in the sandbox environment, and optimized status data is collected after the repair plan is executed. The effectiveness of the repair solution is verified by comparing the initial running data with the optimized state data using a verification test script. When the verification result does not meet expectations, the inference model is triggered to regenerate the optimization strategy based on the data collected in the sandbox, and then return to the sandbox for re-verification; When the verification results meet expectations, the remediation plan is deployed to the target environment for execution using the deployment tool.

[0011] An automated operation and maintenance system based on heterogeneous multi-agent and sandbox verification includes: The knowledge base construction module is used to build a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; The root cause analysis module is used to identify the root causes of multi-source anomalies and decompose complex tasks based on the three-in-one dynamic knowledge base by BossAgent. The confidence decision module is used to calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, several ExpertAgents are activated and the ExpertAgents are used to collaboratively generate a remediation plan through the MCP protocol. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. The sandbox testing module is used to deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, the repair solution is sent to the target operating system for repair and a repair process report is generated. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

[0012] One of the above technical solutions has the following advantages or beneficial effects: This invention breaks down traditional information silos by constructing a three-in-one dynamic knowledge base integrating KPI trees, topology graphs, and trace call stacks. It integrates scattered monitoring metrics, component dependencies, and call chains into a structured, traceable, multi-dimensional anomaly correlation chain, laying a data foundation for accurate diagnosis. Then, BossAgent uses this knowledge base to perform root cause identification and complex task decomposition of multi-source anomaly data, replacing traditional manual experience-based judgment. This upgrades the diagnostic process from fragmented log matching to systematic correlation analysis, significantly improving the logic and completeness of root cause tracing. Based on this, intelligent triage is achieved by calculating the confidence level of root cause identification and comparing it with a preset threshold. When the confidence level exceeds the threshold, multiple ExpertAgents are activated to collaboratively generate a remediation plan via the MCP protocol. If the threshold is not reached, the SeniorAgent tackles highly complex anomalies, dynamically matching the complexity of the problem with the capabilities of the Agent. This ensures both efficient parallel processing of routine anomalies and in-depth, accurate analysis of complex anomalies, overcoming the imbalance between efficiency and effectiveness in a single-Agent model. Subsequently, the generated remediation plan is deployed to a dual-iteration low-latency simulation sandbox for safety and applicability testing. Leveraging its ability to quickly simulate real-world environments, potential risks are fully exposed before the plan enters the production environment, completely avoiding secondary failures caused by direct execution. Finally, through a result triage mechanism—if the results meet expectations, the remediation is executed and a report is generated; if they do not meet expectations, the results are returned to the BossAgent for re-decision—a closed-loop iterative process of "diagnosis-verification-execution-feedback" is formed, continuously optimizing decision-making quality. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a flowchart of an automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification provided in an embodiment of the present invention; Figure 2 This is a comparison chart of the running time and stability of KylinSandbox I, KylinSandbox II, and nsjail sandbox provided in the embodiments of the present invention; Figure 3 This is a flowchart of the heterogeneous multi-agent collaborative division of labor system provided in the embodiments of the present invention; Figure 4 This is a flowchart of heterogeneous multi-agent collaborative execution under disk I / O bottleneck anomalies provided by the present invention; Figure 5 This is a flowchart of the KylinSandbox dual-iteration low-latency simulation sandbox provided by the present invention; Figure 6 This is a schematic diagram of the structure of an automated operation and maintenance system based on heterogeneous multi-agent and sandbox verification provided in an embodiment of the present invention. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] In this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0017] An automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification, such as Figure 1 As shown, a preferred embodiment of the present invention includes the following steps: S1: Construct a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; It should be noted that the aforementioned three-in-one dynamic knowledge base refers to a dynamic, interconnected knowledge network formed by the structured fusion of data from three types of heterogeneous systems. Its essence is to construct a comprehensive anomaly localization system covering the vertical, horizontal, and in-depth dimensions of the system. Specifically, the KPI tree (KeyPerformanceIndicatorTree) is a vertical indicator chain structure. By organizing system performance indicators at various levels (such as CPU utilization, memory usage, and disk I / O wait time) into a tree structure according to hierarchical relationships, it is used to locate anomaly indicator nodes and capture cross-layer dependency chains, thereby deriving causal propagation paths. The topology graph is a horizontal component chain structure. Using a graph theory model, it represents component nodes such as processes, services, and hardware resources in the operating system and their dependency edges. It is used to map anomaly indicators to specific components and analyze the propagation links of anomalies between components at the same level. The Trace call stack is a deep causal chain structure. By dynamically capturing the function call sequence and stack information of the system kernel when an anomaly occurs, it provides low-level execution evidence to reconstruct the code-level execution trajectory. The knowledge base adopts a dynamic update mechanism, which refreshes the KPI sampling values, topology connection status and Trace capture records in real time when the system's operating status changes, ensuring that the knowledge base stays synchronized with the production environment.

[0018] Understandably, the KPI tree provides information on the magnitude and hierarchy of performance anomalies, the topology diagram reveals the scope and propagation path of the anomaly's impact, and the Trace call stack fills in the microscopic evidence of the underlying execution logic. These three elements complement each other, forming a complete chain of evidence from phenomenon to essence. The dynamic nature of the knowledge base ensures its ability to adapt to the continuous evolution of distributed heterogeneous systems, avoiding analytical failures caused by lagging static environmental descriptions. Therefore, this invention, by constructing a multi-dimensional data fusion foundation, lays the prerequisite for accurate anomaly diagnosis and can significantly reduce the risk of misjudgment or missed judgment caused by incomplete data or missing correlations.

[0019] S2: Utilize BossAgent to perform root cause identification and complex task decomposition of multi-source abnormal data based on the aforementioned three-in-one dynamic knowledge base; It should be noted that the BossAgent is a centrally scheduled intelligent agent built based on a large language model (such as the DeepSeek-R1 series 14 billion parameter model). Its role is to act as the decision-making center of the diagnostic process, responsible for parsing heterogeneous data in the three-in-one dynamic knowledge base and performing root cause reasoning. The multi-source abnormal data refers to unstructured or semi-structured data simultaneously originating from three monitoring sources: KPI tree, topology graph, and Trace call stack. Its characteristics are time synchronization but different semantic dimensions. Root cause identification is a logical reasoning process that traces from apparent anomalies to the root cause of the fault by analyzing the causal dependencies between abnormal data. Its output is a unique root cause node identifier (such as the name of the faulty component or the location of the fault code). Complex task decomposition refers to breaking down the complete sequence of operations required for root cause repair (such as parameter adjustment, service restart, and configuration change) into several independently executable sub-task units. Each sub-task has a clearly defined execution object, operation instructions, and expected effect.

[0020] Understandably, BossAgent, by loading a three-dimensional dynamic knowledge base with spatiotemporal correlation capabilities, transforms previously scattered monitoring signals into a unified representation with causal semantics. Leveraging the cross-domain understanding capabilities of a large model, it simulates the diagnostic thinking of a seasoned operations expert: first, it quantifies the severity and level of anomalies using KPI trees; then, it expands the scope of anomaly impact analysis using topology graphs; and finally, it verifies the underlying execution logic using trace stacks. This forms a progressive reasoning approach from "macro to meso to micro," overcoming the shortcomings of traditional log analysis that relies solely on text matching and cannot understand the semantics of system architecture. This enables automated, penetrating analysis from symptoms to root causes. Simultaneously, after identifying the root cause, BossAgent proactively breaks down tasks, transforming the abstract "fixing disk I / O bottlenecks" into concrete, executable steps such as "cleaning page cache → adjusting cache reclamation pressure → monitoring iowait metrics." This provides a clear action blueprint for subsequent agent collaborative execution, avoiding execution deviations caused by vague task descriptions.

[0021] S3: Calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, activate several ExpertAgents and use the MCP protocol to enable the ExpertAgents to collaboratively generate a remediation plan. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. It should be noted that the confidence level refers to the quantitative evaluation value of the correctness of the identification result after BossAgent completes the root cause identification in step S2. Its calculation is based on the consistency of multi-source evidence, typically obtained by weighted averaging of the support of three types of evidence: KPI tree, topology graph, and Trace call stack (using multiple tests to calculate the average as described in the document). The value ranges from 0 to 1, with higher values ​​indicating more reliable root cause identification. The preset threshold is a critical value used to determine the anomaly complexity and Agent scheduling strategy, set to 0.7 according to the document's example. This value distinguishes the handling boundary between regular anomalies and high-complexity anomalies. The ExpertAgents are a collection of specialized processing agents built based on lightweight large language models (such as the DeepSeek-R1 series 8 billion parameter model). Each ExpertAgent performs tool invocation and solution generation for a specific operational domain (such as CPU, memory, disk), characterized by fast response speed, low resource consumption, but limited processing depth. The MCP (Multi-Agent Collaboration Protocol) is a standardized multi-agent collaborative communication protocol. Its core mechanisms include information sharing, role allocation, unified data format, multi-round evidence exchange, and consensus formation, ensuring effective collaboration among heterogeneous agents to avoid decision-making conflicts. The SeniorAgent is a high-level expert agent built on a large model of the same level as the BossAgent (such as the DeepSeek-R1 series 14 billion parameter model). Its characteristics include deep reasoning ability and cross-domain knowledge integration ability, specifically designed to handle complex scenarios with low confidence caused by insufficient evidence or excessively long abnormal chains.

[0022] Understandably, by introducing a confidence quantification mechanism, the system can dynamically assess the reliability of root cause identification and intelligently select the optimal execution architecture: for routine anomalies with sufficient evidence and clear propagation paths, high confidence triggers ExpertAgents to process in parallel, leveraging the rapid response characteristics of lightweight models to achieve collaborative division of labor and significantly improve solution generation efficiency; for complex anomalies with ambiguous evidence and involving multi-system coupling, low confidence triggers SeniorAgents to process centrally, utilizing the deep understanding capabilities of large-parameter models for global reasoning to ensure diagnostic accuracy in complex scenarios. The MCP protocol acts as a collaborative glue in this process; its multi-round evidence exchange mechanism can simulate the consultation process of human experts. Each ExpertAgent independently analyzes and shares intermediate conclusions, correcting individual biases through cross-validation. The resulting consensus evidence chain effectively reduces the risk of solution errors caused by misjudgments by a single Agent. This confidence-driven heterogeneous division of labor system achieves adaptive scheduling of "rapid processing of simple problems and in-depth solution of complex problems," overcoming the resource waste and capability mismatch problems caused by fixed roles or rigid scheduling in traditional multi-Agent systems.

[0023] S4: Deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, send the repair solution to the target operating system for repair and generate a repair process report. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

[0024] It should be noted that the aforementioned dual-iteration low-latency simulation sandbox (KylinSandbox) is an isolated testing environment built based on containerization technology (such as Docker) or lightweight virtualization technology. Its "dual-iteration" characteristic refers to two levels of iterative optimization: the first level is environment simulation iteration, which continuously synchronizes the kernel version, library files, configuration parameters, and process status of the target operating system to ensure a high degree of consistency between the sandbox environment and the production environment; the second level is solution optimization iteration, where after executing the remediation plan within the sandbox, the large language model's reflection mechanism is triggered based on performance metric changes, automatically generating optimization strategies and re-verifying them, forming a closed loop of "execution-feedback-optimization." Low latency refers to compressing the single test execution time to 6-10 seconds by optimizing the sandbox startup process (such as using pre-set snapshots and lazy initialization), resource scheduling logic (such as CPU binding and memory pre-allocation), and I / O paths (such as using tmpfs to simulate disks). Security testing refers to detecting whether the remediation plan contains high-risk commands (such as...) through static and dynamic analysis. Unauthorized operations (such as modifying read-only system files) or illegal path access (such as traversing to...) The applicability test verifies whether the remediation solution can achieve the expected performance optimization effect (such as reduced iowait) under specific configurations in the target environment (e.g., KylinOS kernel compilation options and security module policies). The deployment tool in this solution specifically refers to the Ansible automated operation and maintenance tool, which communicates with the sandbox and target system via the SSH protocol to distribute and execute the solution. The target operating system in this embodiment is KylinOS, a domestically developed operating system based on the Linux kernel, whose features include customized security enhancement mechanisms and specific system call implementations.

[0025] Understandably, by introducing a dual-iteration low-latency simulation sandbox, the system constructs a highly realistic "digital twin" testbed before the solution is implemented. This allows for the pre-observation of the actual impact of remediation operations in an isolated environment, effectively mitigating catastrophic consequences caused by environmental differences or solution defects. The dual-iteration mechanism ensures that verification is not just a one-time execution but a continuous optimization process—iterative environmental simulation guarantees the transferability of verification results, while iterative solution optimization activates the reflective capabilities of the large model through sandbox feedback data, enabling the remediation strategy to adaptively adjust until acceptance criteria are met. The low-latency characteristic makes frequent iterative verification possible, significantly shortening the verification cycle and overcoming the shortcomings of traditional sandboxes, which are difficult to integrate into automated processes due to slow startup. The closed-loop feedback mechanism (returning to BossAgent if verification fails) tightly couples the diagnosis and remediation phases. When the solution fails, it can quickly trace back to the decision-making source to regenerate the strategy, avoiding the time overhead of manual intervention and achieving a fully automated closed-loop operation and maintenance process.

[0026] like Figure 2 As shown, KylinSandbox II, the dual-iteration low-latency simulation sandbox proposed in this invention, achieves significant optimization in runtime compared to the previous KylinSandbox I, reducing the average runtime from 30-40 seconds to 6-10 seconds, and about 18% lower than Google's nsjail sandbox. Furthermore, in terms of stability, KylinSandbox II did not exhibit any unstable peaks in hundreds of tests, while KylinSandbox I and nsjail showed a higher frequency of unstable peaks, fully demonstrating the technical advantages of this invention in low latency and stability.

[0027] Preferably, in step S1, the construction of the three-in-one dynamic knowledge base includes: using KPI trees to vertically correlate system performance indicators to locate anomalies and deduce causal chains, combining topology graphs to horizontally map system components and their dependencies to analyze anomaly propagation paths, and simultaneously using Trace call stacks to dynamically capture system kernel call sequences to provide underlying execution evidence, forming a full-dimensional anomaly localization system covering vertical indicator chains, horizontal component chains, and deep causal chains, thus obtaining the three-in-one dynamic knowledge base.

[0028] It should be noted that vertical correlation system performance indicators refer to the organic organization of performance indicators scattered across different system abstraction layers according to causal dependencies through a tree-like hierarchical structure. This constructs a transmission chain from application-layer business indicators (such as transaction response time) to system-layer resource indicators (such as CPU utilization), then to kernel-layer scheduling indicators (such as context switch counts), and finally to hardware-layer device indicators (such as disk queue depth). Its function is to enable abnormal signals to penetrate downwards or be traced upwards along the hierarchical path, thereby connecting surface phenomena with deep-seated root causes through quantitative relationships. Causal chain derivation refers to using time-series causal inference algorithms (such as Granger causality tests based on vector autoregression models or causal strength calculations based on Bayesian networks) to analyze the leader-lag relationships of indicators at different levels in time series, calculating the causal probability weights between indicators. Its function is to distinguish between true causal transmission and false correlations, ensuring the logical interpretability of the abnormal propagation path. Lateral mapping system components refer to the use of graph theory to abstract runtime entities in the operating system (including user-mode processes, kernel threads, network services, storage volumes, and device drivers) into topological nodes. Dependency discovery techniques (such as system call auditing or dynamic link library analysis) are used to identify control flow, data flow, and resource flow dependencies between nodes. The purpose is to transform physical or logical call relationships into a computable graph structure, providing a network topology foundation for anomaly propagation. Analyzing anomaly propagation paths involves executing graph propagation algorithms (such as anomaly diffusion simulation based on the SIR model or shortest path impact range calculation) on the topological graph. This quantitatively assesses the probability and latency of propagation from the anomaly source node along dependency edges to other nodes. Its purpose is to accurately delineate the boundaries of fault impact, avoiding misjudgments or omissions due to ambiguity in dependency relationships. Dynamically capturing system kernel call sequences involves deploying event-driven probes (such as eBPF kprobe or tracepoint) along critical execution paths in kernel mode. When preset KPI anomaly trigger conditions are met, a sampling mechanism is automatically activated to capture the complete function call trajectory after a user-mode process enters kernel mode. Its purpose is to obtain the precise execution context at the moment of the anomaly, rather than relying on asynchronous logs afterward, ensuring the timeliness and accuracy of underlying evidence. Underlying execution evidence refers to the micro-execution state provided by the Trace call stack, including CPU register snapshots, kernel function parameter values, and process switching events. Its purpose is to verify the consistency between macro-level anomalies and micro-level code execution behavior, filling the gaps in kernel execution logic that traditional monitoring methods cannot reach. A comprehensive anomaly localization system aligns and cross-couples the aforementioned vertical causal transmission dimension, horizontal component dependency dimension, and deep code execution dimension in time and space, constructing a three-dimensional anomaly representation model that can be traced along the timeline, traversed in spatial topology, and drilled down in execution depth. Its purpose is to achieve comprehensive root cause localization capabilities, from phenomena to essence, from components to code, and from macro to micro.

[0029] Understandably, by employing a vertical correlation index-based causal chain derivation method, the system organizes previously isolated performance metrics into a hierarchical tree structure based on causal logic. This allows for interpretable causal tracing of abnormal signals along the KPI tree, overcoming the inherent limitation of relying solely on threshold alarms to pinpoint deep-seated root causes. Through a horizontal mapping component analysis of propagation paths, the system transforms static configuration information into a dynamic topological dependency graph. This enables quantitative assessment and visualization of the impact range of anomalies based on graph algorithms, avoiding fault isolation errors caused by ambiguous dependency relationships. By dynamically capturing traces to provide underlying evidence, the system establishes a verifiable bridge between macroscopic metric anomalies and microscopic code execution. This ensures that root cause reasoning is supported not only by data correlation but also by the consistency verification of kernel execution logic, significantly reducing the probability of misjudgments caused by environmental specificity, sporadic fluctuations, or delayed log sampling. After the integration of these three elements to form a comprehensive system, the knowledge base is no longer just a simple collection of data, but has evolved into an intelligent decision support platform with causal reasoning, propagation simulation, and code verification capabilities. This provides structured, traceable, and inferable high-quality knowledge input for the subsequent understanding of multi-source heterogeneous data by AI agents, enabling them to accurately locate the root cause based on a complete chain of evidence rather than relying on fragmented information for subjective guessing. This fundamentally improves the knowledge completeness and reasoning reliability of automated diagnosis.

[0030] Preferably, step S2 includes: Based on the aforementioned three-in-one dynamic knowledge base, KPI nodes associated with anomalies are anchored through BossAgent, and the dependencies between components are traced according to the topology graph to construct the anomaly propagation path. At the same time, the complete execution link process is restored through Trace call stack. Root cause identification is achieved through comprehensive analysis of the KPI nodes, anomaly propagation path, and complete link process, and the operation and maintenance tasks are decomposed according to the identified root causes.

[0031] It should be noted that anchoring KPI nodes associated with anomalies refers to locating indicator nodes in the KPI tree that significantly deviate from the historical baseline using anomaly detection algorithms and marking them as anomaly anchor points. Their function is to provide quantifiable starting coordinates and longitudinal propagation points for root cause identification. In this process, BossAgent uses a sliding window standard deviation algorithm to monitor KPI sample values ​​in real time. When the values ​​of three consecutive sample points all exceed the baseline mean plus three times the standard deviation, the anchoring operation is triggered, and the corresponding tree node is marked as the anomaly source node. The node attributes record the anomaly level (severe, alarm, alert) and the deviation value. Tracing dependencies between components using the topology graph involves starting with the system component to which the KPI anchor node belongs, performing a breadth-first or depth-first search on the topology graph, traversing all reachable upstream and downstream dependent nodes, and constructing directed paths for anomaly propagation. Its purpose is to map anomalies in metrics to specific components and reveal the lateral spread of faults. BossAgent employs a graph traversal algorithm here, setting a search depth limit (usually 3 hops) to avoid path explosion. Simultaneously, it sorts the paths according to edge dependency weights (edge ​​anomaly propagation probabilities trained based on historical fault data), generating a list of anomaly propagation paths arranged in descending order of propagation probability. Reconstructing the complete execution chain through the Trace call stack involves parsing the captured kernel function call sequence, sorting and associating it according to call timestamps and process IDs, and reconstructing the complete call chain from the user-space system call entry point to the kernel-space underlying driver execution. Its purpose is to provide code-level execution evidence for anomaly propagation paths. BossAgent uses symbolic execution technology to perform semantic analysis on the function call parameters in the Trace, identifying blocking points, error return codes, or resource contention hotspots, forming annotated call chain text. Root cause identification based on comprehensive analysis involves weighted fusion of KPI node deviation, topology propagation path probability, and trace call chain semantics. Bayesian inference is used to calculate the posterior probability of each candidate root cause, and the component or code location corresponding to the highest probability is selected as the final root cause. This serves to reduce single-dimensional noise interference through multi-dimensional evidence cross-validation. Decomposing maintenance tasks based on the identified root causes involves BossAgent matching a predefined repair template library according to the root cause type. The complete repair objective (e.g., "resolve disk I / O bottleneck") is decomposed into a sequence of subtasks with dependencies. Each subtask contains a quadruple of execution subject, operation object, action instruction, and acceptance conditions. This transforms the abstract repair objective into a concrete set of instructions executable by the Agent.

[0032] Understandably, by proactively anchoring KPI nodes through BossAgent, the system first quantifies the vague phenomenon of "performance degradation" into locatable anomalies, providing precise vertical starting coordinates for subsequent analysis and avoiding blind investigations caused by overly broad monitoring scope. Then, by tracing dependencies through a topology graph, the system reveals the lateral propagation trajectory of anomalies within the system architecture from a component network perspective. This allows root cause identification to move beyond being limited to a single host or process, enabling the assessment of cascading impacts across services and nodes, significantly improving fault location capabilities in distributed scenarios. Simultaneously, by reconstructing the execution chain through call stack tracing, the system establishes a verifiable causal bridge between macroscopic indicator anomalies and microscopic code execution, allowing root cause judgment to go beyond simply identifying "which component has a problem" and delve into the finer granularity of "which kernel code is executing abnormally," effectively reducing the probability of misjudgments caused by sporadic fluctuations or monitoring delays. Based on the comprehensive analysis of three types of data, BossAgent employs a weighted evidence fusion strategy, transforming inference results from different dimensions into probabilistic confidence levels. Through cross-validation and mutual correction of multi-source evidence, it significantly reduces noise interference and bias from a single evidence source, thereby improving the robustness of root cause identification. After identifying the root cause, BossAgent immediately initiates a task decomposition process, systematically breaking down complex remediation goals into schedulable and verifiable atomic task units. This pre-decomposition mechanism ensures that subsequent Agent execution phases do not require secondary logical reasoning, avoiding execution deviations or resource waste caused by ambiguous task descriptions, and achieving a seamless transition from diagnosis to remediation.

[0033] Preferably, calculating the confidence level of the root cause identification and comparing the confidence level with a preset threshold includes: When the confidence level exceeds the preset threshold: Synchronize the KPI nodes, topology relationships, and Trace call stack information and collaboration targets associated with the anomaly, assign processing roles to each Agent, unify the data interaction format to achieve cross-Agent data interoperability, form a consensus evidence chain through multiple rounds of evidence exchange and judgment correction, and distribute tasks and generate repair solutions based on the consensus evidence chain. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair solution.

[0034] It should be noted that the synchronization of KPI nodes, topology relationships, and trace call stack information refers to the BossAgent broadcasting the key anomaly data identified in step S2 to all active ExpertAgents via a standardized message protocol. This ensures that each participating agent reasones based on consistent and complete multi-source evidence, avoiding decision conflicts caused by information asymmetry. The collaboration objective refers to the specific requirements for the repair solution generation task defined by the BossAgent, including the repair object (e.g., a specific disk device), optimization metrics (e.g., reducing iowait to below 15%), constraints (e.g., not restarting the service during peak business periods), and output format (e.g., Ansible Playbook YAML structure). This clarifies the collaborative direction and delivery standards for each Agent. Assigning processing roles to each Agent means that the BossAgent decomposes the overall objective into several sub-tasks based on the expertise of each ExpertAgent (e.g., DiskAgent specializes in disks, KernelAgent specializes in kernels) and assigns execution entities. This clarifies responsibility boundaries from the initial stages of collaboration, preventing resource waste and collaboration disorder caused by task overlap or unclear responsibilities. A unified data interaction format refers to forcing all agents to convert their analysis conclusions, intermediate evidence, and proposed solutions into a consistent, machine-readable format through predefined interface description languages ​​such as JSON Schema or Protocol Buffers. Its purpose is to eliminate data parsing barriers caused by differences in output structure between heterogeneous agents, enabling seamless data flow across agents. Cross-Agent data interoperability refers to building low-latency, highly reliable multi-agent communication channels based on message middleware technologies such as Redis Pub / Sub or gRPC bidirectional streams. This allows each agent to publish its own analysis results in real time and subscribe to the progress of other agents, maintaining information synchronization during collaboration and supporting the continuity of multi-round interactions. Multi-round evidence exchange and judgment correction refers to ExpertAgent conducting at least three rounds of iterative discussion according to the MCP protocol. In each round, each agent submits its analysis evidence and, after receiving evidence from other agents, corrects its own judgment based on Bayesian updates or Dempster-Shafer evidence theory. Its purpose is to converge to a more robust consensus conclusion through collective wisdom, suppressing misjudgments or biases by individual agents. The consensus evidence chain refers to the set of analytical conclusions formed after multiple rounds of exchange and supported by more than half of the ExpertAgents. Its structure includes root cause identification, propagation path, repair priority, and verification indicators. Its role is to serve as the authoritative basis for generating the final repair solution, ensuring that the solution has the feasibility and effectiveness of being consistently recognized across Agents.Task distribution refers to the BossAgent breaking down the task of generating a repair solution into specific sub-tasks based on a consensus evidence chain. Each sub-task has a clearly defined execution agent, input dependencies, and delivery deadline. This parallelizes and pipelines the solution generation process, shortening the overall response time. Activating the SeniorAgent to handle high-complexity anomalies when the confidence level does not exceed a preset threshold means the BossAgent bypasses the multi-round collaboration process of the MCP, directly triggering the SeniorAgent and granting it full access to the knowledge base and deep reasoning capabilities. This avoids collaboration deadlocks due to insufficient evidence in low-confidence scenarios, instead relying on the single-point deep analysis capabilities of a large-parameter model to tackle complex anomalies.

[0035] Understandably, when the confidence level exceeds a threshold, the system determines that the current anomaly evidence is sufficient and the root cause is clear, making a divide-and-conquer parallel collaboration strategy suitable. By synchronizing complete multi-source anomaly information, each ExpertAgent can leverage its domain expertise on a unified evidence basis, avoiding duplication of effort and biased conclusions caused by information fragmentation. By clearly defining role assignments and standardizing data formats, the system establishes a standardized interaction contract at the initial stage of collaboration, enabling heterogeneous agents to perform their respective duties and cooperate collaboratively like a professional team, significantly reducing collaboration overhead caused by ambiguous responsibilities or communication barriers. Through multiple rounds of evidence exchange and dynamic judgment correction, the system simulates the collective decision-making process of human expert consultation. In each round, each agent outputs its own analysis and absorbs the viewpoints of others, continuously revising the initial judgment based on evidence fusion theory. The final consensus evidence chain is verified from multiple perspectives, possessing higher resistance to noise interference and decision robustness, effectively suppressing the risk of misjudgment caused by model limitations or data bias in a single agent. Task distribution based on a consensus-based evidence chain allows for parallel processing of remediation plan generation. Each ExpertAgent processes different sub-tasks synchronously, and the efficiency of plan generation increases linearly with the number of Agents, meeting the timeliness requirements for rapid handling of common anomalies. Conversely, when the confidence level does not exceed the threshold, the system determines that the current anomaly evidence is ambiguous and the root cause is complex. Forcibly initiating multi-Agent collaboration may lead to a deadlock due to insufficient evidence, resulting in endless arguments among Agents and an inability to converge. In this case, the SeniorAgent is activated and processed centrally, fully leveraging its deep reasoning and cross-domain knowledge integration capabilities of its large-parameter model. Through single-point deep analysis, it penetrates complex appearances to reach the essence, avoiding the ineffective consumption of the collaboration process under low-quality evidence, and ensuring the thoroughness of diagnosis and the reliability of the plan in complex anomaly scenarios.

[0036] Preferably, such as Figure 3As shown, the ExpertAgents include at least one of the following types: DiskAgent, KernelAgent, SystemAgent, ProcessAgent, CPUAgent, MemAgent, and NetAgent; The DiskAgent is used to handle solutions for disk space exhaustion and IO latency anomalies; the KernelAgent is used to handle solutions for kernel deadlock and parameter exhaustion anomalies; the SystemAgent is used to handle solutions for system resource exhaustion and service anomalies; the ProcessAgent is used to handle solutions for process scheduling and resource leakage anomalies; the CPUAgent is used to handle solutions for processor overload and interrupt anomalies; the MemAgent is used to handle solutions for memory exhaustion and swapping anomalies; and the NetAgent is used to handle solutions for network connection and transmission anomalies. ExpertAgent invokes the appropriate tools to generate repair scripts based on the assigned exception type.

[0037] It should be noted that DiskAgent is a lightweight expert agent specifically designed for disk subsystem anomalies. Its scope is limited to the block device and file system layers. It has built-in interfaces for disk performance analysis tools (such as iostat, blktrace, and smartctl). When it receives disk-related anomaly tasks, the agent uses these tools to collect data such as disk queue depth, I / O response time, and SMART health status. It then combines this data with preset repair templates to generate specific repair scripts for adjusting I / O scheduler parameters, cleaning log files, and expanding logical volumes. KernelAgent, on the other hand, is a deep expert agent focusing on kernel-mode runtime anomalies. Its scope covers core subsystems such as kernel scheduling, memory management, and synchronization mechanisms. It has built-in kernel debugging toolchain calling capabilities (such as perf, bpftrace, and sysctl). When handling kernel deadlocks or parameter exhaustion issues, the agent uses perf to analyze kernel function hotspots, bpftrace to trace lock contention events, and sysctl to read current kernel parameter configurations, generating scripts to adjust adjustable kernel parameters (such as...). , The system agent is a macro-level expert intelligent agent that manages the overall resource and service status of the operating system. Its role is to coordinate the comprehensive scheduling and service health management of various resources such as CPU, memory, disk, and network. It has built-in system-level monitoring tool call interfaces (such as systemd-analyze, systemctl, and sar). When system resources are exhausted or services are abnormal, the agent uses systemd-analyze to locate and start slow services, uses systemctl to restart abnormal services, and uses sar to analyze resource usage trends, generating comprehensive repair scripts for service restart strategies, resource quota adjustments (such as cgroup restrictions), or system configuration optimization. ProcessAgent is a micro-level expert agent specializing in user-space process behavior analysis. Its targets include scheduling, memory usage, and resource leaks in single or multiple process groups. It has built-in capabilities for calling process tracing and diagnostic tools (such as strace, lsof, and gdb). When process scheduling delays or memory leaks occur, this agent uses strace to trace process system call times, lsof to analyze file handle leaks, and gdb to attach to and analyze the process's memory allocation stack, generating process-level repair scripts to adjust process priorities (nice value), limit process resource usage (ulimit), or inject memory leak repair code. CPUAgent is a performance expert agent specializing in processor cores and interrupt subsystems. Its scope includes CPU load balancing, interrupt distribution, and frequency tuning. It has built-in interfaces for calling CPU performance analysis tools (such as mpstat, irqbalance, and cpufreq-info). When processor load is too high or interrupt storms occur, this agent uses mpstat to analyze per-core load distribution, and... Analyze the problem of uneven interrupt distribution, and through The system checks frequency regulation strategies and generates processor-level repair scripts to adjust interrupt affinity (binding interrupts to specific CPU cores), modify the CPU frequency regulator (e.g., switching to performance mode), or optimize process CPU affinity masks. MemAgent is a resource management expert agent specializing in the memory subsystem and swapping mechanisms. Its scope covers physical memory allocation, page cache strategies, swap partition scheduling, etc. It has built-in memory analysis tool invocation capabilities (such as free, vmstat, slabtop). When memory is exhausted or swapping is abnormal, this agent analyzes memory usage composition using free, observes page swapping rates using vmstat, and checks kernel object cache usage using slabtop, generating scripts to adjust page cache reclamation pressure. Limit memory usage cgroups ( This refers to memory-level repair scripts that clean up memory leak processes (OOM Killer policy adjustments). NetAgent is a communication expert agent focused on network protocol stacks and connection states. Its scope covers TCP / IP protocols, socket management, network device drivers, etc. It has built-in network diagnostic tool call interfaces (such as ss, tcpdump, ethtool). When network connections are abnormal or transmission delays occur, this agent analyzes connection states and queue backlogs through ss, captures and parses retransmissions and out-of-order packets through tcpdump, and checks network card driver parameters through ethtool. It then generates repair scripts to adjust TCP congestion control algorithms (such as switching to BBR), optimize connection queue length (net.core.somaxconn), or restart the network protocol stack. Corresponding tools refer to the command-line tools or library function interfaces built into each ExpertAgent that are compatible with specific operating systems (such as KylinOS). Repair scripts refer to automated task descriptions generated by ExpertAgent that can be directly executed by Ansible. These can take the form of shell command sequences, sysctl parameter modification commands, systemd service unit configurations, or cgroup resource limit configurations.

[0038] like Figure 4 This invention describes the execution flow under disk I / O bottleneck anomalies. It is understood that, by... Figure 3The ExpertAgents shown are subdivided into seven specialized intelligent agents: Disk, Kernel, System, Process, CPU, Mem, and Net. The system constructs a finely divided division of labor system covering the entire operating system stack, enabling each agent to focus on in-depth knowledge accumulation and toolchain optimization in a specific technical field. This avoids the knowledge overload and shallow reasoning problems caused by a single general-purpose agent having to handle anomalies in all scenarios. This heterogeneous design allows the system to accurately match agents with corresponding expertise based on the root cause type, achieving precise mapping from task to capability. Common anomalies can be quickly routed to lightweight ExpertAgents that are proficient in that field, utilizing their pre-built professional tools and template libraries to achieve sub-second response times, significantly improving processing efficiency. At the same time, by limiting the scope and toolset of each agent, the system can call highly specialized diagnostic data when generating repair solutions (such as DiskAgent calling blktrace to obtain block-level I / O timing, and KernelAgent calling bpftrace to trace kernel lock contention). These in-depth technical details are difficult for general-purpose agents to capture, thus ensuring the accuracy and effectiveness of the repair solutions. When BossAgent assigns an exception type, it actually activates the optimal processing engine for that domain. The process by which ExpertAgent calls the corresponding tools to generate repair scripts based on the assignment result essentially transforms the abstract root cause description into concrete, verifiable execution instructions tightly coupled with the operating system kernel. This transformation relies on the Agent's mastery of the parameters of specific domain-specific tools (such as MemAgent's familiarity with these parameters). The reasonable range of values ​​ensures the availability and security of the generated scripts. The classification system also supports flexible combination and expansion. When a new type of anomaly occurs, only the corresponding Agent type (such as GPUAgent) needs to be added and its toolchain needs to be registered. There is no need to reconstruct the overall architecture, which significantly enhances the system's evolvability.

[0039] Preferably, step S4 includes: The repair solution is deployed to a dual-iteration low-latency simulation sandbox using a deployment tool. The dual-iteration simulation capability of the sandbox is used to simulate the operating environment to verify the safety and applicability of the repair solution. At the same time, the resource scheduling logic of the sandbox is optimized to reduce runtime latency. Based on the test results of the sandbox, the process is divided: when the test results meet expectations, the repair plan is sent to the target operating system to perform the repair operation, and the execution results are synchronized to ReportAgent to generate a total repair process report; When the test results do not meet expectations, a feedback mechanism is triggered to return the abnormal information to the BossAgent to regenerate the decision.

[0040] It should be noted that the deployment tool in this embodiment refers to the Ansible automated operation and maintenance engine. Its function is to establish a secure communication channel with the sandbox management node and the target operating system through the SSH protocol, transfer the repair solution package (including shell scripts, configuration files, and Ansible task templates) from the control node to the sandbox execution environment, and remotely call the command execution interface in the sandbox to complete the solution deployment. This tool defines the transmission task through a YAML format playbook and uses its idempotency to ensure the consistency of repeated deployments. The "dual iteration" of the dual-iteration low-latency simulation sandbox refers to the first layer of environment simulation iteration. By dynamically synchronizing the kernel version number of the target operating system, the MD5 checksum of system library files, the snapshot of the running process state, and the network connection state, the sandbox environment maintains real-time image consistency with the production environment. Its technical implementation relies on Docker's volume mounting mechanism to mount key directories such as / etc, / proc, and / var from the production environment to the sandbox container in read-only mode, and using the criu tool to hot migrate the process state from the production environment to the sandbox. Simulation capability refers to the sandbox loading the same kernel modules as KylinOS, applying the same security policies (such as SELINUX rules and AppArmor configuration), and simulating the same hardware devices (using the sysfs virtual file system to disguise block device parameters). This provides a runtime context that is indistinguishable from the production environment at the system call layer, driver layer, and policy layer for the remediation solution, thereby verifying whether the solution will cause unpredictable side effects due to environment specificity. Security verification refers to the pre-built static and dynamic analysis engines within the sandbox performing high-risk command detection (such as AST syntax tree analysis to check for destructive operations like `rm -rf / `), unauthorized resource access detection (such as monitoring whether the `open()` system call attempts to read sensitive files like ` / etc / shadow`), and kernel parameter validity verification (such as checking whether the values ​​modified by `sysctl` are within the kernel's allowed range) on the repair script to ensure that the execution of the solution will not compromise system integrity. Applicability verification refers to the actual execution of the repair script in a simulated environment within the sandbox. By comparing performance metrics (such as iowait and CPU load), functional status (such as service liveness and port listening), and response time under simulated business load before and after execution, it verifies whether the repair solution achieves the expected optimization effect and does not introduce new performance degradation. Its technical implementation relies on the performance acquisition probe (based on eBPF) and business traffic replay tool (tcpreplay) integrated within the sandbox.Resource scheduling logic optimization refers to reducing the total startup and running time of the sandbox to 6-10 seconds through techniques such as CPU binding (binding sandbox processes to specific physical cores to avoid scheduling jitter), memory pre-allocation (using mmap to pre-allocate large page memory to reduce page faults), and I / O path optimization (placing the sandbox root file system in the tmpfs memory file system to avoid disk I / O interference), achieving an order-of-magnitude improvement compared to the traditional 30-40 seconds. Meeting expectations in the verification results means that the security scan has no alarms, performance indicators are optimized to within the target threshold range, business simulation tests are passed, and the service status is normal; all four conditions must be met simultaneously for the test to be considered passed. The target operating system specifically refers to KylinOS, characterized by its custom kernel, integrated domestic security modules, and specific driver support, resulting in compatibility differences with standard Linux distributions. Therefore, the sandbox's simulation capabilities must accurately simulate these specificities. ReportAgent is a dedicated intelligent agent responsible for collecting verification logs, execution output, and performance comparison data to generate structured HTML / PDF reports containing timelines, operation records, effect evaluations, and risk warnings. Its role is to provide traceable audit credentials and knowledge accumulation. The feedback mechanism refers to the process where, when the test results do not meet expectations, ReportAgent encapsulates the abnormal phenomena (such as script execution returning a non-zero exit code, performance indicators not improving, or business test failure) and the diagnostic data collected in the sandbox (such as kernel log dmesg and process coreedump files) into a structured feedback message, and sends it back to BossAgent through a message queue (such as RabbitMQ) or gRPC streaming interface, triggering it to re-execute root cause identification and task decomposition. This mechanism adopts an exponential backoff retry strategy (waiting 2 seconds after the first failure, 4 seconds after the second failure, and retrying up to 3 times) to avoid frequent retries that could cause system instability.

[0041] Preferably, such as Figure 5 As shown, the verification process of the dual-iteration low-latency simulation sandbox includes: The remediation scheme is subjected to a security scan, including static syntax risk scanning, high-risk command behavior interception, unauthorized command execution, and illegal access to sensitive paths detection; Configure environment synchronization using deployment tools, complete environment consistency verification, and then start the sandbox environment. Initial runtime data is collected in the sandbox environment, and optimized status data is collected after the repair plan is executed. The effectiveness of the repair solution is verified by comparing the initial running data with the optimized state data using a verification test script. When the verification result does not meet expectations, the inference model is triggered to regenerate the optimization strategy based on the data collected in the sandbox, and then return to the sandbox for re-verification; When the verification results meet expectations, the remediation plan is deployed to the target environment for execution using the deployment tool.

[0042] It should be noted that, in this embodiment, static syntax risk scanning refers to using an abstract syntax tree parser to perform syntactic structure analysis on the Shell script or Ansible Playbook in the remediation plan, identifying coding defects that may lead to execution failure, such as unclosed branch statements, undefined variable references, and grammatically incorrect pipe character usage. Its purpose is to eliminate human errors at the syntax level before the plan is executed, ensuring the script has basic executability. High-risk command behavior interception refers to intercepting high-risk commands based on a predefined behavior blacklist (such as...). The script performs pattern matching on command calls (such as those involving destructive commands). If a high-risk operation is detected, execution is immediately blocked and a security alert is reported. Its purpose is to prevent the remediation plan itself from becoming a source of system damage. Unauthorized command execution detection involves using kernel-level security mechanisms such as seccomp (secure computing mode) or AppArmor to restrict the script process's permission boundaries and monitor its system call sequence. If the script attempts to execute privileged operations beyond its authorized scope (such as using the mount system call to modify the file system mount state or setuid to elevate process privileges), interception is triggered and a violation log is recorded. Its purpose is to ensure that the remediation plan follows the principle of least privilege. Unauthorized access to sensitive paths detection uses path access control lists and file system monitoring technologies (such as inotify or Fanotify) to perform real-time review of file paths that the script attempts to access. If an attempt to read or access sensitive paths is detected... Sensitive documents or travel to If the kernel interface is accessed illegally, the operation is terminated. This is to protect critical system data and credentials from being leaked or tampered with. Deployment tool configuration environment synchronization refers to using Ansible's template rendering and configuration comparison functions to synchronize the production environment's kernel parameter configuration files (such as...) ), service unit files (such as Files in the directory) and environment variable definition files (such as...) The entire system configuration is copied to the corresponding path in the sandbox to ensure that the system configuration in the sandbox is consistent with the production environment byte by byte. This is to eliminate the distortion of verification results caused by differences in configuration files. Environment consistency verification refers to executing a series of health check scripts after the sandbox starts, comparing the kernel version number (output of uname -r), key dynamic library versions (ldd --version), and system call table integrity (via...) between the sandbox and the production environment. The test includes basic commands and process tree structure. If the differences exceed a preset threshold (such as inconsistent version numbers or missing critical services), the environment is deemed inconsistent and resynchronized. Its purpose is to verify the reliability of the sandbox simulation environment and prevent solutions that pass testing in a faulty environment from failing in the production environment. Initial runtime data collection refers to the process where, before the remediation plan is executed, the eBPF performance probe embedded in the sandbox collects baseline performance metrics of the system at a preset sampling period (e.g., once per second), including CPU utilization (obtained per core via mpstat), memory page cache usage (obtained via...). The Cached field in the data), disk I / O latency (via the wa field in vmstat), and network throughput (via the cached field in vmstat). Statistical analysis of the data serves as a benchmark for subsequent effect evaluation, providing quantifiable before-and-after comparisons to make the remediation effect measurable. Collecting optimized state data after the remediation plan is executed involves immediately repeating the same performance data collection process after the script finishes execution, obtaining the instantaneous and sustained values ​​of the same indicators after optimization (e.g., continuously collecting data for 30 seconds to observe stability). Its purpose is to objectively assess whether the remediation plan produces the expected performance improvement and the sustainability of the improvement. Verification test scripts are pre-built test programs within the sandbox that simulate real business loads. These include I / O-intensive tests (using fio to perform random read / write), computation-intensive tests (using sysbench for prime number calculations), and network-intensive tests (using iperf3 for streaming). Their purpose is to verify the stability and effectiveness of the remediation plan under real pressure by applying controlled loads, avoiding spurious optimizations under no-load conditions. Comparing initial running data with optimized state data involves using statistical methods to calculate the rate of change of indicators and combining hypothesis testing (e.g., paired-samples t-test) to determine whether the change is significant (p-value < 0.05). Its purpose is to scientifically verify the remediation effect rather than relying on subjective observation. Triggering the inference model to regenerate optimization strategies based on sandbox-collected data means that when verification results do not meet expectations, ReportAgent encapsulates performance data, service log fragments, and test failure information collected from the sandbox into prompt words, which are then input into BossAgent's large language model. The model analyzes the defects of the original solution through thought chain analysis (e.g., "cleaning the page cache was too aggressive, causing process blocking"), and combines this with sandbox data to generate new parameter adjustment suggestions (e.g., "to..."). The adjustment to 120 instead of 150" allows the solution to self-reflect and optimize, avoiding manual intervention. Re-verification involves resubmitting the newly generated optimization strategy to the sandbox for execution, repeating the above collection, execution, and comparison process until the results meet expectations or the maximum number of retries is reached. This ensures the optimized solution has undergone complete verification and possesses reliability. Deploying the solution to the target environment for execution means that after successful verification, Ansible connects to the production KylinOS host via SSH, using the copy module to transfer scripts, the shell or sysctl module to execute commands, and the systemd module to manage service status. This securely applies the rigorously verified and reliable solution to the real system.

[0043] Automated operation and maintenance systems based on heterogeneous multi-agent and sandbox verification, such as Figure 6 As shown, it includes: The knowledge base construction module is used to build a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; The root cause analysis module is used to identify the root causes of multi-source anomalies and decompose complex tasks based on the three-in-one dynamic knowledge base by BossAgent. The confidence decision module is used to calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, several ExpertAgents are activated and the ExpertAgents are used to collaboratively generate a remediation plan through the MCP protocol. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. The sandbox testing module is used to deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, the repair solution is sent to the target operating system for repair and a repair process report is generated. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

[0044] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0045] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. An automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification, characterized in that, Includes the following steps: S1: Construct a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; S2: Utilize BossAgent to perform root cause identification and complex task decomposition of multi-source abnormal data based on the aforementioned three-in-one dynamic knowledge base; S3: Calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, activate several ExpertAgents and use the MCP protocol to enable the ExpertAgents to collaboratively generate a remediation plan. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. S4: Deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, send the repair solution to the target operating system for repair and generate a repair process report. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

2. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification according to claim 1, characterized in that, In step S1, the construction of the three-in-one dynamic knowledge base includes: using KPI trees to vertically correlate system performance indicators to locate anomalies and deduce causal chains, combining topology graphs to horizontally map system components and their dependencies to analyze anomaly propagation paths, and dynamically capturing system kernel call sequences through Trace call stacks to provide underlying execution evidence, forming a full-dimensional anomaly localization system covering vertical indicator chains, horizontal component chains, and deep causal chains, thus obtaining the three-in-one dynamic knowledge base.

3. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification according to claim 1, characterized in that, Step S2 includes: Based on the aforementioned three-in-one dynamic knowledge base, KPI nodes associated with anomalies are anchored through BossAgent, and the dependencies between components are traced according to the topology graph to construct the anomaly propagation path. At the same time, the complete execution link process is restored through Trace call stack. Root cause identification is achieved through comprehensive analysis of the KPI nodes, anomaly propagation path, and complete link process, and the operation and maintenance tasks are decomposed according to the identified root causes.

4. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification according to claim 1, characterized in that, Calculating the confidence level of the root cause identification and comparing the confidence level with a preset threshold includes: When the confidence level exceeds the preset threshold: Synchronize the KPI nodes, topology relationships, and Trace call stack information and collaboration targets associated with the anomaly, assign processing roles to each Agent, unify the data interaction format to achieve cross-Agent data interoperability, form a consensus evidence chain through multiple rounds of evidence exchange and judgment correction, and distribute tasks and generate repair solutions based on the consensus evidence chain. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair solution.

5. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox inspection according to claim 1, characterized in that, The ExpertAgents include at least one of the following types: DiskAgent, KernelAgent, SystemAgent, ProcessAgent, CPUAgent, MemAgent, and NetAgent; The DiskAgent is used to handle solutions for disk space exhaustion and IO latency anomalies; the KernelAgent is used to handle solutions for kernel deadlock and parameter exhaustion anomalies; the SystemAgent is used to handle solutions for system resource exhaustion and service anomalies; the ProcessAgent is used to handle solutions for process scheduling and resource leakage anomalies; the CPUAgent is used to handle solutions for processor overload and interrupt anomalies; the MemAgent is used to handle solutions for memory exhaustion and swapping anomalies; and the NetAgent is used to handle solutions for network connection and transmission anomalies. ExpertAgent invokes the appropriate tools to generate repair scripts based on the assigned exception type.

6. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification according to claim 1, characterized in that, Step S4 includes: The repair solution is deployed to a dual-iteration low-latency simulation sandbox using a deployment tool. The dual-iteration simulation capability of the sandbox is used to simulate the operating environment to verify the safety and applicability of the repair solution. At the same time, the resource scheduling logic of the sandbox is optimized to reduce runtime latency. Based on the test results of the sandbox, the process is divided: when the test results meet expectations, the repair plan is sent to the target operating system to perform the repair operation, and the execution results are synchronized to ReportAgent to generate a total repair process report; When the test results do not meet expectations, a feedback mechanism is triggered to return the abnormal information to the BossAgent to regenerate the decision.

7. The automated operation and maintenance solution based on heterogeneous multi-agent and sandbox verification according to claim 1, characterized in that, The verification process for the dual-iteration low-latency simulation sandbox includes: The remediation scheme is subjected to a security scan, including static syntax risk scanning, high-risk command behavior interception, unauthorized command execution, and illegal access to sensitive paths detection; Configure environment synchronization using deployment tools, complete environment consistency verification, and then start the sandbox environment. Initial runtime data is collected in the sandbox environment, and optimized status data is collected after the repair plan is executed. The effectiveness of the repair solution is verified by comparing the initial running data with the optimized state data using a verification test script. When the verification result does not meet expectations, the inference model is triggered to regenerate the optimization strategy based on the data collected in the sandbox, and then return to the sandbox for re-verification; When the verification results meet expectations, the remediation plan is deployed to the target environment for execution using the deployment tool.

8. An automated operation and maintenance system based on heterogeneous multi-agent and sandbox testing, characterized in that, include: The knowledge base construction module is used to build a three-in-one dynamic knowledge base, which integrates KPI tree, topology graph and Trace call stack; The root cause analysis module is used to identify the root causes of multi-source anomalies and decompose complex tasks based on the three-in-one dynamic knowledge base by BossAgent. The confidence decision module is used to calculate the confidence level of the root cause identification and compare the confidence level with a preset threshold. When the confidence level exceeds the preset threshold, several ExpertAgents are activated and the ExpertAgents are used to collaboratively generate a remediation plan through the MCP protocol. When the confidence level does not exceed the preset threshold, the SeniorAgent is activated to handle high-complexity anomalies, and the SeniorAgent generates a repair plan. The sandbox testing module is used to deploy the repair solution to a dual-iteration low-latency simulation sandbox for security and applicability testing. If the test results meet expectations, the repair solution is sent to the target operating system for repair and a repair process report is generated. If the test results do not meet expectations, the abnormal information will be returned to the BossAgent to regenerate the decision.

Images