An Intelligent Self-Healing System and Method for Operation and Maintenance Based on LLM Agent

Abstract

This invention discloses an intelligent self-healing system and method for operation and maintenance based on an LLM Agent, belonging to the fields of information technology and operation and maintenance automation technology. The system includes a data acquisition and preprocessing module, a rule scheduling module, a large language model intelligent agent module, a work order generation module, and an approval and execution module. It collects and standardizes multi-source monitoring data through dual channels, and the rule scheduling module intelligently distributes data according to single alarm or multi-alarm aggregation conditions: simple scenarios directly trigger the automatic operation and maintenance mechanism; complex scenarios submit the alarm context and predefined operation types to the LLM Agent to output optimal execution parameters. The work order generation module automatically generates work orders; the execution engine automatically completes the repair operation. This invention solves the problems of rigidity in traditional rule engines and the uncontrollability and difficulty in integration of pure AI models, achieving an end-to-end self-healing closed loop that balances security and intelligence, significantly improving the automation coverage and decision-making accuracy in complex scenarios.

Description

Technical Field

[0001] This invention belongs to the fields of information technology and operation and maintenance automation technology, and relates to an intelligent self-healing system and method for operation and maintenance based on LLM Agent. Background Technology

[0002] As digital transformation deepens, enterprise IT architectures are rapidly evolving towards greater complexity, cloud-native architecture, and multi-technology stack integration. The widespread adoption of technologies such as containerized deployment, microservice architecture, and distributed data centers has led to an exponential increase in the scale and complexity of IT infrastructure and business systems. Simultaneously, the coverage of operation and maintenance monitoring systems, such as Prometheus, Zabbix, and Nagios, continues to expand, enabling real-time collection of operational metrics across the entire value chain from infrastructure to business applications, resulting in an explosive growth in the number of alarm events generated.

[0003] However, the growth of enterprise operations and maintenance (O&M) human resources is far from keeping pace with the surge in alarm volume, posing a serious challenge to the traditional O&M model that heavily relies on manual operations. O&M personnel must manually review massive numbers of alarms, analyze complex root causes, and manually execute remedial operations. This model is not only inefficient in response, failing to meet the stringent requirements of high availability and stability, but is also highly susceptible to errors and secondary failures, especially late at night, on holidays, or when personnel are fatigued. Against this backdrop, automated O&M technology has become a core direction for resolving the contradiction between "massive alarms and limited manpower." However, existing mainstream automated O&M solutions still have significant shortcomings and struggle to cope with complex and ever-changing real-world O&M scenarios.

[0004] Current automated operation and maintenance solutions mainly fall into two categories, both of which have significant shortcomings: 1. Automated scripts based on predefined rules (such as RPA, Ansible Playbook): This method typically requires operations personnel to pre-write detailed processing logic and scripts for each specific scenario. Its drawbacks and limitations are: Poor flexibility: It cannot handle fault scenarios that are not predefined by rules, are unknown, or are complex. When multiple alarm combinations occur, simple "IF-THEN" rules struggle to make effective and accurate decisions. High maintenance costs: Frequent changes in business and system architecture require constant updates and debugging of a large number of scripts, making the rule base bloated and difficult to manage; Low level of intelligence: It lacks true cognitive and reasoning abilities. For example, when faced with multiple alarms occurring simultaneously, such as "QPS surge", "RT increase", and "CPU utilization exceeding threshold", the rule system may only mechanically execute the expansion operation, but it cannot intelligently determine the specific number of replicas that need to be expanded, which can easily lead to resource waste or insufficient expansion.

[0005] 2. Solutions purely based on artificial intelligence (such as early machine learning models): These solutions attempt to predict failures or provide suggestions through models. Their limitations and shortcomings are: Poor interpretability and low credibility: The model is like a "black box", and its decision-making process and basis are difficult for operation and maintenance personnel to understand and trust, making it difficult for automatically executed actions to be approved.

[0006] "Illusion" and risk of misoperation: When training data is insufficient or abnormal operating conditions are encountered, the model may make wrong decisions, which may lead to more serious production accidents if executed automatically.

[0007] Integration difficulties with existing processes: It is difficult to seamlessly integrate with rigorous enterprise IT service management (ITSM) processes (such as work order approval), and it can usually only serve as an auxiliary recommendation system, unable to achieve an end-to-end closed loop.

[0008] Therefore, the industry urgently needs an automated self-healing solution that can not only have highly intelligent decision-making capabilities, but also be closely integrated with existing operation and maintenance rules and approval processes, and whose behavior is reliable and explainable. Summary of the Invention

[0009] This invention addresses the shortcomings of existing O&M automation technologies in handling complex scenarios, including insufficient flexibility of traditional automation tools, limited scenario coverage, coarse decision-making, uncontrollable risks of pure AI models, poor interpretability, and difficulty in integrating into existing O&M processes. It proposes an intelligent self-healing system and method for O&M based on an LLM Agent (Large Language Model Intelligent Agent). Through a hybrid architecture of a "rule scheduling framework + LLM Agent intelligent kernel," rules are responsible for establishing security boundaries and scenario routing, providing fundamental guarantees for automation. Within the security scope defined by the rules, the LLM Agent fully leverages its intelligent advantages in natural language understanding, multi-factor correlation analysis, and complex situational decision-making. This architecture evolves O&M automation from the traditional "rule-based automation" to a new stage of "cognitive intelligent decision-making," significantly improving the intelligence level for handling complex and dynamic O&M scenarios while ensuring system controllability.

[0010] Firstly, the present invention provides an intelligent self-healing system for operation and maintenance based on an LLM Agent, and the technical solution adopted to solve the above-mentioned technical problems is as follows: An intelligent self-healing system for operation and maintenance based on an LLM agent includes a data acquisition and preprocessing module, a rule scheduling module, a large language model intelligent agent module, a work order generation module, and an approval execution module, wherein: The data acquisition and preprocessing module acquires multi-source heterogeneous data from the multi-source monitoring system in real time, preprocesses the multi-source heterogeneous data, and generates standardized alarm information. The multi-source heterogeneous data includes alarm events from various monitoring systems. The rule scheduling module is used to receive the standardized alarm information and match and distribute the alarm information according to preset rules. The large language model intelligent agent module is connected to the rule scheduling module and is used to receive detailed context information of alarm events that match preset rules and the type of operation to be performed, which are distributed by the rule scheduling module; parse the alarm content and perform reasoning to complete the decision. The work order generation module automatically generates work orders that conform to IT service management standards based on the decision results output by the large language model intelligent agent module. The approval and execution module pushes the generated work order to the ITSM system for manual approval. Once approved, the repair operation is automatically executed, the execution result is recorded, and a closed-loop recovery is performed.

[0011] Optionally, the data acquisition and preprocessing module adopts a dual-channel acquisition mechanism, which acquires alarm events from multi-source monitoring systems through push notifications or pull acquisition methods. Through data cleaning and normalization preprocessing, the multi-source heterogeneous data is converted into unified structured data and stored in a unified event queue to generate standardized alarm information. The pull acquisition method is a polling method, and the monitoring system can be one or more different types of monitoring systems.

[0012] Optionally, the rule scheduling module includes a rule matching unit and a scheduling decision unit: The rule matching unit periodically scans the standardized alarm information generated by the data acquisition and preprocessing module at preset time intervals, and matches the standardized alarm information with different preset rules in the rule base; The scheduling decision unit performs traffic distribution and scheduling based on the matching results, directly triggering the automatic operation and maintenance mechanism, or distributing detailed context information of all relevant alarms corresponding to the preset rule, as well as the operation types to be executed as defined by the preset rule, to the large language model intelligent agent module.

[0013] Further, optionally, the preset rules are stored in a rule base in a structured form, including: Matching conditions: Supports single alarm conditions and multi-alarm aggregation conditions, wherein the aggregation conditions include multi-alarm association matching within a time window; Operation type: The type of operation and maintenance operation to be performed, including one or more of the following: node restart, service expansion, resource cleanup, rate limiting, and slow SQL interception; Priority: Can be dynamically configured according to business needs, used to determine the order in which rules are triggered.

[0014] Optionally, the matching result output by the rule matching unit is a structured object that supports logging and persistent storage, including: Matching rule identifier, used to indicate a preset rule that has been successfully matched; The matching condition type specifies whether the matching is based on a single alarm condition or a multi-alarm aggregation condition in the rule base. The associated alarm context set contains standardized information about all relevant alarms that triggered this preset rule; Predefined operation instructions, including the operation type and priority defined by the rules.

[0015] Optionally, the large language model intelligent agent module includes a natural language parsing unit, an inference planning unit, a tool invocation and data acquisition unit, and an inference decision-making unit, wherein: The natural language parsing unit receives structured alarm context information from the rule scheduling module and converts it into a semantic problem description that the reasoning planning unit can directly understand and process. The reasoning and planning unit receives the type of operation to be performed as determined by the rule scheduling module, as well as the semantic problem description generated by the natural language parsing unit. Through a multi-step thinking chain, it analyzes and determines the decision data dimensions required to perform the operation. The tool call and data acquisition unit, based on the required decision data dimensions derived by the reasoning and planning unit, acquires real-time system status data related to operation execution by calling the system's pre-set toolset; The reasoning and decision-making unit, based on the acquired real-time system status data and combined with internal knowledge calculations, arrives at a decision result.

[0016] Further, optionally, the internal knowledge includes: The company's business system's IT architecture, technology stack, operation and maintenance specifications, and standardized operation execution manual; Historical maintenance events, responsible person information, and development specifications; Real-time monitoring, changes, inspections, and other dynamic business status information.

[0017] Optionally, the work order generation module includes a work order generation unit and a format verification unit; The work order generation unit, based on the work order field templates predefined by the rule scheduling module, automatically initiates work orders by calling utility functions and according to the decision results output by the large language model intelligent agent module, wherein: The basic information field is directly filled with the operation intent information determined by the rule scheduling module; the execution parameter field is filled with the specific execution parameters calculated by the large language model intelligent agent module; and the decision basis field is generated based on the reasoning process of the large language model intelligent agent module and real-time query data. The format validation unit performs structured validation on the filled content using the work order field templates predefined by the rule scheduling module, ensuring that the work order format conforms to the interface specifications of the downstream ITSM system.

[0018] Optionally, the approval execution module includes a work order submission unit, an execution engine unit, and a result recording and closure unit: The work order submission unit pushes the generated work orders to the IT service management system for manual approval. The work order content includes alarm context, reasoning logic, and real-time data. The execution engine unit automatically parses and captures the operation instructions and key parameters in the work order after the work order is approved, and calls the corresponding execution interface to complete the operation. The results are recorded and the closed-loop unit records the execution status and results in real time. After the system returns to normal, it automatically triggers resource reclamation operations based on preset conditions or monitoring indicators, updates the work order status and synchronizes it to the configuration management database of IT service management, forming a complete operation and maintenance closed loop.

[0019] Secondly, this invention provides an intelligent self-healing method for operation and maintenance based on an LLM Agent. The technical solution adopted to solve the above-mentioned technical problems is as follows: An intelligent self-healing method for operation and maintenance based on LLM Agent, applied to an intelligent self-healing system for operation and maintenance based on LLM Agent, includes the following steps: S1: Real-time acquisition of multi-source heterogeneous data from multi-source monitoring systems, preprocessing of the multi-source heterogeneous data and generation of standardized alarm information, where multi-source heterogeneous data includes alarm events from various monitoring systems; S2: Receive the standardized alarm information and match and distribute the alarm information according to preset rules; S3: Receive detailed context information of alarm events that match preset rules and the type of operation to be performed, distributed by the rule scheduling module; parse the alarm content and perform reasoning to complete the decision. S4: Based on the decision results output by the large language model intelligent agent module, automatically generate work orders that conform to IT service management standards; S5: Push the generated work order to the ITSM system for manual approval. Once approved, automatically execute the repair operation, record the execution results, and perform closed-loop recycling.

[0020] Optionally, receiving the standardized alarm information and matching and routing the alarm information according to preset rules includes: S21: When a simple rule is matched, the automatic operation and maintenance mechanism is triggered directly.

[0021] S22: When a complex aggregation rule is matched, the detailed context information of all relevant alarms corresponding to the preset rule, as well as the operation type to be performed as defined by the preset rule, will be distributed to the large language model agent module.

[0022] The present invention provides an intelligent self-healing system and method for operation and maintenance based on an LLM agent, which has the following advantages compared with the prior art: 1. Hybrid Architecture Collaboration and Intelligent Triage: Achieving Organic Unity of Security and Intelligence with Full Scenario Coverage: By constructing a collaborative hybrid architecture of "rule scheduling framework + LLM Agent intelligent kernel" and adopting a triage scheduling logic of "direct execution of simple rules + intelligent decision-making of complex rules," this system achieves a precise balance between security and flexibility. The rule scheduling module is responsible for establishing security boundaries, routing scenarios, and defining clear tasks (such as operation types), thus avoiding the "illusion" and uncontrollable risks of pure AI models in access judgment from the source. At the same time, within the security scope and clear tasks defined by the rules, the LLM Agent fully leverages its cognitive intelligence advantages in natural language understanding, multi-alarm correlation analysis, and complex situational reasoning. This design retains the determinism and security of the rule engine while overcoming its flexibility limitations, enabling the system to efficiently handle simple scenarios covered by automated operation and maintenance mechanisms and deeply address complex scenarios such as multi-alarm aggregation that are difficult for traditional solutions to handle. This achieves a significant expansion of the scope of automation coverage and completes a substantial evolution from "rule-based mechanical automation" to "cognitive intelligent decision-making."

[0023] 2. Deep Fusion and Precise Routing of Multi-Source Heterogeneous Data: Through a dual-channel (push notification / polling retrieval) data acquisition mechanism and standardized cleaning and normalization preprocessing procedures, the system achieves comprehensive and reliable access and unified formatting of multi-source heterogeneous monitoring data from Prometheus, Zabbix, Nagios, and other sources, ensuring the integrity, consistency, and timeliness of alarm information. Combined with the refined definitions of single alarm conditions and multi-alarm aggregation conditions (including time window association) in the rule base, the system can effectively filter and accurately identify scenarios in the early stages of alarm influx, achieving intelligent traffic routing and scheduling. This provides a high-quality data foundation for subsequent rule matching and LLM in-depth analysis, significantly improving the overall efficiency and resource utilization of event processing.

[0024] 3. LLM Agent's structured reasoning and dynamic optimization ensure decision accuracy and business adaptability: Through the collaborative work of natural language parsing, inference planning, tool invocation, data acquisition, and inference decision-making within the LLMAgent module, the system can dynamically plan the required data dimensions after receiving a clearly defined task. It can also acquire system status data in real time by invoking pre-built toolsets and perform multi-step inference and parameter optimization calculations based on rich embedded internal knowledge. This ensures that every self-healing decision not only conforms to semantic logic but also has accurate data support and specific business adaptability, fundamentally overcoming the shortcomings of traditional rule-based decision-making being crude or the uncontrollable output of pure AI models.

[0025] 4. Explainable automatic work order generation and end-to-end closed loop ensure process compliance and operational economy: The work order generation module adopts a template-driven, content-filling mechanism with clearly sourced fields and an automated format validation mechanism. Operation types are directly derived from rule base definitions, execution parameters are derived from LLM intelligent calculations, and decision-making is based on a complete inference chain and real-time data. This generates highly explainable work orders that conform to IT service management standards. Format validation ensures compatibility with downstream system interfaces, achieving "one-time generation, direct usability" for work orders. The approval and execution module, through a seamless end-to-end process of work order review, automatic instruction parsing and execution, result recording, and adaptive resource recycling, not only ensures that maintenance activities are deeply integrated into the enterprise ITSM process, possessing complete auditability and traceability, but also automatically triggers resource recycling based on monitoring indicators after system recovery, achieving economical and efficient maintenance operations and flexible resource management.

[0026] 5. Clear division of labor reduces system overhead and improves stability and maintenance convenience: Through a clear division of labor between the rule scheduling module and the LLMA Agent module—specifically, "admission judgment and task definition" versus "intelligent calculation and parameter optimization"—the system achieves risk isolation and complementary capabilities. The rule module, based on rules stored in a table structure, is responsible for reliable judgments of "whether to do it and what to do," significantly reducing the context load and illusion risks of LLM; the LLM Agent focuses on high-value intelligent calculations of "how to do it and how much to do." This division of labor not only reduces model processing time and computational overhead, improving system stability in production environments, but also makes the rule base easier for operations personnel to maintain, update, and transfer.

[0027] 6. Deeply compatible with existing technology systems, providing a highly implementable comprehensive solution: This solution does not require the reconstruction of existing enterprise monitoring, ITSM, Kubernetes, and other operation and maintenance toolchains. Through modular design, it achieves seamless integration, demonstrating high implementability and adaptability. It not only compensates for the shortcomings of automated operation and maintenance mechanisms in decision-making in complex scenarios but also solves the problem of integrating pure AI models with rigorous approval processes. Ultimately, it achieves a comprehensive effect of "rapid perception, intelligent decision-making, process compliance, precise execution, and automatic closed-loop" in complex IT environments, significantly improving operation and maintenance response speed, reducing the burden of manual intervention and operational risks, enhancing system self-healing capabilities and business continuity, and providing a practical technical path for building an intelligent, reliable, and secure next-generation operation and maintenance system. Attached Figure Description

[0028] Appendix Figure 1 This is a system architecture diagram of an intelligent self-healing system for operation and maintenance based on an LLM Agent; Appendix Figure 2 This is a flowchart of the rule scheduling module of an intelligent self-healing operation and maintenance system based on an LLM Agent. Appendix Figure 3 This is a flowchart of the workflow of a large language model intelligent agent module in an intelligent self-healing system for operation and maintenance based on an LLM Agent. Appendix Figure 4 This is a flowchart of an intelligent self-healing method for operation and maintenance based on an LLM Agent. Detailed Implementation

[0029] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0030] Example 1: Combined with appendix Figure 1 This embodiment proposes an intelligent self-healing system for operation and maintenance based on LLM Agent, including a data acquisition and preprocessing module, a rule scheduling module, a large language model intelligent agent module (hereinafter referred to as the LLM Agent module), a work order generation module, and an approval execution module.

[0031] (I) Data Acquisition and Preprocessing Module This module serves as the system's "sensory nerves" and "data cleaning center," deployed on the enterprise's unified operations and maintenance platform. Its core responsibility is to achieve real-time, reliable, and lossless collection of massive, heterogeneous alarm events generated by various monitoring systems in IT infrastructure and software system operations and maintenance scenarios through a standardized processing pipeline. It transforms the raw and messy "data raw materials" into standardized alarm information with a unified format and reliable quality, laying the data foundation for subsequent rule matching and intelligent decision-making.

[0032] 1. Multi-source monitoring data acquisition: Dual-channel mechanism ensures full coverage and high timeliness. To address the challenges posed by the diverse technology stacks and inconsistent interfaces of enterprise monitoring systems, this module employs a dual-channel hybrid acquisition mechanism combining real-time push and periodic pull, ensuring full compatibility with mainstream monitoring systems such as Prometheus, Zabbix, and Nagios, and guaranteeing complete and accurate access to multi-source heterogeneous alarm data.

[0033] The push-based data acquisition channel features high timeliness and passive reception: For modern monitoring systems that support real-time alarm pushes (such as Prometheus with Alertmanager configured, or Zabbix with Webhook notifications), a highly available API receiving endpoint is deployed. When the monitoring system detects that an indicator exceeds a threshold or an abnormal state, it proactively and in real-time pushes alarm event data to the designated receiving port of this module. This method achieves alarm access latency in the millisecond to second range, making it particularly suitable for core business monitoring scenarios with extremely high timeliness requirements.

[0034] The pull-based data acquisition channel features integrity assurance and proactive polling: it provides proactive data acquisition capabilities for monitoring systems that require active data acquisition or for metrics that do not have open push interfaces. This channel initiates data query requests periodically at configurable time intervals (e.g., every 5 minutes), extracting all newly generated alarm events within a specified time period by calling the monitoring system's native data interface. This channel serves as a supplementary and backup mechanism, ensuring no blind spots in monitoring and supporting a failure retry mechanism to guarantee the reliability of the data acquisition process.

[0035] 2. Multi-source heterogeneous data preprocessing workflow: Transformation from messy to unified structured data The raw alarm data collected exhibits significant differences in format and semantics due to variations in the type of monitoring system. This module transforms it into a unified format that can be directly used by subsequent modules through a process of "parsing-cleaning-conversion-enhancement".

[0036] Data parsing and field mapping: The system first automatically selects the corresponding parser based on the data source to extract key fields from the raw data. Then, through a predefined and maintained "global field mapping table", heterogeneous fields from different monitoring systems (such as different naming conventions for "alarm name", "target object", and "timestamp") are accurately mapped to the corresponding fields in the "global field mapping table", thus achieving data semantic unification.

[0037] Data cleaning and quality enhancement are performed on the parsed data: 2.1 Deduplication and invalid filtering: Based on alarm characteristics, identify and discard alarms that are completely duplicated within a short period of time, and at the same time remove invalid data such as missing key fields and abnormal indicator values.

[0038] 2.2 Contextual Association and Information Enhancement: Using "target object" information (such as hostname, service name, IP) as keys, the enterprise configuration management database (CMDB) or service catalog is queried in real time to automatically attach contextual metadata to each alarm event, such as: business line, operation and maintenance manager, service / equipment importance level, cluster and environment (production / test), etc., to enrich the semantic information of the alarm.

[0039] 2.3 Format Normalization and Structured Output: The cleaned and enhanced data is transformed into internally defined, standardized alarm information with a unified structure. All events are ultimately converted into unified structured data. Each message explicitly contains a series of core fields, such as: unique event ID, alarm name, target object, monitoring metrics and values, trigger threshold, standardized timestamp, severity level, data source system, and business context information obtained from the CMDB.

[0040] 3. Standardized data storage and output: Event queues enable reliable buffering and ordered provisioning. The preprocessed structured data does not directly impact downstream processing modules; instead, it is immediately published to a high-performance, persistent event queue that functions as both a "data bus" and a "buffer pool." This system effectively isolates potentially sudden surges in front-end data collection traffic from the relatively complex and time-consuming rule matching and analysis processes at the back-end, preventing downstream systems from being overwhelmed and improving overall system stability. The queue's persistent nature ensures that alarm information is not lost even during brief module restarts or downstream processing anomalies, guaranteeing the reliability of the data processing flow. It provides a stable and orderly data retrieval point for the downstream rule scheduling module. The rule scheduling module can scan this queue at preset intervals (e.g., every minute) to batch retrieve standardized alarm information within a specified time window for subsequent processing, achieving controllable processing rhythm. The queue itself supports high-concurrency read and write operations and can dynamically adjust resources based on historical alarm volumes to avoid data overflow.

[0041] Through the implementation of this module, raw alarm events of varying formats originating from different monitoring systems within the enterprise, such as Prometheus, Zabbix, and Nagios, are collected reliably and in real time. After deep cleaning and business enhancement, they are transformed into standardized alarm information with unified format, rich semantics, controllable quality, and orderly buffering. This provides high-quality, highly consistent data input for the subsequent rule scheduling module to perform complex scenario matching and intelligent routing distribution, providing a data foundation for the entire intelligent self-healing process.

[0042] (II) Rule Scheduling Module Combined with appendix Figure 2 This module serves as the system's "intelligent scheduling hub" and "security decision gateway." Its core responsibility is to receive standardized alarm information from the data acquisition and preprocessing modules, and through a configurable and maintainable rule base, perform real-time pattern recognition, scenario determination, and task routing for alarm events. This module embodies the core concept of the "rule-based scheduling framework," achieving an intelligent hierarchical processing logic of "rapid response in simple scenarios and precise routing in complex scenarios," connecting data perception and intelligent decision-making.

[0043] 1. The structure of the rule base and the configuration of preset rules The rule base uses structured storage (such as database tables or configuration files) to facilitate maintenance, updates, and transfer by operations and maintenance personnel. It serves as the core knowledge carrier and decision-making basis for this module. Each preset rule is a clearly defined policy, containing three parts: matching conditions, operation type, and priority. Together, they achieve precise description and security definition of various operational scenarios. 1.1 Matching conditions: Defines the logical conditions required to trigger the rules, which is the basis for implementing "scene routing".

[0044] (a) Single alarm condition: Based on the threshold or status judgment of a single monitoring indicator, used to handle simple and clear fault scenarios. For example: "The monitoring indicator is CPU utilization, and the indicator value is >85%" or "The monitoring indicator is service HTTP error rate, and the indicator value is >5%".

[0045] (b) Multiple Alarm Aggregation Conditions: Defines a complex pattern where multiple related alarms must occur simultaneously or according to specific logic within a specific time window, used to identify complex faults involving multiple intertwined factors. For example: "The same service (service name: Order Service) simultaneously triggers three alarms within 5 minutes: 'QPS>1000', 'Average Response Time>2 seconds', and 'CPU Utilization>85%'", or "The same node consecutively triggers two alarms within 10 minutes: 'Memory Utilization>90%' and 'Critical Process Abnormal Exit'". This condition provides a clear and pre-audited input scenario for in-depth analysis by the LLM Agent.

[0046] 1.2 Operation Types: This section clearly defines the type of operational operation the system should initiate upon successful rule matching. This provides a clear and secure task instruction passed from the rule to subsequent modules, ensuring that the LLM Agent can focus on "how to do it well" without needing to determine "what to do." Operation types can be flexibly bound based on matching conditions. For example, "CPU utilization exceeding threshold" corresponds to "service scaling up," "disk I / O overload" corresponds to "resource cleanup," and "slow query alarm" corresponds to "slow SQL interception." It also supports node restart, service scaling down, rate limiting interception, and failover, comprehensively covering core operational scenarios.

[0047] 1.3 Priority: A dynamically configurable numerical or level field (e.g., set to four levels from P0 to P3) based on business importance, used to handle conflicts and scheduling order when multiple rules may be triggered simultaneously. For example, the priority of the "core database master node unavailable" rule can be set to the highest (P0), and the priority of the "non-core statistics service CPU utilization high" rule can be set to a lower (P2). The scheduling decision unit will determine the processing order based on priority to ensure that critical business failures receive the highest priority response.

[0048] 2. Working mechanism and process of the rule matching unit The rule matching unit runs periodically at configurable preset time intervals (e.g., every 30 seconds or 5 minutes), and its workflow is as follows: 2.1 Data Snapshot Acquisition: Extract all standardized alarm information generated in the past period from the unified event queue upstream in batches to form an "event snapshot" to be analyzed.

[0049] 2.2 Priority-Based Rule Traversal and Matching Calculation: The matching engine traverses all preset rules in the rule base from high to low priority to avoid low-priority rules unnecessarily consuming processing resources. For single-alarm condition rules, the core fields such as monitoring indicators, values, and target objects in the alarm information are directly compared. For multi-alarm aggregation condition rules, the alarm information is first grouped according to dimensions such as service identifier and node identifier, and then it is verified whether there are multiple related alarms within the same group that meet all the conditions defined in the rule.

[0050] 2.3 Matching Result Output: After completing the scan, output all matching results for the current period. Each result is a structured object, which, in addition to containing the identifier of the successfully matched rule, also includes the rule condition type on which the match was based and its corresponding information: Matching condition type: Explicitly indicate whether the match is based on a "single alarm condition" or a "multiple alarm aggregation condition" in the rule base; Related alarm context set: A detailed set of context information for all related alarms that triggered this preset rule (i.e., all structured alarm data that constitutes this scenario). Predefined operation instructions: The operation type and priority defined by this preset rule.

[0051] All matching processes and results support logging and persistent storage, providing a basis for subsequent traffic routing, scheduling, and operational auditing.

[0052] 3. Intelligent routing logic of the scheduling decision unit The scheduling decision unit executes traffic allocation and scheduling decisions based on the matching results. When the matched rule is a single alarm condition, and the corresponding processing flow is simple and the risk is controllable, the scheduling decision unit will directly trigger the corresponding automation mechanism. When the matched rule is a multi-alarm aggregation condition, the complex scenario is submitted to the LLM Agent for intelligent decision-making, which specifically reflects the design concept of "rule-based filtering and security-defined tasks". 3.1 Diversion Path 1: Directly trigger known automated mechanisms (fast track) If the matching condition is a single alarm condition, it is determined to be a simple and deterministic operation and maintenance scenario. The scheduling decision unit will directly generate a handling instruction and call the corresponding automated script or platform native execution interface (such as Kubernetes HPA) according to the operation type defined in the rule to achieve rapid self-healing. Typical example: If the rule "the system service status of a certain node has been stopped for more than 3 minutes" is matched (operation type: restart service), the standard operation and maintenance automation interface of that node or region will be directly called to perform the restart. If the rule "Pod memory utilization > 80%" is matched (operation type: trigger elastic scaling), the API of Kubernetes HPA (horizontal Pod auto-scaling) or VPA (vertical Pod auto-scaling) will be directly called to perform scaling based on preset indicators, making full use of the efficiency and determinism of cloud-native infrastructure.

[0053] 3.2 Triage Path Two: Submit to LLM Agent for Intelligent Decision Making If the matching condition type is a multi-alarm aggregation condition, it is determined to be a complex operation and maintenance scenario requiring comprehensive analysis. The scheduling decision unit will encapsulate the detailed context information of all related alarms that trigger the preset rule, along with the explicit operation type defined in the rule, to generate a structured decision task package, which is then distributed to the Large Language Model Agent (LLM Agent) module for intelligent decision-making and parameter optimization. In this process, the rule base ensures that the scenario entering the intelligent decision-making stage is reasonable and expected through pre-approved matching conditions, and provides clear action boundaries through explicit operation types, fundamentally limiting the decision-making scope of the LLM Agent and effectively avoiding its "illusion" risk.

[0054] 3.3 Handling of Unmatched Scenarios: For alarm information that does not match any preset rules, the scheduling decision unit marks it as an "unmatched alarm", stores it in an independent pending queue, and can remind operation and maintenance personnel to manually intervene and analyze it through a notification mechanism to ensure that the system remains open to unknown or emerging fault scenarios and avoid omissions.

[0055] Through the implementation of this module, the system achieves intelligent and hierarchical processing of alarm flows. For example, when faced with a set of related alarms such as "Order Service - Instance A QPS surge, RT increase, CPU strain," the module can quickly identify it as a "high-load complex scenario" through aggregation conditions, extract the "execute service expansion" instruction from the rules, and then pass the scenario context to the LLM Agent for fine-grained calculation. For a single "disk space utilization alarm," it may directly trigger a preset log cleanup script. This mechanism, while ensuring operational security and process controllability, accurately directs AI computing power to the most challenging problems requiring complex cognition, while allowing simple problems to be solved through efficient and low-cost automated channels, thereby improving the overall efficiency, coverage, and intelligence level of operation and maintenance response.

[0056] (III) Large Language Model Intelligent Agent Module Combined with appendix Figure 3 This module, serving as the core of the system's intelligent decision-making, is closely connected to the rule scheduling module, establishing stable data interaction. Its core responsibility is to receive detailed contextual information of all relevant alarms corresponding to the triggering rules, as well as the types of operations to be executed as defined by preset rules, within the safety boundaries and clear task framework defined by the rule scheduling module. By simulating the cognitive reasoning process of human experts, it completes in-depth analysis of complex faults, parameter optimization, and forms the final executable decision.

[0057] 1. This module follows the principle of "rules define what to do, AI makes decisions on how to do it," specifically implemented through a collaborative workflow of four internal logical units. The input received by this module is a clearly structured decision task package encapsulated by the rule scheduling module, containing two main elements: 1) the type of operation to be performed (e.g., "service expansion" or "resource cleanup + slow SQL interception"); 2) detailed contextual information (structured data) of all relevant alarms that triggered the preset rule. The module then sequentially launches the four logical units for processing: 1.1 Natural Language Parsing Unit: Converts structured alarm contexts into semantic problem descriptions that are more conducive to understanding and reasoning by large language models.

[0058] 1.2 Reasoning and Planning Unit: Based on the type of operation to be performed and the semantic problem description, the key information required to achieve the decision is planned through a multi-step thinking chain.

[0059] 1.3 Tool Call and Data Acquisition Unit: Through the function call interface of the large language model, it dynamically calls the system's pre-built tool functions to obtain the latest and most accurate system status data from external systems in real time.

[0060] 1.4 Reasoning and Decision-Making Unit: Integrates all information, combines domain-specific knowledge, performs quantitative calculations and trade-offs, outputs specific and optimized execution parameters, and forms structured decision results.

[0061] 2. Specific functions and implementation examples of each unit The following two typical complex operation and maintenance scenarios illustrate the working details of each unit under different tasks: 2.1 Example of scaling up decisions for "high load on order services": 2.1.1 Natural Language Parsing Unit Input: Multiple structured alarm data points (such as QPS, response time, and CPU utilization exceeding thresholds) related to the "Order Service" from the rule scheduling module. Processing: Integrate the discrete field data into a coherent semantic description. Output (Semantic Problem Description): "Problem Scenario: Multiple instances of the microservice 'Order Service' simultaneously exhibit the following anomalies within a short period: QPS exceeds the threshold of 1000, average response time exceeds the threshold of 2000 milliseconds, and CPU utilization exceeds the threshold of 85%. Related Task: Scaling up this service is required to handle the high load." 2.1.2 Reasoning Programming Unit Input: Operation type "Service Expansion" and the semantic description above. Processing: Decompose the task through chain thinking and perform multi-step reasoning. Output (Decision Data Dimension): To make a reasonable expansion decision, the following needs to be obtained: 1) Current number of service replicas; 2) Current real-time QPS; 3) Single replica health carrying capacity (capacity baseline); 4) Cluster available resources.

[0062] 2.1.3 Tool Invocation and Data Acquisition Unit Processing: Based on the planning dimensions, call the preset tools in sequence.

[0063] Output (real-time data): (a) Calling the "Query current replica count" tool returns: 2.

[0064] (b) The "Query real-time monitoring metrics (qps)" tool was called and returned: 1200.

[0065] (c) Call the “Query Service Capacity Baseline” tool and it returns: The recommended QPS for a single replica is no more than 600 and the target CPU utilization is less than 70%.

[0066] (d) Call the "Check Cluster Resources" tool and return a status indicating sufficient resources.

[0067] 2.1.4 Reasoning and Decision-Making Unit Processing: Calculations were performed using a combination of real-time data and internal knowledge (Operational guidelines, historical experience, SOPs). Basic assessment: Only 2 replicas were needed to meet the QPS capacity requirement, but the CPU was severely overloaded (92%). Quantitative calculation: To reduce the CPU usage to the 70% target, approximately 3 replicas (ceil(2*(92 / 70)) ≈ 3) were needed. Comprehensive optimization: Based on historical experience and redundancy requirements, the final decision was to expand to 4 replicas.

[0068] Output (structured decision): { Operation: "Expand Capacity" "Target Service": "Order Service" "Current number of copies": 2, "Target number of replicas": 4 "Decision Basis": "Based on comprehensive real-time data (QPS: 1200, CPU: 92%), capacity baseline (600 QPS, target CPU <70%), and internal specifications, at least 3 replicas are required. Considering business redundancy and deployment practices, it was decided to expand to 4 replicas." } 2.2 Examples of complex operational decisions for "multi-dimensional anomalies in payment services": 2.2.1 Natural Language Parsing Unit Input: Multiple structured alerts (memory usage, slow SQL queries, response time exceeding thresholds) related to the "payment service" transmitted by the rule scheduling module. Processing: Integrate the discrete field data into a coherent semantic description. Output (semantic problem description): "Problem scenario: The payment service simultaneously exhibits multi-dimensional anomalies within 5 minutes, including 93% memory usage (exceeding the 90% threshold), 18% slow SQL queries (exceeding the 15% threshold), and a response time of 3.2 seconds (exceeding the 3-second threshold), suggesting potential resource shortages causing business delays. Related task: 'Resource cleanup + slow SQL interception' operations are required to restore service." 2.2.2 Reasoning Programming Unit Input: The compound operation type "resource cleanup + slow SQL interception" and the semantic description above. Processing: Decompose the task through chain thinking, perform multi-step reasoning, and analyze the key information required for the compound task. Output (decision data dimension): Required information includes: 1) Redundant resource details (e.g., cache, temporary files); 2) Specific slow SQL statements and frequency; 3) Current resource usage distribution of the service; 4) Valid parameters from similar historical scenarios.

[0069] 2.2.3 Tool Invocation and Data Acquisition Unit Processing: Based on the planning dimension, invoke the corresponding toolset.

[0070] Output (real-time data): (a) Calling the "Resource Query Tool" returns: Cache usage 20GB, temporary files 15GB.

[0071] (b) Call the "SQL Analysis Tool" and return: 3 high-frequency slow SQL statements and their details.

[0072] (c) Call the "Service Status Query Tool" and it returns: Connection count 800 / 1000, memory usage 93%.

[0073] (d) Call the "Historical Operation and Maintenance Data Query Tool" to return: the effect data after cleaning up specific resources and intercepting slow SQL in similar scenarios.

[0074] 2.2.4 Reasoning and Decision-Making Unit Processing: Decisions are made based on a combination of real-time data and internal knowledge (architecture information, operational guidelines, SOPs, historical events). Resource cleanup decisions: Based on the specification that "cache cleanup should retain hot data," it was decided to clean up 15GB of non-hot data cache; based on the rule that "temporary file cleanup should not affect core business," it was decided to clean up 15GB of temporary files in full. Slow SQL interception decisions: Based on the requirement of "prioritizing high-frequency, high-time-consuming statements," two slow SQL statements with a cumulative percentage of 15% and the longest execution time were selected from three slow SQL statements for interception, while one slow SQL statement related to a core process was retained for subsequent optimization.

[0075] Output (structured decision): {"Operation":["Resource Cleanup","Slow SQL Interception"], "Target Service": "Payment Service" "Resource cleanup parameters": {"Clean up non-hotspot cache": "15GB", "Clean up temporary files": "15GB"}, "Slow SQL interception parameters":{"List of intercepted SQL_IDs":["sql_001","sql_002"]}, "Decision Basis": "Based on real-time resource usage, SQL analysis results, the operational guidelines of 'retaining hot data' and 'prioritizing the interception of high-frequency slow SQL queries,' and historical processing experience, the above cleanup and interception plan was formulated." 3. Internal knowledge system The internal knowledge upon which the reasoning and decision-making unit relies is a continuously accumulating and updated comprehensive knowledge base, providing key context and constraints for decision-making, mainly including: (a) Static specifications and architecture knowledge: IT architecture topology, technology stack characteristics, written operation and maintenance management specifications, and standard operating procedures (SOP) manuals for the company's business systems.

[0076] (b) Historical and organizational information: historical records of handling various failures, root cause analysis and effect evaluation; information on the responsible persons / teams for each service and module; relevant development specifications and deployment agreements.

[0077] (c) Dynamic business and environment status: Real-time monitoring of the global health status view output by the system; records of recently executed or planned system changes; conclusions of periodic or special system inspection reports.

[0078] This module demonstrates its general intelligent decision-making capabilities in handling various complex operation and maintenance scenarios. Within a rule-based security framework, the module can transform structured alarms into understandable semantic questions, dynamically plan and acquire key data, and ultimately integrate enterprise-specific static and dynamic knowledge to output accurate, interpretable, and executable parameterized decisions. This provides a solid and reliable intelligent core for subsequently generating operation and maintenance work orders that can be directly used for manual approval and automated execution.

[0079] (iv) Work Order Generation Module This module, serving as a link between intelligent decision-making and operational execution processes, has the core responsibility of automatically converting the intelligent decision-making results, which include specific execution parameters, output by the large language model intelligent agent module into structured work orders that conform to the enterprise IT Service Management (ITSM) process standards and have standardized formats. Through the collaborative work of these two main units, this module ensures that intelligent decision-making can be seamlessly integrated into the enterprise's existing rigorous operational approval and execution processes, making it a crucial link in achieving a closed loop of "intelligent decision-making - manual approval - automated execution."

[0080] 1. Module Structure and Core Logic This module consists of two logical units that work together sequentially to automate and standardize the generation of work orders: Work Order Generation Unit: Responsible for automatically extracting and filling work order content based on decision results and predefined templates, realizing the conversion from structured decisions to standardized work order drafts. Format Verification Unit: Responsible for automatically verifying the generated work order drafts for compliance, ensuring they fully conform to the data interface specifications of downstream ITSM systems.

[0081] To automate and structure the generation of work order content and ensure seamless integration with downstream ITSM systems, this system employs a set of work order field templates predefined by the rule scheduling module and tightly bound to the rule base. This template serves as a crucial link between rule scheduling, LLM Agent intelligent decision-making, and ITSM processes. Its design adheres to the principle of "rule-defined framework, AI-filled details," ensuring that the generated work orders not only conform to enterprise process specifications but also possess high interpretability and executability.

[0082] 1.1 Binding relationship between templates and rule base The work order field template does not exist independently, but rather as a structured attribute of each rule in the rule base, strongly bound to the rule. When operations personnel define or modify a rule in the rule base, in addition to setting matching conditions, operation type, and priority, they also need to associate or define the corresponding work order field template. This design ensures: scenario-based adaptation, different types of operations operations (such as expansion, restart, and cleanup) correspond to different work order field templates, and the templates contain the specific fields necessary for that operation type (such as the "current replica count" and "target replica count" fields in the expansion template); Figure 1 Consistency: The "operation type" defined in the rules directly determines the tone and core fields of the work order, ensuring the consistency between the work order content and the operation and maintenance intent from the source; Security and controllability: The work order format and field range of all operations allowed to be initiated through the intelligent self-healing system are strictly controlled by predefined templates, avoiding format errors or missing information that may be caused by AI's free play.

[0083] 1.2 Each work order field template is a structured data object that explicitly defines the following elements: 1.2.1 Field List: Contains all fields required for this type of work order. Each field has: Field Name: such as "Title", "Operation Type", "Target Service", "Current Copy Count", "Decision Basis", etc. Data Type: String, Number, Date, List, etc. Constraints: Required, Length Limit, Value Range, Formatting Requirements (e.g., Regular Expressions). Source of Population: Clearly indicates which module should provide the value for this field.

[0084] The sources of data are divided into three categories: Basic information fields: directly taken from the rule scheduling module, such as operation intent information like "operation type," "target service," and "priority." Execution parameter fields: taken from the decision results of the LLM Agent module, such as specific execution parameters like "target replica count," "cleanup volume," and "intercepted SQL_ID list." Decision basis fields: an explanatory text automatically generated by the work order generation module based on the inference results and real-time data information, explaining the reasons, basis, and logic of the decision.

[0085] 1.2.2 Field Mapping Relationships: Defines which specific field in the work order field template each parameter in the structured decision results output by the LLM Agent should be mapped to. For example, "target_replicas" in the decision results is mapped to the "target number of replicas" field in the work order.

[0086] 1.2.3 Output Format Specification: Defines the final data format (such as JSON Schema, XML template) submitted to the ITSM system to ensure interface compatibility.

[0087] 1.3 Based on the above template design, the automated filling process of the work order generation module is as follows: 1.3.1 Template Triggering and Loading: After the decision is made, the create_ticket() utility function is called through Function Calling to automatically initiate a work order. The rule scheduling module will pass the template ID of the work order field bound to the preset rule to the work order generation module, and the work order generation unit will load the corresponding template according to the ID.

[0088] 1.3.2 Multi-Source Data Acquisition and Mapping: Basic Information Fields: Extracted and filled directly from the context passed by the rule scheduling module. Execution Parameter Fields: Parsing the structured decision results output by the LLM Agent module, and filling the corresponding fields with the parameter values ​​according to the mapping relationship defined in the template. Decision Basis Fields: Integrating the alarm context, the results of LLM inference, and the referenced real-time data, automatically generating a coherent and readable explanatory text according to the pre-defined narrative logic in the template, focusing on generating content with strong interpretability and decision support.

[0089] 1.3.3 Structured Validation and Correction: The format validation unit uses the loaded template as the standard to perform field-by-field validation on the completed work order draft. Validation not only checks data type and format but also performs simple business logic checks (such as ensuring the target number of copies is greater than the current number of copies). If validation fails, the system can attempt automatic correction based on template rules (such as formatting the date) or trigger an alarm for maintenance personnel to intervene.

[0090] 1.3.4 Standardized Output: Work orders that pass verification are serialized into standard messages (such as JSON conforming to ITIL standards) that can be received by downstream ITSM systems according to the output format specifications defined in the template, and marked as "pending approval".

[0091] 2. Automated filling logic of the work order generation unit The work order generation unit receives the structured decision results output by the large language model intelligent agent module as input, and calls the system's pre-defined work order field templates based on the rule scheduling module to automatically extract multi-source information and fill in the core fields of the work order.

[0092] The precise mapping and population logic for the core fields is as follows: 2.1 Basic Information Field Population: The operation intent information determined by the rule scheduling module during the initial traffic distribution and scheduling is directly adopted, including the operation type (such as "service expansion" or "resource cleanup + slow SQL interception"), target service (such as "order service"), and associated alarm indicators, to ensure that the work order is consistent with the original intent of the fault scenario and operation and maintenance decision. Figure 1 To.

[0093] 2.2 Execution Parameter Field Population: Extract the specific execution parameters calculated by the large language model intelligent agent module and populate them one by one according to the template field specifications. For example, for expansion operations, populate "current number of replicas" and "target number of replicas"; for compound operations, populate "resource cleanup amount", "list of intercepted slow SQL IDs", etc., to clarify the specific content and quantitative indicators of the operation execution.

[0094] 2.3 Decision Basis Field Population: Integrating the reasoning process of the large language model intelligent agent module with real-time query data, detailed and traceable decision basis text is generated. This part aims to provide transparent decision logic, typically including a description of the triggered fault scenario, fault analysis, the internal knowledge on which it is based (such as operation and maintenance specifications and historical experience), and the final reasoning conclusion.

[0095] 2.4 Additional Field Population: Automatically supplement auxiliary information such as work order creation time, scope of impact, and required approval roles to ensure the completeness of work order information and meet the requirements of ITSM process flow and management.

[0096] 3. Structured validation process for the format validation unit The format verification unit starts immediately after the work order generation unit fills in the content. It uses the same set of work order field templates predefined by the rule scheduling module as the standard to automatically verify the compliance of the work order draft.

[0097] 3.1 Validation basis: The field template clearly defines the name, data type (string, number, date, etc.), format requirements (such as time format, naming conventions), whether it is required, value range and all other constraints of each field.

[0098] 3.2 Verification process: 3.2.1 Field Integrity Validation: Check whether all fields marked as "required" (such as operation type, target service, core execution parameters) have been filled in without omissions or empty values.

[0099] 3.2.2 Data Type and Format Validation: Verify whether the field content conforms to the template definition. For example, check whether "Target Replica Count" is an integer, whether "Cleanup Quantity" is a valid value, and whether the "Time" field conforms to the predefined format.

[0100] 3.2.3 Business Logic Consistency Verification: Perform simple business logic consistency checks. For example, verify whether the "target number of replicas" is greater than the "current number of replicas" in a scaling scenario; and verify whether the "cleanup amount" does not exceed the reported "current occupancy" in a resource cleanup scenario.

[0101] 3.3 Verification Result Processing: 3.3.1 Verification passed: The draft work order is marked as "ready" and can be pushed to the downstream ITSM system by the approval execution module.

[0102] 3.3.2 Validation Failure: Generate a detailed error report (specifying which field and the reason for the validation failure) and trigger a correction process. For example, if the report states "Field 'Estimated Completion Time' format is incorrect," the system can automatically revert to the work order generation unit for correction or issue an alarm. This mechanism eliminates process interruptions caused by incorrect work order format or content, ensuring the smooth operation of the automated process.

[0103] Through the intelligent content filling of the work order generation unit and the compliance check of the format verification unit, this module transforms abstract intelligent decisions into a standardized work order that is detailed, formatted correctly, and ready for direct submission. The work order clearly demonstrates "what to do" (from the rules), "how to do it and how much" (from AI calculations), and "why do it this way" (from AI reasoning), providing approvers with highly transparent and persuasive decision support, greatly shortening the approval cycle, and serving as a key guarantee for the efficient and reliable implementation of the intelligent self-healing process.

[0104] (V) Approval and Execution Module This module serves as the closed-loop core and final execution control layer of the self-healing operation and maintenance process, connecting the work order generation module with downstream execution stages. Its core responsibility is to securely and reliably integrate standardized work orders into the enterprise IT Service Management (ITSM) process. After obtaining manual authorization, it automatically completes the execution of repair operations, status tracking, and resource reclamation through work order submission, operation execution, and result closure. This achieves end-to-end automation of "work order flow - manual approval - automatic execution - status reclamation," forming a complete operation and maintenance closed loop and ensuring the compliance, traceability, and efficient use of resources in operation and maintenance operations.

[0105] 1. Work Order Submission Unit The work order submission unit receives standardized work orders that have passed format validation from the work order generation module. This unit automatically pushes the work order to the downstream IT service management system via a standard interface, creating a formal process document (such as a change request work order) awaiting approval. When pushing the work order, the urgency level can be automatically set according to the priority defined by the rule scheduling module, and preset approval permission groups and notification mechanisms can be associated to initiate and optimize the approval process.

[0106] 2. Execution Engine Unit The execution engine unit monitors the approval status of work orders in the ITSM system. When the work order status changes to "Approved," the unit automatically triggers and performs the following operations: 2.1 Command parsing: Automatically parse the operation type (such as service_scale_out) and key execution parameters (such as target service name and target number of replicas) from the approved work orders.

[0107] 2.2 Interface Call and Execution: Based on the parsed operation type, the pre-defined execution interface that interfaces with the underlying infrastructure or operation and maintenance platform is called to complete the repair operation. The call process follows existing security authentication mechanisms. The execution process supports status monitoring and pre-defined retry strategies for failures.

[0108] 3. Results Recording and Closed-Loop Unit The results recording and closed-loop unit is used to achieve full traceability of the operation and adaptive recycling management of resources, including the following aspects: 3.1 Execution Log: Record the status (e.g., "in execution", "success", "failure"), time and result of operation execution in real time, and write them back to the original ITSM work order.

[0109] 3.2 Effect Verification and Closed-Loop Judgment: After the operation is completed, continuously monitor the health indicators of the target system (such as service CPU utilization and response time). When it is confirmed that the indicators have returned to the normal range and remain stable, the repair is deemed successful, and the closed-loop process is triggered.

[0110] 3.3 Resource Reclamation and Status Synchronization: For temporary operations (such as expansion during a fault), resource reclamation tasks (such as scaling down) are automatically generated and executed through the execution engine unit. After reclamation is completed, the ITSM work order status is updated to "closed loop," and the final resource configuration information is synchronized in the configuration management database. The valid parameters and results of this processing can be fed back as a new "historical operation and maintenance event" to the internal knowledge base of the large language model intelligent agent module for optimizing similar decisions in the future.

[0111] Through the collaboration of the three units in the approval and execution module, intelligent operation and maintenance (O&M) achieves fully automated management of the entire process from "decision-making" to "implementation" and "recycling." It not only integrates essential manual approval and control processes to ensure operational security and compliance, but also significantly improves O&M efficiency and reduces resource waste through automated execution, detailed recording, and intelligent resource recycling. Furthermore, it generates complete and auditable O&M process records, achieving an end-to-end closed loop of self-healing O&M.

[0112] The system demonstrated in this embodiment achieves intelligent operation and maintenance across the entire chain, from multi-source alarm perception, intelligent scene judgment, cognitive analysis and decision-making, integration of standardized processes, to automated execution and adaptive recovery, through the organic coordination and process-oriented connection of five major modules: data acquisition and preprocessing, rule scheduling, large language model intelligent agent, work order generation, and approval execution.

[0113] The core innovation of this system lies in the construction of a hybrid collaborative architecture of "rule scheduling framework + LLM Agent intelligent kernel": the rule scheduling module establishes security boundaries and task routing to ensure that the automated process is deterministic and controllable, efficiently handles known simple scenarios, and avoids the "illusion" risk of pure AI models; the LLM Agent module performs semantic understanding, multi-factor association and dynamic reasoning within the scope defined by the rules to achieve cognitive decision-making and parameter optimization in complex and unknown scenarios.

[0114] Through the above architecture, the system not only overcomes the shortcomings of traditional rule engines, such as rigidity and limited scenario coverage, but also solves the pain points of pure AI models, such as poor interpretability and difficulty in integrating into existing processes. Ultimately, the system generates standardized work orders with complete content and transparent logic, and seamlessly connects to the enterprise IT Service Management (ITSM) system through the approval and execution module. While respecting human approval and control, it achieves closed-loop, traceable, automated repair and resource recovery.

[0115] Therefore, this system not only improves the automation and intelligence of operation and maintenance response, but also provides comprehensive value in ensuring security, improving efficiency, reducing resource costs, and enhancing business continuity, providing a solid and feasible technical path for building a next-generation intelligent, reliable, and implementable operation and maintenance self-healing platform.

[0116] Example 2: Combined with appendix Figure 4 This embodiment provides an intelligent self-healing method for operation and maintenance based on an LLM Agent, applied to the system described in Embodiment 1. This method achieves intelligent perception, decision-making, and closed-loop repair of operation and maintenance faults through the collaboration of rule scheduling and the LLM Agent. The specific steps and processes of the method are illustrated below using two typical operation and maintenance scenarios.

[0117] 1. Overview of methods and steps: The method includes the following steps: S1: Collect alarm events from multi-source monitoring systems in real time, clean, normalize, and supplement the context to generate standardized alarm information and store it in the event queue.

[0118] S2: Match and distribute alarm information according to preset rules: S21. When a simple rule is matched, the automatic operation and maintenance mechanism is directly triggered to perform repairs. S22. When a complex aggregation rule is matched, the relevant alarm context and predefined operation type are submitted to LLMAgent.

[0119] S3: The LLM Agent receives tasks, parses alarm content, obtains real-time data through tool calls, performs reasoning based on internal knowledge, and outputs structured decision results.

[0120] S4: Based on the decision results and predefined work order field templates, automatically generate and verify work orders that comply with IT service management standards.

[0121] S5: Push the work order to the ITSM system for approval. Once approved, the repair operation is automatically executed, the results are recorded, and resource reclamation is triggered based on monitoring indicators to achieve a closed loop.

[0122] In this system, "simple rules" typically refer to rules that match a single alarm condition, while "complex aggregation rules" refer to rules that match multiple alarm aggregation conditions. The scheduling decision unit allocates traffic accordingly: for simple rules (single alarm condition), the automation mechanism is triggered directly; for complex aggregation rules (multiple alarm aggregation conditions), the decision is submitted to LLMAgent for decision-making.

[0123] 2. Scenario 1: Order service single metric anomaly (simple scenario). This scenario demonstrates rapid response to simple, identifiable faults without the need for LLM Agent intervention. The steps are as follows: (Data Acquisition and Standardization): The system collects the alarm event "Order Service CPU Utilization 92%" from the Nagios monitoring system by pulling data. After cleaning, mapping fields, and supplementing CMDB information, standardized alarm information is generated and stored in the queue.

[0124] (Rule Matching and Traffic Distribution): The rule scheduling module matches a simple rule with the condition "single alarm condition: CPU utilization > 85%" and the operation type is "trigger elastic scaling". The scheduling decision unit determines it as a simple scenario and directly calls the KubernetesHPA API to execute automatic scaling, and the process ends.

[0125] 3. Scenario 2: Multi-dimensional anomalies in payment services (complex scenario) (Data Acquisition and Standardization): The system collects alarms from the Zabbix and Prometheus monitoring systems, such as "Memory Usage 93%", "Slow SQL Percentage 18%" and "Response Time 3.2 seconds" for the payment service. After standardization, the data is stored in a queue.

[0126] (Rule Matching and Traffic Distribution): The rule scheduling module matches a complex rule that meets the "multi-alarm aggregation condition: the same service simultaneously triggers memory, slow SQL, and response time exceeding the threshold within 5 minutes" and whose operation type is "resource cleanup + slow SQL interception". The scheduling decision unit encapsulates the complete alarm context set that triggers the preset rule and the clear "resource cleanup + slow SQL interception" operation type into a decision task package and distributes it to the LLM Agent module.

[0127] (LLM Agent Intelligent Decision Making): Natural Language Parsing and Inference Planning: The LLM Agent converts alarm context into semantic description and plans the data dimensions required for decision-making: details of redundant resources, specific information on slow SQL queries, current service status, and historical processing parameters.

[0128] Tool usage and data acquisition: The preset tools were called in sequence and the query results showed that the cache occupied 20GB (of which about 15GB was non-hotspot), the temporary files were 15GB, and 3 high-frequency slow SQL queries and their IDs were identified.

[0129] Comprehensive reasoning and decision-making: Combining query data and internal knowledge (such as the "clean up and retain hot data" and "prioritize intercepting high-frequency slow SQL" specifications), the decision output is: clean up 15GB of non-hot data cache and 15GB of temporary files, and intercept 2 high-frequency and time-consuming slow SQL queries (ID: sql_001, sql_002).

[0130] (Automatic Work Order Generation): The work order generation module automatically fills in the fields based on the "Resource Cleanup and Interception" work order field template bound to the rules: basic information (operation type, target service) is taken from the rules; execution parameters (cleanup volume, SQL ID list) are taken from the S3 decision results; the decision basis field is automatically generated, integrating alarms, inference, and real-time data. After the format validation unit passes the validation, the work order is marked as "Pending Approval".

[0131] (Approval, Implementation, and Closed-Loop Recovery): The work order was automatically pushed to the ITSM system and approved.

[0132] The execution engine parses the work order instructions and parameters, calls the corresponding resource management and database interfaces, and performs cleanup and interception operations.

[0133] The results are recorded in both the result log and the closed-loop unit log, and payment service metrics are continuously monitored. When memory usage drops below 80% and the percentage of slow SQL queries falls below 5% and stabilizes, resource reclamation is automatically triggered (e.g., canceling slow SQL interception), the ITSM work order status is updated to "closed loop," and information is synchronized in the CMDB. The processing parameters and results are fed back to the knowledge base.

[0134] Through the above steps, this method can complete the process from alarm detection, intelligent analysis, generation of highly interpretable work orders, to initiation of automated repair within one minute when facing complex and interconnected faults such as "multi-dimensional anomalies in payment services." For simpler scenarios such as "order service single indicator exceeding threshold," it directly triggers more efficient native automation mechanisms (such as Kubernetes HPA) through rules to achieve rapid response. This method organically integrates the efficiency and security of rule scheduling with the cognitive flexibility of LLMAgent. Under the premise of fully integrating into the enterprise's existing ITSM approval process, it achieves intelligent, closed-loop, efficient, and auditable self-healing of operations and maintenance, significantly improving operation and maintenance response speed, decision-making quality, and resource utilization efficiency.

[0135] In summary, the core innovation of the intelligent self-healing system and method for operation and maintenance based on LLM Agent of the present invention lies in the construction of a hybrid collaborative architecture of "rule scheduling framework + LLM Agent intelligent kernel", which realizes the complementarity and risk isolation of the "security boundary and task definition" capability of rules and the "semantic understanding and complex reasoning" capability of AI.

[0136] The system efficiently integrates multi-source heterogeneous alarm data through dual-channel acquisition and standardized preprocessing. With the help of the rule base's single and multi-alarm aggregation and matching capabilities, it achieves intelligent aggregation and precise routing of alarm events. In simple scenarios, it directly triggers the automatic operation and maintenance mechanism to achieve rapid self-healing. In complex scenarios, the LLM Agent dynamically acquires real-time data within the security scope defined by the rules through multi-step thinking chain planning and tool calls, generating intelligent decisions with accurate parameters and traceable logic.

[0137] Based on this decision, the system automatically generates complete IT service management standard work orders, seamlessly integrating them into the enterprise's existing approval processes. After manual review, automated execution is triggered, and a closed-loop operation and maintenance mechanism is formed through continuous monitoring and intelligent recycling. This method effectively solves key problems such as the lack of flexibility of traditional rule engines, the difficulty of "black box" decision-making and process integration of pure AI models, and at the same time makes up for the shortcomings of automated operation and maintenance mechanisms in decision-making for complex business scenarios.

[0138] It significantly improves the automation coverage and decision-making accuracy in complex operation and maintenance scenarios; greatly enhances the interpretability and credibility of operations through the complete logical chain of work orders; achieves secure and seamless integration with the existing ITSM system; while significantly shortening the mean time to recovery (MTTR), it avoids resource waste and reduces operation and maintenance costs and human risks by dynamically optimizing parameters, providing enterprises with a feasible and highly practical technical path to build a secure, intelligent and efficient next-generation automated operation and maintenance system under complex IT architectures.

[0139] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A self-healing intelligent operation and maintenance system based on LLM Agent, characterized in that, It includes a data acquisition and preprocessing module, a rule scheduling module, a large language model intelligent agent module, a work order generation module, and an approval execution module, among which: The data acquisition and preprocessing module acquires multi-source heterogeneous data from the multi-source monitoring system in real time, preprocesses the multi-source heterogeneous data, and generates standardized alarm information. The multi-source heterogeneous data includes alarm events from various monitoring systems. The rule scheduling module is used to receive the standardized alarm information and match and distribute the alarm information according to preset rules. The large language model intelligent agent module is connected to the rule scheduling module and is used to receive detailed context information of alarm events that match preset rules and the type of operation to be performed, which are distributed by the rule scheduling module; parse the alarm content and perform reasoning to complete the decision. The work order generation module automatically generates work orders that conform to IT service management standards based on the decision results output by the large language model intelligent agent module. The approval and execution module pushes the generated work order to the ITSM system for manual approval. Once approved, the repair operation is automatically executed, the execution result is recorded, and a closed-loop recovery is performed.

2. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 1, characterized in that, The data acquisition and preprocessing module adopts a dual-channel acquisition mechanism, which acquires alarm events from multi-source monitoring systems through push notifications or pull acquisition methods. Through data cleaning and normalization preprocessing, the multi-source heterogeneous data is converted into unified structured data and stored in a unified event queue to generate standardized alarm information. The pull acquisition method is a polling method, and the monitoring system can be one or more different types of monitoring systems.

3. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 1, characterized in that, The rule scheduling module includes a rule matching unit and a scheduling decision unit: The rule matching unit periodically scans the standardized alarm information generated by the data acquisition and preprocessing module at preset time intervals, and matches the standardized alarm information with different preset rules in the rule base; The scheduling decision unit performs traffic distribution and scheduling based on the matching results, directly triggering the automatic operation and maintenance mechanism, or distributing detailed context information of all relevant alarms corresponding to the preset rule, as well as the operation types to be executed as defined by the preset rule, to the large language model intelligent agent module.

4. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 3, characterized in that, The preset rules are stored in a structured format in a rule base, including: Matching conditions support single alarm conditions and multi-alarm aggregation conditions. The aggregation conditions include multi-alarm association matching within a time window. Operation type: The type of operation and maintenance operation to be performed, including one or more of the following: node restart, service expansion, resource cleanup, rate limiting, and slow SQL interception; Priority, dynamically configured based on business needs, determines the order in which rules are triggered.

5. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 4, characterized in that, The matching results output by the rule matching unit are structured objects that support logging and persistent storage, including: Matching rule identifier, used to indicate a preset rule that has been successfully matched; The matching condition type specifies whether the matching is based on a single alarm condition or a multi-alarm aggregation condition in the rule base. The associated alarm context set contains standardized information about all relevant alarms that triggered this preset rule; Predefined operation instructions, including the operation type and priority defined by the preset rules.

6. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 3, characterized in that, The large language model intelligent agent module includes a natural language parsing unit, an inference planning unit, a tool invocation and data acquisition unit, and an inference decision-making unit, among which: The natural language parsing unit receives structured alarm context information from the rule scheduling module and converts it into a semantic problem description that the reasoning planning unit can directly understand and process. The reasoning and planning unit receives the type of operation to be performed as determined by the rule scheduling module, as well as the semantic problem description generated by the natural language parsing unit. Through a multi-step thinking chain, it analyzes and determines the decision data dimensions required to perform the operation. The tool call and data acquisition unit, based on the required decision data dimensions derived by the reasoning and planning unit, acquires real-time system status data related to operation execution by calling the system's pre-set toolset; The reasoning and decision-making unit, based on the acquired real-time system status data and combined with internal knowledge calculations, arrives at a decision result.

7. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 6, characterized in that, The internal knowledge includes: the IT architecture, technology stack, operation and maintenance specifications, and standardized operation execution manuals of the company's business systems; historical operation and maintenance events, responsible person information, development specifications, and dynamic business status information.

8. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 1, characterized in that, The work order generation module includes a work order generation unit and a format verification unit. The work order generation unit, based on the work order field templates predefined by the rule scheduling module, automatically initiates work orders by calling tool functions and according to the decision results output by the big language model intelligent agent module. Among them: the basic information field is directly filled with the operation intention information determined by the rule scheduling module; the execution parameter field is filled with the specific execution parameters calculated by the big language model intelligent agent module; and the decision basis field is generated based on the reasoning process and real-time query data of the big language model intelligent agent module. The format validation unit performs structured validation on the filled content using the work order field templates predefined by the rule scheduling module, ensuring that the work order format conforms to the interface specifications of the downstream ITSM system.

9. The intelligent self-healing system for operation and maintenance based on LLM Agent according to claim 1, characterized in that, The approval execution module includes a work order submission unit, an execution engine unit, and a result recording and closure unit. The work order submission unit pushes the generated work orders to the IT service management system for manual approval. The work order content includes alarm context, reasoning logic, and real-time data. The execution engine unit automatically parses and captures the operation instructions and key parameters in the work order after the work order is approved, and calls the corresponding execution interface to complete the operation. The results are recorded and the closed-loop unit records the execution status and results in real time. After the system returns to normal, it automatically triggers resource reclamation operations based on preset conditions or monitoring indicators, updates the work order status and synchronizes it to the configuration management database of IT service management, forming a complete operation and maintenance closed loop.

10. A self-healing method for operation and maintenance based on an LLM Agent, applied to the self-healing system for operation and maintenance based on an LLM Agent as described in any one of claims 1-9, characterized in that, Includes the following steps: S1: Real-time acquisition of multi-source heterogeneous data from multi-source monitoring systems, preprocessing of the multi-source heterogeneous data and generation of standardized alarm information, where multi-source heterogeneous data includes alarm events from various monitoring systems; S2: Receive the standardized alarm information and match and distribute the alarm information according to preset rules; S3: Receive detailed context information of alarm events that match preset rules and the type of operation to be performed, distributed by the rule scheduling module; parse the alarm content and perform reasoning to complete the decision. S4: Based on the decision results output by the large language model intelligent agent module, automatically generate work orders that conform to IT service management standards; S5: Push the generated work order to the ITSM system for manual approval. Once approved, automatically execute the repair operation, record the execution results, and perform closed-loop recycling.

11. The intelligent self-healing method for operation and maintenance based on LLM Agent according to claim 10, characterized in that, Receiving the standardized alarm information and matching and routing the alarm information according to preset rules includes: S21: When a simple rule is matched, the automatic operation and maintenance mechanism is triggered directly; S22: When a complex aggregation rule is matched, the detailed context information of all relevant alarms corresponding to the preset rule, as well as the operation type to be performed as defined by the preset rule, will be distributed to the large language model agent module.

Images