An operation and maintenance root cause analysis method based on a large model intelligent agent and a fault tree guide

Abstract

The application discloses an operation and maintenance root cause analysis method based on a large model intelligent agent and a fault tree guide, and belongs to the technical field of intelligent operation and maintenance. The method comprises the following steps: in response to a fault event, obtaining multi-modal operation and maintenance data such as logs, indexes, links and codes; constructing a multi-level fault tree reasoning framework, taking the fault event as a top event, decomposing the fault event into intermediate events and bottom events layer by layer from top to bottom, and determining the fault propagation path and the logic gate relationship; based on the large model intelligent agent, taking the fault tree as a reasoning path constraint, iteratively executing an 'action-feedback-reasoning' cycle, querying multi-modal data by calling a preset tool, and gradually verifying fault nodes; aggregating the evidence of multi-step reasoning, calculating the confidence of each candidate root cause, locating the final root cause entity, and outputting a natural language analysis report containing a complete reasoning process. The application combines domain knowledge (fault tree) with large model reasoning capability, realizes multi-modal data fusion, controllable reasoning and fine-grained root cause positioning, significantly improves the accuracy and efficiency of fault analysis in complex operation and maintenance scenarios, and reduces the dependence on manual work.

Description

Technical Field

[0001] This invention belongs to the field of intelligent operation and maintenance technology, specifically involving an operation and maintenance root cause analysis method guided by a large model intelligent agent and fault tree. Background Technology

[0002] In recent years, with the popularization of cloud computing and microservice architecture, the complexity of modern operation and maintenance systems has increased exponentially, and the coupling between system components has significantly improved. A failure is often caused by the synergy of multiple factors, and there is a non-linear, cross-level complex relationship between the failure phenomenon and the root cause, which greatly increases the difficulty of failure localization. Root Cause Analysis (RCA), as a core task to ensure high system availability, aims to accurately locate the root cause of failures from massive amounts of multi-source data such as alarms, logs, and monitoring metrics, providing support for rapid failure repair.

[0003] Traditional root cause analysis methods heavily rely on the professional experience of operations and maintenance (O&M) personnel. The process typically involves O&M personnel manually collecting multi-source O&M data, speculating on possible root causes based on their expert knowledge, and then verifying and investigating each possibility. This approach is cumbersome, time-consuming, and limited by human experience, making it difficult to handle complex and dynamic system environments. Especially in large-scale distributed systems, it is prone to root cause localization errors and missed cases, failing to meet the high availability requirements of the system.

[0004] In recent years, machine learning technology has been gradually introduced into the field of root cause analysis, leading to automated analysis methods based on causal inference and association rule mining, attempting to reduce reliance on manual intervention. However, these methods generally have significant drawbacks: on the one hand, they require a large amount of labeled data for model training, resulting in high data acquisition costs and long cycles; on the other hand, they struggle to effectively process semantic information in unstructured logs, fail to capture hidden fault clues in log text, and have limited applicability.

[0005] In the prior art, patent CN115022153B proposes a root cause analysis method based on knowledge graphs, which reduces analysis complexity by simplifying alarm events. However, the knowledge graph construction of this method relies heavily on manual definition by experts, making it difficult to cover unknown fault modes and understand the implicit semantics in logs, resulting in poor generalization ability. Patent CN118981389A attempts to use a large language model for root cause analysis, but its scheme focuses on resampling mechanisms and lacks effective guidance for the reasoning process. Large models are prone to "reasoning illusions" when lacking constraints, and cannot combine knowledge in areas such as system architecture and fault propagation paths for controllable reasoning, leading to insufficient accuracy and interpretability of the analysis results.

[0006] Furthermore, existing root cause analysis solutions typically handle different modalities of operational data in a fragmented manner: log analysis tools focus only on textual data, monitoring tools focus only on time-series metrics, and tracing tools focus only on service call relationship data. However, in real-world operational scenarios, faults often manifest as abnormal linkages across multiple modalities. Analysis of single-modal data cannot capture cross-modal causal relationships. For example, a sudden spike in CPU performance may highly overlap with the timestamp of a certain error log, and a single tool cannot establish this correlation, leading to inaccurate root cause localization.

[0007] In summary, existing root cause analysis methods suffer from low automation, weak multimodal data fusion capabilities, uncontrollable reasoning in large models, poor interpretability, and difficulty in locating fine-grained root causes, failing to meet the demands for efficient and accurate root cause localization in complex operational scenarios. Therefore, there is an urgent need for a root cause analysis method that can fuse multimodal data, possess controllable reasoning capabilities, offer strong interpretability, and provide accurate root cause localization. Summary of the Invention

[0008] This invention aims to solve the aforementioned technical problems of existing root cause analysis methods and provides an operation and maintenance root cause analysis method based on a large model intelligent agent and fault tree guidance. This method achieves effective fusion of multimodal data and controllable reasoning of the large model, improves the accuracy, efficiency and interpretability of root cause localization, and enables fine-grained root cause localization, providing precise guidance for fault repair.

[0009] To achieve the above-mentioned objectives, this invention provides a root cause analysis method for operations and maintenance based on large-scale intelligent agents and fault trees, specifically including the following steps: S1. Acquisition of multimodal fault data: In response to a received fault event, the system automatically retrieves multimodal data associated with the fault event from the operations and maintenance data platform. The multimodal data includes at least two or more of the following: log data, monitoring indicator data, link tracing data, and code coverage reports, to achieve comprehensive collection of fault clues from multiple sources.

[0010] S2. Dynamic construction and instantiation of fault trees: First, a general fault tree template is constructed based on a preset fault classification system. Second, based on the preliminary description of the fault event (including but not limited to fault type, affected services, and fault occurrence time), the general fault tree template is instantiated into a specific fault tree that matches the current fault event. The fault tree uses the fault event as the top event, system components or fault levels as intermediate events, and specific root cause entities as bottom events. Logic gates (including AND gates and OR gates) are set for the intermediate events to clarify the fault propagation path and hierarchical relationship.

[0011] S3, Multi-step inference for large-scale intelligent agents: The instantiated fault tree is used as a reasoning path constraint and input into the large model agent. The large model agent performs multi-step analysis in a top-down fault tree node order using an iterative loop of "action-feedback-reasoning" until it reaches the leaf node of the fault tree (i.e., the candidate root cause entity). Each reasoning step specifically includes: Action generation: The large model agent generates action instructions to be executed based on the current fault tree node to be analyzed and historical reasoning records. The action instructions mainly include instructions to call preset tools to query specific multimodal data. Tool execution: Invoke the corresponding operation and maintenance tools according to the action instructions, perform query operations, and obtain structured or unstructured tool feedback results (i.e., fault-related data). History Update: The action instructions executed this time, the tool feedback results, and the reasoning logic judgments explicitly recorded using the mind chain technology are spliced ​​into the reasoning history as the context for the next round of reasoning, ensuring the coherence and traceability of the reasoning process.

[0012] S4. Evidence aggregation and root cause determination: Aggregate the fault evidence obtained from all reasoning steps, combine the logical gate relationships of the fault tree (including but not limited to AND gates and OR gates), and calculate the probability or confidence of each bottom event (candidate root cause entity) as the root cause of the fault using a preset algorithm (such as weighted voting or Bayesian network algorithm); finally output one or more root cause entities with the highest confidence, and generate a natural language analysis report containing the complete reasoning process, supporting evidence, root cause conclusions and repair suggestions.

[0013] Preferably, in addition to log data, monitoring indicator data, link tracing data, and code coverage reports, the multimodal data may also include API response data, configuration change records, network packet capture data, server hardware status data, etc., to further enrich the data sources for fault analysis and improve the comprehensiveness of root cause localization.

[0014] Preferably, the preset fault classification system adopts a four-layer structure, specifically a performance layer, a service layer, a database layer, and a code layer. This classification system is consistent with the classification system of the AnoMod dataset, ensuring that the fault tree can cover the entire chain of fault scenarios from infrastructure to code implementation, thereby improving the universality and applicability of the fault tree.

[0015] Preferably, the large model agent is built based on a fine-tuned large language model. The fine-tuning data uses publicly available root cause analysis datasets (such as LEMMA-RCA and AnoMod). Through fine-tuning, the model fully grasps the log characteristics, indicator characteristics, and fault propagation patterns of common failure modes (including but not limited to memory overflow, high CPU load, DDoS attack, database connection pool exhaustion, code logic errors, etc.), thereby improving the model's inference accuracy and generalization ability.

[0016] Preferably, the preset toolset includes, but is not limited to: network connectivity detection tools (such as Ping tools), log query tools, indicator query tools, link tracing analysis tools, code coverage analysis tools, configuration query tools, server status query tools, etc., and the types of tools can be flexibly expanded according to the actual operation and maintenance scenarios.

[0017] Preferably, the reasoning loop adopts the chain-of-thought technique, which explicitly records the reasoning process and logical judgment of each step in the reasoning history. This not only constrains the reasoning direction of the large model, but also significantly enhances the interpretability of the root cause analysis process, making it easier for maintenance personnel to verify and understand the reasoning logic.

[0018] Preferably, the confidence level of the underlying event can be calculated using a weighted voting method or a Bayesian network algorithm, and probability propagation can be performed based on the logical gate relationships of the fault tree to ensure the rationality and accuracy of the confidence level calculation and provide a reliable basis for root cause determination.

[0019] Compared with the prior art, the beneficial effects of the present invention are: (1) By actively calling multi-source operation and maintenance tools through large model intelligent agents, it can automatically acquire and associate multi-modal data such as logs, indicators, links, and code, establish cross-modal causal relationships, overcome the defects of existing technologies in processing multi-modal data in a fragmented manner, capture fault clues that cannot be found by single-modal analysis, and improve the comprehensiveness and accuracy of root cause localization.

[0020] (2) By introducing fault trees as a reasoning framework, knowledge in areas such as system architecture and fault propagation modes is embedded into the analysis process, which effectively constrains the exploration space of large models, guides large models to reason along a clear path, avoids aimless guessing and reasoning "illusion", and significantly improves the accuracy and interpretability of the analysis results.

[0021] (3) The large model fine-tuned based on the public dataset can generalize to the unseen fault mode, and the fault tree template supports flexible expansion. It can be customized according to the operation and maintenance needs of different industries and systems, and adapt to various complex operation and maintenance scenarios. Attached Figure Description Figure 1 This is a schematic diagram of the intelligent agent framework in this invention; Figure 2 This is a schematic diagram of the root cause analysis process of the present invention; Figure 3 This is an example diagram of the fault tree breakdown of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1: This embodiment uses a typical failure case from the AnoMod dataset as an example to illustrate in detail the complete process of an operation and maintenance root cause analysis method based on a large model intelligent agent and fault tree provided by this invention. The AnoMod dataset is a multimodal benchmark dataset for anomaly detection and root cause analysis of microservice systems. It is based on real microservice architectures (such as TrainTicket) and injects real failures. It integrates five modalities: logs, metrics, tracing, code coverage reports, and API response data, providing full-chain data support for root cause analysis.

[0024] S1. Acquisition of multimodal fault data: First, the operations and maintenance system responds to a fault event (e.g., a user reports "service order failed"). The method of this invention automatically pulls multimodal data related to this fault event from the data platform, including: Log data: Application logs and system logs related to the "order placement service", including error stacks, exception information, etc.

[0025] Monitoring metrics data: Time-series data such as CPU utilization, memory usage, request latency, and throughput corresponding to the time period of the failure.

[0026] Link tracing data: The complete call chain of the "Order Placement Service" is displayed, showing the call relationships and time consumption between various microservices.

[0027] Code coverage report: When a failure occurs, the code coverage report of the relevant microservice shows the execution status of each part of the source code.

[0028] These data together constitute the multi-source clues for subsequent root cause analysis.

[0029] S2. Dynamic construction and instantiation of fault trees: This invention first constructs a general fault tree template based on a pre-defined multi-level fault classification system. This fault classification system, referencing the design of the AnoMod dataset, is divided into four layers: performance layer, service layer, database layer, and code layer. For example, a general fault tree template for "service unavailable" has "service unavailable" as the top event, and intermediate events can be decomposed into "service itself is abnormal," "dependent services are abnormal," or "infrastructure is abnormal," etc.

[0030] Subsequently, based on the preliminary fault description obtained from S1 ("order service failed"), the general template is instantiated into a specific fault tree, such as... Figure 3 As shown. In this example: Top event (T): Service order failed.

[0031] Intermediate event (M1): The order placement service itself encountered an error.

[0032] Intermediate event (M2): Database access error occurs as the order placement service depends on it.

[0033] Bottom event (B1): Database connection pool exhausted.

[0034] Bottom event (B2): Database deadlock.

[0035] B3: Null pointer exception (NPE) exists in the code.

[0036] This fault tree clarifies the possible propagation paths of "order service failure", providing a precise navigation path for subsequent large model inference.

[0037] S3, Multi-step inference for large-scale intelligent agents: The fault tree instantiated in S2 is used as the inference path constraint and input into the large model agent. The large model agent initiates an iterative loop of "action-feedback-inference" according to the top-down node order.

[0038] First round of reasoning: Action generation: The large model agent reads the current node to be analyzed—the top event "Order placement service failed" and its child nodes (M1 and M2). Combining the fault description and historical records, the agent generates the first action instruction: "Invoke tool: Link analysis tool, parameter: query 'Order placement service' call chain". Tool Execution: The agent invokes the call chain analysis tool to obtain feedback results. The results show that the time taken from "order placement service" to "order database" in the call chain has increased abnormally, and ultimately timed out and failed.

[0039] History Update: The agent records this action and feedback in the inference history and uses the thought chain technology to record the inference logic: "The link analysis results show that the database call is the bottleneck, so the failure may be related to the database access anomaly (M2)", and determines that the next node to be analyzed is M2.

[0040] Second round of reasoning: Action generation: The agent's current node is M2 (database access error). Combining the underlying events B1 (connection pool exhaustion) and B2 (database deadlock) under M2, the agent generates a new action: "Call tool: indicator query tool, parameters: query the number of active connections in the connection pool of the order database and the length of the waiting queue; at the same time, call tool: log query tool, parameters: query the error log of the order database, with the keyword 'deadlock'." Tool execution: The metrics query tool shows that the number of active connections in the database connection pool has reached its maximum, and the waiting queue length continues to increase. The log query tool shows that no "deadlock" related records were found in the database logs.

[0041] History Update: The agent updates the current action and feedback to the history and infers that: "There are no deadlock records in the database log, but the connection pool status is abnormal, and it is highly suspected that the failure is caused by the database connection pool exhaustion (B1)." At this point, the inference has reached the leaf node B1 of the fault tree.

[0042] Third round of reasoning: Action generation: To further verify whether B1 (connection pool exhaustion) is the root cause, the agent generates the action: "Invoke tool: code coverage analysis tool, parameter: check the coverage of database connection acquisition and release related code blocks in the order placement service during execution." Tool execution: The code coverage report shows that when the failure occurred, the code segment related to database connection acquisition was executed very frequently, but the code segment for connection release had an abnormally low execution coverage.

[0043] Historical Update: The agent ultimately concluded that "the root cause of the failure was 'database connection pool exhaustion,' and the direct cause was the failure to properly release database connections in the code, leading to the exhaustion of connection pool resources." S4. Evidence aggregation and root cause determination: After all reasoning steps are completed, the evidence aggregation module of this invention begins to function. It gathers the evidence obtained from all the aforementioned reasoning steps: Evidence 1: Link tracing shows that the database call timed out.

[0044] Evidence 2: Indicator data shows that the number of active connections in the connection pool has reached its peak.

[0045] Evidence 3: Log data shows no deadlock.

[0046] Evidence 4: The code coverage report shows that the connection release code was not executed.

[0047] Based on the AND / OR logic of the fault tree (in this example, M2 is caused by either B1 or B2, and the evidence excludes B2, strongly pointing to B1), the aggregation module calculates the confidence level of each underlying event: B1 (database connection pool exhaustion) has a confidence level of 95%, B2 (database deadlock) has a confidence level of 5%, and B3 (code null pointer) has a confidence level of 0%. Ultimately, the system determines B1 (database connection pool exhaustion) as the final root cause of this fault and automatically generates a complete reasoning process (e.g., ...). Figure 2 The flowchart steps shown are presented in the Natural Language Analysis report, along with supporting evidence for each step, the final root cause conclusion, and remedial recommendations (e.g., "Check and fix the code block in the order placement service where the database connection is not properly released").

[0048] Through the above process, this embodiment successfully demonstrates how to combine the domain knowledge guidance of fault trees with the reasoning capabilities of large models to accurately, efficiently, and interpretably locate the root cause of a fault in the complex multimodal data environment provided by the AnoMod dataset. This solves the problems of difficulty in multimodal data fusion, the tendency of large model reasoning to produce "illusions," and the lack of precision in root cause localization in existing technologies.

[0049] Example 2: One of the key innovations of this invention lies in the design of structured prompt words. Through reasonable prompt word constraints, it ensures that large-scale intelligent agents can strictly follow fault tree guidance for controllable reasoning, avoiding deviations from the intended direction. The prompt words are divided into two types: initial prompt words and iterative prompt words. Specific templates are as follows: Initial prompt template Iteration prompt word template With the structured prompts mentioned above, the large model can clearly identify the current analysis's position in the fault tree, available tools, and inference rules. It can also use thought chains for explicit reasoning, avoiding aimless exploration and ensuring the controllability and interpretability of the reasoning process. Figure 2 It demonstrates the interaction process between large model agents and multimodal data sources, clearly presenting the closed-loop process of "prompt input - action generation - tool invocation - feedback reception - inference update".

[0050] The above description is only for the purpose of helping to understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, any equivalent substitutions or modifications made to the technical solution and inventive concept disclosed in the present invention within the scope of the technology disclosed in the present invention should be covered within the scope of protection of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A root cause analysis method for operation and maintenance based on large-scale intelligent agents and fault trees, characterized in that, include: In response to a fault event, multimodal operation and maintenance data is acquired, which includes at least two of the following: log data, indicator data, link tracing data, and code data. A fault tree reasoning framework is constructed. Based on the type of the fault event, a general fault tree template is constructed according to a preset multi-level fault classification system, and the template is instantiated into a specific fault tree. The fault tree contains at least two levels from top to bottom, with its leaf nodes representing candidate root cause entities and intermediate nodes representing fault propagation paths. Based on the large model agent, multi-step reasoning is performed, and the fault tree is used as a constraint for the reasoning path. The large model agent is driven to iteratively execute the "action-feedback-reasoning" loop along the node sequence of the fault tree. The action includes calling a preset tool to query the multimodal operation and maintenance data. By aggregating evidence from multi-step reasoning, the root cause entity corresponding to the failure event is located, and a root cause analysis report containing the reasoning process is generated.

2. The method according to claim 1, characterized in that, The multimodal operation and maintenance data includes: log event text, monitoring indicator time-series data, microservice call chain tracing data, and code coverage reports.

3. The method according to claim 1, characterized in that, The fault tree reasoning framework includes: A general fault tree template is constructed based on a preset multi-level fault classification system, which includes a performance layer, a service layer, a database layer, and a code layer. Based on the preliminary description of the fault event, the general fault tree template is instantiated into a specific fault tree, where the top event is the fault event, the intermediate events correspond to the anomalies at each level, and the bottom events correspond to the specific root cause entities.

4. The method according to claim 1, characterized in that, The multi-step inference performed by the large-model-based intelligent agent includes: The fault tree node to be analyzed, fault description, fault clues, descriptions of available tools, and historical reasoning records are concatenated into prompt words and input into the large language model. Parse the action instructions output by the large language model, which include the name of the tool to be invoked and its parameters; The tool is invoked to execute the command and retrieve feedback results from the corresponding multimodal data source. The feedback results are updated to the reasoning history, and the logical judgments of the reasoning process are explicitly recorded using the thinking chain technology to determine the next node to be analyzed and enter the next round of reasoning.

5. The method according to claim 4, characterized in that, The large language model is a fine-tuned large language model. The fine-tuning data is constructed based on the LEMMA-RCA dataset and / or the AnoMod dataset. The fine-tuning is used to enable the model to learn the ability to identify preset failure modes from log events and monitoring metrics.

6. The method according to claim 1, characterized in that, The evidence for the aggregated multi-step reasoning includes: fusing the evidence obtained in each step according to the logical gate relationship of the fault tree, calculating the confidence level of each base event as the root cause, and selecting the base event with the highest confidence level as the final root cause.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images