A fault root cause positioning method and system based on double-agent cooperation

Abstract

The application discloses a fault root cause positioning method and system based on double-agent cooperation, and relates to the technical field of intelligent operation and maintenance. The method determines a data collection time window according to the occurrence time of a target fault, acquires multi-modal operation and maintenance data in the window, and generates a multi-modal evidence report; inputs the multi-modal evidence report into a diagnosis agent to generate a candidate root cause list including multiple candidate root causes and diagnosis scores thereof; inputs the candidate root cause list into a verification agent to perform historical case matching and rule checking on each candidate root cause to obtain historical case matching scores and rule checking scores; determines a comprehensive credibility score according to the diagnosis scores, the historical case matching scores and the rule checking scores, and determines a target root cause according to the comprehensive credibility score to output a root cause positioning result. The method can reduce the risk of large model hallucination and false attribution, and improve the reliability of the root cause positioning result.

Description

Technical Field

[0001] This application relates to the field of intelligent operation and maintenance technology, and in particular to a fault root cause localization method and system based on dual-agent collaboration. Background Technology

[0002] With the widespread adoption of microservice architectures and cloud-native systems, the number of services, call chains, and operational status data in target systems are rapidly increasing. Root cause analysis of faults typically requires comprehensive analysis of multi-source operational information, including metric data, log data, and call chain data. In recent years, related technologies have begun to incorporate large language models, enhanced retrieval generation, thought chain reasoning, or intelligent agent frameworks for root cause analysis. Large language models perform semantic understanding and reasoning on alarm information, monitoring metrics, log content, and historical cases to generate possible causes of faults. These methods improve the automation of fault analysis to some extent, assisting operations personnel in quickly identifying candidate root causes from complex operational data.

[0003] Existing root cause localization methods based on large language models typically focus on directly understanding the fault phenomenon and generating root cause conclusions through the large language model. The diagnosis process and the result output process are often completed by the same model or the same inference chain, lacking a verification mechanism independent of the diagnosis process. This makes it difficult to verify in a timely manner whether the candidate root causes generated by the model are supported by historical fault cases and whether they conform to the causal logic, temporal relationship and component dependency relationship in the operation and maintenance scenario. Therefore, when the input evidence is incomplete, multimodal operation and maintenance data contains noise interference, or the fault propagation chain is complex, existing methods are prone to generating large model illusions or incorrect attributions without supporting evidence, resulting in insufficient reliability of the final root cause localization results. Summary of the Invention

[0004] This application provides a method and system for fault root cause localization based on dual-agent collaboration. The aim is to generate candidate root causes by a diagnostic agent and to perform historical case matching and rule verification on the candidate root causes by a verification agent. This ensures that the root cause localization results are independently verified before output, thereby reducing the risk of large model illusion and false attribution and improving the reliability of root cause localization results.

[0005] To achieve the above objectives, the first aspect of this application provides a fault root cause localization method based on dual-agent cooperation, comprising: Obtain the time of failure of the target system at the time of the target fault, and determine the data acquisition time window; Acquire multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data; The multimodal evidence report is input into the diagnostic agent to generate a candidate root cause list, wherein the candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause. The candidate root cause list is input into the verification agent, so that the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause. Based on the diagnostic score, historical case matching score, and rule verification score corresponding to each candidate root cause, a comprehensive credibility score is determined for each candidate root cause. Based on the comprehensive credibility score, the target root cause is determined from the candidate root cause list, and the root cause localization result corresponding to the target fault is output.

[0006] By dividing the root cause localization process into steps such as multimodal evidence report generation, diagnostic agent candidate root cause generation, verification agent historical case matching and rule verification, and comprehensive credibility scoring decision, compared with the existing technology that directly relies on a large model to output root cause conclusions in a single inference, this application enables candidate root causes to undergo independent verification and comprehensive scoring before output, thereby reducing the risk of large model illusion and false attribution, and improving the reliability of root cause localization results.

[0007] Further, a multimodal evidence report corresponding to the target fault is generated based on the multimodal operation and maintenance data, including: The multimodal operation and maintenance data includes indicator data, log data, and call chain data; Anomalies are identified in the indicator data, the log data, and the call chain data respectively to obtain indicator anomaly events, log error events, and call chain anomaly events. The abnormal events, log error events, and call chain abnormal events are sorted according to their occurrence time to obtain an abnormal event list. Based on the list of abnormal events, a multimodal evidence report corresponding to the target fault is generated.

[0008] By acquiring indicator data, log data, and call chain data within the data acquisition time window, and performing anomaly identification, time sorting, initial evidence report generation, and noise reduction screening on different types of data, this application can reduce the interference of irrelevant noise on the diagnostic process and retain the temporal correlation between abnormal events, thereby improving the correlation between multimodal evidence reports and target faults, compared to the existing technology of simply splicing or directly inputting multi-source operation and maintenance data into the model.

[0009] Furthermore, anomaly identification is performed on the indicator data, the log data, and the call chain data respectively to obtain indicator anomaly events, log error events, and call chain anomaly events, including: The abnormal event of the indicator is determined based on the degree of deviation of the indicator value in the indicator data from the preset indicator benchmark value; The log error event is determined based on the occurrence of the log level and log content template in the log data within the data collection time window; The call chain exception event is determined based on the call status in the call chain data and the degree of deviation of the call delay in the call chain data from the preset call baseline value.

[0010] By determining abnormal events in metrics, log errors, and call chains based on the degree of deviation of metric values, log levels and log content templates, and call status and call delay deviations, respectively, this application can extract more accurate abnormal evidence from three dimensions—metrics, logs, and call chains—compared to the existing technology that relies on only a single type of operational data or coarse alarm information for root cause analysis. This improves the evidentiary basis for the subsequent generation of candidate root causes.

[0011] Further, the multimodal evidence report is input into the diagnostic agent to enable the diagnostic agent to generate a list of candidate root causes, including: The diagnostic agent determines multiple target templates and multiple target examples that match the target fault from a preset template library and example library based on the multimodal evidence report. The diagnostic agent generates diagnostic prompts based on the multimodal evidence report, the multiple target templates, and the multiple target examples; The diagnostic agent generates multiple candidate root causes and corresponding diagnostic scores for each candidate root cause based on the diagnostic prompt information. The diagnostic agent sorts the multiple candidate root causes according to the diagnostic scores corresponding to each candidate root cause, thereby obtaining the candidate root cause list.

[0012] By identifying multiple target templates and examples matching the target fault from a pre-set template and example library, and generating diagnostic prompts by combining multimodal evidence reports, this application can utilize templates and examples matching the current fault to constrain the reasoning direction of the diagnostic agent, thereby improving the targeting and stability of candidate root cause generation, compared to the existing technology that directly inputs the original fault information into a large model for open reasoning.

[0013] Further, the candidate root cause list is input into the verification agent, so that the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause, including: The verification agent performs similarity matching between each candidate root cause and historical failure cases to obtain similar historical failure cases corresponding to each candidate root cause, and determines the historical case matching score corresponding to each candidate root cause based on the similarity between the similar historical failure cases and the corresponding candidate root cause. The verification agent performs causal rationality verification, temporal consistency verification, and topological rationality verification on each candidate root cause, and obtains the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause. The verification agent determines the rule verification score corresponding to each candidate root cause based on the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause.

[0014] By verifying the historical case matching and causal rationality, temporal consistency and topological rationality checks of each candidate root cause by the intelligent agent, compared with the existing technology that lacks an independent verification link for diagnostic results, this application can cross-validate candidate root causes from two dimensions: historical experience and operation and maintenance rules. This allows for timely identification of candidate root causes that lack historical support or do not conform to engineering logic, thereby improving the reliability of candidate root cause verification.

[0015] Furthermore, a weighted sum is performed based on the diagnostic score, historical case matching score, and rule verification score corresponding to each candidate root cause to obtain the comprehensive credibility score corresponding to each candidate root cause. Based on the comprehensive credibility score, the multiple candidate root causes in the candidate root cause list are sorted. When the comprehensive confidence score corresponding to the candidate root cause ranked first meets the preset high confidence condition, the candidate root cause ranked first is determined as the target root cause, and the root cause localization result is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset review conditions, the candidate root cause ranked first is determined as the root cause to be reviewed, and the root cause location result including manual review prompts is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset correction conditions, the verification information obtained by the verification agent is fed back to the diagnostic agent so that the diagnostic agent can iteratively correct the candidate root cause list.

[0016] By weighting and summing diagnostic scores, historical case matching scores, and rule validation scores to obtain a comprehensive credibility score, and then sorting, outputting, providing review prompts, or iteratively correcting based on the comprehensive credibility score, this application can quantitatively assess the credibility of candidate root causes compared to the existing technology that directly uses model output results as the final conclusion. It can also trigger manual review or feedback correction when the credibility is insufficient, thereby improving the interpretability and controllability of root cause localization results.

[0017] Furthermore, after outputting the root cause localization result corresponding to the target fault, the method further includes: Obtain confirmation results of the repair of the target fault; Based on the fault information corresponding to the target fault, multimodal evidence report, target root cause, comprehensive credibility score and repair confirmation result, a diagnostic experience record is generated; The diagnostic experience records are stored in a vector experience library so that the vector experience library can be used as an example library for diagnostic agents and to verify the historical case matching of agents. The diagnostic experience records in the vector experience base are subjected to self-optimization processing.

[0018] By obtaining repair confirmation results after outputting root cause localization results, and generating diagnostic experience records based on fault information, multimodal evidence reports, target root causes, comprehensive credibility scores, and repair confirmation results, and storing them in a vector experience library, this application can improve the efficiency of locating similar faults compared to the existing technology where each fault requires re-reasoning and historical diagnostic experience is difficult to reuse.

[0019] Furthermore, the self-optimization process includes one or more of the following methods: Based on the comparison between the quality evaluation results of the diagnostic experience records and the preset quality threshold, diagnostic experience records with quality evaluation results lower than the preset quality threshold are identified as low-quality diagnostic experience records, and the low-quality diagnostic experience records are deleted or marked as invalid. Alternatively, based on the storage and retrieval status of the diagnostic experience records, the diagnostic experience records that meet the archiving conditions can be archived. Alternatively, duplicate diagnostic experience records can be merged based on the degree of similarity and root cause consistency between different diagnostic experience records.

[0020] By performing self-optimization processes such as cleaning, archiving, or merging diagnostic experience records in the vector experience base, compared to the existing methods of long-term static accumulation of experience bases or knowledge bases, which are prone to containing low-quality or duplicate records, this application can continuously maintain the quality of experience in the vector experience base, reduce the interference of low-value experience on subsequent diagnosis and verification processes, and thus improve the accuracy and stability of experience base reuse.

[0021] Secondly, this application provides a fault root cause localization system based on dual-agent collaboration, including: a time window determination module, an evidence report generation module, a diagnosis module, a verification module, and a root cause output module; The time window determination module is used to obtain the time of occurrence of the target system's fault and determine the data acquisition time window. The evidence report generation module is used to acquire multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data. The diagnostic module is used to input the multimodal evidence report into the diagnostic agent and generate a candidate root cause list through the diagnostic agent. The candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause. The verification module is used to input the candidate root cause list into the verification agent, and the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause. The root cause output module is used to determine the comprehensive credibility score corresponding to each candidate root cause based on the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause, and to determine the target root cause from the candidate root cause list based on the comprehensive credibility score, and output the root cause location result corresponding to the target fault. Attached Figure Description

[0022] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. The drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a fault root cause localization method based on dual-agent collaboration provided in an embodiment of this application. Figure 2 This is a schematic diagram of a dual-agent collaborative processing flow provided in an embodiment of this application; Figure 3 This is a schematic diagram of a fault root cause localization process based on dual-agent collaboration provided in an embodiment of this application; Figure 4 This is a schematic diagram illustrating the implementation of a fault root cause localization method based on a three-layer collaborative architecture, provided in another embodiment of this application. Figure 5 This is a schematic diagram of a fault root cause localization system based on dual-agent collaboration, provided in another embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] In the description of this application, the terms "first," "second," "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first," "second," "third," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0026] In the description of this application, it should be noted that, unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. The terms "vertical," "horizontal," "left," "right," "upper," "lower," and similar expressions used herein are for illustrative purposes only and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting this application. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0027] In the description of this application, it should be noted that, unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in the specification of this application is for the purpose of describing specific embodiments only and is not intended to limit the application. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0028] This application focuses on root cause localization in the field of intelligent operations and maintenance. In complex systems using microservice architectures, such as e-commerce, financial systems, and power grids, the number of services can easily reach hundreds, with intricate call chains. When system failures occur, traditional operations and maintenance relies on engineers to check logs and monitoring indicators one by one, which is time-consuming and prone to missing key clues. This application automatically aggregates three types of data—Metric, Log, and Trace—through a multimodal evidence fusion device. A large language model diagnostic agent quickly understands the semantics of the failure and generates candidate root cause hypotheses. The verification agent then confirms the hypothesis through historical case matching and rule validation, outputting a high-confidence root cause conclusion.

[0029] Example 1: To address the problem that existing root cause localization processes based on large models lack independent verification mechanisms, which can easily lead to unsupported large model illusions or incorrect attributions, resulting in insufficient reliability of root cause localization results, such as... Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating a fault root cause localization method based on dual-agent cooperation provided in an embodiment of this application. The fault root cause localization method based on dual-agent cooperation in this embodiment includes steps S101 to S105, which are detailed below: Step S101: Obtain the time of the target system's failure at the time of the target fault, and determine the data acquisition time window.

[0030] Specifically, by failure time Using W as the anchor point, the data acquisition time window is the backtracking window W. Multimodal operation and maintenance data before the failure occurs are obtained according to the backtracking window.

[0031] Preferably, the backtracking window W can be set to 5 minutes. Optionally, depending on the actual operation and maintenance scenario, the backtracking window can be configured to be 1 minute to 15 minutes.

[0032] It's important to note that a 5-minute window was chosen because empirical research in the operations and maintenance (O&M) field shows that in many O&M scenarios, precursory anomalies can be detected within 5 minutes before symptoms manifest. Therefore, a window that is too large introduces a lot of irrelevant noise and increases the burden on the LLM context, while a window that is too small may miss crucial early warning signals.

[0033] Step S102: Obtain multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data.

[0034] In some preferred embodiments, anomaly identification is performed on the indicator data, the log data, and the call chain data respectively to obtain indicator anomaly events, log error events, and call chain anomaly events, including: The abnormal event of the indicator is determined based on the degree of deviation of the indicator value in the indicator data from the preset indicator benchmark value; The log error event is determined based on the occurrence of the log level and log content template in the log data within the data collection time window; The call chain exception event is determined based on the call status in the call chain data and the degree of deviation of the call delay in the call chain data from the preset call baseline value.

[0035] Implementing the above-mentioned preferred scheme can extract more accurate anomaly evidence from three dimensions: metrics, logs, and call chains, thereby improving the evidentiary basis for subsequent candidate root cause generation.

[0036] In a preferred embodiment, when identifying anomalies in indicator data, a sliding window statistical process can be performed on the indicator time series corresponding to each indicator data. Specifically, for any indicator time series, sliding windows are constructed sequentially according to a preset window length and a preset step size, and the mean μ and standard deviation σ of the indicator values ​​within each sliding window are calculated.

[0037] Furthermore, the preset window length can be 60 seconds, and the preset step size can be 30 seconds. When the metric value x at a certain moment satisfies |x-μ|>3σ, the metric value corresponding to that moment is marked as an anomaly, and an metric anomaly event is generated based on the anomaly point. For the metric anomaly event marked as an anomaly, the corresponding metric name, the service to which it belongs, the container group Pod name, the time of the anomaly occurrence, the direction of the anomaly, and the deviation magnitude are recorded; wherein, the direction of the anomaly includes an upward direction or a downward direction.

[0038] Furthermore, the abnormal events of each indicator are sorted according to the time of occurrence of the abnormality, resulting in a list of abnormal indicator events, E_metric. The above method identifies indicator anomalies through a sliding window statistical threshold. This statistical threshold method has a much lower computational complexity than deep anomaly detection methods based on autoencoders or Isolation Forests, eliminating the need to build complex deep anomaly detection models and reducing computational complexity while ensuring anomaly identification efficiency.

[0039] In a preferred embodiment, when identifying anomalies in log data, the log data can first be filtered according to log levels. Specifically, log entries with log levels of ERROR, FATAL, and WARN are retained, while log entries with log levels of INFO and DEBUG are filtered out to reduce noisy logs that are less correlated with the target fault.

[0040] Further, the filtered error log entries undergo log template processing. For example, the Drain3 algorithm can be used to cluster and parse the error log entries; where the maximum parsing depth `max_depth` of the Drain3 algorithm can be set to 3, and the similarity threshold `sim_th` can be set to 0.4. Through this log template processing, specific log messages containing variable parameters are generalized into corresponding log content templates. Subsequently, the frequency of each log content template within the data collection time window is counted, and multiple log content templates are sorted from high to low frequency, retaining a preset number of high-frequency log content templates; where the preset number N_log can be 20. Log error events are generated based on the retained high-frequency log content templates, forming a log error event list E_log. Each log error event may include a template identifier, template content, service to which it belongs, first occurrence time, number of occurrences, and associated call chain identifier.

[0041] In a preferred embodiment, when identifying anomalies in the call chain data, status code anomaly detection and delay anomaly detection can be performed on the call chain data respectively.

[0042] Specifically, for status code anomaly detection, the link span record in the call chain data where the call status code status.code is not equal to 0 can be directly identified, and the corresponding link span record is identified as a status code anomaly event.

[0043] For latency anomaly detection, a threshold detection method based on historical baselines can be used. Specifically, for any service pair consisting of a calling service and a called service, the normal call latency data of the service pair within the same time period in the past preset historical period is obtained, and the mean latency μ_latency and the standard deviation σ_latency corresponding to the normal call latency data are calculated; wherein, the preset historical period can be 7 days. When the current call latency satisfies latency > μ_latency + 3σ_latency, the call record corresponding to the current service pair is marked as a latency anomaly event.

[0044] Furthermore, the detected status code exception events and delay exception events are sorted according to the time of occurrence of the exception to obtain the call chain exception event list E_trace.

[0045] Furthermore, after extracting the list of abnormal events E_metric, the list of log error events E_log, and the list of abnormal events E_trace, the three types of event lists are merged and sorted according to the event timestamps corresponding to each abnormal event to obtain a unified event list E_all.

[0046] In one alternative implementation, before generating a multimodal evidence report, the real-time topology and dependencies between nodes in the target system can be extracted based on the currently occurring target fault.

[0047] In some preferred embodiments, generating a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data includes: The multimodal operation and maintenance data includes indicator data, log data, and call chain data; Anomalies are identified in the indicator data, the log data, and the call chain data respectively to obtain indicator anomaly events, log error events, and call chain anomaly events. The abnormal events, log error events, and call chain abnormal events are sorted according to their occurrence time to obtain an abnormal event list. Based on the list of abnormal events, a multimodal evidence report corresponding to the target fault is generated.

[0048] Implementing the above-mentioned preferred scheme can reduce the interference of irrelevant noise on the diagnostic process and preserve the temporal correlation between abnormal events, thereby improving the correlation between multimodal evidence reports and target faults.

[0049] In a preferred embodiment, after obtaining the unified event list E_all, the unified event list E_all can be processed for evidence vectorization to convert the time-ordered anomalous events into structured natural language descriptions suitable for understanding and reasoning by large language models.

[0050] Specifically, the unified event list E_all can be converted into a four-part evidence description, including a summary of the fault phenomenon, a summary of the abnormal indicators, a summary of the error log, and a summary of the abnormal call chain, according to a preset evidence description template.

[0051] The fault phenomenon overview describes the overall context of the target fault and may include the fault occurrence time, the name of the affected service, the fault phenomenon, and the alarm level. This fault phenomenon overview enables the diagnostic agent to obtain the overall fault context.

[0052] The anomaly indicator summary is used to describe abnormal changes in the indicator data. Specifically, anomaly events can be grouped and aggregated according to their respective services, and the anomaly indicator name, original indicator value, and relative percentage change can be described within each service's corresponding group.

[0053] The error log summary is used to describe key error information in the log data. Specifically, log error events can be arranged in descending order of frequency of occurrence of log content templates, and the number of times each log content template appears and representative log examples can be described.

[0054] The exception call chain summary is used to describe the exception call situations in the call chain data. Specifically, exception events in the call chain can be grouped according to exception type to describe the call relationships between services and the direction of the exception.

[0055] Furthermore, the fault phenomenon overview, abnormal indicator summary, error log summary, and abnormal call chain summary are concatenated in a preset order to generate a multimodal evidence report R_evidence corresponding to the target fault.

[0056] For example, the multimodal evidence report R_evidence can be in Markdown format, and different types of evidence can be described in segments through titles, lists and field key-value pairs.

[0057] By using the above methods, the token consumption can be significantly reduced while preserving the temporal correlation between metrics, logs, and call chains.

[0058] In some preferred embodiments, after obtaining the unified event list E_all and before generating a multimodal evidence report based on the unified event list E_all, evidence denoising and filtering can be performed on the abnormal events in the unified event list E_all to filter out noisy events that are irrelevant to the current fault.

[0059] In a preferred embodiment, a two-layer denoising mechanism can be used to filter out abnormal events in the unified event list E_all.

[0060] Specifically, the first layer is anomaly score filtering. For anomaly events in the indicators, an anomaly score is calculated. and retain only Abnormal events in the indicators; among them, The following relationship must be satisfied: Where x represents the observed value of the metric at the current abnormal moment. For example, the actual values ​​of CPU utilization, memory utilization, QPS, request latency, etc. of a certain service / Pod at the abnormal time point; μ represents the mean of the metric within the sliding window, which is used to characterize the normal baseline level of the metric in the current period. In the preferred embodiment, the sliding window can be set to 2h; σ represents the standard deviation of the metric within the same sliding window, which is used to characterize the normal fluctuation range of the metric.

[0061] Furthermore, for log error events, a relative frequency score is calculated. and retain only Log error events; among them, The following relationship must be satisfied: in, This indicates the frequency of the current error log template within the time window, that is, the number of times a specific log template appears within the fault time window. For example, the number of times a specific log template (such as "Connection timeoutto backend service [service] after [timeout]ms") appears within the fault time window. This represents the maximum frequency of all log templates appearing within the same time window, i.e., the frequency of the log template that appears most frequently within the window.

[0062] Furthermore, for call chain exception events, a latency deviation score is calculated. and retain only The call chain exception event; among which, The following relationship must be satisfied: Here, latency represents the actual observed latency value of a service pair consisting of a calling service and a called service in the current call chain exception event; This indicates the average normal latency for the service over the same period in the past 7 days; This indicates the standard deviation of the delay for the same period over the past 7 days.

[0063] Furthermore, the second layer involves keyword matching and filtering. A set of keywords is extracted from the description of the fault symptoms. And for each abnormal event, a set of keywords is generated. Calculate matching degree Only retain Abnormal events; among them, The following relationship must be satisfied: It should be noted that known noisy alarms in the pre-configured whitelist are directly excluded before the first layer of filtering. After two layers of filtering and two layers of noise reduction, the number of events in the evidence report is further reduced, and the correlation between the retained events and the current fault is significantly improved.

[0064] Step S103: Input the multimodal evidence report into the diagnostic agent so that the diagnostic agent generates a candidate root cause list, wherein the candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause.

[0065] In some preferred embodiments, the multimodal evidence report is input into a diagnostic agent to enable the diagnostic agent to generate a list of candidate root causes, including: The diagnostic agent determines multiple target templates and multiple target examples that match the target fault from a preset template library and example library based on the multimodal evidence report. The diagnostic agent generates diagnostic prompts based on the multimodal evidence report, the multiple target templates, and the multiple target examples; The diagnostic agent generates multiple candidate root causes and corresponding diagnostic scores for each candidate root cause based on the diagnostic prompt information. The diagnostic agent sorts the multiple candidate root causes according to the diagnostic scores corresponding to each candidate root cause, thereby obtaining the candidate root cause list.

[0066] By implementing the above preferred scheme, the reasoning direction of the diagnostic agent can be constrained by templates and examples that match the current fault, thereby improving the relevance and stability of candidate root cause generation.

[0067] In some preferred embodiments, the diagnostic agent first extracts core features for identifying the target fault from the multimodal evidence report. These core features may include fault type, fault scope, severity, and fault timeline. The fault type may include one or more of performance degradation, service unavailability, data errors, or resource exhaustion; the fault scope may include a single service, multiple services, or the entire cluster; and the severity may include P0 fatal, P1 severe, or P2 moderate.

[0068] Furthermore, the diagnostic agent analyzes the correlation between different modalities of abnormal events to determine their temporal proximity, topological correlation, and semantic consistency. For example, when the time difference between different abnormal events is less than 30 seconds, it is determined that the different abnormal events are temporally close; the topological correlation is used to characterize whether there is a call relationship between the services or nodes corresponding to different abnormal events; and the semantic consistency is used to characterize whether the abnormal meanings corresponding to different abnormal events match each other.

[0069] Furthermore, the diagnostic agent can also associate abnormal signals with pre-defined domain knowledge to enhance the fault context. For example, when evidence in the multimodal evidence report indicates the existence of an "upgrade anomaly," the diagnostic agent can prioritize analysis in conjunction with version compatibility issues.

[0070] The above approach enables the diagnostic agent to gain a structured understanding of the fault phenomena, abnormal relationships, and fault context before generating candidate root causes, thereby improving the relevance of subsequent candidate root cause generation.

[0071] In some preferred embodiments, the diagnostic agent can use a pre-defined causal knowledge template combined with ICL (Information-Based Learning) to guide the large language model to generate candidate root causes, without relying on online causal graph discovery algorithms. The pre-defined causal knowledge template is a pre-designed and encoded set of causal patterns in the operations and maintenance domain, which can be represented in the form of triples of "condition → root cause → phenomenon". The pre-defined causal knowledge template can include multiple fault templates covering the infrastructure layer, middleware layer, and application layer, and supports user customization and extension according to the actual operations and maintenance environment. The context learning examples are used to provide few-shot learning references for the large language model. For each type of fault, 2 to 3 examples including evidence reports and corresponding root cause analysis conclusions can be configured, and a total of approximately 15 to 20 examples constitute the example library. During each inference, 3 to 5 examples most similar to the current fault characteristics can be selected from the example library as ICL examples.

[0072] Specifically, the diagnostic agent can perform similarity matching between the current multimodal evidence report and a preset fault template library, selecting the top N fault templates with the highest similarity as matching templates. Subsequently, the matching templates, the selected ICL examples mentioned above, and the multimodal evidence report are concatenated to generate structured prompt information; wherein, the ICL examples can be pre-constructed based on historical root cause analysis cases.

[0073] Furthermore, the structured prompt information is input into a large language model, which generates a list of candidate root cause hypotheses. Each candidate root cause hypothesis entry in the list includes at least a root cause description field, a confidence score field, and a reasoning basis field; wherein, the root cause description field is used to characterize the natural language explanation of the cause of the failure, the confidence score field is used to characterize the probability value or level of the corresponding candidate root cause hypothesis being true, and the reasoning basis field is used to characterize the evidence chain or logical deduction process supporting the corresponding candidate root cause hypothesis.

[0074] Furthermore, the diagnostic agent can also perform post-processing optimization on the candidate root cause hypothesis list. Specifically, the semantic similarity between any two candidate root cause hypotheses can be calculated. When the semantic similarity is greater than a preset similarity threshold, the corresponding two candidate root cause hypotheses are merged into a single hypothesis entry; wherein, the preset similarity threshold can be 0.8.

[0075] Furthermore, the candidate root cause hypotheses are sorted in descending order of confidence score from highest to lowest, and the top M candidate root cause hypotheses are retained as the candidate root cause list for output.

[0076] Furthermore, when generating the candidate root cause list, the diagnostic agent can adopt the thought chain (CoT) output format to enable the large language model to explicitly output the reasoning process related to the generation of candidate root causes, thereby enhancing the interpretability and auditability of the diagnostic results.

[0077] In one specific implementation, the thought chain output may include an evidence analysis chain, a causal reasoning chain, a hypothesis evaluation chain, and a conclusion summary chain. The evidence analysis chain is used to analyze the meaning of key anomalies in the multimodal evidence report, and the possible root causes that each key anomaly might point to. The causal reasoning chain is used to construct a reasoning link from the fault phenomenon to candidate root causes based on a pre-defined causal knowledge template obtained through matching. The hypothesis evaluation chain is used to list supporting evidence and counter-evidence for each candidate root cause to assess the validity of the corresponding candidate root cause. The conclusion summary chain is used to synthesize the evaluation results of multiple candidate root causes and provide a final ranking and recommendation result for each candidate root cause.

[0078] By adopting the above-mentioned thought chain output format, the diagnostic agent can not only provide clear and understandable diagnostic logic to operation and maintenance personnel, but also provide verification materials for the subsequent verification agent, enabling the verification agent to perform logical consistency checks on candidate root causes and their reasoning processes based on the evidence analysis chain, causal reasoning chain, hypothesis evaluation chain, and conclusion summary chain.

[0079] Step S104: Input the candidate root cause list into the verification agent so that the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause.

[0080] In some preferred embodiments, the candidate root cause list is input into a verification agent, so that the verification agent performs historical case matching and rule verification on each candidate root cause to obtain a historical case matching score and a rule verification score corresponding to each candidate root cause, including: The verification agent performs similarity matching between each candidate root cause and historical failure cases to obtain similar historical failure cases corresponding to each candidate root cause, and determines the historical case matching score corresponding to each candidate root cause based on the similarity between the similar historical failure cases and the corresponding candidate root cause. The verification agent performs causal rationality verification, temporal consistency verification, and topological rationality verification on each candidate root cause, and obtains the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause. The verification agent determines the rule verification score corresponding to each candidate root cause based on the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause.

[0081] In some preferred embodiments, the verification agent can perform dual-channel verification on each candidate root cause. The dual-channel verification may include historical case matching verification and rule validation verification.

[0082] Specifically, the historical case matching verification serves as the first verification channel, used to determine the degree of similarity between the current fault and historical faults in terms of fault phenomena, evidence, and root causes. When the current fault and historical faults have a high degree of similarity in terms of fault phenomena, evidence, and root causes, it is determined that the corresponding historical fault case can provide support for the current candidate root cause.

[0083] Furthermore, the problem descriptions, evidence summaries, and root cause results from historical failure cases can be concatenated into case text, which is then vectorized to generate case vectors and stored in a vector experience base. For the current candidate root cause, the multimodal evidence report corresponding to the current failure and the current candidate root cause can be vectorized to obtain the current query vector. Based on the current query vector, an approximate nearest neighbor search is performed in the vector experience base to retrieve the top K similar historical failure cases with the highest similarity. The approximate nearest neighbor search can employ an ANN algorithm or an HNSW algorithm, and the similarity can be measured using cosine similarity.

[0084] Furthermore, based on the retrieved K similar historical fault cases, the historical case matching score S_history corresponding to the current candidate root cause is calculated, and the historical case matching score S_history satisfies the following relationship: in, , Indicates the ranking decay weight; Indicates the first Verification score for each historical failure case; Indicates the current query; Indicates the first 1 historical failure case; s Indicates the current query and the first Cosine similarity between historical failure cases.

[0085] When the historical cases match the score When the current candidate root cause is determined, it is verified by matching historical cases. If the historical failure case with the highest similarity obtained is a confirmed case, an additional confidence bonus of 0.2 is added to the current candidate root cause.

[0086] Furthermore, the rule verification serves as a second verification channel, used to perform consistency verification on candidate root causes based on general causal logic rules in the field of operation and maintenance.

[0087] In some preferred embodiments, the rule verification may include causal rationality verification, temporal consistency verification, and topological rationality verification.

[0088] Specifically, for causal rationality verification, the verification agent can verify whether the causal relationship between the candidate root cause and the fault phenomenon conforms to the physical common sense and engineering constraints of the operation and maintenance domain. The rule base can be pre-configured with rules covering common causal patterns, including positive and negative rules. For example, positive rules may include: database connection pool exhaustion leads to an increase in the request waiting queue; connection pool exhaustion leads to an increase in service response timeouts; negative rules may include: connection pool exhaustion does not lead to a decrease in CPU utilization; connection pool exhaustion does not lead to a surge in CPU usage for other services. Specifically, the causal rationality score can be quantified using the following formula: in, This represents the number of positive rules that were hit. 3 represents the number of reverse rules that were hit; 3 represents the expected number of hit rules corresponding to sufficient causal support, which is an adjustable parameter; 0.5 represents the deduction weight corresponding to each reverse rule, which is an adjustable parameter.

[0089] Specifically, for timing consistency verification, the verification agent can check whether the timestamp of a candidate root cause is earlier than the fault phenomenon it caused. A lenient strategy can be adopted during the check, allowing a 5-second time tolerance to account for acquisition delays and clock drift from different data sources. The timing consistency score can be quantified using the following formula: in, , Indicates the timestamp corresponding to the candidate root cause. The timestamp corresponding to the fault phenomenon.

[0090] Specifically, for topology rationality verification, the verification agent can verify whether there is a reasonable call dependency relationship between the components involved in the candidate root cause and the components involved in the fault phenomenon. Specifically, a decay function of call path distance can be used to quantify the topology rationality score. For example, if service B is the root cause service, service A is the service involved in the fault phenomenon, and service A directly calls service B, then... If service A indirectly calls service B via service C, i.e., A→C→B, then Specifically, the following formula can be used for calculation: in, This represents the number of hops in the shortest call path from the root cause component service to the faulty component service. This indicates the preset maximum number of hops for the call path. When no call path exists, the topology rationality score is 0.

[0091] Based on the above causal reasonableness score, temporal consistency score, and topological reasonableness score, the rule verification score can be determined according to the following formula: in, Score the reasonableness of causation. For time series consistency scoring, Score the topological rationality.

[0092] By implementing the above-mentioned preferred scheme, candidate root causes can be cross-validated from two dimensions: historical experience and operational rules. This allows for the timely identification of candidate root causes that lack historical support or do not conform to engineering logic, thereby improving the reliability of candidate root cause verification.

[0093] Step S105: Based on the diagnostic score, historical case matching score, and rule verification score corresponding to each candidate root cause, determine the comprehensive credibility score corresponding to each candidate root cause, and based on the comprehensive credibility score, determine the target root cause from the candidate root cause list, and output the root cause location result corresponding to the target fault.

[0094] Furthermore, after obtaining the historical case matching score and rule verification score corresponding to each candidate root cause, the verification agent can determine the comprehensive credibility score corresponding to each candidate root cause based on the diagnostic score, historical case matching score and rule verification score output by the diagnostic agent.

[0095] In some preferred embodiments, for any candidate root cause, the overall credibility score can be calculated according to the following formula: in, This represents the raw confidence score output by the diagnostic agent. Indicates the historical case matching score. This indicates the rule validation score.

[0096] It should be noted that the weight design follows the principle that LLM confidence is the core of diagnosis and has the highest weight (0.4), while the two verification channels have a balanced impact (0.3 each).

[0097] Furthermore, based on the comprehensive credibility score corresponding to each candidate root cause... Multiple candidate root causes are reordered from high to low based on their comprehensive credibility score, and a predetermined number of the top-ranked candidate root causes are retained as verified candidate root causes; wherein the predetermined number can be 3. Through this method, based on the initial ranking by the diagnostic agent, candidate root causes can be re-evaluated by combining the support level of historical cases and rule verification results, thereby improving the reliability of the candidate root cause ranking results.

[0098] In some preferred embodiments, a comprehensive reliability score is determined for each candidate root cause based on its diagnostic score, historical case matching score, and rule validation score. Then, based on the comprehensive reliability score, a target root cause is determined from the candidate root cause list, and the root cause localization result corresponding to the target fault is output, including: The comprehensive credibility score corresponding to each candidate root cause is obtained by weighted summation of the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause. Based on the comprehensive credibility score, the multiple candidate root causes in the candidate root cause list are sorted. When the comprehensive confidence score corresponding to the candidate root cause ranked first meets the preset high confidence condition, the candidate root cause ranked first is determined as the target root cause, and the root cause localization result is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset review conditions, the candidate root cause ranked first is determined as the root cause to be reviewed, and the root cause location result including manual review prompts is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset correction conditions, the verification information obtained by the verification agent is fed back to the diagnostic agent so that the diagnostic agent can iteratively correct the candidate root cause list.

[0099] Furthermore, after reordering multiple candidate root causes based on the comprehensive credibility score, the output method of the root cause localization result can be determined based on the comprehensive credibility score corresponding to the first ranked candidate root cause.

[0100] In one specific implementation, if the comprehensive credibility score corresponding to the candidate root cause ranked first... If so, the candidate root cause is marked as a high-confidence root cause, and the root cause localization result containing the high-confidence root cause is directly output.

[0101] If the comprehensive credibility score corresponding to the candidate root cause ranked first satisfies If the candidate root cause is marked as a medium-confidence root cause, the root cause localization result with a manual review prompt will be output to prompt the operations and maintenance personnel to manually review the candidate root cause.

[0102] If the comprehensive credibility score of the candidate root cause ranked first If the low-confidence verification information is fed back to the diagnostic agent, the agent can iteratively revise the candidate root cause list based on the verification information. After multiple iterations, if the preset iteration limit is reached and the overall confidence score of the top-ranked candidate root cause is still less than 0.5, the candidate root cause is marked as a low-confidence root cause, and a root cause localization result including diagnostic uncertainty hints and human intervention suggestions is output.

[0103] To more clearly illustrate the relationship between the two agents in this embodiment, see [link to documentation]. Figure 2 , Figure 2 This is a schematic diagram illustrating the collaborative processing of a diagnostic agent and a verification agent according to an embodiment of this application. Figure 2 As shown, in the collaborative processing between the diagnostic agent and the verification agent, after the structured evidence report is input into the diagnostic agent, the agent first performs fault phenomenon understanding and anomaly correlation analysis to determine the fault type, fault scope, severity, and timeline. It then analyzes the correlation between anomalous events based on temporal proximity, topological correlation, and semantic consistency. Subsequently, the diagnostic agent generates a list of candidate root cause hypotheses based on a causal knowledge template library and contextual learning examples.

[0104] After receiving the candidate root cause hypothesis list, the verification agent performs historical case matching based on the vector experience base to retrieve similar historical cases. Simultaneously, it performs rule verification based on the rule base, including causal rationality verification, temporal consistency verification, and topological rationality verification. Further, the verification agent calculates a comprehensive confidence score based on the confidence score output by the diagnostic agent, the historical case matching score, and the rule verification score, and determines the root cause output method based on the comprehensive confidence score. When the comprehensive confidence score is high, a high-confidence root cause and remediation suggestions are output; when the comprehensive confidence score is in the middle range, a medium-confidence root cause is output and manual review is suggested; when the comprehensive confidence score is low, a low-confidence warning is output or low-confidence verification information is fed back to the diagnostic agent for iterative correction. After diagnosis, relevant diagnostic experience can be stored in the vector experience base for subsequent similar case retrieval and incremental optimization.

[0105] In some preferred embodiments, after outputting the root cause localization result corresponding to the target fault, the method further includes: Obtain confirmation results of the repair of the target fault; Based on the fault information corresponding to the target fault, multimodal evidence report, target root cause, comprehensive credibility score and repair confirmation result, a diagnostic experience record is generated; The diagnostic experience records are stored in a vector experience library so that the vector experience library can be used as an example library for diagnostic agents and to verify the historical case matching of agents. The diagnostic experience records in the vector experience base are subjected to self-optimization processing.

[0106] By implementing the above-mentioned preferred scheme, the verified diagnostic experience can be transformed into a sample library for subsequent diagnostic agents and a basis for matching historical cases of the verified agents, thereby improving the efficiency of locating similar faults.

[0107] In some preferred embodiments, the self-optimization process includes one or more of the following: Based on the comparison between the quality evaluation results of the diagnostic experience records and the preset quality threshold, diagnostic experience records with quality evaluation results lower than the preset quality threshold are identified as low-quality diagnostic experience records, and the low-quality diagnostic experience records are deleted or marked as invalid. Alternatively, based on the storage and retrieval status of the diagnostic experience records, the diagnostic experience records that meet the archiving conditions can be archived. Alternatively, duplicate diagnostic experience records can be merged based on the degree of similarity and root cause consistency between different diagnostic experience records.

[0108] In some preferred embodiments, the vector experience base can employ an incremental learning mechanism to continuously update and reuse diagnostic experience records.

[0109] Specifically, after each diagnostic process is completed, the system can store the complete information of this diagnosis in the vector experience base, forming a new diagnostic experience record. The diagnostic experience record may include a problem description, evidence summary, root cause results, verification score, diagnostic timestamp, list of affected services, and repair confirmation result. During vectorized storage, a delayed vectorization strategy can be adopted, directly storing the original text fields into a database text column and asynchronously triggering a vectorization task. The original text fields are then converted into vectors using an embedding model and stored in the corresponding vector column.

[0110] Specifically, in the similar fault retrieval stage, when a new fault occurs, the vector experience base can provide similar case retrieval services for both the diagnostic agent and the verification agent. The retrieved similar cases can not only be used for historical case matching verification by the verification agent, but also serve as examples for dynamic context learning when the diagnostic agent generates candidate root causes. When the similarity between a retrieved similar case and the current fault exceeds a preset similarity threshold, the similar case is preferentially used to replace the preset static example; wherein, the preset similarity threshold can be 0.85.

[0111] Specifically, in the self-optimization phase of the vector experience base, a scheduled task can be used to maintain the quality of diagnostic experience records in the vector experience base. Specifically, diagnostic experience records with a verification score below 0.3 or whose repair confirmation result is incorrect are automatically deleted or marked as invalid after 30 days of retention; diagnostic experience records stored for more than 180 days and not retrieved in the past 90 days are automatically migrated to cold storage for archiving; duplicate diagnostic experience records with a problem description similarity exceeding 0.95 and the same root cause result are merged into one diagnostic experience record, and the corresponding occurrence frequency and most recent occurrence time are updated. Through the above incremental learning mechanism, the vector experience base can continuously accumulate and optimize along with the diagnostic process, providing reusable historical experience for subsequent fault root cause localization.

[0112] By implementing the above-mentioned preferred scheme, the quality of experience in the vector experience base can be continuously maintained, the interference of low-value experience on subsequent diagnosis and verification processes can be reduced, thereby improving the accuracy and stability of experience base reuse.

[0113] See Figure 3 To illustrate this embodiment more clearly, Figure 3 This is a schematic diagram of a fault root cause localization process based on dual-agent collaboration. Figure 3 As shown in the embodiments of this application, the fault root cause localization method based on dual-agent collaboration can first acquire multimodal operation and maintenance data of the target system within the data acquisition time window. The multimodal operation and maintenance data includes indicator data, log data, call chain data, and topology dependency data. Subsequently, multimodal evidence fusion is performed on the multimodal operation and maintenance data to obtain a unified event list and topology information, and a structured multimodal evidence report is generated.

[0114] Further, the multimodal evidence report is input into the diagnostic agent, which combines pre-set causal knowledge templates and contextual learning examples to generate a candidate root cause list. Then, the candidate root cause list is input into the verification agent, which performs historical case matching verification and rule validation, and integrates multi-dimensional verification signals. Afterwards, a comprehensive scoring decision is made based on the diagnostic score, historical case matching score, and rule validation score, determining high-confidence, medium-confidence, or low-confidence root causes based on the comprehensive confidence score. For low-confidence cases, verification information can be fed back to the diagnostic agent for iterative correction. After the diagnostic process is completed, the diagnostic experience can also be stored in a vector experience base to support subsequent similar case retrieval and experience base self-optimization.

[0115] By implementing the embodiments of this application, the root cause localization process is divided into steps such as multimodal evidence report generation, diagnostic agent candidate root cause generation, verification agent historical case matching and rule verification, and comprehensive credibility scoring decision. Compared with the existing technology that directly relies on a large model to output root cause conclusions in a single inference, this application enables candidate root causes to undergo independent verification and comprehensive scoring before output, thereby reducing the risk of large model illusion and false attribution and improving the reliability of root cause localization results.

[0116] Example 2: Based on the above embodiments, this application also provides an implementation of a fault root cause localization method based on a three-layer collaborative architecture. It should be noted that the "layer" in this embodiment refers to the division of different processing functions during method execution and is not used to limit the module structure in the subsequent system or device embodiments.

[0117] See Figure 4 , Figure 4 This is a schematic diagram illustrating the implementation of a fault root cause localization method based on a three-layer collaborative architecture, as shown below. Figure 4 As shown, this embodiment can divide the root cause localization process into an evidence collection layer, a diagnostic reasoning layer, and a verification decision layer. Each layer is connected through structured evidence descriptions and candidate root cause hypotheses. Data flow can be transmitted in a bottom-up, layer-by-layer refinement manner. Data between layers can be transmitted in a structured format, such as using JSON format to describe multimodal evidence, candidate root causes, and verification results, thereby facilitating independent upgrades and troubleshooting at each processing stage.

[0118] The evidence collection layer is used to extract, compress, and structure fault evidence from multi-source heterogeneous operation and maintenance data. Specifically, the evidence collection layer can use the fault occurrence time T_fault as the time anchor, trace back a preset time window W, and aggregate abnormal indicator events, log error events, and call chain abnormal events within the preset time window W into a time-sorted event list. The preset time window W can be 5 minutes, or it can be configured from 1 minute to 15 minutes depending on the actual operation and maintenance scenario. In this way, multimodal alignment processing can be transformed into unified event aggregation processing based on time windows, reducing the complexity of evidence preprocessing. Furthermore, the evidence collection layer can also extract real-time topology dependency data and output a structured multimodal evidence report; the multimodal evidence report can include a fault phenomenon description, an abnormal indicator list, an error log summary, and abnormal call chain information.

[0119] The diagnostic inference layer deploys a diagnostic agent that receives multimodal evidence reports output from the evidence collection layer and generates a candidate root cause list based on the semantic understanding and reasoning capabilities of the large language model. Specifically, the diagnostic agent can employ a large model-guided diagnostic approach, directly analyzing the multimodal evidence reports and combining them with pre-set causal knowledge templates and contextual learning examples to generate a ranked list of candidate root cause hypotheses. The pre-set causal knowledge templates guide the large language model in causal inference, and the contextual learning examples provide a few-sample learning reference for the large language model. Each candidate root cause in the candidate root cause list output by the diagnostic inference layer can include a root cause description, confidence score, and reasoning basis.

[0120] The verification decision layer deploys a verification agent, which performs cross-validation and comprehensive scoring on candidate root causes generated by the diagnostic agent. Specifically, the verification agent can employ a dual-channel verification method to verify candidate root causes: on the one hand, it performs similarity matching between the candidate root cause and historical fault cases in the vector experience base to obtain a historical case matching score; on the other hand, it performs rule verification on the candidate root cause based on preset verification rules, which may include causal rationality verification, temporal consistency verification, and topological rationality verification. Further, the verification agent can determine the comprehensive credibility score corresponding to the candidate root cause based on the confidence score, historical case matching score, and rule verification score output by the diagnostic agent, and output the verified root cause localization result. The root cause localization result may include root cause conclusions, comprehensive credibility scores, verification basis, and remediation suggestions.

[0121] By implementing this embodiment, the evidence collection, candidate root cause generation, and candidate root cause verification in the root cause localization process are processed in layers through the above three-layer collaborative processing method. This decouples the diagnosis process from the verification process, thereby reducing the risk of the diagnosis results lacking evidence support or erroneous attribution, and improving the reliability and auditability of the root cause localization results.

[0122] Example 3: Based on the above method embodiments, this application provides corresponding system embodiments.

[0123] like Figure 5 As shown, one embodiment of this application provides a fault root cause localization system based on dual-agent collaboration, including: a time window determination module 100, an evidence report generation module 200, a diagnosis module 300, a verification module 400, and a root cause output module 500; The time window determination module 100 is used to obtain the time of occurrence of the target system's fault and determine the data acquisition time window. The evidence report generation module 200 is used to acquire multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data. The evidence report generation module 200 includes: a data acquisition unit, an anomaly identification unit, an event sorting unit, and a report generation unit; The data acquisition unit is used to acquire the target system's indicator data, log data, and call chain data within the data acquisition time window; The anomaly identification unit is used to identify anomalies in the indicator data, the log data, and the call chain data respectively, and obtain indicator anomaly events, log error events, and call chain anomaly events. The event sorting unit is used to sort the indicator abnormal events, the log error events, and the call chain abnormal events according to the event occurrence time to obtain an abnormal event list; The report generation unit is used to generate a multimodal evidence report corresponding to the target fault based on the list of abnormal events.

[0124] The diagnostic module 300 is used to input the multimodal evidence report into the diagnostic agent and generate a candidate root cause list through the diagnostic agent, wherein the candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause. The diagnostic module 300 includes: a template example determination unit, a diagnostic prompt generation unit, a candidate root cause generation unit, and a candidate root cause sorting unit; The template example determination unit is used to determine, through the diagnostic agent, multiple target templates and multiple target examples that match the target fault from a preset template library and example library based on the multimodal evidence report; The diagnostic prompt generation unit is used to generate diagnostic prompt information based on the multimodal evidence report, the multiple target templates, and the multiple target examples through the diagnostic intelligent agent; The candidate root cause generation unit is used to generate multiple candidate root causes and a corresponding diagnostic score for each candidate root cause based on the diagnostic prompt information through the diagnostic agent. The candidate root cause sorting unit is used by the diagnostic agent to sort the multiple candidate root causes according to the diagnostic scores corresponding to each candidate root cause, thereby obtaining the candidate root cause list.

[0125] The verification module 400 is used to input the candidate root cause list into the verification agent, and to perform historical case matching and rule verification on each candidate root cause through the verification agent to obtain the historical case matching score and rule verification score corresponding to each candidate root cause. The verification module 400 includes: a historical case matching unit, a rule sub-score determination unit, and a rule verification score determination unit; The historical case matching unit is used to perform similarity matching between each candidate root cause and historical failure cases through the verification agent to obtain similar historical failure cases corresponding to each candidate root cause, and determine the historical case matching score corresponding to each candidate root cause based on the similarity between the similar historical failure cases and the corresponding candidate root cause. The rule sub-score determination unit is used to perform causal rationality verification, temporal consistency verification, and topological rationality verification on each of the candidate root causes through the verification agent, so as to obtain the causal rationality score, temporal consistency score, and topological rationality score corresponding to each of the candidate root causes. The rule verification scoring determination unit is used to determine the rule verification score corresponding to each candidate root cause by the verification agent based on the causal rationality score, temporal consistency score and topological rationality score corresponding to each candidate root cause.

[0126] The root cause output module 500 is used to determine the comprehensive credibility score corresponding to each of the candidate root causes based on the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause, and to determine the target root cause from the candidate root cause list based on the comprehensive credibility score, and output the root cause location result corresponding to the target fault.

[0127] The root cause output module 500 includes: a comprehensive score calculation unit, a root cause rearrangement unit, a target root cause output unit, a review prompt output unit, and an iterative correction feedback unit; The comprehensive scoring calculation unit is used to perform a weighted summation of the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause to obtain the comprehensive credibility score corresponding to each candidate root cause. The root cause rearrangement unit is used to sort multiple candidate root causes in the candidate root cause list according to the comprehensive confidence score. The target root cause output unit is used to determine the candidate root cause ranked first as the target root cause and output the root cause location result when the comprehensive confidence score corresponding to the candidate root cause ranked first meets the preset high confidence condition. The review prompt output unit is used to determine the candidate root cause ranked first as the root cause to be reviewed when the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset review conditions, and output the root cause location result including manual review prompts. The iterative correction feedback unit is used to feed back the verification information obtained by the verification agent to the diagnostic agent when the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset correction conditions, so that the diagnostic agent can iteratively correct the candidate root cause list.

[0128] It is understood that the above-described device embodiments correspond to the method embodiments of this application, and can implement the fault root cause localization method based on dual-agent collaboration provided by any of the above-described method embodiments of this application.

[0129] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0130] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, or improvements made by those skilled in the art within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A fault root cause localization method based on dual-agent cooperation, characterized in that, include: Obtain the time of failure of the target system at the time of the target fault, and determine the data acquisition time window; Acquire multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data; The multimodal evidence report is input into the diagnostic agent to generate a candidate root cause list, wherein the candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause. The candidate root cause list is input into the verification agent, so that the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause. Based on the diagnostic score, historical case matching score, and rule verification score corresponding to each candidate root cause, a comprehensive credibility score is determined for each candidate root cause. Based on the comprehensive credibility score, the target root cause is determined from the candidate root cause list, and the root cause localization result corresponding to the target fault is output.

2. The fault root cause localization method based on dual-agent cooperation according to claim 1, characterized in that, The step of generating a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data includes: The multimodal operation and maintenance data includes indicator data, log data, and call chain data; Anomalies are identified in the indicator data, the log data, and the call chain data respectively to obtain indicator anomaly events, log error events, and call chain anomaly events. The abnormal events, log error events, and call chain abnormal events are sorted according to their occurrence time to obtain an abnormal event list. Based on the list of abnormal events, a multimodal evidence report corresponding to the target fault is generated.

3. The fault root cause localization method based on dual-agent cooperation according to claim 2, characterized in that, The step involves identifying anomalies in the metric data, log data, and call chain data to obtain metric anomaly events, log error events, and call chain anomaly events, including: The abnormal event of the indicator is determined based on the degree of deviation of the indicator value in the indicator data from the preset indicator benchmark value; The log error event is determined based on the occurrence of the log level and log content template in the log data within the data collection time window; The call chain exception event is determined based on the call status in the call chain data and the degree of deviation of the call delay in the call chain data from the preset call baseline value.

4. The fault root cause localization method based on dual-agent cooperation according to claim 1, characterized in that, The step of inputting the multimodal evidence report into the diagnostic agent to enable the diagnostic agent to generate a list of candidate root causes includes: The diagnostic agent determines multiple target templates and multiple target examples that match the target fault from a preset template library and example library based on the multimodal evidence report. The diagnostic agent generates diagnostic prompts based on the multimodal evidence report, the multiple target templates, and the multiple target examples; The diagnostic agent generates multiple candidate root causes and corresponding diagnostic scores for each candidate root cause based on the diagnostic prompt information. The diagnostic agent sorts the multiple candidate root causes according to the diagnostic scores corresponding to each candidate root cause, thereby obtaining the candidate root cause list.

5. The fault root cause localization method based on dual-agent cooperation according to claim 1, characterized in that, The step of inputting the candidate root cause list into the verification agent, so that the verification agent performs historical case matching and rule verification on each candidate root cause, and obtains the historical case matching score and rule verification score corresponding to each candidate root cause, includes: The verification agent performs similarity matching between each candidate root cause and historical failure cases to obtain similar historical failure cases corresponding to each candidate root cause, and determines the historical case matching score corresponding to each candidate root cause based on the similarity between the similar historical failure cases and the corresponding candidate root cause. The verification agent performs causal rationality verification, temporal consistency verification, and topological rationality verification on each candidate root cause, and obtains the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause. The verification agent determines the rule verification score corresponding to each candidate root cause based on the causal rationality score, temporal consistency score, and topological rationality score corresponding to each candidate root cause.

6. The fault root cause localization method based on dual-agent cooperation according to claim 1, characterized in that, The process involves determining a comprehensive reliability score for each candidate root cause based on its diagnostic score, historical case matching score, and rule validation score, and then determining the target root cause from the candidate root cause list based on the comprehensive reliability score, and outputting the root cause localization result corresponding to the target fault. This includes: The comprehensive credibility score corresponding to each candidate root cause is obtained by weighted summation of the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause. Based on the comprehensive credibility score, the multiple candidate root causes in the candidate root cause list are sorted. When the comprehensive confidence score corresponding to the candidate root cause ranked first meets the preset high confidence condition, the candidate root cause ranked first is determined as the target root cause, and the root cause localization result is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset review conditions, the candidate root cause ranked first is determined as the root cause to be reviewed, and the root cause location result including manual review prompts is output. When the comprehensive credibility score corresponding to the candidate root cause ranked first meets the preset correction conditions, the verification information obtained by the verification agent is fed back to the diagnostic agent so that the diagnostic agent can iteratively correct the candidate root cause list.

7. The fault root cause localization method based on dual-agent cooperation according to claim 1, characterized in that, After outputting the root cause localization result corresponding to the target fault, the method further includes: Obtain confirmation results of the repair of the target fault; Based on the fault information corresponding to the target fault, multimodal evidence report, target root cause, comprehensive credibility score and repair confirmation result, a diagnostic experience record is generated; The diagnostic experience records are stored in a vector experience library so that the vector experience library can be used as an example library for diagnostic agents and to verify the historical case matching of agents. The diagnostic experience records in the vector experience base are subjected to self-optimization processing.

8. The fault root cause localization method based on dual-agent cooperation according to claim 7, characterized in that, The self-optimization process includes one or more of the following methods: Based on the comparison between the quality evaluation results of the diagnostic experience records and the preset quality threshold, diagnostic experience records with quality evaluation results lower than the preset quality threshold are identified as low-quality diagnostic experience records, and the low-quality diagnostic experience records are deleted or marked as invalid. Alternatively, based on the storage and retrieval status of the diagnostic experience records, the diagnostic experience records that meet the archiving conditions can be archived. Alternatively, duplicate diagnostic experience records can be merged based on the degree of similarity and root cause consistency between different diagnostic experience records.

9. A fault root cause localization system based on dual-agent cooperation, characterized in that, include: The module includes a time window determination module, an evidence report generation module, a diagnosis module, a verification module, and a root cause output module. The time window determination module is used to obtain the time of occurrence of the target system's fault and determine the data acquisition time window. The evidence report generation module is used to acquire multimodal operation and maintenance data of the target system within the data acquisition time window, and generate a multimodal evidence report corresponding to the target fault based on the multimodal operation and maintenance data. The diagnostic module is used to input the multimodal evidence report into the diagnostic agent and generate a candidate root cause list through the diagnostic agent. The candidate root cause list includes multiple candidate root causes and a diagnostic score corresponding to each candidate root cause. The verification module is used to input the candidate root cause list into the verification agent, and the verification agent performs historical case matching and rule verification on each candidate root cause to obtain the historical case matching score and rule verification score corresponding to each candidate root cause. The root cause output module is used to determine the comprehensive credibility score corresponding to each candidate root cause based on the diagnostic score, historical case matching score and rule verification score corresponding to each candidate root cause, and to determine the target root cause from the candidate root cause list based on the comprehensive credibility score, and output the root cause location result corresponding to the target fault.

Images