Fault root cause positioning method and system

Abstract

This invention discloses a method and system for locating the root cause of a fault, relating to the field of cloud technology. The method includes generating read-only fault analysis data based on multimodal telemetry data and resource topology data within the fault attribution time period; constructing an anomaly relationship graph with abnormal states as graph nodes based on this data, dividing it into multiple fault groups, where nodes within the same group satisfy physical topological reachability and temporal causal relationships; generating candidate fault source location results for each group through retrieval, and selecting the result from the candidate results that satisfies the following conditions: a directed connected path exists between the fault source and the symptom object in the resource topology; the fault source's anomaly start time is earlier than the symptom object's; and at least two different modalities of anomaly data simultaneously support the result. This invention can solve the problem that related technologies cannot accurately determine the root cause of cloud system faults, achieving automated root cause location and evidence verification for complex cloud system faults.

Description

Technical Field

[0001] This invention relates to the field of cloud technology, and in particular to a method and system for locating the root cause of a fault. Background Technology

[0002] As the scale of microservices in cloud-native environments continues to grow, operational data exhibits a multimodal explosion and dynamic topology changes. When related technologies try to locate the root cause of a fault from the massive operational data of cloud systems, the static configuration dependencies are inconsistent with the actual call chain at the time of the fault. Furthermore, metrics, logs, and call chain data are isolated from each other, making it easy to misjudge anomalies that occur after the fault as the root cause of the fault, and making it impossible to accurately determine the root cause of the fault.

[0003] Therefore, effectively addressing dynamic changes in cloud-native topologies and accurately identifying the root cause of faults are technical problems that need to be solved by those skilled in the art.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] This invention provides a method and system for locating the root cause of a fault, enabling automated root cause location and evidence verification for complex distributed faults in cloud systems, and accurately determining the root cause of cloud system faults.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a method for locating the root cause of a fault, comprising: Based on multimodal telemetry data and resource topology data located within the fault attribution time period, generate fault analysis data with read-only attributes; Based on the fault analysis data, an anomaly relationship graph is constructed with anomaly states as graph nodes, and the anomaly relationship graph is divided into at least one fault group. There are physical topological reachable paths and temporal causal relationships between graph nodes in the same fault group, and the association strength of graph nodes between different fault groups is lower than a preset association threshold. The retrieval operation generates candidate fault source location results for each fault group. From each candidate fault source location result, the candidate fault source location result that simultaneously meets the following conditions is selected as the root cause of the fault: there is a directed connected path between the indicated fault source and the symptom object in the resource topology; the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object; and there are at least two different modalities of abnormal data that simultaneously support the location result.

[0007] Another aspect of the present invention provides a root cause localization system for faults, including a root cause inference processor deployed between a data acquisition terminal and an operation and maintenance display terminal; The root cause reasoning processor is used to perform the steps of locating the root cause of a fault as described above when executing a computer program stored in the memory.

[0008] The advantages of the technical solution provided by this invention are that it freezes multimodal telemetry data and dynamic resource topology within the fault attribution time period into read-only fault analysis data, ensuring the consistency and reproducibility of the analysis input and avoiding data inconsistency problems caused by container drift or topology changes. Subsequent inference processes only allow incremental verification based on snapshots, and absolutely prohibit changes to the fixed topology master view, effectively preventing the drift of the final fault diagnosis results. Based on this data, an anomaly relationship graph is constructed and divided into fault groups. Nodes within each group are forced to satisfy physical topology reachability and temporal causality, thereby isolating independent anomaly signals in concurrent faults and reducing interference from irrelevant anomalies in locating the root cause of the fault. Based on this, candidate fault sources are generated through retrieval based on conditions such as whether the call path is connected and whether the anomaly occurred earlier. The true root cause of the fault is selected from the candidate results using three hard constraints supported by topological directed connectivity paths, the order of anomaly occurrence, and at least two modal anomaly data, thus distinguishing between statistical correlation and true causality. Therefore, this invention significantly improves the accuracy and reliability of root cause localization, while the output localization results include traceable paths and modal evidence, reducing the cost of manual review and the false alarm rate. Furthermore, this invention also provides a corresponding implementation system for the root cause localization method, which has corresponding advantages. Attached Figure Description

[0009] To more clearly illustrate the technical solutions of the present invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a method for locating the root cause of a fault provided by the present invention; Figure 2 This is a schematic diagram of the dynamic resource topology graph construction process provided by the present invention; Figure 3 A schematic diagram of the fault analysis data generation process provided by the present invention; Figure 4 A structural framework diagram of an exemplary embodiment of the fault root cause localization device provided by the present invention; Figure 5 A structural diagram of an exemplary embodiment of the fault root cause localization system provided by the present invention; Figure 6 A schematic diagram of the hardware framework applicable to the fault root cause system provided by the present invention. Detailed Implementation

[0011] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. In this specification and the aforementioned drawings, the terms "first," "second," "third," "fourth," etc., are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. The term "exemplary" means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior to or better than other embodiments.

[0012] With the rapid development of cloud computing technology, in private and hybrid cloud environments, a single business application involves hundreds or even thousands of microservice instances. The cloud system generates massive amounts of performance metrics, unstructured logs, and distributed call chain data every moment. This cloud system comprises a cloud management platform, a cloud virtualization environment, and a cloud data center. The cloud management platform runs on top of the cloud virtualization environment, which in turn is built on the physical infrastructure of the cloud data center. The storage formats, sampling frequencies, and retention periods of this data generated by the cloud system vary, lacking unified time alignment and correlation indexes, forming data silos that are difficult to connect. Simultaneously, the creation, destruction, and migration of service instances are commonplace in container orchestration environments. The refresh cycle of traditional configuration management databases is much longer than the container lifecycle, resulting in the inability to accurately capture the true service dependencies at the moment of failure, leaving operations personnel facing an outdated topology view.

[0013] In determining the root cause of cloud system failures, one approach relies on a pre-defined failure mode library and diagnostic decision tree defined by operations experts. In cloud-native environments, service call relationships change frequently, and new failure modes constantly emerge. Maintaining a rule library covering all possible failure scenarios requires a significant and continuous investment of manpower, and the rule update speed lags behind the system change speed, leading to a significant increase in the false negative rate for new failures. Furthermore, fixed threshold rules are difficult to adapt to the dynamic load differences of different tenants or business modules. Another approach inputs all telemetry data into a deep learning model, training it end-to-end to output root cause recommendations. This method performs reasonably well in scenarios with strong statistical correlations, but its black-box nature results in a lack of interpretable causal chains in the output. The model often misjudges subsequent metric fluctuations as root causes or gives conclusions that violate physical topology constraints, such as determining that downstream service anomalies are the cause of upstream service failures. Because it cannot provide a chain of evidence for subsequent investigation, the output of such systems is difficult to trust in real-world production environments.

[0014] When a physical node fails or becomes overloaded, the container orchestration system automatically migrates service instances to other healthy nodes. There is a significant difference between the actual instance topology at the time of the failure and the static configuration snapshot taken seconds later. Traditional root cause analysis tools rely on periodically synchronized configuration management databases, which cannot record instance affiliation and temporary routing information at the moment of failure. Operations personnel may trace back to destroyed container instances along incorrect dependencies, leading to interrupted analysis paths or incorrect conclusions, especially in cases of underlying failures such as network partitions or node outages.

[0015] In view of this, to address the inconsistency between the statically configured dependencies of related technologies and the actual call chain at the time of failure, and the problem that metrics, logs, and call chain data are isolated and cannot distinguish between statistical correlation and true causality, this invention, based on the service dependency topology frozen at the time of failure and aligned multimodal data, uses physical link reachability verification and hard constraints of the chronological order of anomaly occurrence to determine the failure source that satisfies both temporal and topological logic, achieving traceable and verifiable root cause localization of the failure. Various non-limiting embodiments of this invention are described in detail below with reference to the accompanying drawings and specific implementation details. Please refer to [link to previous text]. Figure 1 According to the root cause localization method provided by the present invention, it can be implemented as a computer program product, installed and run in a root cause inference processor of a cloud system or a general-purpose computing device, for the purpose of implementing automated root cause localization and evidence closed-loop verification of cloud-native complex distributed faults. In some embodiments of the method, the method includes the following steps: S101: Generate fault analysis data with read-only attributes based on multimodal telemetry data and resource topology data located within the fault attribution time period.

[0016] The fault attribution time period is a fixed time range defined for analyzing a specific fault, such as including a period before and after the fault, for example, 2 minutes before the fault and 3 minutes after the fault. Multimodal telemetry data is data collected by physical devices or software programs with data acquisition capabilities deployed on the monitored object, and may include performance metrics, log templates, and traces. For example, the data plane is used for passive collection and preprocessing of telemetry data. It resides on the collection side of each computing node, including but not limited to data acquisition agents installed on the host operating system, sidecar proxies within business containers, and metric exporters from various middleware. This data is standardized to obtain standardized performance metrics, structured log templates, and sampled call chain summaries. This data can be transmitted through an efficient centralized telemetry aggregation channel (such as a distributed message queue) instead of being directly polled by the inference engine, thus greatly reducing intrusion on business nodes. Resource topology data is a dynamic dependency graph describing the call relationships between services, instances, and nodes. Read-only attributes mean that once fault analysis data is generated, no write or modification operations are allowed to ensure data consistency during the analysis process. Fault analysis data is a read-only snapshot formed by freezing multimodal telemetry data and resource topology data, and may include indicator summaries, log indexes, call chain summaries, and topology slices.

[0017] In private cloud environments, resource topology changes frequently with container scheduling, and directly performing fault backtracking based on real-time topology often leads to misjudgments due to topology drift. This step establishes a fault attribution time period (e.g., 2 minutes before to 3 minutes after the fault point) based on the trigger signal, and freezes the multimodal telemetry data and resource topology data within this time period as fault analysis data with read-only attributes. This achieves the freezing of performance metrics, logs, and call chain data scattered across various storage backends into an immutable and reproducible snapshot within a unified fault attribution time period.

[0018] S102: Construct an anomaly relationship graph with anomaly states as graph nodes based on fault analysis data, and divide the anomaly relationship graph into at least one fault group. There are physical topological reachable paths and temporal causal relationships between graph nodes in the same fault group, and the association strength of graph nodes between different fault groups is lower than a preset association threshold.

[0019] In this context, an abnormal state refers to the unusual characteristics exhibited by a monitored object within the fault attribution period, such as increased order service latency. A graph node is a vertex in the anomaly graph, representing an abnormal state of a monitored object. The anomaly graph is a graph structure with abnormal states as nodes and anomaly association strength as edges. Fault grouping is a set of graph nodes partitioned from the anomaly graph through constrained clustering. Nodes within the same group possess both physical topological reachability paths and temporal causal relationships. A physical topological reachability path is a directed call chain existing from one monitored object to another in the resource topology. A temporal causal relationship is when one abnormal state occurs earlier than another, with the time difference within a reasonable range.

[0020] In this step, an anomaly relationship graph is constructed based on the fault analysis data from S101, with abnormal states as graph nodes. Each node in the graph represents an abnormal state of a monitored object within the fault attribution time period. Based on the physical topological connectivity between graph nodes, the chronological order of anomalies, and the co-occurrence strength of multimodal data, a constrained clustering algorithm is used to divide the anomaly relationship graph into one or more fault groups. Graph nodes within the same fault group exhibit real call paths and temporal causal relationships, while the correlation strength between different groups is below a preset threshold, thus isolating concurrent faults.

[0021] S103: Generate candidate fault source location results for each fault group through retrieval operations, and select the candidate fault source location results that simultaneously meet the following conditions: the indicated fault source and the symptom object have a directed connected path in the resource topology; the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object; and there are at least two different modalities of abnormal data that simultaneously support the location results as the root cause of the fault.

[0022] The candidate fault source localization result includes the monitored object of the fault source, the time point of the anomaly start, the fault type, and supporting evidence. The symptom object refers to the monitored object corresponding to the observable abnormal behavior caused by the fault, such as a service with response delay. The root cause of the fault is the fault source localization result finally confirmed after verification through three hard constraints.

[0023] In this step, for each fault group, multimodal evidence is collected through iterative retrieval to generate candidate fault source localization results. Then, the root cause of the fault is selected from the candidate results that simultaneously meet three conditions: there is a directed connected path between the fault source and the symptom object in the resource topology; the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object; and there are at least two different modalities of abnormal data that simultaneously support the localization result. This solves the root cause localization problem caused by topology drift and multimodal data silos in cloud-native environments, and realizes interpretable and traceable automated reasoning.

[0024] In the technical solution provided in this embodiment, multimodal telemetry data and dynamic resource topology within the fault attribution time period are frozen as read-only fault analysis data, ensuring the consistency and reproducibility of the analysis input and avoiding data inconsistency issues caused by container drift or topology changes. Subsequent inference processes only allow incremental verification based on snapshots, and never allow changes to the fixed topology master view, effectively preventing the drift of the final fault diagnosis results. Based on this data, an anomaly relationship graph is constructed and divided into fault groups, forcing nodes within the group to satisfy physical topology reachability and temporal causality, thereby isolating independent anomaly signals in concurrent faults and reducing interference from irrelevant anomalies in locating the root cause of the fault. On this basis, candidate fault sources are generated by retrieval based on conditions such as whether the call path is connected and whether the anomaly occurred earlier. The true root cause of the fault is selected from the candidate results using three hard constraints supported by topological directed connectivity path, anomaly time sequence, and at least two modal anomaly data as screening conditions, thereby distinguishing statistical correlation from true causality. Therefore, this invention significantly improves the accuracy and reliability of root cause localization, while the output localization results are accompanied by traceable paths and modal evidence, reducing the cost of manual review and the false alarm rate.

[0025] Based on the above embodiments, the present invention further defines a method for constructing and updating a dynamic resource topology map before generating fault analysis data, which may include the following: Using monitored objects as topological vertices and the call relationships between monitored objects as directed topological edges, a dynamic resource topology graph is constructed in memory. Each directed topological edge is configured with a valid time interval, call intensity decay coefficient, and observation coverage. The event interface of the container orchestration platform is monitored in real time. When the creation, destruction, or migration event of a monitored object is detected, the corresponding topological vertices and their associated directed topological edges in the dynamic resource topology graph are added, deleted, or updated. The traffic data interface of the service mesh is monitored in real time. When a change in call relationships is detected, the corresponding directed topological edges in the dynamic resource topology graph are added or updated, and the call intensity decay coefficient of the corresponding directed topological edge is reset. The monitored objects are independent running units within the cloud system, including services, instances, containers, physical machines, and virtual machines. Topology vertices are nodes in the dynamic resource topology graph, corresponding to a monitored object. Directed topology edges represent the call relationship between one monitored object and another, with the direction from the caller to the callee. The dynamic resource topology graph is a service dependency graph maintained in memory and updated in real time. The valid time interval records the start and end times of call relationships and uses directed topology edges whose failure times fall within the valid time interval when backtracking historical failures. The call heat decay coefficient is a dynamic value that decreases exponentially over time, characterizing the recent activity level of the dependency. If there are no calls between two services for a long period, this coefficient decays exponentially over time, automatically filtering out stale or dead dependencies when constructing candidate subgraphs. Observation coverage represents the completeness of monitoring data at the two ends of a directed topological edge. The confidence weight of a directed topological edge is positively correlated with the corresponding observation coverage value. Edges with low coverage (such as third-party black-box services) will be given lower confidence weights during inference. Event interfaces of container orchestration platforms, such as the Kubernetes API Server, provide events for container creation, destruction, and migration. Service mesh traffic data interfaces provide traffic call relationship data.

[0026] In this embodiment, as Figure 2 As shown, a dynamic resource topology graph is constructed in memory, using the monitored objects as topological vertices and the call relationships between objects as directed topological edges. For example, if service A and service B have a call relationship, a directed topological edge is set between them; if service B and service C have a call relationship, a directed topological edge is set between them; if the database and service C have a call relationship, a directed topological edge is set between them. Internally, an efficient adjacency list structure can be used to store directed edges between storage services, instances, nodes, and even the underlying physical machines. To accurately express the strength and timeliness of dependencies, a valid time interval, call intensity decay coefficient, and observation coverage are configured for each directed edge. The event interface of the container orchestration platform is monitored in real time. When the creation, destruction, or migration event of the monitored object is detected, the corresponding topological vertex and its associated directed edge are added, deleted, or updated. Simultaneously, the traffic data interface of the service mesh is monitored in real time. When a change in call relationship is detected, the corresponding directed topological edge is added or updated, and the call intensity decay coefficient of the corresponding edge is reset. The effective time interval records the start and end times of the call relationship, ensuring that only the topology that existed at that time is used when backtracking a fault; the call heat decay coefficient decreases exponentially over time, which is used to automatically filter low-activity edges; the observation coverage is used to indicate the completeness of the monitoring data at both ends, and low-coverage edges are given lower confidence during inference.

[0027] As can be seen from the above, this embodiment solves the topology drift problem caused by dynamic scheduling of containers, enabling the graph traversal algorithm to automatically ignore expired or low-reliability dependencies during runtime, truly reflecting the system state when a fault occurs, and providing real-time and accurate dependencies for fault analysis.

[0028] Based on the above embodiments, the present invention further specifies how to generate fault analysis data with read-only attributes from multimodal telemetry data and resource topology data, which may include the following: Upon receiving a fault event signal, the fault attribution time period is determined based on the preset fault attribution time and the fault trigger time of the fault event signal. The fault triggering object corresponding to the fault event signal is obtained, along with its performance metrics, log information, and call chain data within the fault attribution time period, and aligned on a unified time axis. For performance metrics with sampling intervals greater than a preset sampling interval threshold, interpolation is applied; log information is downsampled, and missing log segments are linearly filled. The interpolated performance metrics are compressed into a statistical summary, the filled log information is categorized into log templates and a corresponding log index is established, and the aligned call chain data is compressed into a call chain statistical summary. A target topology slice centered on the fault triggering object is cut from the dynamic resource topology graph and assigned a corresponding topology version number. The statistical summary, call chain statistical summary, log index, target topology slice, and topology version number are encapsulated and assigned read-only attributes to obtain the fault analysis data.

[0029] The fault event signal is an alarm or abnormal event issued by the monitoring system, or an alarm event automatically triggered when a key performance indicator (KPI) of a monitored object deviates from the baseline. The fault triggering object is the monitored server that caused the fault event, such as an API interface or service. Performance indicator information is time-series numerical data, such as CPU utilization and IO wait time. Log information is unstructured text logs, which can be processed to extract templates. Call chain data is distributed tracing data, recording request paths and timestamps. A unified timeline aligns indicators, logs, and call chain data with different sampling frequencies to the same time base through interpolation or downsampling. Interpolation fills missing points for indicator data with excessively large sampling intervals using linear interpolation. Downsampling reduces the data volume by sampling the real-time log stream. Linear fill uses linear interpolation to fill missing segments of log information. The indicator statistical summary compresses the original indicator time-series data into statistical features, such as mean, maximum, and quantiles. The log template is a fixed semantic template obtained by merging similar logs. The log index is an inverted index built for fast retrieval of log templates. The call chain statistics summary is statistical information obtained by compressing call chain data, such as average edge latency and error rate. The target topology slice is a local subgraph cut from the dynamic resource topology graph, centered on the fault-triggered object. The topology version number identifies the version corresponding to the topology slice and is used for consistency verification.

[0030] In this embodiment, as Figure 3 As shown, upon receiving a fault event signal, the fault attribution time period is determined based on the preset fault attribution time and the fault trigger time of the signal. Performance metrics, log information, and call chain data of the fault-triggered object within this time period are acquired and aligned on a unified time axis: interpolation is used to fill in points for metrics with large sampling intervals, the real-time log stream is downsampled, and missing segments of log information are linearly filled in. The aligned performance metrics are compressed into a metric statistical summary (e.g., average value, P99 value); the filled log information is categorized into log templates and indexed; the aligned call chain data is compressed into a call chain statistical summary (e.g., average latency and error rate for each side). Simultaneously, a local topology slice centered on the fault-triggered object is cropped from the dynamic resource topology graph and assigned a topology version number. Finally, the metric statistical summary, call chain statistical summary, log index, topology slice, and version number are encapsulated into a read-only data object as fault analysis data.

[0031] As can be seen from the above, this embodiment ensures the alignment and compression of multi-source data in time and space, providing efficient and consistent input for subsequent analysis.

[0032] Furthermore, the present invention also specifies how to read data from the circular buffer before generating fault analysis data, which may include the following: The topological relationship changes between the monitored objects and the performance index statistics of each monitored object are recorded in the order of time windows and stored in the circular buffer in memory. When a fault event signal is received, a fault attribution task is generated according to the preset fault attribution time, the fault triggering object of the fault event signal and the fault triggering time. Based on the fault attribution task, the corresponding starting storage position is determined in the circular buffer according to the fault triggering time, and the data located in the fault attribution time period is read sequentially along the circular buffer from the starting storage position.

[0033] The circular buffer is a fixed-size memory area that stores data cyclically according to time windows, with new data overwriting the oldest data. The topology change record records the addition, deletion, and modification of call relationships between monitored objects within each time window. Performance metric statistics are the statistical values ​​of performance metrics within each time window, such as mean and maximum values. The fault attribution task is an analysis task that includes information such as the fault-triggered object and the attribution time period.

[0034] In this embodiment, to cope with the instantaneous query pressure of massive amounts of data, the topological relationship change records and performance index statistics of each monitored object within each window are stored in a circular buffer in memory, ordered by time window. When a fault event signal is received, a fault attribution task is generated based on the preset fault attribution time, the fault triggering object, and the trigger time. Based on this task, the corresponding starting storage position in the circular buffer is determined according to the fault trigger time, and then, starting from that position, data (including topological relationship change records and performance index statistics) within the fault attribution time period is read sequentially along the circular buffer. Figure 4 As shown, there are N time windows. The fault trigger time can be a fault trigger timestamp. The starting storage position determined by the fault trigger timestamp is window A. The next window in the circular buffer sequence is window A+1. Starting from window A, subsequent window data within the fault attribution time period are read sequentially. This slice reading method does not require loading all data from cold storage, achieving millisecond-level backtracking. This embodiment solves the problem of efficient access to incremental topology data, reduces the computational cost of full topology reconstruction, and effectively avoids unstable inference conclusions caused by topology jitter, such as Pod (container) eviction and reconstruction.

[0035] Based on the above embodiments, the present invention further defines the implementation process of constructing an anomaly relationship graph and dividing it into fault groups, which may include the following: An anomaly relationship graph is constructed using the abnormal states of monitored objects within the fault attribution period as graph nodes and the directed call paths between monitored objects corresponding to two abnormal states in the dynamic resource topology graph as node edges. The edge weights of each node edge in the anomaly relationship graph are determined based on the topology reachability component, temporal correlation component, and cross-modal co-occurrence component. A sparse similarity matrix is ​​constructed based on the edge weights of each node edge in the anomaly relationship graph. Clustering is performed on the anomaly relationship graph with the constraint that there is a directed call path between two graph nodes and that the anomaly start time of one graph node is earlier than the anomaly start time of the other graph node, to obtain at least one fault group.

[0036] In this graph, a node edge is an edge connecting two graph nodes, representing the association between two anomalous states. Edge weights are numerical values ​​quantifying the strength of the association between two anomalous nodes, obtained by weighted summation of three components: topological reachability, temporal correlation, and cross-modal co-occurrence. The topological reachability component is a Boolean value; it is 1 if there is a directed call path between the two graph nodes, and 0 otherwise. The temporal correlation component is the positive peak cross-correlation coefficient calculated through cross-correlation, reflecting the leading relationship between anomalous nodes. The cross-modal co-occurrence component represents the degree of overlap between three modalities of anomalous nodes (indicators, logs, and call chains) within the same time window. The sparse similarity matrix is ​​a matrix with anomalous nodes as rows and columns, and edge weights as elements; it is sparse because most node pairs have no strong association. Clustering is the process of grouping nodes based on the similarity matrix and constraints; any clustering algorithm can be used without affecting the implementation of this invention.

[0037] In this embodiment, firstly, an anomaly relationship graph is constructed using the abnormal states of the monitored objects during the fault attribution period as graph nodes, and the directed call paths between the monitored objects corresponding to two abnormal states in the dynamic resource topology graph as node edges. The three parameters—topology reachability component, temporal correlation component, and cross-modal co-occurrence intensity component—are quantified into numerical values. The topology reachability component (T) can be used as a Boolean hard constraint or a discrete value; for example, reachability is 1, and inaccessibility is 0. The temporal correlation component (C) is the calculated cross-correlation coefficient, typically ranging from [-1, 1], and its positive peak value is extracted. The cross-modal co-occurrence intensity component (M) is normalized to a value between 0 and 1 based on the overlap of the simultaneous occurrence of anomalies in three modalities. The calculation process is as follows: Within the same fault attribution time window, the deviation or rise slope of the performance indicator is mapped to the indicator anomaly degree, the occurrence density of the anomaly log template is mapped to the log anomaly degree, and the latency rise ratio or error rate change of the target call edge is mapped to the call chain anomaly degree. All three are normalized to the range of 0 to 1. Then, the three anomalies are multiplied by their corresponding preset weights (the sum of the three weights is 1) and summed to obtain a weighted sum. If any one of the indicator anomaly degree, log anomaly degree, or call chain anomaly degree is zero, the weighted sum is multiplied by a co-occurrence attenuation coefficient less than 1. The result is the cross-modal co-occurrence component; otherwise, the weighted sum is directly taken as the cross-modal co-occurrence component. This method ensures that the cross-modal co-occurrence component will only obtain a high value when multiple modalities undergo abrupt changes simultaneously within the same time window. The edge weight of each node is determined based on the topological reachability component, temporal correlation component, and cross-modal co-occurrence component: if the topology is unreachable, i.e., there is no actual call path in the physical topology, the edge weight is 0; otherwise, the edge weight = (α × temporal component + β × cross-modal component), where α and β are set weight hyperparameters, for example, α = 0.6, β = 0.4. Next, a sparse similarity matrix is ​​constructed based on the edge weights. Finally, a clustering operation is performed on the anomaly graph with the constraint that there is a directed call path between two graph nodes and that the anomaly start time of one graph node precedes that of the other, to obtain at least one fault group. This embodiment ensures the causal consistency of fault groups through physical topology and temporal hard constraints.

[0038] Based on the above embodiments, this embodiment further defines the method for determining node edges in the defined anomaly relationship graph, which may include the following: Centered on the fault-triggered object, perform a breadth-first search operation upstream and downstream in the dynamic resource topology graph, and select the two monitored objects corresponding to the searched directed call paths as candidate edges; obtain the call heat decay coefficient and observation coverage corresponding to each candidate edge from the dynamic resource topology graph; delete the candidate edges whose call heat decay coefficient is lower than the preset heat threshold or whose observation coverage is lower than the preset observation threshold, and retain the remaining candidate edges as node edges in the anomaly relationship graph.

[0039] Breadth-first search (BFS) is a graph algorithm that uses the fault-triggered object as the center and traverses the topology layer by layer, expanding upstream and downstream. Candidate edges are pairs of monitored objects corresponding to directed call paths obtained through BFS. A preset heat threshold is used to filter out edges with low activity. A preset observation threshold is used to filter out edges with severe data loss.

[0040] In this embodiment, a breadth-first search operation is performed upstream and downstream in the dynamic resource topology graph, centered on the fault-triggered object. This means that the basis for edge selection is the existence of a real call path in the physical topology (i.e., topological reachability). If two monitored objects have no physical call relationship, no connection will be established between their corresponding abnormal state nodes, and the two monitored objects corresponding to the searched directed call paths will be used as candidate edges. Not all physically connected edges will be retained. To control the computational scale and remove noise, the call heat attenuation coefficient and observation coverage of each edge will be evaluated. If two monitored objects have a dependency configuration but lack real traffic for a long time (heat decay) or have severe data loss (low observation coverage), this edge will be considered a low-relevance, low-reliability edge and will be automatically filtered out. In other words, the call heat decay coefficient and observation coverage of each candidate edge can be obtained from the dynamic resource topology graph. Candidate edges with call heat decay coefficients lower than the preset heat threshold or observation coverage lower than the preset observation threshold are deleted. The remaining candidate edges are retained as node edges in the abnormal relationship graph. Only the real and active dependency edges that have been found through the above search and retained through pruning logic will eventually be mapped to the edges between nodes in the abnormal relationship graph.

[0041] As can be seen from the above, this embodiment filters out low-relevance and low-reliability edges that have been without traffic for a long time or have serious data gaps through pruning logic, thereby controlling the computational scale and removing noise.

[0042] Based on the above embodiments, this embodiment further defines the method for calculating edge weights, which may include the following: For each node edge, if there is no directed call path between the first and second monitored objects at both ends of the current node edge in the dynamic resource topology graph, then the edge weight of the current node edge is zero; if there is a directed call path, then cross-correlation is performed on the time series of the first and second monitored objects to obtain the cross-correlation number sequence under different lag times; positive peaks and their corresponding lag times are extracted from the cross-correlation number sequence; the positive peaks are used as the temporal correlation components, and the lag times are used as the temporal sequence constraint basis. If the lag time is positive, it means that the anomaly of the first monitored object is earlier than that of the second monitored object; the deviation of the quantitative performance indicators, the density of log error templates, and the rise in call chain latency of the first and second monitored objects in the same time window are calculated respectively, and the three are weighted and summed and multiplied by the co-occurrence attenuation coefficient to obtain the cross-modal co-occurrence component; the topology reachability component is set to a preset value; the edge weight of the current node edge is determined according to the topology reachability component, the temporal correlation component and the corresponding first weight coefficient, and the cross-modal co-occurrence component and the corresponding second weight coefficient.

[0043] Cross-correlation is used to calculate the similarity between two time series at different time offsets, extracting the peak value and its lag time. The positive peak value is the maximum value (positive value) in the cross-correlation coefficient, representing the strongest positive correlation. The lag time is the offset corresponding to the peak value; a positive number indicates that the first series is earlier than the second series. The temporal correlation component is the value of the positive peak value. The temporal order constraint means that if the lag time is positive, the anomaly time of the first monitored object is earlier than that of the second monitored object. Performance metric deviation is represented by the Z-Score. The log error template occurrence density is the number of times the abnormal log template appears per unit time. The call chain latency increase ratio = (current latency - baseline latency) / baseline latency. The co-occurrence decay coefficient is a coefficient less than 1; when the anomaly degree of a certain modality is 0, this coefficient is multiplied to reduce the total score. The first weight coefficient and the second weight coefficient are the weight hyperparameters of the temporal component and the cross-modal component, respectively.

[0044] In this embodiment, for each node edge, if there is no directed call path between the two monitored objects in the dynamic resource topology graph, the edge weight is zero. If there is, cross-correlation is performed on the time series of the two ends to obtain the cross-correlation number sequence under different lag times. The positive peak and its corresponding lag time are extracted, and the positive peak is used as the time-series correlation component, and the lag time is used as the time-series priority constraint. A positive number indicates that the first object is ahead. At the same time, the deviation of performance indicators, the density of log error templates, and the increase ratio of call chain latency of the two objects within the same time window are calculated respectively. The three are weighted and summed and multiplied by the co-occurrence attenuation coefficient to obtain the cross-modal co-occurrence component. The topology reachability component is set to 1. Finally, the product of the time-series correlation component and the first weight coefficient, and the product of the cross-modal co-occurrence component and the second weight coefficient are summed and multiplied by the topology reachability component (e.g., 1) to obtain the edge weight of the node edge.

[0045] As can be seen from the above, this embodiment provides a quantitative calculation formula for edge weights, which provides a numerical basis for subsequent clustering.

[0046] Furthermore, before determining the edge weights, this embodiment defines a method for generating multimodal anomaly representations for the monitored objects corresponding to each graph node, which may include the following: The fault attribution period is defined as the fault attribution time window. The system acquires a sequence of performance metrics sampling points for the currently monitored object within this time window, extracts the mean and standard deviation under historical normal conditions, and calculates the anomaly significance score for each sampling point within the fault attribution time window. The anomaly significance score can be, for example, equal to the current sample value minus the historical mean, divided by the historical standard deviation. A linear fit is performed on the sampling point sequence to calculate the trend slope. If the trend slope is greater than a preset abrupt change threshold, it is considered a continuous upward trend. The system acquires the log templates for the currently monitored object, using the frequency of each log template within the historical baseline window as the baseline frequency and the frequency within the fault attribution time window as the current frequency. The ratio of the current frequency to the baseline frequency is calculated as the surge ratio. If the surge ratio is greater than a preset multiple threshold and the number of occurrences within the fault attribution time window is greater than a preset noise threshold, it is considered a surge signal. If the baseline frequency is zero and the number of occurrences within the fault attribution time window is greater than a preset noise threshold, it is considered a new template surge signal. The system acquires the target call edges that the currently monitored object participates in and calculates the latency increase ratio of the target call edges within the fault attribution time window. The target call edge refers to a call edge located on the main path of the reverse topology traversal from the fault-triggered object, where the proportion of local call time to the total end-to-end time exceeds a preset threshold.

[0047] The fault attribution time window is one way to implement the fault attribution time period. The anomaly significance score can be calculated as (current value - historical mean) / historical standard deviation. The trend slope can be obtained by fitting the time series using the least squares method; a positive slope indicates an upward trend. The preset mutation threshold is the threshold value for judging whether the slope is a sudden increase. The historical baseline window is a historical time period under normal conditions, used to calculate the baseline frequency. The baseline frequency is the frequency of log template occurrences within the historical baseline window divided by the window duration. The current frequency is the frequency of log template occurrences within the fault attribution time window divided by the window duration. The surge ratio is the ratio of the current frequency to the baseline frequency, with a small constant added to prevent division by zero. The preset multiple threshold is the lower limit of the multiple for judging a surge, such as 10 times. The preset noise threshold is the lower limit of the number of occurrences, such as 5 times. A new template surge signal refers to a log template that has never appeared in history (baseline frequency is zero) and whose current frequency exceeds the noise threshold. The target call edge is a call edge on the main path of the reverse topology traversal of the symptom object, where the proportion of local call time to the total end-to-end time exceeds a preset threshold. The latency increase ratio can be calculated using the formula: (current latency - baseline latency) / baseline latency.

[0048] In this embodiment, the following operations are performed on the metrics side: The performance metrics sampling point sequence of the monitored object within the fault attribution time window is obtained, the historical mean and standard deviation are extracted, the anomaly significance score (Z-Score) of each sampling point is calculated, and the trend slope is calculated by linear fitting of the sequence. If the slope is greater than a preset mutation threshold, it is determined to be a continuous upward trend. On the log side: The associated log templates are obtained, and the frequency of each template appearing within the historical baseline window is used as the baseline frequency. The frequency within the fault attribution time period is used as the current frequency, and the surge ratio is calculated. If the surge ratio is greater than a preset multiple threshold and the number of occurrences is greater than a preset noise threshold, it is determined to be a surge signal; if the baseline frequency is zero and the number of occurrences is greater than the noise threshold, it is determined to be a new template surge signal. On the call chain side: The target call edges in which the monitored object participates (edges located in the reverse traversal trunk and whose local time consumption exceeds the threshold) are obtained, and the latency increase ratio is calculated. The anomaly significance score and trend slope on the metric side are used to quantify the degree and direction of performance deviation; the surge ratio and new template signal on the log side are used to identify explosive growth in abnormal logs; and the latency rise ratio on the call chain side is used to measure the performance degradation of the target call edge. These quantitative features are subsequently used in three aspects: first, as input to cross-modal co-occurrence components to calculate the co-occurrence strength between two abnormal nodes; second, to determine the central node (the node with the strongest anomaly) within the fault group; and third, as the basis for generating root cause hypotheses (such as combining the start time of metric mutation, the time of log occurrence, and the start point of latency rise to determine the anomaly time anchor point).

[0049] Furthermore, the present invention also defines the calculation process of the cross-modal co-occurrence component, which may include the following: mapping the anomaly significance score or trend slope to the index anomaly degree, mapping the logarithm of the surge ratio or the flag bit of the new template surge signal to the log anomaly degree, and mapping the delay rise ratio to the call chain anomaly degree; multiplying the index anomaly degree, log anomaly degree, and call chain anomaly degree by their respective preset weights and then summing them to obtain a weighted sum; if any one of the index anomaly degree, log anomaly degree, and call chain anomaly degree is zero, then multiplying the weighted sum by the co-occurrence attenuation coefficient to obtain the cross-modal co-occurrence component; wherein, the co-occurrence attenuation coefficient is less than 1.

[0050] Specifically, the indicator anomaly score maps the significance score or trend slope to a value between 0 and 1; the log anomaly score maps the logarithm of the surge ratio or the new template surge signal flag to a value between 0 and 1; and the call chain anomaly score maps the latency increase ratio to a value between 0 and 1. The weighted sum is the sum of the log, indicator, and call chain anomalies multiplied by preset weights. The co-occurrence decay coefficient is a coefficient less than 1 multiplied by the weighted sum when the anomaly score of a certain modality is zero, reflecting the co-occurrence requirement.

[0051] In this embodiment, the anomaly significance score or trend slope is mapped to the index anomaly degree, the logarithm of the surge ratio or the flag of the new template surge signal is mapped to the log anomaly degree, and the latency rise ratio is mapped to the call chain anomaly degree. Then, these three are multiplied by their respective preset weights and summed to obtain a weighted sum. If any one of the index anomaly degree, log anomaly degree, or call chain anomaly degree is zero, the weighted sum is multiplied by a co-occurrence attenuation coefficient (less than 1) to obtain the cross-modal co-occurrence component; otherwise, the weighted sum is directly taken. This embodiment ensures that the cross-modal co-occurrence component only receives a high score when multiple modalities undergo abrupt changes simultaneously within the same time window, reflecting the overlap characteristic.

[0052] Furthermore, this embodiment also defines the process of generating metadata for each group after dividing the anomaly relationship graph into fault groups, which may include the following: For each fault group, the graph node with the highest degree of anomaly or the graph node at the target position of the fault propagation path is selected from the graph nodes contained in the current fault group as the center node of the current fault group; the anomaly start time of all graph nodes in the current fault group is obtained, and the anomaly start time is weighted and averaged according to the anomaly degree of each graph node to obtain the time centroid of the current fault group; the average edge weight of all node edges in the current fault group is used as the intra-group cohesion, and the average edge weight of the boundary node edges between graph nodes in the current fault group and graph nodes outside the group is used as the inter-group separation; the boundary confidence of the fault group is determined according to the ratio of intra-group cohesion to inter-group separation, and the larger the ratio, the more stable the division of the fault group; the center node, time centroid, and boundary confidence are used as metadata descriptors of the current fault group and marked.

[0053] In this system, the central node is the graph node with the highest anomaly level within the fault group or located at a critical position in the propagation path. The anomaly start time is the moment when the monitored object corresponding to the node begins to exhibit anomalies. The temporal centroid is the weighted average of all anomaly start times within the fault group, calculated according to the anomaly level of each node. Intra-group cohesion is the average edge weight of all nodes within the fault group. Inter-group separation is the average edge weight of the boundary node edges between nodes within and outside the fault group. Boundary confidence is the ratio of intra-group cohesion to inter-group separation, reflecting the robustness of the grouping. The metadata descriptor describes the data structure of the fault group characteristics, including the central node, temporal centroid, and boundary confidence.

[0054] In this embodiment, for each fault group, the graph node with the highest anomaly level or the graph node located at a critical position in the fault propagation path is selected as the center node from the graph nodes contained in the group. The anomaly start time of all graph nodes within the group is obtained, and the weighted average of each anomaly start time according to the anomaly level of each graph node is calculated to obtain the time centroid of the group. The average edge weight of all node edges within the group is calculated as the intra-group cohesion, and the average edge weight of the boundary node edges between nodes within and outside the group is calculated as the inter-group separation. The boundary confidence is determined based on the ratio of intra-group cohesion to inter-group separation; the larger the ratio, the more robust the group. The center node, time centroid, and boundary confidence are used as metadata descriptors for this group.

[0055] As can be seen from the above, this embodiment provides a quantitative description for each fault group, which facilitates subsequent reasoning and scheduling.

[0056] Based on the above embodiments, this embodiment further defines the process of generating candidate fault source location results for each fault group through a retrieval operation, which may include the following: Each fault group is configured with an independent state machine. The state machine records the known data types collected and the missing unknown data types for the current fault group, such as performance indicator information, log information, call chain data, or change event data. For each fault group, if the unknown data type is log information, and time series analysis shows an upstream-downstream anomaly relationship but the upstream monitored object lacks log evidence, then a log template retrieval is performed, using an inverted index to retrieve target log templates containing specific error codes or anomaly target words. If the unknown data type is performance indicator information, and call chain data shows that local calls have slowed down but the corresponding performance indicators have not changed significantly, then for the monitored objects associated with the slowed call edges in the call chain, pull and lock waits or thread queue lengths are implemented within the fault attribution time period. The system retrieves performance metrics slices related to degree; if the unknown data type is call chain data, it extracts the target call path from the call chain database that meets the conditions of sudden increase in latency, increase in error rate, or call topology offset within the fault attribution period; if the unknown data type is change event data, it queries the configuration management platform or continuous integration pipeline to obtain the most recent change event record; it stores each retrieval result into the state machine and triggers a re-analysis of the missing data type in the current fault group until the preset iteration termination condition is met, so as to determine the corresponding candidate fault source location result based on the data already stored in the state machine, and the set of missing data types recorded in the state machine is empty; multiple consecutive retrieval operations have not stored any new data type in the state machine; the number of times the retrieval operation is executed reaches the preset upper limit.

[0057] The state machine maintains a logger for each fault group, tracking collected and missing data types. Known data types are performance metrics, logs, call chains, or change data already retrieved through retrieval. Unknown data types are missing data types that have not yet been retrieved. Log template retrieval uses an inverted index to find log templates with specific error codes or keywords. Performance metric slices are fine-grained metrics related to lock wait times or thread queue lengths. Target call paths are the backbone of call chains that meet conditions such as sudden increases in latency, increased error rates, or call topology offsets. Change event logs are the most recent changes retrieved from the configuration center or CI (Continuous Integration) / CD (Continuous Deployment / Continuous Delivery) pipelines. Iteration termination conditions are an empty set of missing data types, no new data retrieved after multiple consecutive searches, or reaching a preset limit on the number of searches.

[0058] In this embodiment, an independent state machine is configured for each fault group to record the collected data types and missing unknown data types (performance metrics, logs, call chains, or change events). Based on the missing type, corresponding searches are performed: if logs are missing and time-series analysis shows upstream and downstream anomalies but upstream logs are missing, a log template search is performed, using an inverted index to find the target log template; if performance metrics are missing and the call chain slows down but the metrics remain unchanged, a metric slice related to lock waiting or thread queue length is retrieved; if call chains are missing, target call paths satisfying sudden increases in latency, increased error rates, or topology shifts are extracted from the call chain database; if change events are missing, the configuration center or CI / CD pipeline is queried to obtain the most recent change records. Each search result is stored in the state machine, triggering a re-analysis of the missing data type until termination conditions are met (e.g., the missing set is empty, no new data is found consecutively, or the maximum number of attempts is reached). Finally, the candidate fault source location results are determined based on the data stored in the state machine.

[0059] As can be seen from the above, this embodiment realizes iterative retrieval prioritizing evidence gaps, thereby improving reasoning efficiency.

[0060] Based on the above embodiments, the present invention further defines how to extract the target call path from the call chain database, which may include the following: Starting with the symptom object, the call chain is traced hop by hop in reverse. During the tracing process, if the latency increase ratio of the current call edge exceeds a preset latency threshold, the current call edge and its associated objects are connected to form a backbone link as the target call path. If the error code ratio of the current call edge exceeds a preset error rate threshold, the current call edge and its associated objects are connected to form a backbone link as the target call path. If the traffic allocation ratio offset of the current call edge exceeds a preset offset threshold, the current call edge and its associated objects are connected to form a backbone link as the target call path.

[0061] Among these, reverse hop-by-hop tracing involves tracing upstream nodes from the symptom object along the call chain. A preset latency threshold is used to determine the lower limit of the percentage increase in latency spikes. The error code percentage is the proportion of error codes returned by the calling edge out of the total number of calls. A preset error rate threshold is used to determine the lower limit of the percentage increase in the error rate. The traffic allocation ratio offset is the difference between the current proportion of traffic allocated to a path and the historical pattern. A preset offset threshold is used to determine the lower limit of the offset amount for topology offsets.

[0062] In this embodiment, starting from the symptom object, hop-by-hop tracing is performed in reverse along the call chain. During the tracing process, if the latency increase ratio of the current call edge exceeds a preset latency threshold, the call edge and its associated objects are connected into a backbone link as the target call path; if the error code ratio exceeds a preset error rate threshold, the same connection is made; if the traffic allocation ratio offset exceeds a preset offset threshold, the same connection is made. This process can satisfy multiple conditions simultaneously.

[0063] As can be seen from the above, this embodiment extracts the key call paths with significant abnormal characteristics, which identifies the key links for root cause analysis and helps to improve the accuracy and efficiency of root cause analysis.

[0064] Based on the above embodiments, this embodiment further defines the process of determining the location result of candidate fault sources based on the data stored in the state machine, which may include the following: For each fault group, during the process of generating at least one candidate fault source location result based on the data already stored in the corresponding state machine, the candidate fault triggering object identifier, the abnormal start time point of the candidate fault triggering object, the fault cause type of the candidate fault triggering object, and the supporting evidence data of the fault source determined by the current candidate fault source location result are recorded.

[0065] The candidate fault triggering object identifier refers to the monitored object ID of the candidate fault source, and the anomaly start time is the moment when the anomaly of the candidate fault source begins. Fault cause types include resource saturation, code defects, and network congestion. Supporting evidence data includes indicator curves, log samples, and call chain summaries that support this localization result.

[0066] In this embodiment, for each fault group, based on the data stored in the state machine, including indicators, logs, call chains, and changes, when generating the candidate fault source location result, the candidate fault triggering object identifier, abnormal start time point, fault cause type, and supporting evidence data corresponding to the fault source determined by the result are recorded.

[0067] As can be seen from the above, this embodiment ensures that each candidate result has clear evidence to cite, which facilitates subsequent verification.

[0068] Based on the above embodiments, this embodiment further defines the process of selecting the root cause of a fault from the candidate fault source location results, which may include: For each candidate fault source location result, query the dynamic resource topology graph of the current candidate fault source location result to see if there is a directed connected path between the fault source and the symptom object. If not, mark the verification as failed. If a directed connected path exists between the fault source and the symptom object, compare the abnormal start time of the fault source with the abnormal start time of the symptom object. If the abnormal start time of the fault source is not earlier than the abnormal start time of the symptom object, or if the difference between their abnormal times exceeds a preset lag interval, mark the verification as failed. If the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object, then... The system checks whether the abnormal data supporting the current candidate fault source location result contains at least two of the following: abnormal performance index data, abnormal log data, and abnormal call chain data. If not two, the verification fails. If at least two are present, the current candidate fault source location result is used as the fault source location result. If any one of the verifications fails, the current candidate fault source location result is marked as pending verification, and a corresponding supplementary retrieval operation is triggered based on the failure type. If the verification still fails after a preset number of supplementary retrievals and re-verifications, the result with the most verified items is selected from all candidate fault source location results as the root cause of the fault, and the failure reason for the failed item is recorded.

[0069] Here, a directed connected path is a call chain with a consistent direction from the source node of the fault to the symptom object. The preset lag interval is a time difference range preset based on the fault type. The pending verification status is the state requiring supplementary retrieval after verification failure. Supplementary retrieval operations are additional searches performed for failure types (such as time sequence reversal or path unreachable). Degradation selection is the result that passes the most verification items selected as the root cause of the fault after multiple supplementary verification attempts fail.

[0070] In this embodiment, for each candidate result, the system first queries the dynamic resource topology graph to determine if a directed connected path exists between the fault source and the symptom object. If no such path exists, the verification fails. If a path exists, the system compares the anomaly start time: if the fault source is not earlier than the symptom or the time difference exceeds a preset lag interval, the system marks the path as failed; if it is earlier, the system checks if the supporting data simultaneously contains at least two of the following: indicators, logs, and call chains. If less than two, the system marks the path as failed. If both conditions are met, the result is used as the fault source location result. If any item fails, the result is marked as pending verification, and supplementary searches are triggered based on the failure type (e.g., expanding the time window or resetting cluster boundaries). If the system still fails after a preset number of supplementary searches and re-verifications, the result with the most verified items is selected from all candidate results as the root cause of the fault, and the failure reason for the failed item is recorded. This embodiment implements a strict causal consistency verification and degradation strategy.

[0071] Furthermore, the present invention also defines a feedback loop following the triggering of a supplementary retrieval operation based on the failure type, which may include the following: If the failure is due to time sequence reversal, the search time window is expanded and the corresponding search operation is re-executed. If the failure is due to path verification failure, and the analysis finds that the root cause is outside the current fault group boundary, nodes that meet the abnormal conditions outside the boundary are included in the current fault group, and the fault group division is re-executed. If the verification still fails after supplementary search and regrouping, the ranking score of the current candidate fault source location result is reduced according to the score adjustment rules, and the ranking score of the candidate fault source location result that passed the verification is increased.

[0072] Among these, time sequence reversal means the abnormal start time of the fault source is later than the symptom, or the lag time is unreasonable. Expanding the retrieval time window involves extending the retrieval time range in both directions to capture earlier or later evidence. Path verification failure means there is no directed connected path between the fault source and the symptom. Resetting the cluster boundary involves including nodes outside the boundary that meet the abnormal conditions into the current fault group and re-dividing the cluster. Ranking and scoring is a comprehensive scoring of candidate results based on the verification results, with higher scores for those that pass the verification.

[0073] In this embodiment, if the failure is due to time sequence reversal, the retrieval time window is expanded and the retrieval operation is re-executed. If the failure is due to path verification failure and analysis reveals that the root cause is outside the current fault group boundary, nodes outside the boundary that meet the abnormal conditions are included in the current fault group, and the fault group division is re-executed. If the verification still fails after supplementary retrieval and regrouping, the ranking score of the current candidate fault source location result is reduced according to the score adjustment rules, and the ranking score of the candidate results that pass the verification is increased. This embodiment achieves adaptive adjustment of inference through feedback loop.

[0074] Furthermore, the present invention also defines a sorting fusion and report generation process, which may include the following: For each fault group, a comprehensive score is determined based on the anomaly salience of graph nodes within the fault group, the topological distance between graph nodes and symptom objects, and the hysteresis correlation strength between graph nodes. A reverse random walk is performed on the anomaly relationship graph, starting from the symptom object and hopping backward along the call relationship, increasing the score weight of graph nodes that are upstream in the call chain and have high anomaly salience. The candidate fault source location result with the highest comprehensive score in each fault group is taken as the final location result for the corresponding fault group, and a fault report is generated that includes the fault source identifier, anomaly start time point, fault type, propagation path highlighting, and supporting data summary. When multiple fault groups exist, it is checked whether each fault group shares the same change event record or the same underlying resource anomaly. If so, multiple fault groups are merged into a common cause view; otherwise, their independent outputs are maintained. The fault report, the window identifier corresponding to the fault attribution time period, and the filtering rules used in the retrieval process are permanently saved.

[0075] Here, anomaly significance refers to the degree of anomalousness of a node, such as its Z-score. Topological distance is the number of hops or path length between a graph node and a symptom object in the topological graph. Lagged correlation strength is the numerical value of the time-series correlation component. The overall score is a weighted score combining anomaly significance, topological distance penalty, and lag correlation strength. Reverse random walk is a random walk that starts from the symptom object and proceeds in the reverse direction of the call relationship, increasing the weight of upstream nodes. Propagation path highlighting highlights the fault propagation links in the topological graph. The common cause view merges multiple fault groups into a unified view when they share the same change or underlying resource.

[0076] In this embodiment, for each fault group, a comprehensive score for each candidate result is determined based on the anomaly salience of nodes in the graph within the group, the topological distance between the node and the symptom, and the strength of the hysteresis correlation. A reverse random walk is performed on the anomaly relationship graph, starting from the symptom object and moving backward hop-by-hop along the call relationship, increasing the score weight of upstream nodes with high anomaly salience. The candidate result with the highest comprehensive score in each group is used as the final location result, generating a fault report containing the fault source identifier, anomaly start time point, fault type, highlighted propagation path, and supporting data summary. When multiple fault groups exist, it is checked whether each group shares the same change event or underlying resource anomaly; if so, they are merged into a common cause view; otherwise, they are output independently. Finally, the fault report, attribution time window identifier, and filtering rules used in the retrieval process are permanently saved. This embodiment provides a final output visualization report and knowledge archive.

[0077] Furthermore, the present invention also specifies how to handle abnormal situations, which may include the following: When a first-type directed topological edge with an observation coverage lower than a preset coverage threshold is detected in the dynamic resource topological graph, the weight of the first-type directed topological edge in the clustering and verification processes is reduced according to a preset weight adjustment rule. When more than a preset number of fault event signals are received, the priority of the business links corresponding to each fault event signal is identified, and the fault event signals of the higher-priority business links are grouped first. When a second-type directed topological edge with an occurrence frequency exceeding a preset threshold and an alternation frequency of occurrence and disappearance exceeding a preset flip threshold is detected in the dynamic resource topological graph, a topological uncertainty marker is added to the verification result corresponding to the second-type directed topological edge. If multiple occurrence frequencies are higher than the highest threshold, a topological uncertainty marker is added. If the difference between the current number of attempts is less than a preset difference threshold for a third-type target directed topology edge, then the fault propagation path corresponding to each third-type target directed topology edge is listed in the fault report. When the retrieval service is detected to be unavailable, the step of generating candidate fault source location results for each fault group through the retrieval operation is paused, and the telemetry snapshot summary of the most recent preset number of time windows is read from the local circular buffer. The telemetry snapshot summary includes the statistical characteristics of performance indicators, the index of log templates, and the aggregated data of the call chain. Based on the telemetry snapshot summary, the preliminary fault location result carrying the low confidence mark is determined. When the retrieval service is detected to be restored to normal, the step of generating candidate fault source location results for each fault group through the retrieval operation is re-executed, and the preliminary fault location result is updated.

[0078] Among them, business link priority is the processing priority divided according to business importance; occurrence count is the total number of times a directed topology edge appears within the statistical time window; alternation frequency is the switching frequency of the appearance and disappearance of directed edges; preset flip threshold is the lower limit of the alternation frequency for judging frequent flips; topology uncertainty marker is the uncertainty information recorded in the verification ledger. Retrieval service unavailable indicates a failure of downstream storage or retrieval services. The local circular buffer is a summary of telemetry snapshots from the most recent multiple time windows cached in the edge node's memory. Low confidence marker is a preliminary result marker for output in degraded mode.

[0079] In this embodiment, when a first-type target directed topological edge with an observation coverage lower than a preset coverage threshold is detected in the dynamic resource topology graph, the weight of this edge in clustering and verification is reduced according to preset rules. When more than a preset number of fault event signals are received, the priority of the service links corresponding to each signal is identified, and fault groups of high-priority links are processed first. The activity status of each directed topological edge in the dynamic resource topology graph is monitored with a preset statistical time window as the length, and the directed topological edge with the highest occurrence frequency is selected as a stable dependency edge, while temporary dependency edges with an occurrence frequency lower than a preset threshold are ignored. When the occurrence frequency of a directed topological edge exceeds a preset threshold and the alternation frequency exceeds a preset flip threshold, a topological uncertainty mark is added to the corresponding verification result; if there are multiple target edges whose occurrence frequency differs from the highest frequency by less than a preset difference threshold, the fault propagation path corresponding to each edge is listed in the fault report. That is, when multiple dependency edges with similar occurrence frequencies cannot be uniquely determined, multiple possible fault propagation paths are provided in the final output report. When the retrieval service is unavailable, the generation of candidate fault source location results is paused. Instead, telemetry snapshot summaries (including indicator statistical features, log indexes, and call chain aggregation data) from multiple recent time windows are read from the local circular buffer. Based on these summaries, preliminary location results marked with low confidence are determined. Once the retrieval service is restored, the normal retrieval process is re-executed, and the preliminary results are updated.

[0080] As can be seen from the above, this embodiment ensures the robustness of the system under extreme conditions such as network jitter, high concurrency, and service unavailability.

[0081] It should be noted that there is no strict order of execution between the steps in this invention. As long as they conform to the logical order, these steps can be executed simultaneously or in a certain preset order. Figure 1 This is just an illustrative example and does not mean that this is the only possible execution order.

[0082] This invention also provides a corresponding apparatus for the root cause localization method, further enhancing the practicality of the method. The apparatus can be described from both a functional module perspective and a hardware perspective. The following describes the root cause localization apparatus provided by this invention, which is used to implement the root cause localization method provided by this invention. In this embodiment, the root cause localization apparatus may include or be divided into one or more program modules. These program modules are stored in a storage medium and executed by one or more processors to complete the root cause localization method disclosed in the embodiments. The program module referred to in this embodiment is a series of computer program instruction segments capable of performing specific functions, which are more suitable than the program itself for describing the execution process of the root cause localization apparatus in the storage medium. The following description will specifically introduce the functions of each program module in this embodiment. The root cause localization apparatus described below can be referred to in correspondence with the root cause localization method described above.

[0083] From the perspective of functional modules, see Figure 4 , Figure 4 This is a structural diagram of the fault root cause location device provided in this embodiment under one specific implementation. The device may include: The telemetry snapshot generation module 401 is used to generate fault analysis data with read-only attributes based on multimodal telemetry data and resource topology data located within the fault attribution time period. The multi-fault group decomposition module 402 is used to construct an anomaly relationship graph with anomaly states as graph nodes based on fault analysis data, and divide the anomaly relationship graph into at least one fault group. There are physical topological reachable paths and temporal causal relationships between graph nodes in the same fault group, and the association strength of graph nodes between different fault groups is lower than a preset association threshold. The root cause analysis module 403 is used to generate candidate fault source location results for each fault group through retrieval operations, and select the candidate fault source location results that simultaneously meet the following conditions: there is a directed connected path between the indicated fault source and the symptom object in the resource topology; the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object; and there are at least two different modalities of abnormal data that simultaneously support the location results as the root cause of the fault.

[0084] The aforementioned fault root cause localization device is described from the perspective of functional modules. Furthermore, the present invention also provides an electronic device, described from a hardware perspective. This electronic device includes a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-described fault root cause localization method embodiments.

[0085] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the root cause localization method for faults when it is run.

[0086] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0087] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described embodiments of the root cause fault location method.

[0088] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described embodiments of the root cause fault location method.

[0089] Finally, the present invention also provides a root cause fault location system, see [link to documentation]. Figure 5 The root cause localization system includes a root cause inference processor deployed between the data acquisition terminal and the operation and maintenance display terminal; the root cause inference processor is used to implement the steps of root cause localization as described in any of the above embodiments when executing a computer program stored in the memory.

[0090] Among them, such as Figure 6 As shown, the data acquisition end includes data processors deployed on various computing nodes. For example, there are M computing nodes and M data processors. The first data processor is deployed on the first computing node, the second data processor on the second computing node, and the Mth data processor on the Mth computing node. These data processors are used to collect and report telemetry data. For instance, the data acquisition end can collect and report telemetry data through a collection agent. The collection agent can be deployed on one of the data processors, such as the Mth data processor. Other data processors can send their collected data to the collection agent through a distributed message queue. The operation and maintenance display end is a user interface or monitoring dashboard used to display fault reports. The root cause inference processor is an intermediate layer server or container deployed between the data acquisition end and the operation and maintenance display end, executing methods for locating the root cause of faults.

[0091] The root cause analysis system supports parallel operation with existing observable systems. As long as the existing monitoring system provides standard metrics, log templates, and trace summary interfaces, it can be integrated into this system. In traditional brownfield scenarios, users can initially enable a lightweight mode with snapshots, clustering, and conservative verification to achieve basic root cause assistance at low cost. As data becomes more comprehensive, it can be gradually expanded to a full-featured cluster-level agent and a supplementary evidence closed-loop mode. During system upgrades, the isolation mechanism between snapshot versions and rule versions allows for the parallel implementation of historical and new strategies, ensuring cross-version comparison and verification during fault replay and review.

[0092] For example, the root cause inference processor is deployed in the management cluster or runs as a software-as-a-service in a separate operations domain; the root cause inference processor receives fault event signals from the data acquisition end and the corresponding multimodal telemetry data by subscribing to message queues or streaming channels.

[0093] The management cluster is the server cluster responsible for the control plane in the cloud system. Software as a Service (SaaS) provides operational and inference capabilities to multiple tenants as an independent service, eliminating the need for independent deployment in each tenant environment. Message queues are middleware used for asynchronous transmission of events and data, such as Kafka (message queue name). Streaming channels are channels for continuously transmitting data streams.

[0094] In this embodiment, the root cause inference processor runs as an independent RCA (Root Cause Analysis) inference service cluster. This service can be deployed within a management cluster or run as a SaaS (Software as a Service) in an independent operations domain. It receives trigger events (such as alarm signals) and corresponding time-window data snapshots from the data plane in real time by subscribing to message queues or streaming channels. Internally, the root cause inference processor does not directly rely on the real-time query capabilities of the underlying database. Instead, it performs high-intensity inference computation based on dynamic graphs in memory and cached snapshots, ensuring high throughput and low latency in the inference process.

[0095] Furthermore, the root cause reasoning processor has a snapshot storage area and a feature cache area, and the snapshot storage area and the feature cache area are isolated from each other by an independent locking mechanism. The snapshot storage area is configured as a read-only mode during the fault location reasoning process and is used to store fault analysis data. The feature cache area supports read and write operations and is used to store feature data extracted from the fault analysis data and intermediate results of retrieval operations.

[0096] The snapshot storage area is a read-only memory region used to store fault analysis data (frozen snapshots). The feature cache area is a read-write memory region used to store feature data extracted from the fault analysis data and intermediate results of retrieval operations. The read-write locking mechanism is an independent locking mechanism that isolates the snapshot storage area from the feature cache area, ensuring that the snapshot storage area is not written to during inference.

[0097] This embodiment further defines the internal storage structure of the root cause inference processor, which includes a snapshot storage area and a feature cache area, with read / write isolation between the two achieved through independent locking mechanisms. The snapshot storage area is configured in read-only mode during fault location inference and is used to store fault analysis data (i.e., frozen telemetry snapshots). The feature cache area supports read and write operations and is used to store feature data extracted from the fault analysis data and intermediate results of retrieval operations. This embodiment ensures the consistency of the view during inference and prevents interference from external data writes.

[0098] Finally, based on some possible application environments for the root cause fault location method, this invention also provides an exemplary application scenario embodiment to describe the entire process, which may include the following: In a private cloud Kubernetes (open-source container orchestration system) cluster environment, the underlying physical resources include several high-performance host machines, and the upper layer runs a group of business microservices consisting of gateway services, order services, inventory services, payment services, etc., as well as a middleware cluster (caching, database, message queue). The root cause inference processor of this invention can be deployed as a group of independent containers in a dedicated operation and maintenance management namespace. The data plane reuses the existing metric collection monitored objects, log collection monitored objects, and tracing monitored objects of the cluster as the underlying data source, and interfaces with them through adapters.

[0099] For application scenarios with high network traffic, users reported that the order submission interface was extremely slow to respond, and even timed out. The monitoring system detected that the order service's P99 (meaning that 99% of requests or operations take less than or equal to this value) latency exceeded the preset threshold (2 seconds) and lasted for 3 minutes, triggering an alarm and sending a trigger signal to the control plane.

[0100] The root cause inference processor immediately establishes a fault attribution task, setting the fault time window to 2 minutes before and 3 minutes after the fault (e.g., 13:58 to 14:03). The snapshot manager slices the topology of this time period from the memory circular buffer and constructs an accurate service dependency graph at that time by querying the historical event interface of the container orchestration platform and the traffic data interface of the service mesh. The system discovers that a container instance of an inventory service drifted after restarting within the fault time window. The snapshot manager accurately fixes the IP address mapping before and after the drift in the snapshot, avoiding log misalignment due to IP reuse. Subsequently, the system initiates concurrent queries to various data sources to retrieve key metric summaries (CPU, memory), error log template identifier sequences, and call chain statistics (such as the average time of each call edge) within this window. All data is tagged with a unique snapshot version number, and all subsequent inference is performed only on this read-only copy.

[0101] At the moment the fault was triggered, the system simultaneously received dozens of alarms, including order delays, payment service errors, slow response times in the recommendation service, and high disk I / O on a certain underlying host machine. The system mapped all abnormal signals onto the resource topology graph and calculated the proximity between abnormal nodes. Topology reachability analysis revealed a strong dependency between the order service and the inventory service, while the recommendation service had no direct call chain with the order service. Lag correlation calculations showed that the latency spike in the inventory service (13:58:10) occurred slightly earlier than the latency spike in the order service (13:58:15), with a correlation coefficient as high as 0.9, while the abnormal time point of the recommendation service had no significant statistical correlation with them. The clustering algorithm clustered the order service, inventory service, and the underlying host machine A into fault group one, and assigned the recommendation service to fault group two, successfully isolating irrelevant noise.

[0102] The orchestrator initiates an inference agent for each faulty group. The agent identifies the group center as the inventory service and initiates multiple rounds of investigation. The first round retrieves call chain summaries, revealing that the remote call time from order to inventory has spiked from 20 milliseconds to 3000 milliseconds, accompanied by numerous timeouts. The second round retrieves resource metrics for the inventory service container, finding normal CPU utilization but significantly increased network I / O latency. The third round retrieves the inventory service logs, finding no application-layer error logs but numerous connection timeout warnings. The fourth round retrieves related facilities, checking host machine A where the inventory service resides, discovering that its disk I / O utilization reached 100% saturation at 13:58. Further investigation of change events reveals that another non-business process on this host machine initiated a full backup at that time.

[0103] The agent generates candidate fault source localization results, concluding that disk I / O congestion on host A is causing slow response times for the inventory service running on it, thus dragging down the upstream order service. The verification module performs three hard constraint checks: a topology consistency check confirms that the inventory service is scheduled on host A and the physical path is reachable; a timing consistency check confirms that the backup task started (13:58:00) before the host I / O spike (13:58:05), before the inventory service slowed down (13:58:10), and before the order service timed out (13:58:15), with the lag within a reasonable range; a multimodal consistency check confirms that the evidence chain includes underlying indicators (IO saturation), intermediate layer logs (connection timeout), and upper layer call chains (increased latency), with the three modalities mutually corroborating each other. The verification passes.

[0104] After successful verification, the root cause inference processor executes an evidence minimization strategy, removing a large number of duplicate timeout logs and retaining only one representative log sample; it also removes normal CPU and memory curves, highlighting only the comparison chart of disk I / O and application interface latency; and generates a natural language description: "Host A was found to have exhausted its I / O resources due to a backup task, causing delays in inventory service processing and ultimately resulting in order service timeouts." This report is pushed to the operations and maintenance dashboard and automatically suggests limiting the I / O bandwidth of backup tasks or migrating backup times.

[0105] If, during the verification process, the error time in the inventory service's logs is found to be earlier than the host machine's IO spike time (timing inversion), the verifier will determine that the candidate result fails. The control plane will trigger a backtracking mechanism, the agent will expand the search scope, re-examine whether the problem is caused by a code defect in the inventory service itself (such as a deadlock), and restart a new round of hypothesis verification until an explanation that meets all physical constraints is found.

[0106] It should be noted that the above application scenarios are shown only to facilitate understanding of the ideas and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0107] As shown above, this embodiment freezes multimodal telemetry data and dynamic resource topology within the fault attribution time period into read-only fault analysis data, ensuring the consistency and reproducibility of the analysis input and avoiding data inconsistency issues caused by container drift or topology changes. Based on this data, an anomaly relationship graph is constructed and divided into fault groups, forcing nodes within the group to satisfy physical topology reachability and temporal causality, thereby isolating independent anomaly signals in concurrent faults and reducing interference from irrelevant anomalies on root cause localization. Candidate fault sources are generated through retrieval, and the true root cause of the fault is selected from the candidate results using three hard constraints supported by topological directed connectivity path, anomaly time sequence, and at least two modal anomaly data. This distinguishes between statistical correlation and true causality, significantly improving the accuracy and reliability of root cause localization. At the same time, the output localization results are accompanied by traceable paths and modal evidence, reducing the cost of manual review and the false alarm rate. Furthermore, this invention adopts an architecture of passive data acquisition on the data plane and centralized inference on the control plane. All complex calculations are performed in an independent intermediate layer without consuming business resources. It also significantly reduces network transmission bandwidth and storage pressure through the minimum evidence set strategy, demonstrating excellent engineering performance.

[0108] The foregoing has provided a detailed description of a method and system for locating the root cause of a fault, as provided by this invention. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Whether the units and algorithm steps of the various examples described in the disclosed embodiments are executed in electronic hardware or computer software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, and such implementations should not be considered beyond the scope of this invention. Several improvements and modifications can be made to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A method for locating the root cause of a fault, characterized in that, include: Based on multimodal telemetry data and resource topology data located within the fault attribution time period, generate fault analysis data with read-only attributes; Based on the fault analysis data, an anomaly relationship graph is constructed with anomaly states as graph nodes, and the anomaly relationship graph is divided into at least one fault group. There are physical topological reachable paths and temporal causal relationships between graph nodes in the same fault group, and the association strength of graph nodes between different fault groups is lower than a preset association threshold. The retrieval operation generates candidate fault source location results for each fault group. From each candidate fault source location result, the candidate fault source location result that simultaneously meets the following conditions is selected as the root cause of the fault: the indicated fault source and the symptom object have a directed connected path in the resource topology; the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object; and there are at least two different modalities of abnormal data that simultaneously support the location result.

2. The method for locating the root cause of a fault according to claim 1, characterized in that, Prior to the multimodal telemetry data and resource topology data located within the fault attribution time period, the following is also included: Using the monitored objects as topological vertices and the call relationships between the monitored objects as directed topological edges, a dynamic resource topology graph is constructed in memory, and each directed topological edge is configured with a valid time interval, a call heat decay coefficient, and an observation coverage. The event interface of the container orchestration platform is monitored in real time. When the creation, destruction or migration event of the monitored object is detected, the topological vertices and their associated directed topological edges of the dynamic resource topology graph are added, deleted or updated accordingly. The system monitors the traffic data interface of the service mesh in real time. When a change in the call relationship is detected, the system adds or updates the directed topology edges of the dynamic resource topology graph and resets the call heat decay coefficient of the corresponding directed topology edges. The effective time interval records the start and end times of the call relationship, and when tracing back historical faults, the directed topological edge whose fault time point falls within the effective time interval is used; the call heat decay coefficient decreases exponentially over time; the observation coverage is used to represent the completeness of the monitoring data of the two topological vertices of the directed topological edge, and the confidence weight of the directed topological edge is positively correlated with the value of the corresponding observation coverage.

3. The method for locating the root cause of a fault according to claim 1, characterized in that, Based on multimodal telemetry data and resource topology data located within the fault attribution time period, fault analysis data with read-only attributes is generated, including: When a fault event signal is received, the fault attribution time period is determined based on the preset fault attribution time and the fault trigger time of the fault event signal. Obtain the fault triggering object corresponding to the fault event signal, its performance index information, log information and call chain data within the fault attribution time period, and align them on a unified time axis. For performance index information with a sampling interval greater than a preset sampling interval threshold, interpolation is used to fill in the points; the log information is downsampled; and missing segments of the log information are filled in linearly. The interpolated performance metrics are compressed into a metric statistical summary, the filled log information is classified into log templates and corresponding log indexes are established, and the aligned call chain data is compressed into a call chain statistical summary. Cut out the target topology slice centered on the fault-triggered object from the dynamic resource topology map and assign it a corresponding topology version number; The statistical summary of the indicators, the statistical summary of the call chain, the log index, the target topology slice, and the topology version number are encapsulated and given read-only attributes to obtain fault analysis data.

4. The method for locating the root cause of a fault according to claim 3, characterized in that, Prior to the multimodal telemetry data and resource topology data located within the fault attribution time period, the following is also included: The topological relationship changes between the monitored objects within each time window and the performance index statistics of each monitored object are recorded and stored in a circular buffer in memory in the order of time windows. When a fault event signal is received, a fault attribution task is generated based on the preset fault attribution time, the fault triggering object of the fault event signal, and the fault triggering time. Based on the fault attribution task, the corresponding starting storage position is determined in the circular buffer according to the fault trigger time, and data within the fault attribution time period is read sequentially along the circular buffer starting from the starting storage position.

5. The method for locating the root cause of a fault according to any one of claims 1 to 4, characterized in that, Based on the fault analysis data, an anomaly relationship graph is constructed with abnormal states as nodes, and the anomaly relationship graph is divided into at least one fault group, including: An anomaly relationship graph is constructed using the abnormal states of the monitored objects during the fault attribution period as graph nodes and the directed call paths between the monitored objects corresponding to two abnormal states in the dynamic resource topology graph as node edges. The edge weights of each node edge in the anomaly graph are determined based on the topological reachability component, the temporal correlation component, and the cross-modal co-occurrence component. A sparse similarity matrix is ​​constructed based on the edge weights of each node in the abnormal relationship graph. Clustering is performed on the abnormal relationship graph with the constraints that there is a directed call path between two graph nodes and that the abnormal start time of one graph node is earlier than the abnormal start time of the other graph node, to obtain at least one fault group.

6. The method for locating the root cause of a fault according to claim 5, characterized in that, Based on the fault analysis data, an anomaly relationship graph is constructed with anomaly states as nodes, including: Centered on the fault-triggered object, perform a breadth-first search operation upstream and downstream in the dynamic resource topology graph, and take the two monitored objects corresponding to the searched directed call path as candidate edges. From the dynamic resource topology graph, obtain the call popularity decay coefficient and observation coverage corresponding to each candidate edge; Candidate edges whose call popularity decay coefficient is lower than a preset popularity threshold or whose observation coverage is lower than a preset observation threshold are deleted, and the remaining candidate edges are retained as node edges in the abnormal relationship graph.

7. The method for locating the root cause of a fault according to claim 5, characterized in that, Determining the edge weights of each node edge in the abnormal relationship graph includes: For each node edge, if the first monitored object and the second monitored object at both ends of the current node edge do not have a directed call path in the dynamic resource topology graph, then the edge weight of the current node edge is zero. If a directed call path exists, cross-correlation is performed on the time series of the first monitored object and the second monitored object to obtain a cross-correlation number sequence under different lag times; positive peaks and their corresponding lag times are extracted from the cross-correlation number sequence; the positive peaks are used as the values ​​of the time-series correlation components, and the lag times are used as the basis for time-series order constraints. If the lag time is positive, it indicates that the anomaly of the first monitored object is earlier than that of the second monitored object. The deviation of the quantitative performance index, the density of log error templates, and the increase in call chain latency of the first monitored object and the second monitored object within the same time window are calculated respectively. The three are weighted and summed and then multiplied by the co-occurrence attenuation coefficient to obtain the cross-modal co-occurrence component. The topology reachability component is set to a preset value. The edge weight of the current node edge is determined based on the topological reachability component, the temporal correlation component and its corresponding first weight coefficient, and the cross-modal co-occurrence component and its corresponding second weight coefficient.

8. The method for locating the root cause of a fault according to claim 5, characterized in that, The fault attribution time period is the fault attribution time window. Before determining the edge weights of each node edge in the anomaly graph, a multimodal anomaly representation is generated for the monitored object corresponding to each graph node in the anomaly graph, including: Obtain the performance index sampling point sequence of the currently monitored object within the fault attribution time window, extract the mean and standard deviation under historical normal conditions, and calculate the abnormal significance score of each sampling point in the fault attribution time window; perform linear fitting on the performance index sampling point sequence, calculate the trend slope, and if the trend slope is greater than the preset mutation threshold, it is determined to be a continuous upward trend. Obtain the log template of the currently monitored object, take the frequency of each log template in the historical baseline window as the baseline frequency, take the frequency of each log template in the fault attribution period as the current frequency, and calculate the ratio of the current frequency to the baseline frequency as the burst ratio. If the surge ratio is greater than a preset multiple threshold and the number of occurrences within the fault attribution time window is greater than a preset noise threshold, it is determined to be a surge signal; if the reference frequency is zero and the number of occurrences within the fault attribution time window is greater than a preset noise threshold, it is determined to be a new template surge signal. Obtain the target call edge that the currently monitored object participates in, and calculate the latency increase ratio of the target call edge during the fault attribution period.

9. The method for locating the root cause of a fault according to claim 8, characterized in that, The calculation process for the cross-modal co-occurrence components includes: The abnormal significance score or trend slope is mapped to the indicator abnormality degree; the logarithm of the surge ratio or the flag bit of the new template surge signal is mapped to the log abnormality degree; and the latency rise ratio is mapped to the call chain abnormality degree. After multiplying the anomaly scores of the metrics, logs, and call chains by their respective preset weights, the results are summed to obtain a weighted sum. If any one of the metric anomaly degree, log anomaly degree, or call chain anomaly degree is zero, then the weighted sum is multiplied by the co-occurrence attenuation coefficient to obtain the cross-modal co-occurrence component; wherein the co-occurrence attenuation coefficient is less than 1.

10. The method for locating the root cause of a fault according to claim 5, characterized in that, After dividing the anomaly relationship diagram into at least one fault group, it also includes: For each fault group, select the graph node with the highest degree of abnormality or the graph node at the target position of the fault propagation path from the graph nodes contained in the current fault group, and use it as the center node of the current fault group. Obtain the abnormal start time of all graph nodes in the current fault group, and calculate the weighted average of each abnormal start time according to the abnormality degree of each graph node to obtain the time centroid of the current fault group. The average edge weight of all nodes within the current fault group is used as the intra-group cohesion, and the average edge weight of the boundary nodes between graph nodes within the current fault group and graph nodes outside the group is used as the inter-group separation. The boundary confidence of the current fault group is determined based on the ratio of the intra-group cohesion to the inter-group separation. The central node, the time centroid, and the boundary confidence are used as metadata descriptors for the current fault group and then marked.

11. The method for locating the root cause of a fault according to any one of claims 1 to 4, characterized in that, The retrieval operation generates candidate fault source location results for each fault group, including: Each fault group is configured with an independent state machine, which records the known data types collected and the missing unknown data types of the current fault group; For each fault group, if the unknown data type is log information type, and the time series analysis shows that there is an abnormal relationship between upstream and downstream but the upstream monitored object lacks log evidence, then log template retrieval is performed, and the target log template containing specific error codes or abnormal target words is retrieved using the inverted index. If the unknown data type is performance metric information, and the call chain data shows that local calls are slower but the corresponding performance metrics do not change significantly, then for the monitored objects associated with the slower call edges in the call chain, pull out slices of performance metrics related to lock waiting or thread queue length during the fault attribution period. If the unknown data type is call chain data, then the target call path that meets the conditions of sudden increase in delay, increase in error rate, or call topology offset within the fault attribution period is extracted from the call chain database. If the unknown data type is change event data, then query the configuration management platform or continuous integration pipeline to obtain the most recent change event record; Each retrieval result is stored in the state machine, and the missing data types in the current fault group are re-analyzed until the preset iteration termination condition is met, so as to determine the corresponding candidate fault source location result based on the data stored in the state machine.

12. The method for locating the root cause of a fault according to claim 11, characterized in that, Extract target call paths from the call chain database that meet the conditions of sudden increase in latency, increased error rate, or call topology offset within the fault attribution period, including: Starting from the symptom object, trace back hop by hop along the reverse call chain; During the tracing process, if the latency increase ratio of the current calling edge exceeds the preset latency threshold, the current calling edge and its associated objects are connected to form a backbone link as the target calling path. If the percentage of error codes on the current calling edge exceeds a preset error rate threshold, then the current calling edge and its associated objects are connected to form a backbone link, which serves as the target calling path. If the traffic allocation ratio of the current calling edge deviates beyond a preset offset threshold, the current calling edge and its associated objects are connected to form a backbone link, which serves as the target calling path.

13. The method for locating the root cause of a fault according to claim 11, characterized in that, Based on the data stored in the state machine, the corresponding candidate fault source location results are determined, including: For each fault group, during the process of generating at least one candidate fault source location result based on the data already stored in the corresponding state machine, the candidate fault triggering object identifier, the abnormal start time point of the candidate fault triggering object, the fault cause type of the candidate fault triggering object, and the supporting evidence data of the fault source determined by the current candidate fault source location result are recorded.

14. The method for locating the root cause of a fault according to any one of claims 1 to 4, characterized in that, The process of determining the root cause of the failure includes: For each candidate fault source location result, query the dynamic resource topology graph of the current candidate fault source location result to see if there is a directed connected path between the fault source and the symptom object. If not, mark the verification as failed. If a directed connection path exists between the fault source and the symptom object in the dynamic resource topology graph of the current candidate fault source location result, the abnormal start time of the fault source in the current candidate fault source location result is compared with the abnormal start time of the symptom object. If the abnormal start time of the fault source is not earlier than the abnormal start time of the symptom object, or the abnormal time difference between the two exceeds a preset lag interval, the verification is marked as failed. If the abnormal start time of the fault source is earlier than the abnormal start time of the symptom object, it is checked whether the abnormal data supporting the current candidate fault source location result contains at least two of the following: abnormal performance index data, abnormal log data, and abnormal call chain data. If less than two, the verification is marked as failed. If at least two, the current candidate fault source location result is taken as the final selected fault source location result. If any verification fails, the current candidate fault source location result will be marked as pending verification, and the corresponding supplementary search operation will be triggered according to the failure type. If the verification still fails after a preset number of supplementary searches and re-verifications, the one with the most verified items will be selected from all candidate fault source location results as the root cause of the fault, and the failure reasons for the failed items will be recorded.

15. The method for locating the root cause of a fault according to claim 14, characterized in that, After triggering the corresponding supplementary retrieval operation based on the failure type, it also includes: If the failure is due to time sequence reversal, the search time window will be expanded and the corresponding search operation will be re-executed. If the failure is due to path verification failure, and the analysis finds that the root cause is outside the current fault group boundary, then the nodes outside the boundary that meet the abnormal conditions will be included in the current fault group, and the fault group division will be re-executed. If the verification still fails after supplementary retrieval and regrouping, the ranking score of the current candidate fault source location result will be reduced according to the score adjustment rules, and the ranking score of the candidate fault source location result that passed the verification will be increased.

16. The method for locating the root cause of a fault according to claim 14, characterized in that, Also includes: For each fault group, a comprehensive score for the localization results of each candidate fault source is determined based on the anomalous significance of graph nodes within the fault group, the topological distance between graph nodes and symptom objects, and the hysteresis correlation strength between graph nodes. Perform a reverse random walk on the anomaly graph, starting from the symptom object and hopping backward along the call relationship, increasing the scoring weight of graph nodes that are upstream in the call chain and have high anomaly significance; The candidate fault source with the highest comprehensive score in each fault group is used as the final fault source location result for the corresponding fault group, and a fault report is generated that includes fault source identifier, abnormal start time point, fault type, propagation path highlighted mark and supporting data summary. When multiple fault groups exist, check whether each fault group shares the same change event record or the same underlying resource anomaly. If so, merge multiple fault groups into a common cause view; otherwise, keep their independent output. The fault reports, the window identifiers corresponding to the fault attribution time periods, and the filtering rules used in the retrieval process are permanently saved.

17. The method for locating the root cause of a fault according to any one of claims 1 to 4, characterized in that, Also includes: When a first type of directed topological edge with an observation coverage lower than a preset coverage threshold is detected in the dynamic resource topology graph, the weight value of the first type of directed topological edge in the clustering and verification processes is reduced according to the preset weight adjustment rules. When more than the preset number of concurrent fault event signals are received, the priority of the business link corresponding to each fault event signal is identified, and the fault event signals of the business link with higher priority are processed first. When a second type of directed topological edge is detected in the dynamic resource topology graph, and the frequency of its appearance and disappearance exceeds a preset threshold, a topological uncertainty mark is added to the verification result corresponding to the second type of directed topological edge; if there are multiple third type of directed topological edges whose difference between the number of appearances and the highest number of appearances is less than a preset difference threshold, the fault propagation path corresponding to each third type of directed topological edge is listed in the fault report. When the retrieval service is detected to be unavailable, the step of generating candidate fault source location results for each fault group through the retrieval operation is paused. The telemetry snapshot summary of the most recent preset number of time windows is read from the local circular buffer. The telemetry snapshot summary includes the statistical characteristics of performance indicators, the index of log templates, and the aggregated data of call chains. Based on the telemetry snapshot summary, a preliminary fault location result carrying a low confidence marker is determined. Once the retrieval service is detected to have returned to normal, the step of generating candidate fault source location results for each fault group through the retrieval operation is re-executed, and the preliminary fault location results are updated.

18. A root cause fault location system, characterized in that, This includes a root cause inference processor deployed between the data acquisition end and the operation and maintenance display end; The root cause reasoning processor, when executing a computer program stored in the memory, implements the steps of the fault root cause localization method as described in any one of claims 1 to 17.

19. The root cause fault location system according to claim 18, characterized in that, The root cause inference processor is deployed in the management cluster or runs as software as a service in an independent operations domain. The root cause inference processor receives fault event signals and corresponding multimodal telemetry data from the data acquisition terminal by subscribing to message queues or streaming channels.

20. The root cause fault location system according to claim 18, characterized in that, The root cause reasoning processor has a snapshot storage area and a feature cache area, and the snapshot storage area and the feature cache area are isolated from each other by an independent locking mechanism; The snapshot storage area is configured in read-only mode during the fault location reasoning process to store fault analysis data; The feature cache supports read and write operations and is used to store feature data extracted from the fault analysis data and intermediate results of retrieval operations.

Images