A fault root cause positioning method based on a two-horizontal-one-vertical CMDB topology diagram

Abstract

The application discloses a fault root cause positioning method based on a two-horizontal-one-vertical CMDB topology diagram, comprising the following steps: acquiring transaction data flow of a business system to construct transaction business calling topology and output business layer horizontal topology; acquiring calling log and link tracking data of an application service to construct application service calling topology and output application layer horizontal topology; acquiring configuration data and running state data of underlying IT resources to construct support resource topology and output resource layer vertical topology; performing full-link correlation on the three topologies to establish a data correlation model and output a multi-dimensional application operation and maintenance graph with causal logic; accessing alarm and operation and maintenance data, performing fault root cause positioning based on the graph, and outputting fault root cause, influence range and disposal suggestions in combination with a root cause analysis algorithm. Through construction of a business-application-resource three-dimensional topology model and fusion of multi-source operation and maintenance data, the application realizes rapid and accurate positioning of faults, and significantly improves operation and maintenance efficiency and business continuity.

Description

Technical Field

[0001] This invention relates to the field of information technology operation and maintenance management technology, and in particular to a fault root cause localization method based on a two-horizontal-one-vertical CMDB topology diagram. Background Technology

[0002] In the existing field of IT operations and maintenance management, the Configuration Management Database (CMDB) is a core tool supporting fault location and system monitoring. Traditional CMDBs are mainly designed for monolithic applications or simple distributed architectures. The definition and relationship modeling of configuration items revolve around physical machines, virtual machines, and static application instances, and configuration data is maintained through scheduled scanning and periodic updates. In terms of fault location, operations and maintenance personnel usually rely on manual experience, combining scattered monitoring indicators, alarm information, and log data to troubleshoot one by one, lacking a systematic causal correlation mechanism.

[0003] However, with the widespread adoption of emerging technologies such as cloud-native, microservices, containerization, and serverless, enterprise IT architectures are exhibiting highly dynamic, elastic, and multi-component collaborative characteristics. Traditional CMDBs cannot natively model cloud-native components such as containers, Pods, ServiceMesh, and Ingress, and are also ill-suited to the real-time requirements of second-level resource scaling. A more prominent issue is that existing solutions generally lack configuration data related to business transaction call chains and application service call chains, resulting in data fragmentation between the business layer, application layer, and resource layer, with no causal logical relationship between the discrete metadata. When a system failure occurs, a large number of redundant alarms emerge simultaneously, making it difficult for operations personnel to quickly pinpoint the root cause from massive amounts of information, leading to a persistently high Mean Time To Repair (MTTR) and posing a severe challenge to business continuity.

[0004] Therefore, there is an urgent need for a method that can adapt to the dynamic cloud-native environment, connect the entire link between business, application, and resources, and enable rapid and accurate location of the root cause of failures. Summary of the Invention

[0005] This application provides a fault root cause localization method based on a two-horizontal-one-vertical CMDB topology map, including the following steps: S1: Obtain the transaction data stream of the business system, construct the transaction business call topology based on the transaction data stream, and output the horizontal topology of the business layer; S2: Obtain application service call logs and tracing data, construct application service call topology based on call logs and tracing data, and output application layer horizontal topology; S3: Obtain configuration data and runtime status data of underlying IT resources, construct supporting resource topology based on configuration data and runtime status data, and output the vertical topology of the resource layer; S4: Connect the horizontal topology of the business layer, the horizontal topology of the application layer, and the vertical topology of the resource layer in the entire link, establish a data association model, and output a multi-dimensional application operation and maintenance graph with causal logic. S5: Integrates alarm and operation and maintenance data, performs fault root cause localization based on multi-dimensional application operation and maintenance map, and outputs fault root cause, impact scope and handling suggestions by combining root cause analysis algorithm.

[0006] In some embodiments, the transaction business call topology covers the core business system, transaction process and user request path, and marks business priority, traffic weight and dependency relationship by business scenario to form a business-level panoramic call view.

[0007] In some embodiments, the application service call topology supports microservice and service mesh environments. By collecting service registration information, interface call logs and tracing data, a three-level dependency topology of service-method-instance is constructed, and the dynamic call relationship is updated in real time based on the heartbeat detection and event notification mechanism of the service registry.

[0008] In some embodiments, the application service call topology is based on an event-driven mechanism, responding to service instance creation, destruction, and scaling events, and updating the service-method-instance three-level dependency topology in real time.

[0009] In some embodiments, the system supports resource topology compatibility with physical machines, virtual machines, containers and cloud platform full-stack resources, constructs a vertical support link of "infrastructure-middleware-application instance", and supports configuration item modeling and dynamic adaptation of containers, Pods, namespaces, configuration mappings, entry gateways and service mesh components.

[0010] In some embodiments, the data association model takes CMDB configuration items as the core, associates monitoring indicators, alarm information, log data and call chain data, establishes a one-to-one mapping between topology nodes and operation and maintenance data, and establishes a correspondence between nodes in the business layer horizontal topology, application layer horizontal topology and resource layer vertical topology and CMDB configuration items.

[0011] In some embodiments, the root cause analysis algorithm employs correlation analysis, link backtracking, and anomaly propagation path deduction, and combines topology weights and alarm priorities to filter redundant alarms and locate the underlying fault source.

[0012] In some embodiments, the steps of performing fault root cause localization include: alarm access, topology mapping, abnormal path backtracking, root cause verification and result output, forming a closed loop of full-link localization.

[0013] In some embodiments, the step of performing fault root cause localization further includes: visually presenting the fault propagation path and the scope of affected services based on a multi-dimensional application operation and maintenance map, and outputting recommended handling solutions.

[0014] In some embodiments, the supporting resource topology also supports dynamic resource lifecycle management triggered by resource change events, responding to creation, destruction and migration events of underlying resources, and updating the vertical topology of the resource layer in real time.

[0015] This application constructs a three-dimensional CMDB topology model consisting of transaction business call topology, application service call topology, and supporting resource topology, forming a "two horizontal and one vertical" structure. It then establishes a multi-dimensional application operation and maintenance graph with causal logic by linking the three topologies across the entire chain. Based on this, root cause analysis algorithms are used to achieve automatic fault tracing and precise fault location. This method effectively solves the problems of poor adaptability of traditional CMDBs in cloud-native environments, missing data across the entire chain, and reliance on manual experience for fault location. It significantly improves the efficiency of fault root cause location, reduces mean time to repair (MTTR), and provides complete context for operation and maintenance data, reducing misjudgments and omissions, thus strongly ensuring business continuity. Attached Figure Description

[0016] Figure 1 This is a flowchart of the adaptive energy-saving control method based on thyristors proposed in this invention. Detailed Implementation

[0017] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0018] This application provides a fault root cause localization method based on a two-horizontal-one-vertical CMDB topology map, such as... Figure 1 As shown, it includes the following steps: S1: Obtain the transaction data stream of the business system, construct the transaction business call topology based on the transaction data stream, and output the horizontal topology of the business layer; S2: Obtain application service call logs and tracing data, construct application service call topology based on call logs and tracing data, and output application layer horizontal topology; S3: Obtain configuration data and runtime status data of underlying IT resources, construct supporting resource topology based on configuration data and runtime status data, and output the vertical topology of the resource layer; S4: Connect the horizontal topology of the business layer, the horizontal topology of the application layer, and the vertical topology of the resource layer in the entire link, establish a data association model, and output a multi-dimensional application operation and maintenance graph with causal logic. S5: Integrates alarm and operation and maintenance data, performs fault root cause localization based on multi-dimensional application operation and maintenance map, and outputs fault root cause, impact scope and handling suggestions by combining root cause analysis algorithm.

[0019] This embodiment provides a specific implementation of a fault root cause localization method based on a two-horizontal-one-vertical CMDB topology diagram. This method aims to solve the problems of data dispersion and difficulty in fault localization in traditional CMDBs in cloud-native environments. By constructing a three-layer topology of business, application, and resources and establishing causal relationships, it achieves automatic source tracing and accurate root cause localization of alarms.

[0020] First, in step S1, the system retrieves transaction data streams from the core business systems via the business monitoring platform or API gateway. These data streams record the complete path of a user request from initiation to completion, including the call relationships and interaction order between various business systems. By parsing information such as transaction serial numbers and request identifiers, a horizontal topology of the business layer can be automatically drawn. This topology, based on business scenarios, clearly shows the dependencies and data flows between different business systems; for example, the payment system depends on the order system and inventory system. The reason for setting up this topology is that traditional CMDBs often only focus on IT resources, ignoring the connections at the business level, making it impossible to quickly determine the scope of affected business when a failure occurs. By constructing a horizontal topology of the business layer, operations personnel can intuitively see which business process has an anomaly, thereby quickly assessing the impact.

[0021] Next, in step S2, the system collects application service call logs and distributed tracing data under the microservice architecture. This data comes from service meshes (such as Istio), APM tools, or custom tracking points, recording the number of calls, response time, success or failure, and span information of each service interface. By aggregating and analyzing this data, an application-layer horizontal topology can be constructed. This topology delves into the service, method, and even instance levels, showcasing the dynamic dependencies between services. For example, service A's method A calls service B's method B, and service B depends on a database instance. The reason for setting this topology is that service call relationships in a microservice environment are complex and change frequently, and traditional static configurations cannot reflect real-time call chains. The application-layer horizontal topology can be updated in real time, helping operations personnel quickly locate which upstream service or downstream dependency is causing the problem when a service call fails.

[0022] Subsequently, in step S3, the system retrieves configuration and operational status data of underlying IT resources from cloud platform APIs, container orchestration platforms (such as Kubernetes), and infrastructure monitoring systems. This data includes the model, IP address, CPU utilization, and memory usage of resources such as physical machines, virtual machines, containers, Pods, databases, and middleware. Based on this data, a supporting resource topology, i.e., a vertical resource layer topology, can be constructed. This topology extends upwards from the underlying infrastructure to middleware and then to application instances, forming a clear vertical support chain. For example, an application instance runs in a Pod, which is scheduled on a physical machine, and the application depends on a Redis cluster. This topology is designed because the root cause of many failures lies in the resource layer, such as CPU exhaustion, high disk I / O, or network latency. However, traditional alerting systems can only display upper-layer application anomalies and cannot quickly trace resource bottlenecks. The supporting resource topology fills this gap by directly linking the state of the resource layer with the performance of the application layer.

[0023] After constructing the three topologies mentioned above, the system proceeds to step S4. The system will establish a unified data association model by fully linking the business layer horizontal topology, application layer horizontal topology, and resource layer vertical topology. Specifically, the system uses configuration items in the CMDB as the core, mapping nodes in the three topologies one by one to the corresponding configuration items, while also associating monitoring metrics, alarm information, log data, and call chain data. For example, an order service node in a business system will correspond to the order service interface in the application layer topology, and then to the Pods and servers in the resource layer topology. Through this association, metadata originally scattered across various systems is integrated into a multi-dimensional application operation and maintenance graph with causal logic. This graph is crucial: it provides a complete context for subsequent fault localization. When an alarm occurs, the system no longer looks at an alarm point in isolation, but can trace along the path in the graph to understand the causal relationship between alarms.

[0024] Finally, in step S5, the system receives alarm events and operational data from various monitoring sources in real time and maps them to corresponding nodes in the multi-dimensional application operation and maintenance graph. Then, combining root cause analysis algorithms, such as correlation analysis, link tracing, and anomaly propagation path deduction, the system automatically calculates the causal relationships between alarm events. The algorithm considers topology weights and alarm priorities, filtering out redundant alarms and retaining only the most likely underlying fault sources. Ultimately, the system outputs the root cause of the fault (e.g., CPU resource exhaustion on a server), the scope of impact (which business systems and services are affected), and recommended remedial measures (e.g., expanding CPU or restarting Pods). The entire localization process achieves automated closed-loop processing, significantly reducing the time spent on manual troubleshooting.

[0025] Through the collaborative work of the above five steps, this embodiment achieves rapid and accurate fault location in a cloud-native environment. Compared with traditional methods, this method not only fills in the data gaps at the business and application layers but also establishes causal logic between the three layers, making alarms no longer isolated noise but traceable clues. Operations personnel no longer need to blindly search through massive amounts of alarms; the system can directly provide the root cause and scope of impact, thereby significantly reducing the mean time to repair (MTTR) and improving business continuity. Simultaneously, the multi-dimensional application operations graph also provides a data foundation for daily capacity planning and performance optimization, truly upgrading the CMDB from a static configuration repository to the core engine of intelligent operations.

[0026] In some embodiments, the transaction call topology covers core business systems, transaction processes, and user request paths, and labels business priorities, traffic weights, and dependencies based on business scenarios, forming a business-level panoramic call view. The construction of the transaction call topology can be achieved by connecting to an enterprise's internal business monitoring platform or API gateway. The system first collects transaction flow data from these sources, such as transaction order numbers, user session IDs, request sources, and target system identifiers. Then, it uses a graph database or relational database to organize this information into nodes and edges. Nodes represent different business systems (such as order systems, payment systems, and inventory systems), and edges represent the call relationships and data flow between systems. To form a business-level panoramic call view, the system also introduces a business scenario dimension, grouping multiple system calls involved in the same transaction or the same type of user request into a single scenario. For example, in an e-commerce platform, a complete order placement process will trigger the shopping cart system, order system, payment system and logistics system in sequence. The system will link these call relationships together and mark the business priority of the scenario (such as the core transaction process being of high priority), traffic weight (such as the traffic of this scenario accounting for 80% during Double Eleven), and dependency relationship (such as the payment system depending on the return result of the order system).

[0027] In practical implementation, two specific technical solutions can be adopted. The first is based on the extension of a distributed tracing system, such as adding business tags to Jaeger or Zipkin, so that each RPC call carries a business scenario ID, thereby automatically generating a call topology with business semantics. The second is to parse key fields in business logs, aggregate the logs using offline or real-time stream processing engines (such as Flink), extract the call frequency and response time between business systems, and then combine this with the pre-defined business system relationships in the CMDB to dynamically generate a panoramic view. For example, a bank's core system can identify different business scenarios such as personal online banking transfers and corporate bulk payroll by parsing the channel code and transaction code in transaction messages, and then establish an independent call topology for each scenario, marking its priority and SLA level.

[0028] The beneficial effects of this embodiment are reflected in several aspects. By labeling priorities and traffic weights according to business scenarios, operations and maintenance personnel can immediately determine the importance of affected businesses when a fault occurs and prioritize handling high-priority scenarios. The introduction of traffic weights helps distinguish between normal traffic and sudden traffic surges, avoiding misjudging normal traffic fluctuations as faults. Explicit labeling of dependencies makes the assessment of the scope of fault impact more accurate. For example, when the order system experiences an anomaly, the system can automatically calculate that the payment system and logistics system that depend on it will also be affected, thus providing early warning. Finally, the business-level panoramic call view integrates the originally scattered system call information into a hierarchical and focused graph, significantly reducing the cognitive burden on operations and maintenance personnel, shortening the time for fault location and impact assessment, and effectively ensuring the continuity of core businesses.

[0029] In some embodiments, the application service call topology supports microservice and service mesh environments. It constructs a three-tiered service-method-instance dependency topology by collecting service registration information, interface call logs, and distributed tracing data. The dynamic call relationships are updated in real-time based on heartbeat detection and event notification mechanisms from the service registry. The construction of the application service call topology relies on infrastructure components within the microservice architecture. The system first pulls service registration information from the service registry (e.g., Consul, Nacos, or Eureka) to obtain the IP addresses, port numbers, and health status of all current service instances. Simultaneously, the system connects to a distributed distributed tracing system (e.g., SkyWalking or Zipkin) to collect Span data for each RPC call, including the caller's service name, the callee's service name, the method name, the call duration, and the status code. Furthermore, the system collects interface call logs from the application server and extracts the call relationships between services using regular expressions or structured log parsing. Based on these three types of data, the system can construct a three-tiered service-method-instance dependency topology: the top layer is the service nodes, the middle layer is the method interfaces exposed by the services, and the bottom layer is the specific instances running the methods. For example, in an e-commerce system, the createOrder method of the order service may be deployed on three container instances at the same time. The topology will clearly show the three instances of the method and their respective downstream dependencies.

[0030] To achieve real-time updates of dynamic call relationships, the system employs two complementary mechanisms. The first is a heartbeat detection mechanism: each service instance periodically sends heartbeat packets to the monitoring center. If no heartbeat is received three times consecutively, the system determines that the instance is offline and immediately removes the node and its associated edges from the topology. The second is an event notification mechanism: the service registry proactively pushes events to the topology building module when instances are registered, deregistered, or when metadata changes. Upon receiving the event, the module incrementally updates the topology without requiring a full rebuild. For example, when Kubernetes performs a rolling update on a Deployment, old Pod instances are gradually destroyed, and new Pod instances are successively registered. The event notification mechanism can capture these changes and refresh the topology within seconds, ensuring that the topology always reflects the true runtime state.

[0031] The beneficial effects of this embodiment are significant. By constructing a three-level dependency topology of service-method-instance, operations and maintenance personnel can drill down from a macro-level service call overview to the specific method and instance granularity. When an interface response slows down, they can quickly locate which method timed out or which instance is overloaded, avoiding needle-in-a-haystack troubleshooting. Based on a real-time update mechanism using heartbeats and event notifications, the topology can adapt to the dynamic scaling and failover of the microservice environment. Even with frequent instance changes, alarm correlations remain consistent. In practical applications, after introducing this solution, an internet finance company reduced the average fault location time of the service call chain from 30 minutes to less than 5 minutes, reduced redundant alarms by more than 60%, and greatly improved operational efficiency. Simultaneously, this topology provides a precise data foundation for subsequent capacity planning and anomaly detection, truly realizing a shift from reactive firefighting to proactive prevention.

[0032] In some embodiments, the application service call topology is based on an event-driven mechanism, responding to service instance creation, destruction, and scaling events, and updating the service-method-instance three-level dependency topology in real time. Real-time updates of the application service call topology rely on an event-driven mechanism, which is implemented by listening to various events emitted by the infrastructure and middleware layers. In practical implementation, the following technical solutions can be adopted. The first solution is based on the Kubernetes Watch API, where the system continuously listens for change events of resources such as Pods, Services, and Endpoints. When a new Pod instance is created and registered with the service registry, the Watch API immediately pushes an Add event. Upon receiving this event, the topology building module adds an instance node to the service-method-instance three-level dependency topology and connects it to the corresponding method node. When a Pod is destroyed or scaled down, the Delete event is also captured, and the system then removes the corresponding instance node and related edges. The second approach utilizes the control plane components of a service mesh (such as Istio). Pilot or Citadel continuously tracks the routing rules and endpoint information of all Sidecar proxies. The online or offline status of any instance triggers Envoy's xDS protocol update, and the system can drive topology refresh by subscribing to these control plane change streams. The third approach uses a message queue (such as Kafka) as an event bus. Each service instance actively sends lifecycle events to a designated Topic when starting or stopping. The topology consumer listens to the Topic and performs incremental updates. This approach is not dependent on a specific orchestration platform and is more versatile.

[0033] The beneficial effects of this embodiment are mainly reflected in real-time performance and accuracy. Traditional timed polling methods typically require scanning intervals of tens of seconds or even minutes. During this time, the topology is outdated, which may cause alarms to be associated with incorrect instances, leading to misjudgments. The event-driven mechanism compresses update latency to the millisecond level, refreshing the topology almost simultaneously with instance changes, ensuring the accuracy of alarm mapping. For example, during a sudden traffic surge, Kubernetes automatically scales up multiple Pod replicas. The event-driven mechanism immediately incorporates these new instances into the topology, and subsequent alarm analysis can correctly allocate traffic to the new instances, avoiding misjudging normal new instances as abnormal due to topology lag. In addition, the timely response to scaling events allows operations personnel to observe the changing trends of the number of service instances in real time. Combined with monitoring metrics, this allows for quick judgment on whether scaling is reasonable, providing first-hand data for capacity planning. Overall, this mechanism significantly improves the timeliness and reliability of fault location, reduces the cost of secondary investigations caused by outdated topologies, and allows the operations team to focus more on the true root cause of the fault.

[0034] In some embodiments, the supporting resource topology is compatible with physical machines, virtual machines, containers, and cloud platform resources across the entire stack, constructing a vertical support link from infrastructure to middleware to application instances. It also supports configuration item modeling and dynamic adaptation for containers, Pods, namespaces, configuration mappings, ingress gateways, and service mesh components. The construction of the supporting resource topology requires compatibility with various heterogeneous resources such as physical machines, virtual machines, containers, and cloud platforms, forming a vertical support link from infrastructure to middleware to application instances. To achieve this, the system typically integrates multiple data acquisition channels. The first example is for physical machines and virtual machines, where hardware specifications, operating system versions, and network configurations can be obtained by deploying an agent or calling out-of-band management interfaces (such as IPMI or VMware vSphere API). The second example is for containers and cloud-native components, where the system connects to the Kubernetes API Server to periodically or in real-time pull the status of resource objects such as Nodes, Pods, Namespaces, ConfigMaps, and Ingresses. Simultaneously, it obtains routing rules and endpoint information for service mesh components through the Service Mesh control plane (such as Istio's Pilot). The third example concerns middleware. The system can utilize JMX, Redis INFO commands, or database connection pool monitoring interfaces to collect the running parameters and health status of components such as MySQL, Redis, and Kafka. All collected data is normalized and stored in a CMDB, establishing a hierarchical relationship according to the structure of "physical machine / virtual machine → container / Pod → middleware → application instance," forming a vertical support chain. For example, a Java process for an order service runs in a Pod scheduled on a physical machine, and this process is connected to a Redis cluster and a MySQL master database. The vertical chain clearly shows the support relationship between these four components.

[0035] For dynamic adaptation, the system listens for Kubernetes Watch events and cloud platform resource change notifications. When a Pod is rescheduled to another physical machine due to a failure, the system automatically updates the association edges between the Pod and the new physical machine, while maintaining the associations between its upper-layer application instances and lower-layer middleware. For configuration resources such as ConfigMaps or Ingresses, the system treats them as independent configuration item nodes and establishes dependency edges with the application instances that use them. When the configuration changes, the system marks the relevant instances as "pending confirmation," prompting operations personnel to check whether the configuration change has caused an anomaly.

[0036] The benefits of this embodiment are comprehensive. First, the compatibility of full-stack resources eliminates the blind spots of traditional CMDBs, which can only manage a portion of resources. This allows fault location to drill down from the application layer all the way to the physical hardware layer, without interrupting the tracing path due to missing data at a certain layer. Second, the construction of the vertical support chain allows operations personnel to intuitively see whether "the application crashed because its dependent database went down, or because its physical machine experienced a memory overflow," clearly presenting the causal relationship of multi-layered faults. Third, the native modeling of containers, Pods, namespaces, ConfigMaps, Ingress, and service mesh components enables dynamic resource changes in the cloud-native environment to be captured and reflected in the topology in a timely manner, avoiding configuration data distortion caused by the inability of traditional CMDBs to recognize these components. In actual deployment, after adopting this solution, a large internet company found that cross-layer faults that previously required multiple teams and several hours to locate (such as application performance degradation due to underlying network jitter) can now be rooted in just over ten minutes, improving operational efficiency by nearly 80%. Meanwhile, the vertical support link also provides a precise data foundation for capacity management and cost optimization. For example, it can quickly identify which application instances share the same physical machine, thereby avoiding resource contention during expansion.

[0037] In some embodiments, the data association model uses CMDB configuration items as its core, associating monitoring metrics, alarm information, log data, and call chain data to establish a one-to-one mapping between topology nodes and operational data. It also establishes a correspondence between nodes in the business layer horizontal topology, application layer horizontal topology, and resource layer vertical topology and CMDB configuration items. The construction of the data association model uses CMDB configuration items as its core hub. In practical implementation, the system first needs to establish the correspondence between topology nodes and CMDB configuration items. This step can be achieved using unique identifiers for the configuration items. For example, each CMDB configuration item has a CI ID, and each node is assigned an associated CI ID when constructing the business layer, application layer, and resource layer topologies. For example, in the business layer topology, the order system node corresponds to a configuration item named "Order System" in the CMDB; in the application layer topology, the createOrder method of the order service corresponds to a configuration item named "Order Service - createOrder Interface" in the CMDB; and in the resource layer topology, the Pod instance running this service corresponds to a configuration item named "pod-order-svc-xxx" in the CMDB. Through this mapping, all nodes in the three-layer topology establish a bidirectional correspondence with the configuration items in the CMDB.

[0038] Regarding the association of operational data, the following three technical solutions can be adopted. The first solution is through tag or metadata matching. Monitoring metrics, alarm information, and log data carry identifiers of the source objects during collection, such as hostnames, Pod names, and service names. The system matches these identifiers with the names or aliases of CMDB configuration items, thereby mounting the data to the corresponding topology nodes. The second solution utilizes Trace IDs and Span IDs from distributed tracing. Call chain data naturally contains service and method-level information. The system can map each Span in the Trace to the corresponding method node in the application layer topology, and simultaneously associate it with resource layer nodes through the host information carried by the Span. The third solution uses tag indexes from time-series databases. For example, Prometheus metrics contain instance tags. The system parses these tags and associates them with instance configuration items in the CMDB to achieve automatic binding of monitoring metrics to topology nodes. These three solutions can be used in combination to ensure that different types of operational data can be correctly associated with the corresponding topology nodes.

[0039] The beneficial effects of this embodiment are significant. By establishing a one-to-one mapping between topology nodes and operational data, monitoring metrics, alarms, logs, and call chain data, which were originally scattered across multiple systems, are unified into a single context. When a node generates an alarm, operations personnel can view all monitoring curves, recent logs, and call chain snapshots associated with that node with a single click, eliminating the need to switch between multiple tools and greatly reducing information acquisition time. More importantly, this mapping relationship enables subsequent root cause analysis algorithms to traverse all associated data along the topology path, achieving true causal reasoning. For example, when a service node in the application layer generates a response timeout alarm, the algorithm can automatically retrieve the call chain data associated with that node, discovering that the timeout was caused by a slow query on a downstream database node. Simultaneously, checking the database node's monitoring metrics confirms a CPU spike, thus accurately pinpointing the root cause. Furthermore, the CMDB configuration items, acting as a core hub, ensure consistency between topology nodes and real IT assets, avoiding misjudgments caused by data silos. In practical applications, after implementing this data association model, a company reduced the average information collection time for fault diagnosis from 40 minutes to less than 5 minutes, the false alarm rate decreased by 70%, the operations and maintenance team was able to make decisions more quickly, and business stability was significantly improved.

[0040] In some embodiments, the root cause analysis algorithm employs correlation analysis, link tracing, and anomaly propagation path deduction, combined with topology weights and alarm priorities to filter redundant alarms and locate the underlying fault source. The implementation of the root cause analysis algorithm integrates three core mechanisms: correlation analysis, link tracing, and anomaly propagation path deduction, and introduces topology weights and alarm priorities to filter redundant alarms, ultimately pinpointing the underlying fault source. In practical implementation, the following technical solutions can be adopted.

[0041] The first approach is a breadth-first anomaly propagation deduction based on graph theory. After an alarm event is mapped to a multi-dimensional application operation and maintenance graph, the algorithm starts with each alarm node and performs a breadth-first traversal upstream and downstream along the topology edges. During the traversal, the algorithm calculates the anomaly score for each node, which is determined by the node's own alarm severity, the number of associated alarms, and the topology weight. The topology weight is pre-set based on the node's priority in the business scenario, traffic share, and historical failure frequency; for example, nodes on the core transaction chain have a much higher weight than edge nodes. The algorithm prioritizes traversing high-weight nodes and records the anomaly propagation path. When a node's anomaly score exceeds a threshold, it is marked as a candidate root cause. Simultaneously, alarm priority is used to filter low-priority redundant alarms; for example, only P0 and P1 level alarms are retained for calculation, while P2 and P3 level alarms are used as auxiliary references but do not participate in root cause voting.

[0042] The second approach is root cause localization based on random walks. The algorithm treats the multi-dimensional application operation and maintenance graph as a directed weighted graph, where edge weights are determined by call frequency and response time. When a batch of alarms arrives, the algorithm initiates multiple random walks in the graph, starting from all alarm nodes. Each step moves along an outgoing edge to a downstream node with probability p, or jumps to any alarm node with probability 1-p. After a sufficient number of walks, the frequency of each node's visits reflects its probability of being the root cause. Combining topological weights, the visit frequency can be weighted and adjusted, ultimately selecting the node with the highest visit frequency as the root cause. The advantage of this method is that it does not require predefined rules and can automatically adapt to complex call relationships.

[0043] The third approach is Bayesian inference based on causal graphs. The algorithm first uses historical alarm data and a fault case library to learn the conditional probability distribution between alarm events and construct a causal graph. When a real-time alarm occurs, the algorithm uses the observed alarm as evidence and performs Bayesian backpropagation on the causal graph to calculate the posterior probability of each node as the root cause. Topological weights and alarm priorities can be incorporated into the calculation as part of the prior probability; for example, nodes corresponding to higher-priority alarms receive higher prior probabilities. Finally, the node with the highest posterior probability is output as the root cause.

[0044] The beneficial effects of this embodiment are reflected on multiple levels. Through correlation analysis and link tracing, the algorithm can automatically discover the causal chain between alarms, rather than viewing each alarm in isolation, avoiding the common mistake of mistaking symptoms for root causes. Anomaly propagation path deduction allows the algorithm to simulate how a fault spreads in the system, thereby finding the initial disturbance point. The introduction of topology weights and alarm priorities effectively filters out a large number of secondary alarms caused by the same root cause. For example, when a database crashes, all upstream services will generate timeout alarms, but these are all derivative alarms. The algorithm prioritizes database nodes through weight comparison, thereby accurately locating the root cause. In actual deployment, after applying this algorithm, a large e-commerce platform saw its accuracy in locating the root cause of faults increase from 65% to 92%, the average location time decrease from 20 minutes to 3 minutes, and the filtering rate of redundant alarms reach over 85%. The operations and maintenance team no longer needs to search through hundreds of alarms; the system directly provides the most likely root cause and scope of impact, significantly improving the efficiency of emergency response and business continuity.

[0045] In some embodiments, the steps for performing fault root cause localization include: alarm access, topology mapping, abnormal path backtracking, root cause verification, and result output, forming a closed-loop localization process. The steps for performing fault root cause localization are designed as a complete closed-loop process, sequentially including alarm access, topology mapping, abnormal path backtracking, root cause verification, and result output. Each step has a specific technical implementation method to ensure that the entire localization process is efficient and traceable.

[0046] During the alarm ingestion phase, the system aggregates alarm events from various monitoring sources, including alarms from the business layer, application layer, and resource layer, through a unified message queue or event bus (such as Kafka or RabbitMQ). Before entering the processing pipeline, these alarms undergo standardization, converting alarm formats from different sources into a unified schema and supplementing fields such as the timestamp of the alarm's generation, the source object identifier, and the severity level. For example, alarms from Prometheus' Alertmanager, Zabbix triggers, and cloud platform monitoring are all connected to the same Topic and consumed uniformly by the alarm consumer.

[0047] The topology mapping phase is responsible for matching each standardized alert with nodes in the multi-dimensional application operations and maintenance graph. The system searches for the corresponding topology node in the graph based on the source object identifier carried by the alert (such as hostname, Pod name, service name, instance ID, etc.). If the alert source object is a Pod, it is mapped to that Pod node in the resource layer topology; if it's a timeout of a service interface, it's mapped to the corresponding method node in the application layer topology. After mapping, the alert is attached to the graph, possessing complete context information, including the business scenario to which the node belongs, upstream and downstream dependencies, and associated monitoring metrics and logs.

[0048] The anomaly path backtracking phase utilizes root cause analysis algorithms to perform causal reasoning on the graph. The algorithm can employ breadth-first search, starting from all alarm nodes and searching upstream and downstream along the topological edges, calculating the anomaly score for each node by combining topological weights and alarm priorities. Alternatively, it can use random walks or Bayesian inference to identify the node most likely to be the root cause. During the backtracking process, the algorithm records the path of anomaly propagation, for example, from CPU alarms at the resource layer to timeout alarms at the application layer, and then to transaction failure alarms at the business layer.

[0049] The root cause verification phase confirms candidate root causes. The system can automatically compare against a historical failure case library; if the current candidate root cause matches a pattern from a historical case, the confidence level increases. It can also trigger rapid automated verification operations, such as performing a health check on a suspected abnormal node or calling its self-healing script, and observing whether the alarms disappear. Once verification is successful, the candidate root cause is confirmed as the final root cause.

[0050] The results output phase presents the location results in a structured format. The system generates a fault report, including the root cause node, fault type, scope of impact (listing all affected business scenarios and services), an anomaly propagation path diagram, and recommended handling suggestions (such as restarting the Pod, scaling up the node, or rolling back the version). This report can be pushed to the operations and maintenance personnel's instant messaging tools, ticketing system, or visual dashboard, and is automatically archived in the knowledge base for subsequent review.

[0051] The benefits of this closed-loop process are significant. The unified standardization of alarm access eliminates processing obstacles caused by inconsistent alarm formats from multiple sources, enabling seamless integration of subsequent steps. Topology mapping transforms isolated alarms into contextualized events, preventing alarms from being viewed in isolation. Anomaly path backtracking automatically derives causal chains through algorithms, replacing the inefficient method of manual, layer-by-layer troubleshooting. The root cause verification step increases the reliability of the location results and reduces false alarms. The output transforms the location results into actionable guidelines, shortening the response time from problem discovery to resolution. The entire closed loop forms an automated link from alarm generation to root cause elimination. Operations personnel only need to confirm and execute the handling recommendations in the final step, significantly reducing the need for manual intervention. In actual deployment, after adopting this closed-loop process, one enterprise saw its average fault handling time decrease from 45 minutes to 12 minutes, and each location result has a complete path record for auditing and review, resulting in a qualitative improvement in the maturity and efficiency of the operations team.

[0052] In some embodiments, the step of performing root cause localization further includes: visually presenting the fault propagation path and affected business scope based on a multi-dimensional application operation and maintenance graph, and outputting recommended handling solutions. The function of visually presenting the fault propagation path and affected business scope is built upon the multi-dimensional application operation and maintenance graph. The system uses a front-end graphical engine (such as D3.js or AntV G6) to render the nodes and edges in the graph into an interactive topology map. After the root cause analysis algorithm completes the backtracking of the abnormal path, the system highlights the abnormal propagation path on the topology map with different colors: red indicates the fault source node, orange indicates the affected intermediate nodes, and yellow indicates the edge-affected nodes. Simultaneously, the system automatically calculates a list of affected business scenarios based on the business priorities and dependencies marked in the horizontal topology of the business layer, and circles these scenarios with dashed boxes or bubble charts to visually demonstrate the scope of the fault. For example, when a database node fails, the topology graph will start from that database node and extend upstream along the application service call topology, highlighting all services and methods that call that database. At the same time, the transaction processes that depend on these services will be marked in the business layer topology, such as "order creation process" and "payment process", which will both be shown as affected.

[0053] Regarding fault handling recommendations, the system maintains a fault handling knowledge base, which stores historical fault cases and their corresponding handling measures. Once the root cause is identified, the system extracts the feature vector of the current fault (including fault type, impact level, resource type, etc.), performs similarity matching with cases in the knowledge base, and recalls the most similar historical cases and their handling solutions. Simultaneously, the system can also dynamically generate targeted suggestions by combining configuration data in the CMDB and real-time monitoring metrics. For example, if the root cause is that the CPU of the node hosting the Pod is exhausted, the system will recommend "scaling up the node" or "migrating the Pod to an idle node"; if the root cause is that the database connection pool is exhausted, it will recommend "increasing the connection pool limit" or "checking slow queries." These recommended solutions, along with the fault propagation path diagram and the list of impact areas, generate a structured fault report and push it to the operations personnel's ticketing system or collaboration tools.

[0054] The beneficial effects of this embodiment are highly intuitive. By visually presenting the fault propagation path, operations and maintenance personnel no longer need to piece together the causal relationships between alarms in their minds; they can immediately see where the fault started, what steps it went through, and which businesses were affected. The automatic calculation of the affected business scope allows business leaders to know immediately which core processes are impacted, thus enabling rapid activation of emergency plans. The output of recommended handling solutions directly translates the fault location results into actionable actions, reducing the thinking time for operations and maintenance personnel between "knowing what the problem is" and "knowing how to fix it." In practical applications, after introducing this visualization function, one internet company reduced the duration of fault notification meetings from an average of 30 minutes to 10 minutes, because participants no longer needed to repeatedly confirm the scope of impact and handling steps; all information was clearly presented in the report. Simultaneously, the accuracy of the recommended handling solutions continuously improves with the accumulation of the knowledge base, and more and more common faults can achieve one-click self-healing, further reducing the cost of manual intervention. Overall, this embodiment delivers the final results of fault location to the operations and maintenance team in the most intuitive and practical way, truly achieving a closed-loop acceleration from problem discovery to problem resolution.

[0055] In some embodiments, the supporting resource topology also supports dynamic resource lifecycle management triggered by resource change events, responding to creation, destruction, and migration events of underlying resources and updating the vertical topology of the resource layer in real time. Dynamic resource lifecycle management supporting the resource topology is achieved by listening to resource change events of the underlying infrastructure. In specific implementations, the following technical solutions can be adopted. The first solution is based on the Kubernetes Watch API, where the system continuously listens for change events of core resources such as Pods, Nodes, and PersistentVolumes. When a new Pod is created, the Watch API immediately pushes an Add event. Upon receiving this event, the topology building module adds a Pod node to the vertical topology of the resource layer and connects it to the corresponding physical or virtual machine node based on the Pod's nodeName field. When a Pod is destroyed due to failure or scaling down, a Delete event triggers the system to remove the node and its associated edges. For migration events, such as when a Pod is rescheduled to another node, an Update event carries the new nodeName, and the system updates the association between the Pod node and its parent node accordingly. The second approach targets virtual machine environments. It involves monitoring the Event Manager of the virtualization platform (such as VMware vCenter) to capture VM creation, cloning, migration, and destruction events, and synchronously updating the topology. The third approach is a unified event bus in hybrid cloud scenarios. The system deploys event collectors on both the public and private cloud sides, uniformly reporting change events for various resources to a centralized message queue (such as Kafka). The topology consumer executes corresponding add, delete, and modify operations based on the event type and resource identifier. For example, when an ECS instance on Alibaba Cloud is automatically created due to elastic scaling, the event collector encapsulates the instance's ID, specifications, and availability zone information into a standard event and pushes it to Kafka. After consumption by the topology module, it is inserted as a new resource node into the vertical topology.

[0056] The beneficial effects of this embodiment are mainly reflected in real-time performance and automation. Traditional CMDBs rely on periodic scanning or manual entry to update resource information, with cycles typically lasting hours or even days. During this period, the topology is outdated, which may lead to alarms being associated with resources that no longer exist or newly created resources being missed. Event-driven lifecycle management compresses update latency to the second level, synchronizing the topology almost simultaneously with resource changes, ensuring the real-time accuracy of configuration data. For example, in an automatic scaling of a Kubernetes cluster, dozens of Pods are created within minutes. The event-driven mechanism captures these creation events one by one and updates the topology accordingly. Subsequent monitoring and alarm systems can immediately identify these new instances, avoiding monitoring blind spots caused by topology lag. In addition, timely response to resource destruction events also helps clean up zombie nodes in the topology, preventing abandoned resources from interfering with fault location. In actual deployment, after introducing this mechanism, a cloud service provider saw its CMDB configuration data accuracy increase from 82% to 99.5%, and false alarms caused by outdated configurations decreased by more than 90%. Meanwhile, automated topology updates free up the operations team from the manual maintenance of configurations, allowing them to focus more on higher-value fault analysis and optimization. Overall, this implementation enables the supporting resource topology to truly possess dynamic adaptability matching the cloud-native environment, providing a solid and reliable real-time data foundation for upper-layer data association and root cause localization.

[0057] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0058] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for locating the root cause of a fault based on a two-horizontal-one-vertical CMDB topology map, characterized in that, Includes the following steps: S1: Obtain the transaction data stream of the business system, construct the transaction business call topology based on the transaction data stream, and output the business layer horizontal topology; S2: Obtain application service call logs and tracing data, construct application service call topology based on the call logs and tracing data, and output application layer horizontal topology; S3: Obtain the configuration data and operating status data of the underlying IT resources, construct the supporting resource topology based on the configuration data and the operating status data, and output the vertical topology of the resource layer; S4: Connect the horizontal topology of the business layer, the horizontal topology of the application layer, and the vertical topology of the resource layer in the entire link, establish a data association model, and output a multi-dimensional application operation and maintenance graph with causal logic; S5: Access alarm and operation and maintenance data, perform fault root cause localization based on the multi-dimensional application operation and maintenance map, and output the fault root cause, impact range and handling suggestions by combining the root cause analysis algorithm.

2. The method according to claim 1, characterized in that, The transaction service call topology covers the core business system, transaction process and user request path, and marks business priority, traffic weight and dependency relationship by business scenario to form a business-level panoramic call view.

3. The method according to claim 1, characterized in that, The application service call topology supports microservice and service mesh environments. By collecting service registration information, interface call logs and tracing data, a three-level dependency topology of service-method-instance is constructed, and the dynamic call relationship is updated in real time based on the heartbeat detection and event notification mechanism of the service registry.

4. The method according to claim 3, characterized in that, The application service call topology is based on an event-driven mechanism, responding to the creation, destruction, and scaling events of service instances, and updating the service-method-instance three-level dependency topology in real time.

5. The method according to claim 1, characterized in that, The supporting resource topology is compatible with physical machines, virtual machines, containers and cloud platform full-stack resources, constructs a vertical support link of "infrastructure-middleware-application instance", and supports configuration item modeling and dynamic adaptation of containers, Pods, namespaces, configuration mapping, entry gateways and service mesh components.

6. The method according to claim 1, characterized in that, The data association model takes CMDB configuration items as its core, associates monitoring indicators, alarm information, log data and call chain data, establishes a one-to-one mapping between topology nodes and operation and maintenance data, and establishes a correspondence between the nodes in the business layer horizontal topology, the application layer horizontal topology and the resource layer vertical topology and the CMDB configuration items.

7. The method according to claim 1, characterized in that, The root cause analysis algorithm employs correlation analysis, link backtracking, and anomaly propagation path deduction, and combines topology weights and alarm priorities to filter redundant alarms and locate the underlying fault source.

8. The method according to claim 1, characterized in that, The steps for performing fault root cause localization include: alarm access, topology mapping, abnormal path backtracking, root cause verification and result output, forming a closed loop for full-link localization.

9. The method according to claim 1, characterized in that, The step of performing fault root cause localization also includes: based on the multi-dimensional application operation and maintenance map, visually presenting the fault propagation path and the scope of affected services, and outputting recommended handling solutions.

10. The method according to claim 1, characterized in that, The supporting resource topology also supports dynamic resource lifecycle management triggered by resource change events, responding to the creation, destruction, and migration events of underlying resources and updating the vertical topology of the resource layer in real time.

Images