A multi-unit analysis fault determination method based on operation and maintenance data

Abstract

The application relates to a multi-unit analysis fault determination method based on operation and maintenance data, and belongs to the operation and maintenance field.In the method, after an initial abnormal alarm is acquired, a fault locator can first determine a plurality of units related to the alarm and respective analysis indexes of the units, then introduces a fault propagation topological relationship among the units, arranges a to-be-analyzed object into a to-be-analyzed resource block set with a time sequence relationship, and forms a dynamic analysis sequence. Each resource block in the sequence only carries one kind of index of one unit, and each time during the analysis process, only one resource block is focused on, so that the computing power used for analysis is concentratedly used in any time window instead of being dispersed in all indexes, and thus the fault root cause can be accurately located from complicated operation and maintenance data.

Description

Technical Field

[0001] This invention belongs to the field of operation and maintenance, and relates to a multi-unit analysis method for fault determination based on operation and maintenance data. Background Technology

[0002] With the widespread application of information technology in various businesses, modern business systems generally adopt a distributed architecture, consisting of a large number of units such as servers, containers, microservices, and databases. These units call to each other and depend on each other. If any unit fails, it may spread along the call chain, affecting business continuity.

[0003] Against this backdrop, accurately locating the root cause of a failure from complex operational data has become a core technical issue in ensuring system reliability. Summary of the Invention

[0004] In view of this, the present invention provides a multi-unit analysis fault determination method based on operation and maintenance data to solve the above problems.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Firstly, a multi-unit analysis method for fault determination based on operation and maintenance data is provided. This method is applied to a fault locator and includes: the fault locator acquiring initial abnormal alarms of a service and identifying multiple units related to the initial abnormal alarms, as well as one or more analytical indicators for each unit. Each analytical indicator corresponds to a type of operation and maintenance data to be analyzed. The fault locator constructs a dynamic analysis sequence based on the fault propagation topology between the multiple units. The dynamic analysis sequence consists of a set of resource blocks to be analyzed that have dynamic temporal analysis relationships. Each resource block to be analyzed in the set includes an analytical indicator for a corresponding unit. The fault locator performs dynamic temporal analysis based on the dynamic analysis sequence to determine the unit containing the root cause of the service fault from the multiple units.

[0007] Therefore, after acquiring the initial abnormal alarm, the fault locator can first identify multiple units related to the alarm and their respective analytical metrics. Then, it introduces the fault propagation topology between these units, arranging the objects to be analyzed into a set of resource blocks with temporal relationships, forming a dynamic analysis sequence. In this sequence, each resource block carries only one metric of a unit, and the analysis focuses on only one resource block at a time. This ensures that the computing power used for analysis is concentrated within any given time window, rather than being dispersed across all metrics, thus enabling accurate location of the root cause of the fault from complex operational data.

[0008] Optionally, the fault locator constructs a dynamic analysis sequence based on the fault propagation topology relationship between multiple units, including: the fault locator determining the fault propagation topology relationship between multiple units from the global fault propagation topology relationship, the fault propagation topology relationship indicating the dependency direction of fault propagation between various analysis indicators of multiple units; the fault locator constructs a dynamic analysis sequence based on the reverse direction of the dependency direction, the unit that is closer to the downstream in the fault propagation topology relationship is given higher priority in the dynamic analysis sequence as the corresponding resource block to be analyzed.

[0009] Therefore, the fault propagation topology indicates the dependency direction of fault propagation among the analytical indicators of each unit. When constructing the dynamic analysis sequence, the fault locator determines this part of the topology relationship between multiple alarm-related units from the global fault propagation topology and arranges them according to the reverse direction of the dependency direction. This ensures that the unit closer to the downstream in the fault propagation link has its corresponding resource block to be analyzed given priority in the sequence, and the analysis path is opposite to the actual direction of fault propagation, so as to achieve the analysis can orderly approximate the root cause.

[0010] Optionally, for any one of the multiple units, if any one unit has multiple analysis indicators, then multiple resource blocks to be analyzed that contain the multiple analysis indicators of any one unit are arranged consecutively in the dynamic analysis sequence.

[0011] Therefore, when the same unit involves multiple analytical indicators due to an alarm, the multiple resource blocks to be analyzed, containing these indicators, are arranged consecutively in the dynamic analysis sequence. This consecutive arrangement allows the fault locator to perform centralized analysis of different indicators of the unit at adjacent positions in the sequence, rather than processing different indicators of the same unit in a scattered manner at different positions in the sequence, thus maintaining the continuity of the analysis process.

[0012] Optionally, each resource block in the set of resource blocks to be analyzed includes a time window in which dynamic timing analysis is performed on the resource block to be analyzed, and the order of each resource block to be analyzed in the set of resource blocks to be analyzed is consistent with the timing order of the time windows of each resource block to be analyzed.

[0013] Therefore, each resource block in the set of resource blocks to be analyzed contains a time window for which dynamic time-series analysis is performed, and the order of the resource blocks is consistent with the temporal order of their respective time windows. This constraint binds the arrangement of resource blocks in the sequence to the sequential order of analysis, so that the dynamic analysis sequence structurally carries both "what to analyze" and "when to analyze," providing a temporal reference for the orderly advancement of the analysis process, thereby realizing dynamic time-series analysis.

[0014] Optionally, the fault locator performs dynamic time-series analysis based on a dynamic analysis sequence to determine the unit where the root cause of the service fault resides from multiple units. This includes: for the first resource block to be analyzed in the current time window of the dynamic analysis sequence, the first resource block to be analyzed contains a first analysis indicator of the first unit among multiple units; the fault locator collects the operation and maintenance data corresponding to the first analysis indicator within the current time window; the fault locator determines the analysis result of the first unit based on the operation and maintenance data corresponding to the first analysis indicator; the fault locator determines the unit where the root cause of the service fault resides as the first unit based on the analysis result, and / or, the fault locator determines the second resource block to be analyzed in the next time window after the current time window based on the analysis result, until the unit where the root cause of the service fault resides is completely determined, and the second resource block to be analyzed includes another second analysis indicator of the first unit and a third analysis indicator of the second unit among multiple units. Therefore, this execution method, which determines the next analysis target in real time based on the current analysis result, allows the analysis sequence to evolve dynamically during execution, rather than remaining fixed, thus achieving flexible and dynamic determination of the root cause.

[0015] Optionally, the fault locator determines the unit where the root cause of the service fault is located as the first unit based on the analysis results, including: the fault locator indicates that the first unit itself has an abnormal operation based on the analysis results, and the abnormal start time of the first unit is earlier than the units that have been analyzed among multiple units and are located downstream of the first unit, and there are no units located upstream of the first unit in the fault propagation topology, and all the resource blocks to be analyzed corresponding to the first unit have been analyzed, thus determining the unit where the root cause of the fault is located as the first unit.

[0016] Therefore, to determine whether the first unit is the root cause of the fault, four conditions must be met simultaneously: the first unit itself exhibits an anomaly, its anomaly begins earlier than all analyzed downstream units, there are no upstream units in the fault propagation topology, and all resource blocks corresponding to the first unit have been analyzed. These four conditions together constitute a closed-loop verification to ensure the accuracy of root cause determination.

[0017] Optionally, the method further includes: if the analysis results indicate that the first unit itself is operating normally and there is no unit located upstream of the first unit in the fault propagation topology, then the fault locator determines that the unit where the root cause of the fault is located is the unit with the earliest abnormal start time among the units that have been analyzed.

[0018] Therefore, when the analysis results show that the first unit itself operates without abnormalities and there are no upstream units in the fault propagation topology, the fault locator does not directly determine that there is no root cause. Instead, it identifies the unit with the earliest abnormal start time from all the analyzed units as the root cause. This approach can handle scenarios where the root cause is located in the middle of the analysis path rather than at the end of the topology, avoiding omissions and further improving the accuracy of root cause determination.

[0019] Optionally, based on the analysis results, the fault locator determines that the second resource block to be analyzed in the dynamic analysis sequence needs to be analyzed in the next time window after the current time window. This includes: the fault locator indicating, based on the analysis results, that the first unit itself has an abnormal operation, and the abnormal start time of the first unit is later than the units that have been analyzed and are located downstream of the first unit, determining that the second resource block to be analyzed in the next time window after the current time window contains a third analysis indicator; or, the fault locator indicating, based on the analysis results, that the first unit itself has no abnormal operation, and the abnormal situation of the first unit is affected by the second unit located upstream of the first unit, the fault... The locator removes other unanalyzed resource blocks from the dynamic analysis sequence of the first unit and determines that the second resource block to be analyzed in the next time window after the current time window contains the third analysis indicator; or, the fault locator indicates that the first unit itself has an abnormal operation based on the analysis results, and the abnormal start time of the first unit is earlier than the units that have been analyzed and are located downstream of the first unit, and the first unit also has other unanalyzed resource blocks, and determines that the second resource block to be analyzed in the next time window after the current time window contains the second analysis indicator.

[0020] Therefore, when the root cause determination criteria are not met and the next resource block to be analyzed needs to be determined, there are three specific jump rules: If the first unit itself is abnormal but its start time is later than that of the downstream unit, then the next resource block to be analyzed is the resource block corresponding to its upstream unit; if the first unit itself is not abnormal and the abnormality is affected by the upstream unit, then the other unanalyzed resource blocks of the first unit are removed first, and then the analysis jumps to the resource block corresponding to its upstream unit; if the first unit itself is abnormal and its start time is earlier than that of the downstream unit, but there are still other unanalyzed resource blocks, then the analysis continues with the remaining resource blocks of the first unit. These three rules ensure that the analysis path always proceeds in the direction indicated by the evidence, guaranteeing the accuracy of root cause determination.

[0021] Optionally, the fault locator acquires the initial abnormal alarm of the service and determines multiple units related to the initial abnormal alarm and one or more analytical indicators for each of the multiple units, including: the fault locator acquires the initial abnormal alarm; the fault locator determines the multiple units involved in the abnormal situation indicated by the initial abnormal alarm, and generates at least one fault hypothesis for each of the multiple units based on the abnormal situation; the fault locator determines one or more analytical indicators for each of the multiple units based on a preset hypothesis-indicator mapping relationship and at least one fault hypothesis.

[0022] Therefore, when determining the units related to the alarm and their analysis indicators, the fault locator first acquires the initial abnormal alarm, identifies the multiple units involved in the abnormal situation indicated by the alarm, then generates at least one fault hypothesis for each unit based on the abnormal situation, and finally determines one or more analysis indicators for each unit through a preset hypothesis-indicator mapping relationship. This process ensures that the selection of indicators is driven by fault hypotheses, rather than full data collection, and the determination of each resource block to be analyzed has a clear reasoning basis.

[0023] Optionally, the unit can be any of the following: server, container, microservice instance, database instance, middleware node, network device; the analysis metric can be any of the following: CPU utilization, memory utilization, disk I / O wait time, network retransmission rate, interface response time, call error rate, number of slow queries, connection pool utilization, garbage collection pause time.

[0024] In a second aspect, an electronic device is provided, including a memory and a processor, wherein the memory is used to store a computer program and the processor is used to execute the computer program to implement the method of the first aspect.

[0025] The objectives and other advantages of this invention can be realized and obtained through the following description. Attached Figure Description

[0026] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0027] Figure 1 A schematic diagram of the architecture of an operation and maintenance system provided by the present invention;

[0028] Figure 2 A flowchart of a multi-unit analysis fault determination method based on operation and maintenance data provided by the present invention;

[0029] Figure 3 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0030] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other. The accompanying drawings are for illustrative purposes only, representing only schematic diagrams and not actual physical objects, and should not be construed as limiting the present invention. To better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product form.

[0031] Figure 1 This is a schematic diagram of the architecture of an operation and maintenance system provided in an embodiment of this application. The operation and maintenance system includes a fault locator and a business system.

[0032] The business system is the information system that carries out business operations. In one possible implementation, the business system is deployed in a distributed architecture, consisting of various units such as servers, containers, microservice instances, database instances, middleware nodes, and network devices. These units have call and dependency relationships, and each unit continuously generates operational data during operation. This operational data includes, but is not limited to, central processing unit (CPU) utilization, memory usage, disk input / output (IO) latency, network retransmission rate, interface response time, call error rate, number of slow queries, connection pool occupancy, and garbage collection (GC) pause time. When the business system detects an anomaly in its operation, it generates an initial anomaly alarm and reports this alarm to the fault locator. The initial anomaly alarm at least indicates the surface information of the anomaly and the one or more units involved.

[0033] The fault locator is the executing entity of the fault root cause determination method provided in this application. After obtaining the initial abnormal alarm reported by the business system, the fault locator identifies multiple units related to the alarm and one or more analysis indicators for each unit based on the initial abnormal alarm. It then constructs a dynamic analysis sequence based on the fault propagation topology between the multiple units and performs dynamic time-series analysis based on the dynamic analysis sequence to determine the unit containing the root cause of the business fault from among the multiple units, and outputs the determination result. The specific workflow of the fault locator will be described in detail in subsequent method embodiments.

[0034] The fault locator can be deployed on a standalone physical server, a virtual server, or a container instance. It can also be deployed in a cloud environment as Software as a Service (SaaS), providing an interface for external calls as a cloud-based fault location service. The fault locator connects to the business system via a communication network, which can be a wired network, a wireless network, or a combination of both. For example, data communication can be based on Transmission Control Protocol / Internet Protocol (TCP / IP). Internally, the fault locator may include a processor, memory, and a communication interface. The memory stores program instructions, hypothesis-indicator mapping relationships, global fault propagation topology, and other data. The processor is coupled to the memory and is used to call the program instructions and data in the memory to execute the fault root cause determination method provided in this application. The communication interface is used for data interaction with the business system, such as receiving initial anomaly alarms reported by the business system and collecting operational data from various units of the business system. The fault locator obtains operational data from the business system in several ways: the business system can actively report data via an Application Programming Interface (API); the fault locator can initiate collection requests to various units within the business system, and each unit can respond and return data; or the fault locator can subscribe to and obtain data from the data pipeline or message queue connected to the business system. This application does not limit the specific implementation method. In another possible implementation, the fault locator can be a functional module integrated into other operational platforms or systems, rather than a standalone device. When the fault locator is a functional module, it can be embedded into existing automated operations platforms, artificial intelligence for IT operations platforms (AIOps platforms), or monitoring platforms in the form of software plugins, microservice components, or application packages. It can reuse the platform's processor, memory, and communication interface resources to obtain initial anomaly alarms and operational data from the business system through the platform's internal interfaces.

[0035] Based on the above system architecture, the following detailed description of the multi-unit analysis fault determination method based on operation and maintenance data provided in this application will be provided in conjunction with specific embodiments.

[0036] The following detailed description of the multi-unit analysis fault determination method based on operation and maintenance data provided in this application, with reference to specific embodiments, provides a detailed explanation.

[0037] like Figure 2 As shown in the figure, this application provides a multi-unit analysis fault determination method based on operation and maintenance data, which is applied to the system described in the above embodiment.

[0038] The specific process of this method is as follows:

[0039] S201, the fault locator obtains the initial abnormal alarm of the service and determines multiple units related to the initial abnormal alarm and one or more analysis indicators for each of the multiple units, with each analysis indicator corresponding to a type of operation and maintenance data to be analyzed.

[0040] Initial anomaly alerts are alert messages generated by a business system when it detects an anomaly in its operation. An initial anomaly alert must include at least an anomaly description and the identifier of the involved unit. The anomaly description describes what happened, such as "interface response timeout," "increased error rate," "connection pool exhaustion," or "CPU utilization exceeding threshold." This description can be a text description or a structured field, such as a key-value pair like "alert_type: timeout, threshold: 2000ms, current_value: 3500ms." The unit identifier is information that uniquely identifies a unit within the business system, such as a microservice instance name, database instance name, container number, or server hostname. In distributed systems, the unit identifier is usually environment-dependent; for example, in a containerized Kubernetes environment, the unit identifier might be a Pod name or service name; in a traditional physical machine deployment environment, it might be a combination of hostname and process name. It's important to note that the initial anomaly alert typically only directly identifies the unit with the most prominent anomaly description. For example, the alarm "The order timeout rate of gateway A exceeds 15%" directly indicates only gateway A. The subsequent fault locator will use topology expansion to identify other units related to this alarm. Therefore, the units involved in the initial anomaly alarm are not identical in scope to the multiple units ultimately identified in S201; the former is usually a subset of the latter. In one possible implementation, the initial anomaly alarm may also include auxiliary information such as the timestamp of the anomaly occurrence, the duration of the anomaly, and the alarm level. The timestamp of the anomaly occurrence can be a single point in time, such as "2025-06-15 14:23:05," or a time range, such as "2025-06-15 14:23:05 to 14:25:30." The duration of the anomaly indicates how long the anomaly has lasted and is helpful in distinguishing between transient jitter and persistent faults. The alarm level, such as "critical," "warning," or "notification," can be used to help determine the scope of the anomaly's impact. Initial anomaly alarms can be proactively reported to the fault locator by the business system via API, or manually triggered by operations and maintenance personnel based on anomaly indicators on the monitoring panel; this application does not impose any restrictions on this. When proactively reported by the business system via API, the API can be a Representational State Transfer (REST) ​​style HTTP interface or a Remote Procedure Call (RPC) interface, and the alarm data format can be a structured format such as JSON or XML.

[0041] The aforementioned units refer to the components that carry out business operations within a business system. Any one of these units can be any of the following: server, container, microservice instance, database instance, middleware node, or network device. Specifically, a server can be a physical server or a virtual machine running on a physical server; a container can be a Docker container or a container instance managed by another container runtime; a microservice instance refers to a running instance of a microservice within a business system, and a microservice can have one or more running instances; a database instance can include relational database instances (such as MySQL and PostgreSQL) and non-relational database instances (such as Redis and MongoDB); a middleware node can include message queue nodes (such as Kafka Broker), search engine nodes (such as Elasticsearch nodes), etc.; and a network device can include switches, routers, load balancers, etc. The format and collection method of the operational data generated by different unit types may differ. When collecting operational data subsequently, the fault locator can select the corresponding collection protocol or collection proxy based on the unit type.

[0042] Analytical metrics refer to monitoring items used to measure the operational status of a unit. Accordingly, analytical metrics can be any of the following: CPU utilization, memory utilization, disk I / O wait time, network retransmission rate, interface response time, call error rate, number of slow queries, connection pool occupancy, and garbage collection (GC) pause time. Interface response time can be further subdivided into different quantiles such as average, P50, P90, P99, and P999 to accommodate the varying latency sensitivities of different services. Call error rate can be subdivided by error type, such as timeout error rate, connection rejection error rate, and abnormal return code error rate. Each analytical metric corresponds to a type of operational data to be analyzed. For example, the operational data corresponding to the analytical metric "CPU utilization" is time-series data of CPU utilization, consisting of multiple data points, each containing a timestamp and a CPU utilization value. The operational data corresponding to the analytical metric "P99 time spent on downstream calls" is the result of calculating the P99 value from the sampled time data of each call to the downstream interface within the statistical time window. Operation and maintenance data can be stored on the business system side in the form of time-series databases, log files, or message streams. The fault locator retrieves the data from the corresponding storage as needed when performing subsequent steps.

[0043] S201 specifically includes the following three steps:

[0044] Step 1: The fault locator acquires initial anomaly alarms. This can be achieved either through proactive reporting by the business system via an API interface, or through manual reporting triggered by operations personnel based on anomaly indicators on the monitoring panel. When using the API interface for proactive reporting, the fault locator provides an alarm receiving interface. The business system calls this interface after triggering an alarm rule, sending the alarm information to the fault locator in a structured data format. When using manual reporting, operations personnel can manually package the current anomaly information into an alarm message and send it to the fault locator via the "Submit Diagnosis" button on the monitoring panel after observing anomaly indicators. After acquiring the initial anomaly alarm, the fault locator parses the anomaly symptoms and the involved unit identifiers from the alarm content. The parsing method can be: if the alarm is structured data, directly extract the corresponding fields; if the alarm is a text description, extract the anomaly type and unit name through preset keyword matching or regular expressions. After parsing, the fault locator stores the anomaly symptoms and unit identifiers in memory or a cache for use in subsequent steps.

[0045] Step 2: The fault locator identifies the multiple units involved in the abnormal situation indicated by the initial abnormal alarm, and generates at least one fault hypothesis for each of the multiple units based on the abnormal situation.

[0046] Regarding the identified units, it should be noted that the unit directly specified in the initial alarm is usually only the unit with the most obvious abnormal symptoms, while the true root cause of the fault may be located upstream or downstream of that unit. Therefore, the fault locator takes the unit involved in the initial alarm as the starting point and searches for upstream and downstream units that have a direct dependency relationship with it in the global fault propagation topology, including these units in the candidate range to form a set of units related to this alarm.

[0047] Before introducing the specific process of determining the unit, we first explain the two dependency directions and the definition of "upstream / downstream" involved in this application. In the call relationship of a business system, the request is sent from the caller to the callee. For example, if gateway A calls order service B, the call direction is A→B, where A is the caller and B is the callee. However, from the perspective of fault propagation, faults often propagate from the callee to the caller. For example, if database C slows down, it causes order service B, which calls it, to also time out, which in turn causes gateway A, which calls B, to also time out. In this case, the fault propagation direction is C→B→A, which is the opposite of the call direction. The "upstream" and "downstream" referred to in this application are uniformly defined based on the direction of fault propagation. The starting point of fault propagation is upstream, and the destination of fault propagation is downstream. In the above example, C is the upstreammost unit, B is the intermediate unit, and A is the downstreammost unit. That is, the direction of the root cause of the fault is upstream, and the direction of the fault manifestation is downstream. The expressions "reverse direction of dependency direction," "tracing upstream," and "downstream unit" involved in subsequent steps are all based on this definition. This convention helps to naturally form an analytical path that "gradually converges from the representation end to the root cause end" when constructing dynamic analysis sequences.

[0048] Based on the above definition, the fault locator starts with the unit involved in the initial alarm and searches for units with fault propagation dependencies on these units in the global fault propagation topology. The global fault propagation topology is topological data describing the causal chain of fault propagation between units in the business system and between the analytical metrics of each unit. This topology data can be stored in the form of a directed graph, where nodes represent units or their analytical metrics, and directed edges represent the direction of fault propagation. For example, an edge from node "number of slow queries in database C" to node "P99 time consumption of order service B's call to C" indicates that an increase in the number of slow queries in database C (upstream) will lead to an increase in the time consumption of order service B's call to C (downstream). The global fault propagation topology can be pre-built and stored in the fault locator, or it can be dynamically generated by the fault locator after obtaining service call chain information from external systems such as the configuration management database (CMDB) or service mesh through a communication interface after receiving an alarm. When retrieved from the CMDB, static deployment dependencies between services can be obtained; when retrieved from the Service Mesh, dynamic call relationships and call metrics between services can be obtained. The fault locator can extract the actual call direction between units from the call chain information and, combined with known fault propagation patterns, automatically generate metric-level fault propagation topology relationships. When determining candidate units, the fault locator extends along the fault propagation topology relationship upstream (i.e., the root cause direction) and downstream (i.e., the manifestation direction) by a preset number of hops, such as extending upstream by 2 hops and downstream by 1 hop, or it can extend all the way to the topology boundary. The specific extension strategy can be configured according to actual needs.

[0049] The generation of fault hypotheses refers to an inference made by the fault locator based on anomaly symptoms to determine the possible fault types of a given unit, guiding subsequent analysis metrics to be collected. The fault locator generates at least one fault hypothesis for each unit based on anomaly symptoms and the unit's position in the fault propagation topology. Common fault hypothesis types include, but are not limited to: internal resource bottlenecks, internal logic anomalies, downstream dependency drag, sudden surge in upstream call volume, and network communication failures. Internal resource bottlenecks refer to performance degradation caused by insufficient hardware resources such as CPU, memory, and disk I / O; internal logic anomalies refer to problems in the unit's own code or configuration, such as slow queries, deadlocks, infinite loops, and memory leaks; downstream dependency drag refers to a situation where the unit itself is functioning normally, but downstream units it calls respond slowly or erroneously, causing the unit's performance to also be abnormal (here, "downstream" is based on the call direction, i.e., the callee); sudden surge in upstream call volume refers to a sudden increase in the number of requests from upstream units, exceeding the unit's processing capacity (here, "upstream" is based on the call direction, i.e., the request initiator); and network communication failures refer to problems in the network links between units, such as packet loss, retransmission, and increased latency. The specific method for generating fault hypotheses can be as follows: The fault locator maintains a fault hypothesis generation rule base, which defines which fault hypotheses typically correspond to units at different positions in the call chain under different anomaly manifestations. For example, for an alarm about an interface response timeout, the corresponding fault hypotheses in the rule base for the called unit located in the middle of the call chain include "its own CPU or memory resource bottleneck" and "slow response of downstream dependencies"; for the database unit located at the bottom of the call chain, the corresponding fault hypotheses include "its own slow query" and "its own disk I / O bottleneck". The rule base can be built based on the experience of operation and maintenance experts and historical fault cases, and can be customized by operation and maintenance personnel according to the actual business environment. In another possible implementation, fault hypotheses can also be generated by the fault locator based on the unit type and anomaly manifestation using a preset general mapping template, without relying on the rule base. For example, for all units located in the middle of the call chain, hypotheses containing "its own bottleneck" and "downstream drag" are generated.

[0050] Step 3: The fault locator determines one or more analytical indicators for each of the multiple units based on the preset hypothesis-indicator mapping relationship and at least one fault hypothesis.

[0051] The hypothesis-metric mapping relationship refers to the correspondence between fault hypothesis types and key operational metrics used to verify those hypotheses. This mapping relationship is predefined based on prior knowledge in the operations and maintenance domain, ensuring that when verifying a fault hypothesis, only a few of the most relevant metrics need to be collected, rather than all metrics for that unit. The mapping relationship can be stored in the fault locator's memory as a mapping table. Each record in the mapping table contains a fault hypothesis type field and a corresponding list of analytical metrics field. When the same fault hypothesis type corresponds to multiple analytical metrics, these metrics can be parallel or complementary. In one possible implementation, the mapping relationship can also include suggestions for the collection parameters of each analytical metric, such as collection frequency and collection window length. These parameters can be used to configure the collection behavior of each time window during subsequent dynamic time-series analysis. Furthermore, the mapping relationship can also set priorities for multiple analytical metrics under the same fault hypothesis to determine the collection order of metrics when analytical resources are limited. An exemplary hypothesis-metric mapping relationship is shown in Table 1. The fault locator maps each fault hypothesis generated in step 2 to one or more corresponding analytical indicators using the mapping relationship shown in Table 1. Then, it merges and deduplicates the analytical indicators corresponding to all fault hypotheses in the same unit to obtain one or more analytical indicators that the unit ultimately needs to collect. The merging and deduplication process is as follows: the analytical indicators mapped from all fault hypotheses in the unit are placed into a set, and duplicates are automatically removed using the uniqueness of the elements in the set. The final output is an indicator list.

[0052] An exemplary hypothesis-index mapping relationship is shown in Table 1.

[0053] Table 1

[0054] CPU resource bottleneck CPU utilization, CPU run queue length Self-resource bottleneck Memory usage, page swapping frequency, GC pause time Disk I / O bottleneck Disk I / O wait time, disk queue depth The query itself is malfunctioning (slow query). Number of slow queries, average query time The logic itself is flawed (lock conflict). Lock wait time, number of deadlock detections Downstream dependence drags down P99 response time to downstream calls, call success rate, and number of threads waiting for downstream responses. Upstream call volume surge Request arrival rate, number of active connections in the connection pool Network communication failure TCP retransmission rate, network round-trip time, and socket connection timeout count

[0055] To facilitate understanding, the process of determining the above-mentioned units and analysis indicators will be illustrated below using two specific business scenarios. The following steps, S202 and S203, will further explain the construction of the dynamic analysis sequence and the execution process of the dynamic time series analysis based on these two scenarios.

[0056] Scenario 1: Simple chained dependency scenario.

[0057] An e-commerce business system has an order placement chain consisting of the following units: Gateway A, Order Service B, and Product Inventory Database C. Gateway A is responsible for receiving order requests from external users, performing protocol conversion, and routing forwarding. Order Service B is responsible for handling the order placement business logic, including verifying inventory and generating order records. Product Inventory Database C is responsible for storing product inventory data and providing inventory query and deduction operations. The calling dependency relationship among the three is: Gateway A calls the interface of Order Service B, and Order Service B calls the query and update interface of Product Inventory Database C, i.e., the calling direction is A→B→C. From the fault propagation perspective of this application, the fault propagation direction is C→B→A, i.e., C is the upstream, B is the intermediate unit, and A is the downstream. Gateway A and Order Service B are both deployed in a container cluster as containerized microservice instances, while Product Inventory Database C is deployed on a standalone server as a database instance.

[0058] At a certain moment, the business system generates an initial anomaly alarm: "The order placement interface timeout rate of Gateway A exceeds 15%." After receiving this alarm, the fault locator analyzes it and finds the anomaly to be "interface response timeout," involving the unit "Gateway A." Starting from Gateway A, the fault locator searches for units with fault propagation dependencies within the global fault propagation topology. The global fault propagation topology indicates that, according to the fault propagation direction, an anomaly in C can propagate to B, and an anomaly in B can propagate to A. The fault locator extends upstream along the fault propagation direction and finds that A's upstream is B, B's upstream is C, and C has no upstream units. Therefore, the fault locator determines the units related to this alarm as: Gateway A, Order Service B, and Product Inventory Database C.

[0059] Next, the fault locator generates fault hypotheses for each unit. For gateway A, located at the downstream end of the fault propagation chain, its primary responsibility is request forwarding, and it doesn't contain complex business logic. Therefore, the root cause of its response timeout is usually not insufficient processing capacity, but rather the excessively long wait time for the backend service response. Based on gateway A's unit type and topology location, the fault locator generates the fault hypothesis "slow backend service response leading to connection pool accumulation." For order service B, located in the middle of the fault propagation chain, it may experience problems with its own processing logic or be hampered by upstream dependencies. The fault locator generates fault hypotheses for it including "its own CPU or memory resource bottleneck" and "slow response from the called database C." For product inventory database C, located at the upstream end of the fault propagation chain, it is the final bearer of data access. The fault locator generates fault hypotheses for it including "slow queries" and "its own disk I / O bottleneck."

[0060] Then, the fault locator maps the fault assumptions of each unit to analysis indicators according to the hypothesis-indicator mapping relationship shown in Table 1.

[0061] For gateway A, the fault hypothesis "slow backend service response leading to connection pool backlog" is not among the standard fault hypothesis types in Table 1. In one possible implementation, the fault locator can break down or categorize this hypothesis into existing hypothesis types in Table 1. Since the direct cause of "connection pool backlog" is that the request volume exceeds the processing capacity of the backend service, it can be classified as "sudden increase in upstream call volume." It should be noted that for gateway A, which is located at the entry point of the business system, its direct callers are usually outside the scope of this analysis. Therefore, the fault locator detects whether the anomaly is caused by a sudden increase in external request traffic by collecting gateway A's own "request arrival rate" and "number of active connections in the connection pool." Thus, the fault locator maps "slow backend service response leading to connection pool backlog" to two analytical metrics: "request arrival rate" and "number of active connections in the connection pool."

[0062] For order service B, the fault hypothesis "bottlenecks in its own CPU or memory resources" corresponds to the two types of "bottlenecks in its own CPU resources" and "bottlenecks in its own memory resources" in Table 1. The mapped analysis metrics are: CPU utilization, CPU run queue length, memory usage, paging frequency, and GC pause time. Since CPU run queue length and paging frequency may not be easily obtained directly in some deployment environments, in one possible implementation, the fault locator can be tailored according to the actual range of collectable metrics, retaining only CPU utilization, memory usage, and GC pause time. The fault hypothesis "slow response from downstream database C" corresponds to "downstream dependency drag" in Table 1. The mapped analysis metrics are: P99 response time for calls to C, call success rate, and number of threads waiting for C's response. After merging and deduplicating the above metrics, the analysis metrics for order service B are determined as: CPU utilization, memory usage, GC pause time, P99 response time for calls to C, call success rate, and number of threads waiting for C's response.

[0063] For the commodity inventory database C, the fault hypothesis "slow query" corresponds to "self-logical anomaly (slow query)" in Table 1, and the corresponding analytical metrics are: number of slow queries and average query time. The fault hypothesis "disk I / O bottleneck" corresponds to "disk I / O bottleneck" in Table 1, and the corresponding analytical metrics are: disk I / O wait time and disk queue depth. After merging, the analytical metrics for commodity inventory database C are determined as: number of slow queries, average query time, disk I / O wait time, and disk queue depth.

[0064] At this point, the analytical metrics for each unit in Scenario 1 have been determined. They are summarized as follows: For Gateway A, the analytical metrics are request arrival rate and the number of active connections in the connection pool; for Order Service B, the analytical metrics are CPU utilization, memory utilization, GC pause time, P99 response time to C calls, call success rate, and the number of threads waiting for C responses; and for the Product Inventory Database C, the analytical metrics are the number of slow queries, average query time, disk I / O wait time, and disk queue depth.

[0065] Scenario 2: Branch dependency scenario with shared nodes.

[0066] The recommendation and shopping cart workflow of an e-commerce system includes the following units: Shopping Cart Service A1, Recommendation Service A2, Product Details Service B, and Cache Cluster C. Shopping Cart Service A1 handles user add-to-cart operations; Recommendation Service A2 generates a list of recommended products based on user profiles; Product Details Service B provides detailed product information, including price, inventory status, and promotional tags; and Cache Cluster C caches product details data to reduce access pressure on the underlying database. The dependencies between these services are as follows: Shopping Cart Service A1 needs to retrieve product details when processing add-to-cart requests, therefore it calls the interface of Product Details Service B; Recommendation Service A2 also needs to retrieve product details when generating the recommendation list, similarly calling the interface of Product Details Service B; Product Details Service B, when processing requests, prioritizes reading data from Cache Cluster C, and if the cache misses, it retrieves data from the underlying database. The call directions are A1→B→C and A2→B→C. According to the fault propagation perspective of this application, the fault propagation direction is C→B→A1 and C→B→A2, that is, C is the upstream, B is the intermediate shared node, and A1 and A2 are the two downstream branch units. The shopping cart service A1, recommendation service A2, and product detail service B are all deployed as microservice instances, and the cache cluster C is deployed as a Redis cluster instance.

[0067] At a certain moment, the business system simultaneously generates two initial anomaly alarms: "Add to cart interface response timed out" for shopping cart service A1 and "Recommendation interface response timed out" for recommendation service A2. After acquiring these two alarms, the fault locator analyzes them and finds that both alarms exhibit the anomaly of "interface response timeout," involving shopping cart service A1 and recommendation service A2. Starting from A1 and A2 respectively, the fault locator searches for fault propagation dependencies in the global fault propagation topology, discovering that the upstream of A1 is product detail service B, and the upstream of A2 is also product detail service B, with cache cluster C being the upstream of B. Therefore, the fault locator determines that the units related to this alarm are: shopping cart service A1, recommendation service A2, product detail service B, and cache cluster C.

[0068] Next, the fault locator generates fault hypotheses for each unit. For shopping cart service A1 and recommendation service A2, both are at the downstream end of the fault propagation chain and exhibit the same anomaly; therefore, the generated fault hypothesis is "downstream service response slows down." For product detail service B, located in the middle of the fault propagation chain and a shared upstream of A1 and A2, exhibiting an interface timeout anomaly, the fault locator generates fault hypotheses based on B's topological location and unit type, including "its own CPU or memory resource bottleneck," "downstream cache cluster C response slows down," and "overwhelmed by a sudden surge in upstream call volume." The hypothesis of "overwhelmed by a sudden surge in upstream call volume" is based on B's characteristic as a shared node: when the request volume of A1 and A2 increases simultaneously, the total number of requests received by B may exceed its processing capacity. For cache cluster C, located at the upstream end of the fault propagation chain, the fault locator generates fault hypotheses including "insufficient processing capacity" and "slow queries." In the context of caching units, "insufficient self-processing capacity" mainly refers to the bottleneck of a single node's CPU or the overload of some nodes caused by uneven cluster sharding; "slow queries" in caching systems such as Redis usually correspond to commands recorded in the slow log.

[0069] Then, the fault locator maps the fault assumptions of each unit to analysis indicators according to the hypothesis-indicator mapping relationship shown in Table 1.

[0070] For shopping cart service A1, the fault hypothesis "slower downstream service response" corresponds to "downstream dependency drag" in Table 1, and the mapped analysis metrics are: P99 response time for calls to B, call success rate, and number of threads waiting for B's response. After merging, the analysis metrics for A1 are determined to be the above three items. Similarly, for recommendation service A2, the analysis metrics are determined to be: P99 response time for calls to B, call success rate, and number of threads waiting for B's response.

[0071] For product detail service B, the fault hypothesis "its own CPU or memory resource bottleneck" is mapped to: CPU utilization, CPU run queue length, memory usage, paging frequency, and GC pause time. In one possible implementation, based on collectability, CPU utilization, memory usage, and GC pause time are pruned and retained. The fault hypothesis "downstream cache cluster C responds slowly" corresponds to "downstream dependency drag," mapped to: P99 response time for calls to C, call success rate, and number of threads waiting for C's response. The fault hypothesis "overwhelmed by a sudden surge in upstream calls" corresponds to "a sudden surge in upstream calls," mapped to: request arrival rate and number of active connections in the connection pool. After merging and deduplicating the above three sets of indicators, the analysis indicators for product detail service B are determined as: CPU utilization, memory usage, GC pause time, P99 response time for calls to C, call success rate, number of threads waiting for C's response, request arrival rate, and number of active connections in the connection pool.

[0072] For cache cluster C, the fault hypothesis "insufficient processing capacity" corresponds to "its own CPU resource bottleneck," mapped to: CPU utilization and CPU run queue length. After pruning, CPU utilization is retained. The fault hypothesis "slow queries" corresponds to "its own logical anomalies (slow queries)," mapped to: number of slow queries and average query time. Since caching systems like Redis typically record slow logs rather than slow queries in relational databases, in one possible implementation, "number of slow queries" specifically corresponds to the number of slow log entries in Redis, and "average query time" corresponds to the average execution time of commands in the slow logs. After merging, the analytical metrics for cache cluster C are determined as: CPU utilization, number of slow queries, and average query time.

[0073] At this point, the analysis metrics for each unit in Scenario 2 have been determined. They are summarized as follows: For shopping cart service A1 and recommendation service A2, the analysis metrics are P99 response time to call B, call success rate, and the number of threads waiting for B's response; for product details service B, the analysis metrics are CPU utilization, memory usage, GC pause time, P99 response time to call C, call success rate, number of threads waiting for C's response, request arrival rate, and number of active connections in the connection pool; for cache cluster C, the analysis metrics are CPU utilization, number of slow queries, and average query time.

[0074] In both scenarios above, the fault locator accurately determines a small number of key indicators that need to be collected for each unit through fault assumptions and assumption-indicator mapping, rather than collecting all indicators, thus centralizing analytical computing power from the source. It should be noted that the specific methods of indicator pruning and mapping relationships in the above examples are merely illustrative and can be flexibly adjusted according to the specific business environment and the range of collectable indicators in actual implementation. This application does not impose any limitations on these aspects.

[0075] S202, the fault locator constructs a dynamic analysis sequence based on the fault propagation topology relationship between multiple units.

[0076] A dynamic analysis sequence is an ordered set of multiple resource blocks to be analyzed, arranged according to a specific temporal relationship. This set specifies which time windows the fault locator will focus on, and for which unit, which analysis metric, during subsequent dynamic temporal analysis. The dynamic analysis sequence is called "dynamic" because during the execution of S203, the fault locator adjusts the unexecuted portions of the sequence in real time based on the analysis results of each time window. This includes removing resource blocks that no longer need analysis or shifting the focus of the next time window to another resource block in the sequence. In other words, the dynamic analysis sequence is an initial sequence at the beginning of its construction and evolves dynamically during execution.

[0077] The resource block to be analyzed is the basic unit constituting the dynamic analysis sequence and the smallest scheduling unit for the fault locator when performing dynamic time-series analysis. Each resource block in the set to be analyzed includes one analysis metric corresponding to a given unit. For example, the CPU utilization of order service B, the P99 time of order service B's call to C, and the number of slow queries in the product inventory database C are each independent resource blocks to be analyzed. The position of the resource blocks to be analyzed in the sequence determines their analysis order. For simplicity, the term "resource block to be analyzed" will be abbreviated as "resource block" in the following sections; both have the same meaning.

[0078] Each resource block in the set of resource blocks to be analyzed also includes a time window in which dynamic time-series analysis is performed on that resource block. A time window refers to a continuous period of time during which the fault locator performs focused acquisition and analysis on the resource block. Each resource block to be analyzed is configured with its own time window, and the time windows of different resource blocks are continuous; that is, the time window of the previous resource block ends immediately after the time window of the next resource block begins. In one possible implementation, there may also be a preset short interval between time windows, such as a buffer time for data acquisition switching or analysis state saving. This interval is usually much smaller than the length of the time window itself. In offline analysis or scenarios where some data sources have significant latency, a longer waiting interval is allowed between adjacent time windows; this application does not strictly limit this. The order of the resource blocks to be analyzed in the set of resource blocks to be analyzed is consistent with the temporal order of their respective time windows. For example, the first resource block to be analyzed in the sequence corresponds to time window T1, the second corresponds to time window T2, and so on. The length of the time window can be configured according to the characteristics of the corresponding analytical metric. For example, for metrics that change rapidly, such as CPU utilization, the time window length can be set to 10 to 30 seconds; for metrics that require a statistical window, such as the number of slow queries, the time window length can be set to 30 to 60 seconds. In one possible implementation, the length of each time window can be the same, for example, uniformly set to 30 seconds; in another possible implementation, the length of each time window can also be set differently according to different metric types, and this application does not limit this.

[0079] Furthermore, for any given unit among multiple units, if multiple analysis metrics are identified for that unit in S201, then multiple resource blocks to be analyzed, containing those multiple analysis metrics, are arranged consecutively in the dynamic analysis sequence. Consecutive arrangement means that these multiple resource blocks to be analyzed are adjacent in the sequence, without inserting resource blocks from other units in between. For example, if order service B has six analysis metrics, then the six resource blocks corresponding to these six metrics occupy six adjacent positions in the sequence, corresponding to six consecutive time windows. This consecutive arrangement design allows the fault locator to complete the concentrated analysis of different aspects of the same unit coherently within adjacent time windows, avoiding the context switching overhead introduced by frequent cross-unit jumps. Simultaneously, since these resource blocks are scheduled independently, the fault locator can still decide whether to skip the remaining resource blocks of a unit based on the analysis results of the first or first few metrics during the analysis process, thus balancing analysis efficiency.

[0080] Based on the above terminology definitions, S202 specifically includes the following two steps.

[0081] Step A: The fault locator determines the fault propagation topology between multiple units from the global fault propagation topology, which indicates the direction of fault propagation dependency between various analytical indicators of multiple units.

[0082] The global fault propagation topology, introduced in step 2 of S201, describes the causal chain of fault propagation between units and their analytical metrics throughout the entire business system. It is typically stored as a directed graph. In S201, the fault locator has already used the global fault propagation topology to identify multiple units related to the alarm. In step A of S202, the fault locator needs to further extract local fault propagation topology relationships from the global topology, containing only these related units and their analytical metrics.

[0083] Specifically, the fault locator uses the multiple units determined in S201 as a node set, and prunes a subgraph composed of these units and their analysis indicator nodes from the directed graph of the global fault propagation topology. This subgraph retains all directed edges between these nodes in the original graph, with each directed edge indicating a dependency direction of fault propagation. For example, if there is an edge in the global topology pointing from node "slow query count of product inventory database C" to node "P99 time consumption of order service B's call to C", and both product inventory database C and order service B are in the unit set determined in S201, then this edge is retained in the subgraph. If one end of an edge is connected to a unit that is not in the unit set related to this alarm, then the edge is discarded during pruning.

[0084] After the above pruning, the local fault propagation topology obtained by the fault locator only includes the dependency direction between the units directly related to this alarm and their analysis metrics. This local topology will serve as the basis for constructing the dynamic analysis sequence in step B. It should be noted that the fault propagation topology indicates the dependency direction of fault propagation between the analysis metrics of each unit, not just the dependency relationship between units. This means that the nodes in the topology graph can be refined to the analysis metric level. For example, the topology graph contains a metric-level fault propagation causal chain such as "number of slow queries in C → P99 time of B's ​​call to C → interface response time of B → number of active connections in A's connection pool". This metric-level topology provides a more detailed basis for the subsequent precise arrangement of the resource blocks to be analyzed.

[0085] Step B: The fault locator constructs a dynamic analysis sequence based on the reverse direction of the dependency direction. The closer the unit is to the downstream in the fault propagation topology, the higher its priority will be arranged as the corresponding resource block to be analyzed in the dynamic analysis sequence.

[0086] In the local fault propagation topology obtained in step A, the dependency direction indicates the direction of fault propagation, i.e., the fault propagates from the indicators of the upstream unit to the indicators of the downstream unit. For example, according to the upstream and downstream definition in this application, database C is the upstream. An increase in the number of slow queries in C leads to an increase in the time taken for the order service B in the midstream to call C, which in turn leads to a backlog in the connection pool of the downstream gateway A. The fault propagation direction is C→B→A. When constructing the dynamic analysis sequence, the fault locator arranges the sequence in the reverse direction of this dependency direction, i.e., tracing from downstream to upstream, prioritizing the resource blocks of the downstream unit, and then arranging the resource blocks of the upstream unit. In this way, when performing dynamic time-series analysis, the fault locator will first analyze the abnormal manifestation unit located at the downstream end of the fault propagation chain, and then trace upstream step by step along the causal chain, approximating the root cause from the surface to the core.

[0087] The specific construction process is as follows.

[0088] First, the fault locator determines the upstream and downstream positions of each unit in the fault propagation chain based on the local fault propagation topology extracted in step A. The determination of upstream and downstream is based on the fault propagation direction defined in this application: the starting point of fault propagation is upstream, and the destination is downstream. For example, if the index of C in the topology points to the index of B, then C is upstream and B is downstream.

[0089] Secondly, the fault locator maps all the analysis metrics of each unit determined in S201 to resource blocks to be analyzed, one by one. Each resource block to be analyzed contains one analysis metric of a unit and is configured with a corresponding time window. The initial order of the time windows is determined according to the position of the resource blocks to be analyzed in the sequence, and may be dynamically adjusted during subsequent execution in S203.

[0090] Then, the fault locator arranges the resource blocks to be analyzed into an initial dynamic analysis sequence in the reverse direction of the dependency direction. The specific arrangement rules are as follows:

[0091] Rule 1: The closer a cell is to the downstream, the higher its corresponding resource block will be prioritized in the sequence. Here, the degree of "downstream proximity" of a cell is determined by its position in the local fault propagation topology. If cell A is downstream of cell B in the fault propagation topology, then cell A's resource blocks will be arranged before cell B's resource blocks. When there are multiple cells in a fault propagation chain, the resource blocks of each cell are arranged sequentially from downstream to upstream.

[0092] Rule 2: Multiple resource blocks corresponding to various analytical indicators within the same unit are arranged consecutively in the sequence. When unit A precedes unit B, all resource blocks to be analyzed in unit A are arranged before all resource blocks to be analyzed in unit B. The order of multiple resource blocks within the same unit can be determined based on the priority of the analytical indicators, with higher-priority resource blocks placed first to ensure priority analysis within the earliest time windows. The priority of analytical indicators can be obtained from the hypothesis-indicator mapping relationship or set according to default rules, such as placing indicators that directly reflect abnormal phenomena (e.g., P99 time consumption) before other indicators within the same unit (e.g., CPU utilization). In another possible implementation, multiple resource blocks within the same unit can also be arranged randomly or in lexicographical order of indicator names; this application does not limit this.

[0093] Rule 3: For topologies with branch dependencies, where multiple downstream units depend on the same upstream unit, the resource blocks of each branch's downstream units are arranged before the resource blocks of their shared upstream unit. In other words, downstream branch units are arranged first, followed by their shared upstream unit. For example, as defined in this application, if A1 and A2 are both downstream of B, and B is downstream of C, the arrangement order is: resource blocks of A1 and A2, resource blocks of B, and resource blocks of C. The order between A1 and A2 can be determined by the severity of alarms or by the lexicographical order of unit identifiers, since they are at the same level in the topology. When A1 and A2 each have multiple analysis metrics, all resource blocks of A1 and A2 are arranged consecutively, with the same order as above. This arrangement strategy ensures that each branch's downstream units have received sufficient preliminary analysis before analyzing the shared upstream unit, facilitating a unified decision on whether to jump upstream based on the analysis results of each branch.

[0094] Using the above arrangement rules, the fault locator constructs an initial dynamic analysis sequence. This sequence is in the form: T1 → [Resource block 1 of a downstream unit] → T2 → [Resource block 2 of a downstream unit or resource block 1 of the next unit] → … → Tn → [Resource block of the upstream unit]. Each resource block in the sequence contains a corresponding analysis time window.

[0095] To facilitate understanding, the following example illustrates the construction process of the dynamic analysis sequence using two business scenarios given in S201.

[0096] Scenario 1 (continued): Simple chained dependency scenario.

[0097] In S201, the identified relevant units are Gateway A, Order Service B, and Product Inventory Database C. According to the fault propagation definition in this application, the fault propagation direction is C→B→A, meaning C is the upstream unit, B is the intermediate unit, and A is the downstream unit. The analysis metrics for each unit are as follows: Gateway A: Request arrival rate, number of active connections in the connection pool. Order Service B: CPU utilization, memory utilization, GC pause time, P99 response time to calls to C, call success rate, number of threads waiting for C's response. Product Inventory Database C: Number of slow queries, average query time, disk I / O wait time, disk queue depth.

[0098] In step A, the fault locator extracts the local topology between A, B, C, and their analytical metrics from the global fault propagation topology. This local topology indicates the following direction of fault propagation at the metric level: number of slow queries and disk I / O wait time in C → P99 latency of B's ​​call to C and number of threads waiting for C's response → interface response time in B → number of active connections in A's connection pool. In other words, the fault propagates from C to B and then to A.

[0099] In step B, the fault locator constructs a sequence in the reverse direction of the fault propagation (i.e., tracing back from downstream to upstream). The fault propagation direction is C→B→A, and the reverse direction starts from the downstream A, proceeds sequentially to the middle B, and then to the upstream C. Therefore, the downstream A is arranged first, followed by the middle B, and the upstream C is arranged last.

[0100] Next, the analysis metrics of each unit are mapped to resource blocks to be analyzed, and a time window is configured for each resource block. Assume that all time windows are uniformly set to 30 seconds. Multiple resource blocks within the same unit are prioritized: for B, the P99 time spent calling C (the metric that most directly reflects upstream bottlenecks) is prioritized over other metrics in the same unit; for C, the number of slow queries is prioritized over disk I / O wait time.

[0101] The final dynamic analysis sequence is shown in Table 2.

[0102] Table 2

[0103] 1 T1 Gateway A Connection pool active connections 2 T2 Gateway A Request arrival rate 3 T3 Order Service B P99 response time for C calls 4 T4 Order Service B Success rate of call 5 T5 Order Service B Number of threads waiting for C response 6 T6 Order Service B CPU utilization 7 T7 Order Service B Memory usage 8 T8 Order Service B GC pause time 9 T9 Product inventory database C Number of slow queries 10 T10 Product inventory database C Average query time 11 T11 Product inventory database C Disk I / O wait time 12 T12 Product inventory database C Disk queue depth

[0104] As shown in Table 2, Gateway A, as the downstream unit, has its resource blocks listed first (T1-T2); Order Service B is listed in the middle (T3-T8); and Product Inventory Database C, as the upstream unit, is listed last (T9-T12). Multiple resource blocks within the same unit are arranged consecutively. During the subsequent dynamic time-series analysis in S203, the fault locator will start from T1 and analyze the operation and maintenance data corresponding to each resource block sequentially.

[0105] Scenario 2 (continued): Branch dependency scenario with shared nodes.

[0106] In S201, the identified relevant units are shopping cart service A1, recommendation service A2, product details service B, and cache cluster C. According to the fault propagation definition in this application, the fault propagation direction is C→B→A1 and C→B→A2, meaning C is the upstream, B is the intermediate shared node, and A1 and A2 are the two downstream branch units. The analysis indicators for each unit are as follows: Shopping cart service A1: P99 response time to B, call success rate, number of threads waiting for B's response. Recommendation service A2: P99 response time to B, call success rate, number of threads waiting for B's response. Product details service B: CPU utilization, memory usage, GC pause time, P99 response time to C, call success rate, number of threads waiting for C's response, request arrival rate, number of active connections in the connection pool. Cache cluster C: CPU utilization, number of slow queries, average query time.

[0107] In step A, the fault locator extracts a local topology from the global topology, comprising A1, A2, B, C, and their analytical indices. This local topology indicates two fault propagation paths: C→B→A1 and C→B→A2, where C is the upstream node, B is an intermediate shared node, and A1 and A2 are the two downstream branch units.

[0108] In step B, the fault locator constructs a sequence in the reverse direction of fault propagation (i.e., tracing back from downstream to upstream). According to rule one, the downstream units A1 and A2 are arranged first, followed by B, and then C. According to rule three, the downstream units (A1 and A2) of each branch are arranged before the shared upstream unit (B). A1 and A2 are at the same level and arranged in lexicographical order of their unit identifiers, with A1 preceding A2. The final constructed dynamic analysis sequence is shown in Table 3.

[0109] Table 3

[0110] 1 T1 Shopping cart service A1 P99 response time for call to B 2 T2 Shopping cart service A1 Success rate of call 3 T3 Shopping cart service A1 Number of threads waiting for B's response 4 T4 Recommended Service A2 P99 response time for call to B 5 T5 Recommended Service A2 Success rate of call 6 T6 Recommended Service A2 Number of threads waiting for B's response 7 T7 Product Details Service B P99 response time for C calls 8 T8 Product Details Service B Success rate of call 9 T9 Product Details Service B Number of threads waiting for C response 10 T10 Product Details Service B CPU utilization 11 T11 Product Details Service B Memory usage 12 T12 Product Details Service B GC pause time 13 T13 Product Details Service B Request arrival rate 14 T14 Product Details Service B Connection pool active connections 15 T15 Cache cluster C Number of slow queries 16 T16 Cache cluster C Average query time 17 T17 Cache cluster C CPU utilization

[0111] As shown in Table 3, A1 and A2, as the two downstream branch units, have their resource blocks at the very beginning (T1-T6), with A1's resource blocks (T1-T3) and A2's resource blocks (T4-T6) arranged consecutively. Product detail service B, as an intermediate shared node, is in the middle (T7-T14); cache cluster C, as the upstream unit, is at the very end (T15-T17). During the subsequent dynamic timing analysis performed by S203, the fault locator will analyze each resource block sequentially according to this sequence.

[0112] It should be noted that the sequence numbers, time window identifiers, and specific order of resource blocks within each unit given in Tables 2 and 3 are merely illustrative examples. In actual implementation, the duration of the time window and the sorting rules between resource blocks within the same unit can be flexibly configured according to specific needs, and this application does not impose any limitations on this. Furthermore, the sequence may also include an identifier field to mark which resource blocks belong to the same unit and which resource blocks have direct topological dependencies. This identifier information can be used by S203 when performing dynamic jumps.

[0113] S203, the fault locator performs dynamic timing analysis based on dynamic analysis sequences to determine the unit where the root cause of the service fault is located from multiple units.

[0114] The unit containing the root cause of a fault refers to the unit that first exhibits an anomaly in the entire fault propagation chain, and that this anomaly is the fundamental cause of all downstream anomalies. In other words, the anomaly in the root cause unit itself is not propagated from other units, but originates from its own internal factors such as resource bottlenecks, logical errors, or configuration problems. This anomaly then spreads outward along the dependency direction in the fault propagation topology, triggering a series of cascading anomalies involved in this alarm. During dynamic time-series analysis, the goal of the fault locator is to ultimately pinpoint the unit containing the root cause from multiple suspicious units through window-by-window analysis, immediate judgment, and dynamic redirection, and then output this unit. The unit containing the root cause can be a microservice instance, a database instance, a middleware node, etc., with the specific type consistent with the aforementioned unit definition. It should be noted that in the subsequent description of this step, the term "resource block" refers to the "resource block to be analyzed" defined in S202; both have the same meaning and are the smallest scheduling unit used by the fault locator when performing dynamic time-series analysis.

[0115] Based on the above terminology definitions, the specific execution process of S203 is as follows. The following explanation uses the current time window as an example. Let the current time window be the i-th time window Ti. In the dynamic analysis sequence, the resource block to be analyzed in this time window is the first resource block to be analyzed, denoted as RB1. RB1 contains a first analysis index of the first unit among multiple units, denoted as M1. In the dynamic analysis sequence constructed in S202, the fault locator, starting from the first time window T1, executes the following steps S1 to S3 sequentially until the unit containing the root cause of the fault is completely determined.

[0116] Step S1: The fault locator collects the operation and maintenance data corresponding to the first analysis indicator within the current time window.

[0117] At the start of the current time window Ti, the fault locator initiates an operational data collection request to the corresponding data source based on the first unit and the first analytical indicator M1 recorded in RB1. The data source can be a monitoring agent, log collector, time-series database, or message queue deployed on various units within the business system. Internally, the fault locator can maintain a data source routing table, which records the data source address, collection protocol, and authentication credentials for each analytical indicator for each unit. The collection protocol is determined based on the data source type; for example, for a time-series database, PromQL or SQL queries can be used; for a message queue, a subscription-based retrieval method can be used; and for a monitoring agent, an HTTP REST interface call can be used.

[0118] During the data collection process, the fault locator only collects the maintenance data corresponding to M1, not all maintenance data from the first unit. For example, if M1 is "P99 time consumption of the call from B to C by order service B", then only the call time consumption data of B to C is collected within the current time window, and its P99 value is calculated; other indicators such as CPU utilization and memory utilization of B are not collected within the current time window. This focused collection method concentrates diagnostic computing power on a narrow band, achieving high temporal resolution collection of a single indicator, improving the signal strength of the abnormal characteristics of the indicator, and thus making it easier to capture weak abnormal traces from background noise. The duration of the collection window is the length of the time window corresponding to the resource block to be analyzed, for example, 30 seconds. The collected maintenance data is temporarily stored in the memory of the fault locator in time series form for analysis in step S2.

[0119] Step S2: The fault locator determines the analysis results of the first unit based on the operation and maintenance data corresponding to the first analysis indicator.

[0120] After data collection is complete, the fault locator analyzes the collected operational data within the same time window to obtain the interim analysis results of the first unit. The analysis results include at least the following two dimensions:

[0121] Dimension 1: Whether the first unit itself operates abnormally. This is determined by comparing the collected operational data with the normal baseline of the first analytical indicator. The normal baseline can be the historical mean and standard deviation, moving average, or static threshold, etc. If the operational data deviates from the normal baseline by more than a preset deviation threshold, for example, if the P99 latency exceeds three times the baseline standard deviation, then the first unit is considered to be operating abnormally; otherwise, the first unit is considered to be operating normally. The specific value of the deviation threshold can be configured according to business sensitivity; for example, it can be set to 2 times the standard deviation for latency-sensitive businesses and 1.5 times the standard deviation for error-rate-sensitive businesses.

[0122] It should be noted that "no abnormalities in its own operation" here means that, excluding any abnormal behavior caused by dependence on the upstream, the core resource indicators (such as CPU, memory, and processing time of its own logic) of the first unit are within the normal range. If the upstream call indicators of the first unit (such as P99 time and success rate) are abnormal, but its own resource indicators are all normal, and the start time of the abnormality is later than or no earlier than that of the upstream, then it can also be determined that its abnormality is affected by the upstream, and it will be handled as if it were operating without abnormalities in subsequent steps.

[0123] Dimension Two: If the first unit itself exhibits an anomaly, the start time of this anomaly is further determined. The start time of the anomaly refers to the point in time when the operational data first deviates from the normal baseline. This is determined by taking the time series of operational data within the current collection window as input, tracing back from the window's start time, and finding the first data point that meets the anomaly judgment criteria. The timestamp of this data point is the anomaly start time. The tracing range can be limited to the unit's deployment startup time or a preset maximum tracing duration (e.g., 10 minutes) to avoid infinite tracing. Simultaneously, the fault locator will record the anomaly's characteristic pattern, such as a spike, a continuous climb, or an oscillation, for subsequent cross-window pattern comparison.

[0124] After obtaining the above analysis results, the fault locator, in conjunction with information from other analyzed units, makes the following supplementary judgment: It compares the anomaly start time of the first unit with the anomaly start time of the units analyzed downstream of the first unit. The information of the downstream units was obtained during the analysis of the preceding time window and stored in the fault locator's analysis status table. The analysis status table records the identifier of each analyzed unit, a list of analyzed indicators, the analysis results (normal or abnormal) of each indicator, and the anomaly start time. By comparison, it can be determined whether the anomaly of the first unit occurs earlier or later than its downstream units. If the first unit itself operates without anomalies, but the main anomaly manifestation of the first unit (such as high interface latency) is highly positively correlated with the latency of its calls to upstream units, then the fault locator infers that the anomaly in the first unit is affected by the second unit located upstream of the first unit.

[0125] Step S3: Based on the analysis results, the fault locator determines that the unit where the root cause of the service fault is located is the first unit, and / or, based on the analysis results, the fault locator determines that the second resource block to be analyzed in the dynamic analysis sequence needs to be analyzed in the next time window after the current time window, until the unit where the root cause of the service fault is located is completely determined.

[0126] Step S3 is the core decision-making step in dynamic time series analysis. Based on the analysis results obtained in step S2, the fault locator, in accordance with the preset decision rules, executes one of the following three scenarios.

[0127] Case 1 (Termination - Root Cause Determination): The fault locator indicates that the first unit itself has an abnormal operation based on the analysis results, and the abnormal start time of the first unit is earlier than the units that have been analyzed and are located downstream of the first unit. In addition, there are no units located upstream of the first unit in the fault propagation topology, and all the resource blocks to be analyzed corresponding to the first unit have been analyzed. Then the unit where the root cause of the fault is located is determined to be the first unit.

[0128] This scenario satisfies four conditions: the first unit itself is abnormal; the abnormality occurs earlier than all analyzed downstream units; there are no upstream units in the topology; and all indicators for this unit have been analyzed. These four conditions constitute a closed-loop verification, ruling out the possibility that the first unit is the victim or that there is an earlier root cause. Once these conditions are met, the fault locator outputs that the first unit is the unit where the root cause of the fault is located, ending this dynamic time series analysis. The output results may include the root cause unit identifier, key abnormal indicators, the abnormality start time, and the complete propagation chain.

[0129] Case 2 (Continue Tracing - Determine the Next Window Target): When the conditions of Case 1 are not fully met, the fault locator determines the second resource block to be analyzed in the next time window based on different directions of the analysis results. This includes the following three sub-cases:

[0130] Sub-case 1: If the fault locator indicates an anomaly in the operation of the first unit based on the analysis results, and the anomaly in the first unit begins later than the units already analyzed and located downstream of the first unit, then the second resource block to be analyzed in the next time window after the current time window is determined to be the second resource block to be analyzed. This second resource block contains a third analysis indicator. The third analysis indicator is an analysis indicator of the second unit located upstream of the first unit. This sub-case indicates that although the first unit itself has an anomaly, the anomaly occurred later than downstream, suggesting that the earlier source of the anomaly is upstream, requiring tracing back upstream.

[0131] Sub-case 2: If the fault locator indicates, based on the analysis results, that the first unit itself is operating normally, and the anomaly in the first unit is caused by the second unit located upstream of the first unit, then the fault locator removes the other unanalyzed resource blocks of the first unit from the dynamic analysis sequence. It then determines that the second resource block, which contains the third analysis indicator, needs to be analyzed in the next time window following the current one. This sub-case indicates that the first unit is entirely the victim and does not require further analysis. The remaining resource blocks are pruned to release analysis resources, and the next window jumps directly to the upstream second unit. During the pruning operation, the removal scope includes all other unanalyzed resource blocks of the first unit. Since the first unit is identified as the victim, its downstream units have also been determined to be normal or victimized in the preceding analysis (because the investigation direction is from downstream to upstream, and the downstream has already been investigated), therefore no further processing is needed for the downstream.

[0132] Furthermore, when other units at the same level as the first unit in the fault propagation topology and sharing the same upstream unit exist in the dynamic analysis sequence, such as A1 and A2 in scenario two, the fault locator can continue to analyze the core indicators of these other units to confirm whether they are also victims. If the core indicators of these other units also indicate that they are affected by the same upstream unit, the fault locator can remove the remaining resource blocks to be analyzed for these other units, thereby accelerating the jump upstream. If the core indicators of these other units indicate that they have independent anomalies, the fault locator retains the analysis path of that branch. In another possible implementation, the fault locator can also, according to a pre-configured strategy, skip all resource blocks of all other units at the same level after determining that the first unit is a victim, and trace upstream as quickly as possible. The specific strategy adopted can be configured according to the business system's preference for fault location speed and accuracy.

[0133] Sub-case 3: The fault locator indicates an anomaly in the operation of the first unit based on the analysis results. The anomaly in the first unit begins earlier than the units already analyzed and located downstream of it. However, the first unit still has other unanalyzed resource blocks. Therefore, in the next time window following the current one, the second unanalyzed resource block needs to be analyzed, and this second unanalyzed resource block contains a second analysis indicator. This second analysis indicator is another analysis indicator for the first unit. This sub-case indicates that the first unit is currently the earliest anomaly point, but other indicators have not yet been cross-validated, requiring further analysis of the remaining indicators of the same unit to obtain more comprehensive evidence.

[0134] It should be noted that in sub-cases 1 and 2 above, when there are multiple upstream units in the fault propagation topology (i.e., the first unit depends on multiple upstream units), the fault locator needs to determine the jump target from among these upstream units. In one possible implementation, the fault locator can determine which upstream unit's call is most correlated with the first unit's abnormal indicators based on the currently collected operational data, and prioritize jumping to that upstream unit. For example, if the first unit calls both upstream C and upstream D, and the first unit's abnormal indicators show abnormally high call time for C and normal call time for D, then prioritize jumping to C. In another possible implementation, if it is impossible to distinguish from the collected data, the fault locator can sort the dependent edges in the fault propagation topology by weight, prioritizing jumping to the upstream unit with the highest weight; the weight of the dependent edge can be set based on the frequency of failures caused by this dependency in historical failure cases. If neither of the above methods can determine the target, the fault locator can randomly select or select the jump target according to the lexicographical order of the unit identifiers, and backtrack to other upstream units as needed in subsequent analysis.

[0135] Case 3 (Termination-Backtracking to Determine Root Cause): If the analysis results indicate that the first unit itself is operating normally, there is no unit upstream of the first unit in the fault propagation topology, and all resource blocks to be analyzed corresponding to the first unit have been analyzed, then the fault locator determines that the unit where the root cause of the fault is located is the unit with the earliest abnormal start time among the units that have been analyzed.

[0136] This scenario corresponds to an analysis that has traced the topology to its end, where all indicators of the terminal cell show normal behavior, indicating that the root cause is not at the very end of the topology but rather somewhere along the analyzed path. The fault locator backtracks through all cells marked as having anomalies in the analyzed state table, compares their anomaly start times, and selects the cell with the earliest start time as the root cause. If multiple cells have the same anomaly start time, the anomaly intensity is further compared, or a decision is made based on topology location (prioritizing upstream cells). If no cell in the analyzed array has anomalies (i.e., all cells are victims), the fault locator can mark the cell with the most significant anomaly as the root cause and output a warning message, indicating that it may be caused by external factors or dependencies not covered by the topology; this application does not impose any limitations on this.

[0137] To facilitate understanding, the following example illustrates the execution process of dynamic time-series analysis in S203 using two business scenarios given in S201 and S202. For simplicity, the following examples assume that the collection, analysis, and decision-making of each indicator can be completed within the corresponding time window, with a time window length of 30 seconds, and that a normal baseline has been pre-configured.

[0138] Scenario 1 (continued): Simple chained dependency scenario.

[0139] In Table 2 of S202, the dynamic analysis sequence for Scenario 1 is as follows: T1 - Number of active connections in the Gateway A connection pool; T2 - Request arrival rate of Gateway A; T3 - P99 latency of the order service B's call to C; T4 - Call success rate of B; T5 - Number of threads waiting for C in B; T6 - CPU utilization of B; T7 - ​​Memory utilization of B; T8 - GC pause time of B; T9 - Number of slow queries in C; T10 - Average query latency of C; T11 - Disk I / O wait time of C; T12 - Disk queue depth of C. According to the definition in this application, the fault propagation direction is C→B→A, that is, C is upstream, B is in the middle, and A is downstream.

[0140] The fault locator starts executing from T1.

[0141] T1 (Gateway A, Number of Active Connections in Connection Pool): The time series of the number of active connections in the connection pool for Gateway A over the past 30 seconds is collected. Analysis reveals that the number of active connections surged from a baseline of 200 to 950 within 30 seconds, significantly deviating from the normal baseline. The fault locator determines that Gateway A itself is malfunctioning, with the anomaly starting at the 3rd second after the start of window T1 (denoted as 14:23:08). Downstream units analyzed: None. Anomaly start time comparison: No comparable downstream units. Upstream topology: A's upstream units are B and C. A still has unanalyzed metrics (request arrival rate). Sub-case 3 is satisfied: Amalgamation, earlier than downstream units (no downstream units), and unanalyzed metrics. The fault locator determines the next time window T2 to continue analyzing Gateway A's second metric—the request arrival rate.

[0142] T2 (Gateway A, Request Arrival Rate): The request arrival rate time series was collected. Analysis revealed that the request arrival rate remained within the baseline range for 30 seconds without significant fluctuations. The fault locator determined the request arrival rate to be normal. Combined with the results of T1, the connection pool of Gateway A was congested, but the request volume did not surge. This indicates that the connection pool congestion was not caused by increased upstream traffic, but rather by slower upstream response, leading to slower connection release and thus congestion. Therefore, the fault locator inferred that the anomaly in Gateway A was affected by its upstream unit (i.e., Order Service B). This satisfies the conditions of sub-case 2: no anomaly itself (normal request rate, connection pool congestion affected by upstream). The fault locator removed other unanalyzed resource blocks from Gateway A (none, Gateway A has already checked all), determining that the second resource block to be analyzed in the next time window T3 is the P99 latency of Order Service B's call to C.

[0143] T3 (Order Service B, P99 time consumption for calling C): Collect the time consumption data of B's ​​call to C and calculate the P99 value. Analysis shows that the P99 time consumption spiked from the baseline of 50ms to 3500ms, a serious anomaly. The fault locator determined that B itself had an anomaly (the high time consumption for calling C reflects that B is waiting for upstream C), with the anomaly starting at 14:23:05. Downstream unit analyzed: Gateway A (anomaly start time 14:23:08). Comparison: B's anomaly start time of 14:23:05 is earlier than A's 14:23:08. The conditions of sub-case 3 are met: its own anomaly, earlier than the downstream, and there are still unanalyzed indicators (T4-T8). The fault locator determines the next time window T4 to continue analyzing the call success rate of B.

[0144] T4 (Order Service B, Success Rate): Collect the success rate of B's ​​call to C. Analysis reveals that the success rate dropped from 99.9% to 72%, with a large number of calls returning timeout errors. The anomaly started at 14:23:05, consistent with T3. The fault locator determined this metric to be abnormal, further solidifying the conclusion that B's anomaly occurred earlier than A's. Continuing to satisfy sub-case 3, determine the number of threads waiting for C's response in analysis T5.

[0145] T5 (Order Service B, number of threads waiting for C's response): Collect the number of threads waiting for C's response. Analysis revealed that the number of threads surged from the baseline of 50 to 900, with the anomaly starting at 14:23:05. Continue to satisfy sub-case 3 to determine the CPU utilization of T6 analysis B.

[0146] T6 (Order Service B, CPU Utilization): Collect the CPU utilization of B. Analysis shows that the CPU utilization remains around 40%, which is within the normal range. The fault locator determines that B's CPU indicators are normal. Continue to meet sub-condition 3 (there are still remaining indicators such as memory and GC), and determine T7 to analyze memory usage.

[0147] T7 (Order Service B, Memory Usage): Memory usage was collected, and the result is normal (60%). Continue to satisfy sub-case 3 and determine T8 to analyze GC pause time.

[0148] T8 (Order Service B, GC Pause Time): GC pause time was collected, and the result was normal (average 20ms). At this point, all six metrics of B have been analyzed. The overall analysis of B shows that B's own CPU, memory, and GC are normal, but its call time, success rate, and number of waiting threads to C are severely abnormal, and the abnormality started earlier than downstream A. This meets the criterion of "no abnormalities in its own operation" (its own core resource metrics are normal, only the upstream call metrics are abnormal). Therefore, the fault locator infers that B is also a victim, and its abnormality is caused by upstream C. This satisfies the conditions of sub-case 2. The fault locator removes other resource blocks of B that have not yet been analyzed (there are none left), and determines that the second resource block to be analyzed in the next time window T9 is the number of slow queries of its upstream unit C.

[0149] T9 (Product Inventory Database C, Slow Query Count): Collect the number of slow queries for C in the past 30 seconds. Analysis reveals that the number of slow queries surged from a baseline of 5 queries / 30 seconds to 200 queries / 30 seconds, a severe anomaly. It is determined that C itself is experiencing an anomaly, with the anomaly starting at 14:23:02. Downstream units analyzed: B (anomaly start time 14:23:05), A (14:23:08). Comparison: C's anomaly start time of 14:23:02 is earlier than B and A. Upstream Topology: C has no upstream units in the fault propagation topology. However, C still has unanalyzed indicators (T10-T12). Sub-case 3 is satisfied; determine the average query time for C analyzed by T10.

[0150] T10 (Product Inventory Database C, Average Query Time): The average query time for C was collected, and the result increased from the baseline of 20ms to 1500ms, with the anomaly starting at 14:23:02. Sub-case 3 is still satisfied, and T11 is determined to analyze disk I / O wait time.

[0151] T11 (Product Inventory Database C, Disk I / O Wait Time): Disk I / O wait time was collected, and the result was normal (5ms). Sub-case 3 is satisfied, and T12 is used to analyze the disk queue depth.

[0152] T12 (Product Inventory Database C, Disk Queue Depth): Disk queue depth was collected, and the result was normal. At this point, all four metrics for C have been analyzed. C itself has a serious slow query problem, with the anomaly starting at 14:23:02, earlier than all downstream systems, and there are no upstream systems in the topology. All four conditions of Case 1 are met. The fault locator determines the root cause of the fault to be the product inventory database C. The root cause is output as C, the root cause metric is the number of slow queries, the anomaly starting at 14:23:02, and the propagation chain is: slow query in C → increased time spent calling C from B → connection pool overflow in A → timeout in A interface.

[0153] Scenario 2 (continued): Branch dependency scenario with shared nodes.

[0154] In Table 3 of S202, the dynamic analysis sequence for Scenario 2 is as follows: T1-P99 time of A1 calling B, T2-A1 call success rate, T3-Number of threads waiting for B in A1, T4-P99 time of A2 calling B, T5-A2 call success rate, T6-Number of threads waiting for B in A2, T7-P99 time of B calling C, T8-B call success rate, T9-Number of threads waiting for C in B, T10-B CPU utilization, T11-B memory utilization, T12-B GC pause time, T13-B request arrival rate, T14-B number of active connections in the connection pool, T15-C number of slow queries, T16-C average query time, and T17-C CPU utilization. As defined in this application, the fault propagation directions are C→B→A1 and C→B→A2, with C at the top, B in the middle (shared nodes), and A1 and A2 at the bottom.

[0155] The fault locator starts from T1.

[0156] T1 (Shopping cart service A1, P99 time consumption for calling B): Collected the time consumption of A1's call to B, P99 value is 3200ms, abnormal, start time 14:25:10. It has its own abnormality, occurring earlier than the downstream (no downstream). T2-T3 are still unanalyzed. Sub-case 3 is satisfied, determining the success rate of A1's call analysis in T2.

[0157] T2 (A1, call success rate): Success rate dropped to 68%, exception occurred, start time 14:25:10. Sub-case 3 is satisfied, determine the number of threads A1 is waiting for in T3 analysis.

[0158] T3 (A1, number of threads waiting for B): Thread count surged, anomaly, start time 14:25:10. All A1 metrics have been checked. Overall judgment: All abnormal metrics of A1 are related to calling B, and A1 is located at the downstream end of the fault propagation topology. Its anomaly is entirely manifested in high call time and low success rate to B. According to the criterion of "no anomalies in its own operation," A1 only has three metrics allocated to calls to B; its own resource metrics (such as CPU, memory, etc.) are not within the scope of A1's metrics. According to the S203 rule, A1 itself has abnormal metrics (high call time to B), but the anomaly start time is later than or no earlier than the upstream (B has not yet been analyzed, so comparison is not possible). Since all A1 metrics have been checked and upstream B exists, the fault locator should trace upstream. This applies to the boundary cases of sub-case 1 or sub-case 2. In the actual strategy, it can be configured as follows: when a downstream unit only has abnormal call-related metrics and no metrics related to its own resources, sub-case 2 applies, i.e., A1 is determined to be the victim, its remaining resource blocks are removed (since they are no longer available), and the process jumps upstream. However, in the sequence, A2 (another downstream unit at the same level) is listed after A1, not B. According to the branch processing logic supplemented by sub-case 2 in S203, the fault locator continues to analyze the core metrics of other units A2 at the same level. T4 is already in the sequence, so T4 continues.

[0159] T4 (Recommended service A2, P99 time consumption for calling B): The time consumption of A2's call to B was collected, with a P99 value of 3100ms, which is abnormal, starting at 14:25:11. It has its own abnormality, occurring earlier than downstream (none). T5-T6 are still unanalyzed. Sub-condition 3 is met, therefore T5 is determined.

[0160] T5(A2, call success rate): Exception, sub-condition 3 is met, determine T6.

[0161] T6 (A2, waiting for B thread count): Anomaly. All metrics for A2 have been checked. Similar to A1, all anomalies point to B. The fault locator determines that A2 is also a victim. Sub-case 2 applies, processing the remaining resource blocks of A1 and A2 together (both are now empty). At this point, all downstream units at the same level have been analyzed and are all victims, so the fault locator determines the next window to jump to the shared upstream B. The original sequence T7 is B's resource block, so continue with T7.

[0162] T7 (Product Details Service B, P99 time consumption for calling C): The time consumption of B's ​​call to C was collected, with a P99 value of 2800ms, which is abnormal, starting at 14:25:08. It is abnormal itself, occurring earlier than downstream A1 (14:25:10) and A2 (14:25:11). T8-T14 are still unanalyzed. Sub-case 3 is satisfied, therefore T8 is determined.

[0163] T8-T14 (Remaining Metrics for B): Analyze B's call success rate, number of waiting threads for C, CPU, memory, GC, request arrival rate, and number of active connections in the connection pool in sequence. Assume the following results: low success rate, high number of waiting threads, normal CPU utilization, normal memory, normal GC, normal request arrival rate, and high number of active connections in the connection pool. Overall judgment: B's own resources are normal; the anomalies are concentrated in calls to C and connection pool backlog. The request volume did not surge, indicating that B is also a victim, dragged down by upstream C. All of B's ​​metrics have been checked, satisfying sub-case 2. Therefore, the next window, T15, is determined to be the number of slow queries for C.

[0164] T15 (Cache Cluster C, Slow Query Count): The number of slow queries collected for C spiked from a baseline of 3 queries / 30 seconds to 180 queries / 30 seconds, a severe anomaly. It was determined that C itself was experiencing an anomaly, with the anomaly starting at 14:25:05. Downstream units analyzed: B (14:25:08), A1 (14:25:10), A2 (14:25:11). Comparison: C's anomaly start time of 14:25:05 is earlier than B, A1, and A2. Upstream Topology: C has no upstream units in the topology. However, C still has unanalyzed metrics (T16-T17). Sub-case 3 is satisfied; determine the average query time for C analyzed in T16.

[0165] T16 (C, Average Query Time): The average query time increased from the baseline of 10ms to 1200ms, with the anomaly starting at 14:25:05. Continue to satisfy sub-case 3, and determine the CPU utilization of analysis C in T17.

[0166] T17 (C, CPU Utilization): CPU utilization is normal (30%). At this point, all three metrics for C have been analyzed. C itself has a serious slow query problem, with the anomaly starting at 14:25:05, earlier than all downstream nodes, and there are no upstream nodes in the topology. All metrics have been analyzed. All four conditions of Case 1 are met. The fault locator determines the root cause of the fault to be the cache cluster C. The output root cause is C, the root cause metric is the number of slow queries, the anomaly starting at 14:25:05, and the propagation chain is: slow query in C → increased time spent calling C from B and connection pool accumulation → increased time spent calling B from A1 and A2 → timeouts at A1 and A2 interfaces.

[0167] The two scenario examples demonstrate that the fault locator effectively converges to the root cause along the reverse direction of fault propagation through window-by-window focusing, immediate decision-making, and dynamic redirection. It should be noted that the judgment of analysis results, the specific execution of redirection decisions, and the branching strategy in Scenario 2 are all illustrative examples. In actual implementation, these can be flexibly adjusted according to specific business rules and topology complexity, and this application does not impose any limitations on them.

[0168] It should be noted that, for the sake of brevity, the above description refers to the dynamic analysis sequence as an ordered sequence of resource blocks to be analyzed, and the dynamic time-series analysis process revolves around this sequence. In actual implementation, when the fault propagation topology has a branching structure (such as the two paths A1→B→C and A2→B→C in scenario two), the fault locator can construct a complete sequence containing all branches, arranging the resource blocks of the same level of branch units sequentially, and then analyzing them one by one in order. This processing method linearizes multiple investigation paths into a single sequence, and the analysis logic on each branch path is the same, that is, tracing from downstream to upstream and dynamically jumping or pruning based on the analysis results. Therefore, to avoid repetition, this embodiment is described in the form of a single sequence, and scenarios with multiple paths can be deduced similarly, and will not be elaborated on separately in this application.

[0169] In summary, after acquiring initial anomaly alarms, the fault locator can first identify multiple units related to the alarms and their respective analytical metrics. Then, it introduces the fault propagation topology between these units, arranging the objects to be analyzed into a set of resource blocks with temporal relationships, forming a dynamic analysis sequence. In this sequence, each resource block carries only one metric from one unit. During the analysis, only one resource block is focused at a time, ensuring that the computing power used for analysis is concentrated within any given time window, rather than being dispersed across all metrics. This allows for accurate location of the root cause of the fault from complex operational data.

[0170] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Exemplarily, the electronic device may be a terminal, or a chip (system) or other component or assembly that can be disposed in the terminal. Figure 3 As shown, the electronic device 200 may include a processor 201. Optionally, the electronic device 200 may also include a memory 202 and / or a transceiver 203. The processor 201 is coupled to the memory 202 and the transceiver 203, for example, via a communication bus.

[0171] The following is combined Figure 3 A detailed description of each component of the electronic device 200 is provided below:

[0172] The processor 201 is the control center of the electronic device 200. It can be a single processor or a collective term for multiple processing elements. For example, the processor 201 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement the embodiments of this application, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0173] Optionally, the processor 201 can perform various functions of the electronic device 200 by running or executing software programs stored in the memory 202 and calling data stored in the memory 202, such as performing the aforementioned functions. Figure 2 The method shown.

[0174] In a specific implementation, as one example, the processor 201 may include one or more CPUs, for example... Figure 3 CPU0 and CPU1 are shown in the diagram.

[0175] In a specific implementation, as one example, the electronic device 200 may also include multiple processors, for example... Figure 3 The processor 201 shown is an example. Each of the processors 201 can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0176] The memory 202 is used to store the software program that executes the solution of this application, and the processor 201 controls its execution. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0177] Optionally, the memory 202 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 202 may be integrated with the processor 201 or exist independently, and may be accessed through the interface circuit of the electronic device 200. Figure 3 (Not shown in the image) is coupled to processor 201, but this embodiment does not specifically limit this.

[0178] Transceiver 203 is used for communication with other electronic devices. For example, if electronic device 200 is a terminal, transceiver 203 can be used to communicate with a network device or with another terminal device. As another example, if electronic device 200 is a network device, transceiver 203 can be used to communicate with a terminal or with another network device.

[0179] Optionally, transceiver 203 may include a receiver and a transmitter. Figure 3 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0180] Optionally, the transceiver 203 can be integrated with the processor 201, or it can exist independently and be connected via the interface circuit of the electronic device 200. Figure 3 (Not shown in the image) is coupled to processor 201, but this embodiment does not specifically limit this.

[0181] Understandable, Figure 3 The structure of the electronic device 200 shown does not constitute a limitation on the electronic device. Actual electronic devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0182] Furthermore, the technical effects of the electronic device 200 can be referred to the technical effects of the methods described in the above-described method embodiments, and will not be repeated here.

[0183] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0184] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0185] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0186] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0187] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0188] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0189] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0190] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0191] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0192] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0193] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0194] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0195] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A multi-unit analysis fault determination method based on operation and maintenance data, characterized in that, The method is applied to a fault locator, and the method includes: The fault locator acquires the initial abnormal alarm of the service and determines multiple units related to the initial abnormal alarm and one or more analysis indicators for each of the multiple units, with each analysis indicator corresponding to a type of operation and maintenance data to be analyzed. The fault locator constructs a dynamic analysis sequence based on the fault propagation topology relationship between the multiple units. The dynamic analysis sequence consists of a set of resource blocks to be analyzed that have dynamic time-series analysis relationships. Each resource block to be analyzed in the set of resource blocks to be analyzed includes an analysis index of a corresponding unit. The fault locator performs dynamic time-series analysis based on the dynamic analysis sequence to determine the unit where the root cause of the service fault is located from the plurality of units; The fault locator constructs a dynamic analysis sequence based on the fault propagation topology among the multiple units, including: The fault locator determines the fault propagation topology between the multiple units from the global fault propagation topology, which indicates the direction of fault propagation dependency between the various analytical indicators of the multiple units. The fault locator constructs the dynamic analysis sequence according to the reverse direction of the dependency direction. In the fault propagation topology, the unit that is closer to the downstream is given higher priority in the dynamic analysis sequence and is arranged as the corresponding resource block to be analyzed. The fault locator performs dynamic time-series analysis based on the dynamic analysis sequence to determine the unit where the root cause of the service fault resides from the plurality of units, including: For the first resource block to be analyzed in the current time window within the dynamic analysis sequence, the first resource block to be analyzed includes a first analysis index of the first unit among the plurality of units: The fault locator collects the operation and maintenance data corresponding to the first analysis indicator within the current time window. The fault locator determines the analysis result of the first unit based on the operation and maintenance data corresponding to the first analysis indicator. Based on the analysis results, the fault locator determines that the unit where the root cause of the service fault is located is the first unit, and / or, based on the analysis results, the fault locator determines that the second resource block to be analyzed in the dynamic analysis sequence is the next time window after the current time window, until the unit where the root cause of the service fault is located is completely determined. The second resource block to be analyzed includes another second analysis indicator of the first unit and a third analysis indicator of the second unit among the plurality of units. Based on the analysis results, the fault locator determines the unit where the root cause of the service fault lies as the first unit, including: The fault locator indicates, based on the analysis results, that the first unit itself has an abnormal operation, and the abnormal start time of the first unit is earlier than the units that have been analyzed among the multiple units and are located downstream of the first unit, and there are no units located upstream of the first unit in the fault propagation topology, and all the resource blocks to be analyzed corresponding to the first unit have been analyzed, thus determining that the unit where the root cause of the fault is located is the first unit.

2. The method according to claim 1, characterized in that, If any one of the multiple units has multiple analysis indicators, then multiple resource blocks to be analyzed that include the multiple analysis indicators of the unit are arranged consecutively in the dynamic analysis sequence.

3. The method according to claim 1, characterized in that, Each resource block in the set of resource blocks to be analyzed includes a time window in which the dynamic timing analysis is performed on the resource block to be analyzed, and the order of the resource blocks to be analyzed in the set of resource blocks to be analyzed is consistent with the timing order of the time windows of each resource block to be analyzed.

4. The method according to claim 1, characterized in that, The method further includes: If the analysis results indicate that the first unit itself is operating normally and there is no unit upstream of the first unit in the fault propagation topology, then the fault locator determines that the unit where the root cause of the fault is located is the unit with the earliest abnormal start time among the units that have been analyzed among the plurality of units.

5. The method according to claim 4, characterized in that, Based on the analysis results, the fault locator determines that the second resource block to be analyzed in the dynamic analysis sequence needs to be analyzed in the next time window after the current time window, including: The fault locator, based on the analysis results, indicates that the first unit itself has an operational anomaly, and the anomaly start time of the first unit is later than the unit that has already been analyzed and is located downstream of the first unit among the plurality of units. It then determines that the second resource block to be analyzed needs to be analyzed in the next time window after the current time window, and the second resource block to be analyzed contains the third analysis indicator; or... The fault locator, based on the analysis results, indicates that the first unit itself is operating normally and that the abnormality in the first unit is caused by the second unit located upstream of the first unit. The fault locator then removes other unanalyzed resource blocks from the dynamic analysis sequence of the first unit and determines that the second resource block needs to be analyzed in the next time window after the current time window, and that the second resource block contains the third analysis indicator; or... The fault locator indicates, based on the analysis results, that the first unit itself has an abnormal operation, and the abnormal start time of the first unit is earlier than the unit that has been analyzed among the multiple units and is located downstream of the first unit. Furthermore, the first unit also corresponds to other resource blocks that have not yet been analyzed. It determines that the second resource block to be analyzed among the other resource blocks to be analyzed is the one that needs to be analyzed in the next time window after the current time window, and the second resource block to be analyzed contains the second analysis index.

6. The method according to claim 1, characterized in that, The fault locator acquires the initial anomaly alarm of the service and determines multiple units related to the initial anomaly alarm, as well as one or more analytical indicators for each of the multiple units, including: The fault locator acquires the initial abnormal alarm; The fault locator identifies multiple units involved in the abnormal situation indicated by the initial abnormal alarm, and generates at least one fault hypothesis for each of the multiple units based on the abnormal situation. The fault locator determines one or more analytical indicators for each of the plurality of units based on a preset hypothesis-indicator mapping relationship and the at least one fault hypothesis.

7. The method according to claim 1, characterized in that, The unit is any one of the following: server, container, microservice instance, database instance, middleware node, network device; the analysis metric is any one of the following: CPU utilization, memory utilization, disk I / O wait time, network retransmission rate, interface response time, call error rate, number of slow queries, connection pool utilization, garbage collection pause time.

Images