A network fault root cause positioning method and operation and maintenance equipment

Abstract

The application discloses a network fault root cause positioning method and operation and maintenance equipment, relates to the technical field of network fault positioning, and is used for improving the accuracy of network fault root cause positioning.In the method, the operation and maintenance equipment constructs an elastic backtracking window associated with a network topology delay, combines a time-space alignment drift degree to quantify a cause-effect relationship, establishes a more reasonable log searching interval for each candidate root cause node, reduces time axis out-of-range time sequence misplacement of a preset window range, avoids a situation that related technology is difficult to capture the correlation between the two, and improves the accuracy of network fault root cause positioning.

Description

Technical Field

[0001] This application relates to the field of network fault location technology, and in particular to a method and maintenance equipment for locating the root cause of network faults. Background Technology

[0002] With the rapid development of cloud computing and distributed architecture technologies, modern communication networks are becoming increasingly large-scale, and their topologies and dependencies are becoming extremely complex. In daily operations and maintenance (O&M) management, to ensure high service availability, O&M equipment collects a large amount of multi-dimensional observation data, mainly including time-series indicators reflecting system operating status (such as CPU utilization and link latency) and log text recording discrete events (such as error stacks and alarm information). When network failures or performance degradation occur, O&M equipment needs to comprehensively analyze this diverse multi-modal data to pinpoint the root cause of the failure from the large amount of data.

[0003] In related network fault root cause localization techniques, a window-based correlation analysis method based on absolute timestamps is typically used to uncover the causal relationship between abnormal metrics and log events. Specifically, this method first aggregates metric data and log data from different device nodes to an analysis device and maps them onto the same timeline based on their inherent timestamps. Then, a preset time window (e.g., 60 seconds before and after the anomaly) is set. Within this window, the linear correlation between the metric curve shape and log occurrence frequency is calculated using the Pearson correlation coefficient, or a frequent itemset mining algorithm is used to identify the log patterns with the highest co-occurrence frequency during the metric anomaly period. In this way, the technique establishes a correlation between specific metric fluctuations and log events falling within this fixed window on the timeline, thereby filtering out highly correlated logs as candidate root causes.

[0004] However, in large-scale distributed network environments, due to differences in physical link transmission paths, fluctuations in device processing performance, and the existence of asynchronous log write-to-disk mechanisms, data transmission from different nodes to analysis devices often involves non-negligible and time-varying transmission delays and clock deviations. In such scenarios, sudden changes in indicators and log events caused by the same fault source may exhibit temporal misalignments beyond a preset window range on the timeline, making it difficult for related technologies to capture the correlation between the two and reducing the accuracy of fault root cause localization. Summary of the Invention

[0005] This application provides a method and maintenance equipment for locating the root cause of network faults, which can improve the accuracy of locating the root cause of network faults.

[0006] Firstly, a method for locating the root cause of network faults is provided, applied to operation and maintenance equipment. This method includes: responding to a sudden change in performance indicators of a target network element node at the time of anomaly triggering; determining the upstream path of all data transmission paths pointing to the target network element node based on the network topology diagram; identifying each network device located on the upstream path as a candidate root cause node; determining the transmission path segment from each candidate root cause node to the target network element node based on the network topology; using the sum of the link propagation delay of all single-hop links on the transmission path segment and the average queuing processing delay of all intermediate nodes as the cumulative propagation delay of each candidate root cause node relative to the target network element node; and constructing a springboard for each candidate root cause node based on the triggering time of the performance indicator change and the cumulative propagation delay. The elastic backtracking window is defined by the time axis center being the difference between the trigger time and the accumulated propagation delay. The time axis width of the elastic backtracking window is the sum of the preset basic tolerance duration and the jitter extension duration. The jitter extension duration is the product of the number of intermediate nodes in the transmission path segment and the preset single-hop jitter coefficient. Candidate abnormal logs are selected from the operation logs generated by each candidate root cause node within its corresponding elastic backtracking window. The absolute value of the time difference between the recording time of the candidate abnormal log and the time axis center of the elastic backtracking window is used as the spatiotemporal alignment drift. If the log level of a candidate abnormal log is determined to be greater than a preset severity threshold, the candidate root cause node corresponding to the candidate abnormal log with the smallest spatiotemporal alignment drift is selected as the fault root cause node causing the sudden change in performance indicators.

[0007] By adopting the above technical solution, the operation and maintenance equipment constructs an elastic backtracking window associated with network topology latency and combines it with spatiotemporal alignment drift to quantify causal relationships. This establishes a more reasonable log search interval for each candidate root cause node, thereby reducing the occurrence of temporal misalignments on the timeline that exceed the preset window range, which makes it difficult for related technologies to capture the correlation between the two, and thus improving the accuracy of network fault root cause localization.

[0008] In conjunction with some embodiments of the first aspect, in some embodiments, before taking the sum of the link propagation delay of all single-hop links on the transmission path segment and the average queuing processing delay of all intermediate nodes as the cumulative propagation delay of each candidate root cause node relative to the target network element node, the method further includes: obtaining discrete sampling data of each intermediate node on the transmission path segment within a preset sampling period before the triggering time of the performance index mutation, wherein the discrete sampling data includes the number of bytes in the port buffer queue and the port forwarding rate at multiple time points; calculating the ratio of the number of bytes in the port buffer queue to the port forwarding rate at each time point to obtain a queuing delay sampling sequence; and determining the standard deviation of the queuing delay sampling sequence as the delay jitter value of all intermediate nodes on the transmission path segment.

[0009] By adopting the above technical solution, before calculating the cumulative propagation delay, the operation and maintenance equipment dynamically estimates the queuing delay and its delay jitter value of each intermediate node by analyzing the discrete sampling data of the port buffer queue and forwarding rate within a specific time window before the anomaly occurs. This allows the assessment results of delay and jitter to more accurately reflect the instantaneous network state before the fault occurs.

[0010] In conjunction with some embodiments of the first aspect, in some embodiments, the step of constructing an elastic backtracking window for each candidate root cause node based on the trigger time of the performance index mutation and the cumulative propagation delay specifically includes: calculating the total path delay variance based on the sum of squares of delay jitter values; determining the path delay fluctuation level based on the total path delay variance; determining a weighting factor from a preset mapping table based on the path delay fluctuation level; using the product of the weighting factor and a preset single-hop jitter coefficient as the corrected single-hop jitter coefficient; constructing an elastic backtracking window for each candidate root cause node based on the trigger time of the performance index mutation and the cumulative propagation delay, wherein the center of the time axis of the elastic backtracking window is the difference between the trigger time and the cumulative propagation delay, the width of the time axis of the elastic backtracking window is the sum of the preset basic tolerance duration and the jitter extension duration, and the jitter extension duration is the product of the number of intermediate nodes in the transmission path segment and the corrected single-hop jitter coefficient.

[0011] By adopting the above technical solution, the operation and maintenance equipment quantifies the stability of the entire path by calculating the total path delay variance, and dynamically adjusts the jitter coefficient used for calculating the window width accordingly. Thus, for a path that has historically or recently performed stably, its backtracking window automatically narrows to reduce interference from irrelevant logs, while for a path with drastic fluctuations, its window widens accordingly to accommodate greater delay uncertainty, preventing the omission of critical logs due to an excessively narrow window, thereby improving the accuracy of network fault root cause localization.

[0012] In conjunction with some embodiments of the first aspect, in some embodiments, the step of determining the upstream path of all data transmission paths pointing to the target network element node based on the network topology diagram specifically includes: obtaining the inbound traffic quintuple information of the target network element node at the time of abnormal triggering; when multiple parallel equal-cost multi-path routing links are identified based on the network topology diagram, obtaining the load-sharing hash algorithm and hash factor on the equal-cost multi-path routing links; substituting the inbound traffic quintuple information into the load-sharing hash algorithm for simulation calculation to obtain the unique physical transmission link actually traversed by the inbound traffic quintuple information; and determining the unique physical transmission link and the device node connected to the unique physical transmission link as the upstream path pointing to the target network element node.

[0013] By adopting the above technical solution, when facing complex network environments with equivalent multi-path routing, the operation and maintenance equipment can uniquely determine the path taken by the abnormal traffic in multiple parallel physical links by parsing the five-tuple information of the abnormal traffic and simulating the hash algorithm on the load balancer. This avoids wasting troubleshooting resources on parallel paths that do not carry abnormal traffic, thereby narrowing down the set of candidate root cause nodes and improving the efficiency and accuracy of root cause location.

[0014] In conjunction with some embodiments of the first aspect, in some embodiments, before the step of obtaining the inbound traffic quintuple information of the target network element node at the time of the abnormal triggering, the method further includes: receiving a link state announcement message sent by the network control plane; and, if it is determined from the link state announcement message that a link state change event exists and the change time is before the time of the abnormal triggering, updating the network topology diagram based on the link state change event.

[0015] By adopting the above technical solution, before performing path tracing, the maintenance equipment first detects dynamic changes in the network topology by listening to link status announcement messages. When a link status change is detected before the anomaly occurs, the latest information is used to update the network topology map, thereby reducing the possibility of tracing back to the wrong upstream path due to the use of outdated topology information.

[0016] In conjunction with some embodiments of the first aspect, in some embodiments, the step of filtering candidate abnormal logs from the operation logs generated by each candidate root cause node within its corresponding elastic backtracking window specifically includes: sending clock probe messages to each candidate root cause node and recording the sending timestamp, receiving timestamp, and processing dwell time of each candidate root cause node; calculating the clock deviation value of each candidate root cause node relative to the reference clock of the maintenance equipment based on the sending timestamp, receiving timestamp, and processing dwell time; correcting the recording time of the operation logs of each candidate root cause node based on the clock deviation value to obtain a unified time base log; and identifying the logs in the unified time base log that fall within the elastic backtracking window as candidate abnormal logs.

[0017] By adopting the above technical solution, through active clock detection and deviation correction, the log timestamps of all candidate root cause nodes are calibrated to a unified reference time base, reducing the interference caused by clock deviation to timing analysis.

[0018] In some embodiments of the first aspect, after identifying the candidate root cause node corresponding to the candidate abnormal log with the smallest spatiotemporal alignment drift as the root cause node causing the sudden change in performance indicators, the method further includes: identifying all intermediate forwarding nodes between the root cause node and the target network element node based on the network topology diagram; obtaining performance monitoring data generated by the intermediate forwarding nodes within the elastic backtracking window; counting the number of intermediate forwarding nodes whose performance monitoring data deviates from the baseline operating state by a factor greater than a preset propagation threshold; determining the ratio of the number to the total number of intermediate forwarding nodes as the fault propagation chain confidence; generating a root cause analysis report containing the root cause node, spatiotemporal alignment drift, and fault propagation chain confidence when the fault propagation chain confidence is greater than a preset confidence threshold; and marking the root cause node as an isolated abnormal node when the fault propagation chain confidence is less than or equal to the preset confidence threshold, and reviewing the candidate root cause node corresponding to the next hour's spatiotemporal alignment drift.

[0019] By adopting the above technical solution, after identifying the root cause node of the fault, it is possible to check whether there is a chain of performance degradation in the intermediate nodes on the path from the root cause node to the target node, thereby confirming whether the impact of the fault is propagating along the network path. This distinguishes between a true conductive fault and an isolated anomaly that is coincidental in time, thus improving the accuracy of network fault root cause localization.

[0020] In a second aspect, embodiments of this application provide an operation and maintenance device, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the operation and maintenance device to perform the method described in the first aspect and any possible implementation thereof.

[0021] Thirdly, embodiments of this application provide a computer program product containing instructions that, when the computer program product is run on an operation and maintenance device, cause the operation and maintenance device to perform the method described in the first aspect and any possible implementation thereof.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium including instructions that, when executed on an operation and maintenance device, cause the operation and maintenance device to perform the method described in the first aspect and any possible implementation thereof.

[0023] Understandably, the maintenance equipment provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the methods provided in the embodiments of this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0025] 1. By constructing an elastic backtracking window associated with network topology latency and combining it with spatiotemporal alignment drift to quantify causal relationships, the operation and maintenance equipment establishes a more reasonable log search interval for each candidate root cause node. This reduces the occurrence of time-series misalignments on the timeline that exceed the preset window range, making it difficult for related technologies to capture the correlation between the two, thereby improving the accuracy of network fault root cause localization.

[0026] 2. The maintenance equipment quantifies the stability of the entire path by calculating the total path delay variance and dynamically adjusts the jitter coefficient used to calculate the window width accordingly. Thus, for a path that has historically or recently performed stably, its backtracking window automatically narrows to reduce interference from irrelevant logs, while for a path with drastic fluctuations, its window widens to accommodate greater delay uncertainty, preventing the omission of critical logs due to an excessively narrow window, thereby improving the accuracy of root cause localization of network faults.

[0027] 3. When facing complex network environments with equivalent multi-path routing, the operation and maintenance equipment can uniquely determine the path taken by the abnormal traffic by parsing the five-tuple information of the abnormal traffic and simulating the hash algorithm on the load balancer. This avoids wasting troubleshooting resources on parallel paths that do not carry abnormal traffic, thereby narrowing down the set of candidate root cause nodes and improving the efficiency and accuracy of root cause location. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating a method for locating the root cause of a network fault in an embodiment of this application.

[0029] Figure 2 This is another flowchart illustrating a method for locating the root cause of a network fault in an embodiment of this application.

[0030] Figure 3 This is a schematic diagram of the physical device structure of the maintenance equipment in the embodiments of this application. Detailed Implementation

[0031] The terminology used in the following embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. As used in the specification and appended claims of this application, the singular expressions “a,” “an,” “the,” “the,” “the,” and “this” are intended to include the plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this application refers to and includes any or all possible combinations of one or more of the listed items.

[0032] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature, and in the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.

[0033] This application provides a method and maintenance equipment for locating the root cause of network faults, which can improve the accuracy of locating the root cause of network faults.

[0034] Please see Figure 1 This is a flowchart illustrating a method for locating the root cause of a network fault in an embodiment of this application.

[0035] S101. In response to the sudden change in performance indicators of the target network element node at the time of abnormal triggering, determine the upstream path of all data transmission paths pointing to the target network element node based on the network topology diagram.

[0036] In this context, a target network element node refers to a specific physical or virtual device within the network monitoring domain whose performance data exceeds the normal operating range, such as a core switch, aggregation router, or service server. A performance metric mutation refers to a drastic fluctuation or jump in the key operational metrics of the target network element node within a short period, such as a surge in port packet loss rate, a significant increase in round-trip latency, or a sharp drop in throughput. The anomaly trigger time is used to indicate the timestamp of the first capture of this performance metric mutation. A network topology diagram refers to the network-wide device connection relationships stored in a graphical data structure, including nodes (devices) and edges (physical or logical links). An upstream path refers to the set of all paths that can deliver service traffic to the target network element node in the data flow direction. Its direction is consistent with the data flow direction, but it needs to be traversed in the reverse direction of the data flow during root cause tracing.

[0037] Specifically, when monitoring data collected by the maintenance equipment via protocols such as Telemetry or SNMP indicates that the performance indicators of a certain network element node deviate from the baseline and trigger an alarm, the maintenance equipment immediately identifies this node as the target network element node and pinpoints the time of the anomaly trigger. Subsequently, the maintenance equipment invokes the global network topology graph maintained in memory and executes a reverse graph traversal algorithm (such as reverse breadth-first search) with the target network element node as the endpoint. This process aims to identify all source devices and intermediate forwarding devices that directly or indirectly send data to the target network element node, thereby constructing one or more potential fault propagation links pointing to the target network element node, i.e., upstream paths, thus narrowing the search space for the root cause of the fault from the entire network to the upstream range that has a direct data dependency relationship with the target network element node.

[0038] In some embodiments, the upstream path of the target network element node can be determined in several ways: Optionally, to cope with topology changes in a dynamic routing environment, before performing path determination, the operation and maintenance equipment can receive link state advertisement messages (such as OSPFLSA or IS-ISLSP) sent by the network control plane. If a link state change event is determined based on the link state advertisement message and the change time is before the anomaly trigger time, the network topology diagram is updated in real time based on the link state change event. This ensures that subsequent path analysis is based on the topology that reflects the network connectivity at the moment the anomaly occurred, avoiding tracing to the wrong path due to the use of outdated topology. Optionally, for complex network environments with equal-cost multi-paths, if multiple parallel equal-cost multi-path routing links are identified based on the network topology diagram, the operation and maintenance equipment can further obtain the inbound traffic five-tuple information (source / destination IP, source / destination port, protocol number) of the target network element node at the time of the anomaly trigger, as well as the load-sharing hash algorithm and hash factor on the equal-cost multi-path routing links. By substituting the inbound traffic quintuple information into the load-sharing hash algorithm for simulation calculation, the unique physical transmission link that the abnormal traffic actually passed through can be calculated, and the unique physical transmission link and the connected device nodes can be identified as the upstream path, thereby eliminating the interference of parallel paths that do not carry abnormal traffic.

[0039] S102. Identify each network device located on the upstream path as a candidate root cause node.

[0040] Among them, candidate root cause nodes refer to each independent network device on the upstream path determined by S101 that may cause or lead to sudden changes in the performance of the target network element node. Due to their critical positions on the data transmission path, the abnormal states of these devices (such as congestion, hardware failure, and configuration errors) may affect the downstream target network element nodes.

[0041] Specifically, after determining the upstream path in step S101, the maintenance equipment traverses each hop in the upstream path, identifying all network devices (including access layer, aggregation layer, and core layer devices) along the path, except for the target network element node itself, as candidate root cause nodes. This step establishes a set containing all potential fault sources, providing an analysis object for subsequent step-by-step troubleshooting.

[0042] S103. Based on the network topology, determine the transmission path segment from each candidate root cause node to the target network element node.

[0043] The transmission path segment refers to the sub-path in the upstream path that starts from a specific candidate root cause node and ends at the target network element node. It includes the candidate root cause node itself, as well as all intermediate forwarding nodes and connection links between it and the target network element node.

[0044] Specifically, for each candidate root cause node identified in step S102, the maintenance device uses the network topology diagram as the starting point and the target network element node as the ending point to perform a pathfinding algorithm (such as Dijkstra's algorithm or breadth-first search) on the topology diagram. Since the data flow direction has been determined in step S101, the pathfinding here is unidirectional and deterministic. The result of this operation is to generate a unique transmission path segment corresponding to each candidate root cause node.

[0045] S104. The sum of the link propagation delay of all single-hop links on the transmission path segment and the average queuing processing delay of all intermediate nodes is used as the cumulative propagation delay of each candidate root cause node relative to the target network element node.

[0046] In this context, a single-hop link refers to a physical or logical connection between two adjacent network nodes on a transmission path segment, such as an optical fiber, copper cable, or VLAN link. Link propagation delay refers to the time required for an optical or electrical signal to transmit through a single-hop link medium, primarily determined by the link length and signal propagation speed, and is typically a relatively fixed value. Intermediate nodes refer to all forwarding devices on the transmission path segment other than the origin (candidate root cause node) and the destination (target network element node). Average queuing processing delay refers to the average waiting and processing time for a data packet within a network device, from entering through an input port to leaving through an output port, including the sum of queuing delay, lookup table forwarding delay, and dequeue delay. Cumulative propagation delay represents the total time required from an event generated by a candidate root cause node (such as the generation of an error log) to its impact (such as performance degradation) being observed at the target network element node.

[0047] Specifically, for each transmission path segment determined in step S103, the maintenance equipment first obtains the propagation delay of each link on the path segment. This value can typically be obtained from network planning data, equipment specifications, or through probing measurements (such as ping). Then, the maintenance equipment obtains the average queuing processing delay of each intermediate node on the path segment. This value can be obtained as a long-term average from historical monitoring data, or as an estimate based on the equipment model and typical load. Finally, the propagation delay of all single-hop links and the average queuing processing delay of all intermediate nodes are added together, and the sum is the cumulative propagation delay of the candidate root cause node relative to the target network element node.

[0048] In some embodiments, the average queuing processing latency of an intermediate node can be predetermined as follows: First, the maintenance equipment obtains sampled data of the port queue depth and processing throughput of the intermediate node from the historical monitoring database over a preset statistical period (such as the past 24 hours or 7 days). Then, based on the M / M / 1 queuing model in queuing theory, the instantaneous queuing latency at each sampling moment is calculated, specifically the queue length divided by the service rate. Next, all instantaneous queuing latencies are statistically averaged to obtain the average queuing processing latency of the node. Optionally, when historical data is insufficient, the average queuing processing latency can be estimated based on typical processing latency parameters in the equipment specifications or using an empirical lookup method based on equipment type and load level. For example, for a core router, the typical average queuing processing latency can be set to 50-200 microseconds; for an access switch, it can be set to 10-50 microseconds.

[0049] S105. Based on the triggering time of the performance index mutation and the cumulative propagation delay, construct an elastic backtracking window for each candidate root cause node.

[0050] The elastic backtracking window refers to the time interval allocated for retrieving abnormal logs for each candidate root cause node. The window's position and width on the timeline are dynamically calculated to cover log recording time deviations caused by transmission latency and network jitter. The timeline center refers to the theoretical moment when the fault occurred. The preset basic tolerance duration refers to the minimum fixed error range allowed by the maintenance equipment. The jitter extension duration refers to the fault tolerance time dynamically increased based on path length. The preset single-hop jitter coefficient is an empirical value for the latency fluctuations that a single network hop may introduce, used to quantify the uncertainty of latency processing by network devices.

[0051] Specifically, after calculating the cumulative propagation delay relative to the target network element node for each candidate root cause node in step S104, the maintenance equipment first determines the center point of the window. The logic is as follows: if the candidate root cause node is the real source of the fault, then the time when it generates the abnormal log, plus the cumulative propagation delay required for the abnormal impact to propagate to the target network element node, should be approximately equal to the triggering time of the performance mutation on the target network element node. Therefore, it can be deduced that the theoretical occurrence time of the root cause event should be the triggering time minus the cumulative propagation delay, which is the time axis center of the elastic backtracking window. Subsequently, to cope with the inherent latency jitter in the network, the maintenance equipment constructs a flexible window width. This width consists of two parts: one part is a fixed preset basic tolerance duration, ensuring basic fault tolerance capability. The other part is a dynamic jitter extension duration, which multiplies the number of intermediate nodes in the transmission path segment by a preset single-hop jitter coefficient. This means that the longer the path and the more devices it passes through, the greater the potential cumulative effect of latency jitter, and therefore the wider the allocated time window. The elastic backtracking window built in this way can more accurately define the log search range that matches the latency of a specific path.

[0052] In some embodiments, the maintenance equipment can determine a preset single-hop jitter coefficient based on the network environment type, equipment performance level, and historical jitter statistics. Specifically, this coefficient has different typical value ranges for different network environments: in a high-performance data center network environment, the preset single-hop jitter coefficient is typically set to 5-15 milliseconds. In an enterprise campus network environment, it is set to 10-30 milliseconds. In a wide area network or public network environment, it is set to 20-100 milliseconds. In actual deployment, the maintenance equipment can analyze the historical latency jitter data of each device node over a period of time (such as the last 30 days) and calculate the 95th percentile of the single-hop latency jitter as the preset single-hop jitter coefficient.

[0053] S106. Select candidate anomaly logs from the running logs generated by each candidate root cause node within its corresponding elastic backtracking window.

[0054] Among them, the operation log refers to the text data automatically generated by the network device during operation, recording various events (including status changes, error messages, configuration changes, and alarms). The candidate anomaly log refers to all operation log entries on a specific candidate root cause node whose generation time falls within the time range of the elastic backtracking window constructed for it.

[0055] Specifically, this step aims to perform preliminary screening from a large amount of raw log data, narrowing down the analysis scope, based on the elastic backtracking window established for each candidate root cause node in step S105. For each candidate root cause node, the operation and maintenance device accesses its log storage system (which can be located on a local device or a centralized log server) and extracts all log records generated between the start and end times of the corresponding elastic backtracking window. However, since the clocks of various devices in a distributed system may deviate, directly comparing the timestamps recorded in the logs may lead to errors. Therefore, an optimized implementation is to first align the timestamps. Specifically, the operation and maintenance device calculates the clock deviation of each node relative to the unified time base of the operation and maintenance device by interacting with each candidate root cause node through clock probe messages. Then, the original recording time of the running logs obtained from each node is corrected using this deviation value to generate a unified time base log. Finally, based on this corrected, globally comparable timestamp, all logs falling within the corresponding elastic backtracking window are filtered out as candidate anomaly logs for subsequent analysis.

[0056] S107. The absolute value of the time difference between the recording time of the candidate anomaly log and the time axis center of the elastic backtracking window is used as the spatiotemporal alignment drift.

[0057] Among them, the spatiotemporal alignment drift is a quantitative indicator used to represent the degree of deviation between the actual occurrence time of a candidate anomaly log and the time when it should theoretically occur (i.e., the center of the time axis).

[0058] Specifically, after filtering candidate abnormal logs in step S106, the maintenance equipment performs a simple mathematical operation on each log. This operation takes the corrected recording time of the log and the center of the timeline of the elastic backtracking window corresponding to the candidate root cause node that generated the log, and calculates the absolute value of the difference between the two. This calculation result is the spatiotemporal alignment drift. The implication of this indicator is that a log with a drift close to zero means that its occurrence time highly matches the root cause event time predicted based on network topology and latency physics. Therefore, the possibility of a causal relationship between this log and the performance mutation of the target network element node is extremely high. Conversely, a log with a large drift, even if it falls on the edge of the elastic window, is relatively less likely to be a direct cause.

[0059] S108. If the log level of a candidate abnormal log is greater than the preset severity threshold, the candidate root cause node corresponding to the candidate abnormal log with the smallest spatiotemporal alignment drift is taken as the root cause node of the failure that caused the sudden change in performance indicators.

[0060] Log levels refer to the severity labels attached by the logging system to distinguish the importance of events, such as Debug, Info, Warning, Error, and Critical, from low to high. A preset severity threshold is a configurable threshold for log levels used to filter out low-level logs that lack sufficient information or are irrelevant.

[0061] Specifically, first, the maintenance equipment filters all candidate anomaly logs for which the spatiotemporal alignment drift was calculated in step S107. The log level of each log is checked and compared with a preset severity threshold (e.g., set to Warning). Only logs with a level higher than this threshold are retained for the final comparison stage. This method excludes logs that, while temporally aligned, do not inherently indicate a serious problem. After severity filtering, the maintenance equipment searches for the log with the smallest spatiotemporal alignment drift among the remaining candidate anomaly logs. Once found, the candidate root cause node that generated this log is preliminarily identified as the root cause node of the current failure.

[0062] In some embodiments, to further improve the accuracy of the root cause node, a verification process can be introduced. Specifically, after determining the initial root cause node, all intermediate forwarding nodes between it and the target network element node are identified, and the performance data of these intermediate nodes during the fault occurrence is examined. The fault propagation chain confidence is calculated by statistically analyzing the proportion of intermediate nodes whose performance degradation exceeds a threshold. Only when the fault propagation chain confidence is higher than a preset value is the node officially confirmed as the root cause and a report generated. Otherwise, it is marked as an isolated anomaly, and the candidate root cause nodes corresponding to the next hourly spatial alignment drift are reviewed.

[0063] In the above embodiments, the operation and maintenance equipment constructs an elastic backtracking window associated with network topology latency and combines it with spatiotemporal alignment drift to quantify causal relationships. This establishes a more reasonable log search interval for each candidate root cause node, thereby reducing the situation where temporal misalignment occurs on the timeline beyond the preset window range, making it difficult for related technologies to capture the correlation between the two, and thus improving the accuracy of network fault root cause localization.

[0064] However, in scenarios where network load changes drastically, the elastic backtracking window determined in the above embodiments may not be accurate enough, or it may be too wide, leading to the introduction of noise, or too narrow, leading to the omission of the root cause of network failure.

[0065] Please see Figure 2 This is another flowchart illustrating a method for locating the root cause of a network fault in an embodiment of this application.

[0066] S201. In response to the sudden change in performance indicators of the target network element node at the time of abnormal triggering, determine the upstream path of all data transmission paths pointing to the target network element node based on the network topology diagram.

[0067] S202. Identify each network device located on the upstream path as a candidate root cause node.

[0068] S203. Based on the network topology, determine the transmission path segment from each candidate root cause node to the target network element node.

[0069] Step S201 is similar to step S101, step S202 is similar to step S102, and step S203 is similar to step S103, so they will not be described again here.

[0070] S204. Obtain discrete sampling data of each intermediate node on the transmission path segment within a preset sampling period before the triggering time of the sudden change in performance index.

[0071] The preset sampling period refers to a specific time window before an anomaly occurs (e.g., 5 minutes before a fault). Discrete sampling data refers to performance snapshots collected at a certain frequency, including the number of bytes in the port buffer queue and the port forwarding rate at multiple time points.

[0072] Specifically, in order to more realistically recreate the network congestion situation before the failure, the operation and maintenance equipment retrieves historical data from the monitoring database for each intermediate node on the transmission path segment within a preset sampling period before the abnormal triggering time. Then, it extracts the number of bytes in the port buffer queue and the port forwarding rate at each sampling time point from the historical data.

[0073] S205. Calculate the ratio of the number of bytes in the port buffer queue to the port forwarding rate at each time point to obtain the queuing delay sampling sequence.

[0074] Specifically, after S204, the maintenance equipment, for each sample pair collected at each time point, calculates an instantaneous queuing delay estimate by dividing the number of bytes in the port buffer queue by the port forwarding rate, based on Little's rule in queuing theory. The calculation results from all time points are combined to form a queuing delay sampling sequence reflecting the queuing delay variation of that node within that period.

[0075] S206. The standard deviation of the queuing delay sampling sequence is determined as the delay jitter value of all intermediate nodes on the transmission path segment.

[0076] Specifically, after obtaining the queuing delay sampling sequence in S205, the standard deviation of the queuing delay sampling sequence is calculated, and this standard deviation is used as the delay jitter value of the candidate root cause node. This delay jitter value is a direct quantification of the processing delay stability of the node; the larger the value, the more severe the delay fluctuation.

[0077] S207. The sum of the link propagation delay of all single-hop links on the transmission path segment and the average queuing processing delay of all intermediate nodes is taken as the cumulative propagation delay of each candidate root cause node relative to the target network element node.

[0078] Step S207 is similar to step S104, and will not be described again here.

[0079] S208. Based on the triggering time of performance index mutation and the cumulative propagation delay, construct elastic backtracking windows for each candidate root cause node.

[0080] First, the maintenance equipment calculates the sum of squares of the latency jitter values ​​of each intermediate node calculated in step S206 to obtain the total path latency variance, which reflects the instability of the entire path. Second, the path latency fluctuation level is determined based on the total path latency variance (e.g., stable, slightly fluctuating, and severely fluctuating levels). Third, the corresponding weighting factor is looked up from a preset mapping table according to the path latency fluctuation level (the larger the fluctuation, the larger the factor). Next, this weighting factor is multiplied by a preset single-hop jitter coefficient to obtain a corrected single-hop jitter coefficient. Finally, the window center is determined based on the trigger time of the performance index mutation and the cumulative propagation delay calculated in S208 (calculated in the same way as in S105), and the window width is calculated. The window width at this point is the sum of the preset basic tolerance duration and the jitter extension duration, where the jitter extension duration is the product of the number of intermediate nodes in the transmission path segment and the corrected single-hop jitter coefficient. In this way, for paths with severe jitter, the system automatically expands the search window to avoid missed detections; for stable paths, the window is narrowed to reduce false alarms.

[0081] Specifically, the calculation method for the time axis center of the elastic backtracking window is consistent with S105. This step is an improvement on the time axis width of the elastic backtracking window in step S105, aiming to construct an elastic backtracking window that better reflects the true jitter characteristics of the path. First, for a transmission path segment, the sum of the squared jitter values ​​(standard deviations) of all intermediate nodes on the path is obtained to get the total path delay variance. Then, according to a preset variance threshold, the variance value of the path is mapped to a path delay fluctuation level. For example, variances less than V1 are considered stable, and variances between V1 and V2 are considered moderate fluctuations. Next, the corresponding weighting factor is looked up in the preset mapping table using the fluctuation level. For example, 0.8 corresponds to stability, and 1.5 corresponds to severe fluctuations. Subsequently, this weighting factor is multiplied by the global preset single-hop jitter coefficient to obtain a corrected single-hop jitter coefficient for the current path. Finally, this corrected coefficient is multiplied by the number of intermediate nodes to calculate the new jitter extension duration, and then added to the preset basic tolerance duration to obtain the final window width. In this way, for paths with severe jitter, the maintenance equipment automatically expands the search window to avoid missing detections. For stable paths, the maintenance equipment narrows the window to reduce false alarms.

[0082] S209. Select candidate anomaly logs from the running logs generated by each candidate root cause node within its corresponding elastic backtracking window.

[0083] S210. The absolute value of the time difference between the recording time of the candidate anomaly log and the time axis center of the elastic backtracking window is used as the spatiotemporal alignment drift.

[0084] S211. If the log level of a candidate abnormal log is greater than the preset severity threshold, the candidate root cause node corresponding to the candidate abnormal log with the smallest spatiotemporal alignment drift is taken as the root cause node of the failure that caused the sudden change in performance indicators.

[0085] Step S209 is similar to step S106, step S210 is similar to step S107, and step S211 is similar to step S108, so they will not be described again here.

[0086] In the above embodiments, the maintenance equipment quantifies the stability of the entire path by calculating the total path delay variance, and dynamically adjusts the jitter coefficient used to calculate the window width accordingly. Thus, for a path that has historically or recently performed stably, its backtracking window automatically narrows to reduce interference from irrelevant logs, while for a path with drastic fluctuations, its window widens accordingly to accommodate greater delay uncertainty, preventing the omission of critical logs due to an excessively narrow window, thereby further improving the accuracy of network fault root cause localization.

[0087] The above describes a method for locating the root cause of a network fault in an embodiment of this application. The following describes an exemplary operation and maintenance device 300 provided in an embodiment of this application.

[0088] Figure 3 This is an exemplary hardware structure diagram of the maintenance device 300 provided in this application embodiment. In some embodiments, the maintenance device 300 is a computer device. The computer device includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores data. The network interface of the computer device is used to communicate with other external terminals or servers via a network connection. In some embodiments, the network interface can be a wired network interface; in some embodiments, the network interface can also be a wireless network interface. When the computer program is executed by the processor, it implements a method for locating the root cause of a network fault according to an embodiment of this application.

[0089] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0090] In some embodiments of this application, a computer-readable storage medium is also provided, including instructions that, when executed on the maintenance device 300, cause the maintenance device 300 to perform a method for locating the root cause of a network fault according to an embodiment of this application.

[0091] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0092] As used in the above embodiments, depending on the context, the term "when..." can be interpreted as meaning "if...", "after...", "in response to determining...", or "in response to detecting...". Similarly, depending on the context, the phrase "when determining..." or "if (the stated condition or event) is interpreted as meaning "if determining...", "in response to determining...", "when (the stated condition or event) is detected", or "in response to detecting (the stated condition or event)".

[0093] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0094] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for locating the root cause of a network fault, characterized in that, Applied to maintenance equipment, the method includes: In response to the sudden change in performance indicators of the target network element node at the time of abnormal triggering, all data transmission paths are determined to be upstream paths of the target network element node based on the network topology diagram; Each network device located on the upstream path is identified as a candidate root cause node; Based on the network topology, the transmission path segments from each of the candidate root cause nodes to the target network element node are determined respectively; Obtain discrete sampling data of each intermediate node on the transmission path segment within a preset sampling period before the triggering time of the sudden change in the performance index. The discrete sampling data includes the number of bytes in the port buffer queue and the port forwarding rate at multiple time points. At each time point, the ratio of the number of bytes in the port buffer queue to the port forwarding rate is calculated to obtain the queuing delay sampling sequence; The standard deviation of the queuing delay sampling sequence is determined as the delay jitter value of all intermediate nodes on the transmission path segment; The sum of the link propagation delay of all single-hop links on the transmission path segment and the average queuing processing delay of all intermediate nodes is taken as the cumulative propagation delay of each candidate root cause node relative to the target network element node. Calculate the total path delay variance based on the sum of squares of the delay jitter values; The path delay fluctuation level is determined based on the total path delay variance. The weighting factor is determined from the preset mapping table based on the path delay fluctuation level. The product of the weighting factor and the preset single-hop jitter coefficient is used as the corrected single-hop jitter coefficient; Based on the triggering time of the performance index mutation and the cumulative propagation delay, an elastic backtracking window is constructed for each of the candidate root cause nodes. The center of the time axis of the elastic backtracking window is the difference between the triggering time and the cumulative propagation delay. The width of the time axis of the elastic backtracking window is the sum of the preset basic tolerance duration and the jitter extension duration. The jitter extension duration is the product of the number of intermediate nodes in the transmission path segment and the corrected single-hop jitter coefficient. Candidate anomaly logs are selected from the running logs generated by each of the candidate root cause nodes within their respective elastic backtracking windows. The absolute value of the time difference between the recording time of the candidate anomaly log and the time axis center of the elastic backtracking window is used as the spatiotemporal alignment drift degree. If the log level of the candidate abnormal log is determined to be greater than a preset severity threshold, the candidate root cause node corresponding to the candidate abnormal log with the smallest spatiotemporal alignment drift is taken as the root cause node of the failure that caused the sudden change in the performance index.

2. The method according to claim 1, characterized in that, The step of determining the upstream path of all data transmission paths pointing to the target network element node based on the network topology diagram specifically includes: Obtain the inbound traffic quintuple information of the target network element node at the time of the anomaly triggering; If multiple parallel equal-cost multi-path routing links are identified based on the network topology diagram, the load-sharing hash algorithm and hash factor on the equal-cost multi-path routing links are obtained. Substitute the inbound traffic quintuple information into the load sharing hash algorithm for simulation calculation to obtain the unique physical transmission link that the inbound traffic quintuple information actually traverses. The unique physical transmission link and the device node connected to the unique physical transmission link are determined as the upstream path pointing to the target network element node.

3. The method according to claim 2, characterized in that, Before the step of obtaining the inbound traffic five-tuple information of the target network element node at the time of the anomaly triggering, the method further includes: Receive link status announcement messages sent by the network control plane; If a link state change event is determined to exist based on the link state announcement message and the change time is before the time of the anomaly triggering, the network topology diagram is updated based on the link state change event.

4. The method according to claim 1, characterized in that, The step of filtering candidate anomaly logs from the runtime logs generated by each candidate root cause node within its corresponding elastic backtracking window specifically includes: A clock probe message is sent to each of the candidate root cause nodes, and the sending timestamp, receiving timestamp, and processing dwell time of each candidate root cause node are recorded. Based on the sending timestamp, the receiving timestamp, and the processing dwell time, calculate the clock deviation value of each candidate root cause node relative to the reference clock of the maintenance equipment; The recording time of the running log of each candidate root cause node is corrected according to the clock deviation value to obtain a unified time base log; Logs that fall within the elastic backtracking window from the unified time-base log are identified as candidate abnormal logs.

5. The method according to claim 1, characterized in that, After the step of identifying the candidate root cause node corresponding to the candidate anomaly log with the smallest spatiotemporal alignment drift as the root cause node of the failure leading to the sudden change in the performance metric, the method further includes: Based on the network topology diagram, all intermediate forwarding nodes between the root cause node of the fault and the target network element node are identified. Obtain the performance monitoring data generated by the intermediate forwarding node within the elastic backtracking window; The number of intermediate forwarding nodes whose performance monitoring data deviates from the baseline operating state by a factor greater than a preset transmission threshold is counted. The ratio of the stated number to the total number of intermediate forwarding nodes is determined as the fault propagation chain confidence level. If the confidence level of the fault propagation chain is greater than a preset confidence threshold, a root cause analysis report is generated, which includes the root cause node of the fault, the spatiotemporal alignment drift, and the confidence level of the fault propagation chain. If the confidence level of the fault propagation chain is less than or equal to the preset confidence threshold, the fault root cause node is marked as an isolated abnormal node, and the candidate root cause node corresponding to the second smallest spatiotemporal alignment drift is reviewed.

6. A maintenance equipment, characterized in that, The maintenance equipment includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the maintenance equipment to perform the method as described in any one of claims 1-5.

7. A computer program product containing instructions, characterized in that, When the computer program product is run on the maintenance equipment, the maintenance equipment performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium comprising instructions, characterized in that, When the instruction is executed on the maintenance equipment, the maintenance equipment performs the method as described in any one of claims 1-5.

Images