Abnormal root cause determination method and device

Abstract

The embodiment of the invention provides an abnormal root cause positioning method and device. The abnormal root cause positioning method and device are applied to a micro-service system of a cloud-side collaborative hybrid architecture. The method comprises the following steps: determining all calling paths of downstream abnormal nodes in the micro-service system; the calling path indicates a calling path from an upstream node of the downstream abnormal node to the downstream abnormal node; aiming at each calling path, the following steps are executed: obtaining the edge-level quantile delay of each calling edge in the calling path; according to the edge-level quantile time delay of each calling edge, determining the path propagation time delay of the calling path; the path propagation time delay of the calling path indicates the longest time required by the abnormal information to be transmitted to the downstream abnormal node along the target calling path; and determining an abnormal root cause according to the path propagation delay of each calling path. According to the method, the time sequence reachability of the upstream node to the downstream abnormal node is determined through the quantitative calculation of the edge-level quantile delay and the path propagation delay, the inverse causality is reduced from the physical propagation level, and the positioning accuracy of the abnormal root cause is improved.

Description

Technical Field

[0001] This application relates to the field of microservice system technology, and in particular to a method and apparatus for determining the root cause of anomalies. Background Technology

[0002] In cloud-edge hybrid microservice systems, the microservice call chain often needs to travel back and forth between the cloud data center and edge nodes. This means that any upstream node failure can be amplified and propagated to downstream critical Service Level Objective (SLO) nodes in a short period of time via communication middleware. To address this type of cross-domain failure propagation, it is necessary to accurately pinpoint the root cause of the failure.

[0003] Currently, sliding window correlation or threshold alarms can be used to directly identify the most relevant indicator using Pearson correlation coefficient, Granger causality, or rule bases, and this most relevant indicator is then used as the root cause of the anomaly. However, this method suffers from insufficient accuracy in locating the root cause of the anomaly. For cross-domain anomaly propagation, how to accurately locate the root cause of the anomaly remains a technical problem to be solved. Summary of the Invention

[0004] This application provides a method and apparatus for determining the root cause of anomalies, which is used to accurately locate the root cause of anomalies in cross-domain anomaly propagation.

[0005] In a first aspect, embodiments of this application provide a method for determining the root cause of an anomaly, applied to a microservice system, wherein the architecture of the microservice system is a cloud-edge collaborative hybrid architecture, the cloud-edge collaborative hybrid architecture including a cloud data center and edge nodes, and the method includes:

[0006] Determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has experienced an exception; the exception window of the upstream node and the downstream abnormal node is the same;

[0007] For each of the aforementioned call paths, the following steps are performed: The edge-level quantile delay of each call edge in the call path is obtained; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the starting node to the ending node of the call edge; based on the edge-level quantile delay of each call edge, the path propagation delay of the call path is determined; the path propagation delay of the call path indicates the maximum time required for the exception information to propagate along the call path to the downstream exception node.

[0008] The root cause of the exception is determined based on the path propagation delay of each of the aforementioned call paths.

[0009] Optionally, determining the root cause of the exception based on the path propagation delay of each of the call paths includes:

[0010] Determine the optical cone leading edge time corresponding to the path propagation delay of each of the aforementioned call paths; the optical cone leading edge time of the call path indicates the sum of the path propagation delay of that call path and the start time of the anomaly window;

[0011] Based on the leading edge time of the optical cone for each of the calling paths, a candidate region of optical cones is determined; the optical cones in the candidate region are the upstream nodes whose leading edge time is less than or equal to the termination time of the abnormal window.

[0012] The root cause of the anomaly is determined based on the candidate region of the light cone.

[0013] Optionally, if the candidate optical cone region is an initial candidate optical cone region, after determining the candidate optical cone region based on the optical cone leading edge time of each calling path, the method further includes:

[0014] Based on the path propagation delay of each of the aforementioned call paths, determine the mean delay and the standard deviation of the delay;

[0015] A trailing baseline value is determined based on the mean delay and the standard delay; the trailing baseline value is correlated with the mean delay and the standard delay deviation.

[0016] Extreme upstream nodes are removed from the initial candidate light cone domain to obtain the target candidate light cone domain; the light cones in the target candidate light cone domain are upstream nodes whose path propagation delay of the calling path is greater than the tail reference value.

[0017] The step of determining the root cause of the anomaly based on the candidate optical cone region includes:

[0018] The root cause of the anomaly is determined based on the target light cone candidate domain.

[0019] Optionally, the method further includes:

[0020] If an upstream node has at least one of the aforementioned call paths, obtain the target call path from the upstream node to the downstream abnormal node; the target call path indicates the call path with the shortest path propagation delay among at least one of the aforementioned call paths of the upstream node;

[0021] Based on the path propagation delay of each of the aforementioned call paths, the mean delay and standard deviation of the delay are determined, including:

[0022] The mean delay and standard deviation of the delay are determined based on the path propagation delay of each target call path;

[0023] The step of removing extreme upstream nodes from the initial optical cone candidate domain to obtain the target optical cone candidate domain includes:

[0024] For each upstream node, if the path propagation delay of the target call path corresponding to that upstream node is greater than the tailing benchmark value, the upstream node is determined to be the extreme upstream node.

[0025] The target optical cone candidate domain is obtained by removing the extreme upstream node from the initial optical cone candidate domain.

[0026] Optionally, the trailing reference value is linearly positively correlated with the sum of the mean delay and three times the standard deviation of the delay.

[0027] Optionally, determining all target call paths for downstream abnormal nodes in the microservice system includes:

[0028] Using the downstream abnormal node as the sink, the service dependency graph of the microservice system is traversed in reverse. All upstream nodes within k layers upstream of the downstream abnormal node are taken as target upstream nodes to obtain the target call path. The target upstream path is the path from the target upstream node to the downstream abnormal node. k is a positive integer.

[0029] Optionally, each call edge in the service dependency graph is bound to a cross-domain jitter hybrid kernel. The cross-domain jitter hybrid kernel is obtained by convolving the latency propagation kernel of the upstream node with the latency distribution models of the first link, the second link, and the third link, respectively, weighting and fusing them according to the link time proportion, and then pruning them by a cumulative energy threshold. The first link is a link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

[0030] Optionally, obtaining the edge-level quantile delay of each call edge in the target call path includes:

[0031] For each call edge, determine its link attribute; take a preset signal level on the delay distribution model corresponding to the link attribute of the call edge to obtain the edge-level quantile delay of the call edge;

[0032] The link attributes of the calling edge include one of the first link, the second link, and the third link; the first link is a link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

[0033] Optionally, the method further includes:

[0034] From the distributed tracing logs, obtain the latency samples of the first link, the second link, and the third link;

[0035] Determine whether the number of delay samples of the first link is greater than or equal to a preset threshold. If yes, fit the delay samples of the first link with a Gaussian mixture model to obtain the delay distribution model of the first link. If no, fit the delay samples of the first link with a unimodal Gamma distribution to obtain the delay distribution model of the first link.

[0036] Determine whether the number of delay samples of the second link is greater than or equal to a preset threshold. If yes, fit the delay samples of the second link with a Gaussian mixture model to obtain the delay distribution model of the second link. If no, fit the delay samples of the second link with a unimodal Gamma distribution to obtain the delay distribution model of the second link.

[0037] Determine whether the number of delay samples of the third link is greater than or equal to a preset threshold. If yes, perform Gaussian mixture model fitting on the delay samples of the third link to obtain the delay distribution model of the third link. If no, use a unimodal Gamma distribution to fit the delay samples of the third link to obtain the delay distribution model of the third link.

[0038] Secondly, embodiments of this application provide an anomaly root cause determination device applied to a microservice system, wherein the microservice system has a cloud-edge collaborative hybrid architecture, the cloud-edge collaborative hybrid architecture including a cloud data center and edge nodes, and the device includes:

[0039] A path determination unit is used to determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has experienced an exception; the exception window of the upstream node and the downstream abnormal node is the same.

[0040] The delay determination unit is configured to perform the following for each of the call paths: obtain the edge-level quantile delay of each call edge in the call path; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the start node to the end node of the call edge; determine the path propagation delay of the call path based on the edge-level quantile delay of each call edge; the path propagation delay of the call path indicates the maximum time required for the exception information to be transmitted along the call path to the downstream exception node;

[0041] The root cause determination unit is used to determine the root cause of the anomaly based on the path propagation delay of each of the call paths.

[0042] Thirdly, embodiments of this application provide an electronic device, including:

[0043] Memory, used to store computer programs;

[0044] A processor for executing the computer program to implement the method as described in any one of the first aspects.

[0045] Fourthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the method in any of the possible implementations of any of the above aspects.

[0046] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program (also referred to as code or instructions) that, when run on a computer, causes the computer to perform the method in any of the possible implementations of any of the above aspects.

[0047] Sixthly, embodiments of this application provide a chip system including one or more processors for calling and executing instructions stored in memory, causing the methods in any of the above aspects or possible implementations to be executed. The chip system may be composed of chips or may include chips and other discrete devices.

[0048] This application provides an anomaly root cause localization method and apparatus, applied to a cloud-edge collaborative hybrid architecture microservice system, specifically executed by a root cause localization system. The method includes: determining all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system where the anomaly occurred; the upstream node and the downstream abnormal node have the same anomaly window; for each call path, performing the following: obtaining the edge-level quantile delay of each call edge in the call path; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the starting node to the ending node of the call edge; determining the path propagation delay of the call path based on the edge-level quantile delay of each call edge; the path propagation delay of the call path indicates the longest time required for the anomaly information to propagate along the target call path to the downstream abnormal node; and determining the anomaly root cause based on the path propagation delay of each call path. This application clarifies the temporal reachability of upstream nodes to downstream abnormal nodes by quantitatively calculating edge-level quantile delay and path propagation delay, reducing inverse causality from the physical propagation level. Therefore, compared with using sliding window correlation or threshold alarms, this method of anomaly root cause localization can reduce the probability of misjudging downstream symptoms as root causes and improve the accuracy of anomaly root cause localization. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This application provides a flowchart of an abnormal root cause determination method;

[0051] Figure 2 A flowchart illustrating a method for determining the path propagation delay of a call path, provided in an embodiment of this application;

[0052] Figure 3 A flowchart illustrating a method for a health baseline model and actual residuals of nodes provided in this application embodiment;

[0053] Figure 4 This application provides a flowchart of a method for obtaining a cross-domain jitter hybrid core;

[0054] Figure 5A flowchart illustrating a method for locating the root cause of anomalies based on cross-domain jitter hybrid kernels, provided in this application embodiment;

[0055] Figure 6 A flowchart illustrating a method for obtaining a candidate region of light cones, provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of an abnormal root cause determination device provided in an embodiment of this application. Detailed Implementation

[0057] To enable those skilled in the art to better understand the present application, the technical solutions in this embodiment will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0059] First, the technical terms involved in the embodiments of this application will be explained.

[0060] (1) Light cone

[0061] In spacetime geometry, the light cone is used to delineate the region that an event can influence or be influenced by. The light cone includes the past light cone and the future light cone. The past light cone contains all historical events that can transmit information at the speed of light and act upon the event; the future light cone contains all future events that the event can influence within the constraints of information propagation. The region outside the light cone is causally isolated from the current event due to the limited propagation speed.

[0062] In this embodiment, the optical cone is specifically a quantile delay optical cone, which refers to an upstream node whose shortest path delay to a downstream abnormal node does not exceed the span of the abnormal window. The optical cone provided in this embodiment retains the causal reachability principle of the original physical concept and can be aligned with the actual network latency in microservice scenarios. Therefore, the optical cone can accurately locate the root cause of anomalies.

[0063] (2) Microservice system

[0064] A microservice system breaks down a traditional monolithic application into small, loosely coupled, and reusable service units (microservices) based on business functions. Each microservice focuses on a single business scenario and achieves cross-service communication through standardized interfaces to jointly complete complex business processes.

[0065] A microservice system comprises business microservices, communication middleware, and a monitoring and operations platform. Business microservices are the core functional units; for example, in an e-commerce system, business microservices might include order microservices, payment microservices, user management microservices, and data analysis microservices. The communication middleware is responsible for information transmission and request forwarding between the various microservices. The monitoring and operations platform is used to monitor the operational metrics of each microservice, enabling automatic service deployment, elastic scaling, and fault self-healing.

[0066] Current microservice systems often adopt a cloud-edge collaborative hybrid architecture. This architecture includes cloud data centers and edge nodes. Each microservice within a microservice system can be deployed in either the cloud data center or the edge node as needed. For example, taking an e-commerce microservice system, edge nodes can deploy an order microservice (referred to as the edge order microservice), a local communication proxy, and a local cache node, while the cloud data center can deploy a payment microservice (referred to as the cloud payment microservice), a message queue cluster (cloud message queue cluster), and a data analytics microservice.

[0067] The cloud-edge collaborative hybrid architecture allows microservice call chains to travel back and forth between the cloud data center and edge nodes. This means that any anomaly in an upstream node can be amplified and propagated to critical SLO nodes downstream within a short period of time through the communication middleware. For example, continuing with the e-commerce microservice system, after a user sends an order request, the corresponding microservice call chain is as follows: Edge Order Microservice → Local Communication Proxy → Cloud Message Queue Cluster → Cloud Payment Microservice → Cloud Message Queue Cluster → Edge Order Microservice → Critical SLO Node. If the local communication proxy at the edge node experiences a 10ms momentary delay due to network jitter, requests to the order microservice will accumulate on the local cache node, triggering the asynchronous RPC timeout retry mechanism and amplifying the request volume. The accumulated requests are synchronized to the cloud data center through the cloud message queue cluster, causing a surge in pressure on the cloud message queue cluster. This pressure then spreads to the cloud payment microservice, causing its response delay to exceed 50ms, for example, increasing to 200ms. This delay is then propagated back to the edge order microservice through the cloud message queue cluster, increasing order confirmation delays. Therefore, accurately locating the root cause is crucial to address this cross-domain anomaly propagation.

[0068] It should be noted that the cross-domain anomaly propagation provided in this application embodiment refers to the propagation of anomalies between cloud data centers and edge nodes.

[0069] Currently, sliding window correlation or threshold alarms can be used to directly identify the most relevant indicator through Pearson correlation coefficient, Granger causality, or rule base, and this most relevant indicator can be used as the root cause of the anomaly. However, in high-concurrency scenarios, this method is used to misjudge downstream symptoms as the root cause, resulting in insufficient accuracy in locating the root cause of the anomaly.

[0070] In view of this, embodiments of this application provide an anomaly root cause localization method, which is applied to a microservice system with a cloud-edge collaborative hybrid architecture and can be executed by the root cause localization system.

[0071] The root cause localization system can perform the following operations: determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system where the abnormality occurred; the upstream node and the downstream abnormal node have the same abnormal window; for each call path, perform the following operations: obtain the edge-level quantile delay of each call edge in the call path; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the starting node to the ending node of the call edge; determine the path propagation delay of the call path based on the edge-level quantile delay of each call edge; the path propagation delay of the call path indicates the longest time required for the abnormal information to be transmitted along the target call path to the downstream abnormal node; determine the root cause of the abnormality based on the path propagation delay of each call path. This application clarifies the temporal reachability of upstream nodes to downstream abnormal nodes by quantitatively calculating edge-level quantile delay and path propagation delay, reducing inverse causality from the physical propagation level. Therefore, compared with sliding window correlation or threshold alarms, using Pearson correlation coefficient, Granger causality, or rule base to directly lock the most relevant indicators, this abnormal root cause localization method can reduce the probability of misjudging downstream symptoms as root causes and improve the accuracy of abnormal root cause localization.

[0072] In practical applications, a root cause analysis system can include a software system, which can be provided to the user as a software package for self-deployment, such as on a local physical server or in a private cloud. In some possible implementations, the load migration system can also be deployed in a public cloud and provided to the user as a cloud service. For example, a cloud service provider can offer a one-stop system service integrating the functions of the aforementioned core components.

[0073] The method for determining the root cause of anomalies provided in this application will be described below with reference to the accompanying drawings. It should be noted that this method is applied to a microservice system with a cloud-edge collaborative hybrid architecture, and the execution entity of this method will be illustrated using a root cause localization system as an example.

[0074] Appendix Figure 1This application provides a flowchart of an anomaly root cause determination method, which includes the following steps:

[0075] S10, determine all call paths of downstream abnormal nodes in the microservice system.

[0076] The call path indicates the call path from the upstream node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has an exception; the exception window of the upstream node and the downstream abnormal node is the same.

[0077] An upstream node refers to a node in a microservice call chain that precedes a downstream abnormal node in terms of business process and data transmission order. An exception in an upstream node may propagate through the microservice call chain, affecting the operational status of downstream nodes; therefore, it is a candidate for the root cause of an exception in a downstream abnormal node. In this embodiment, the upstream node of the downstream abnormal node can also be referred to as the cause node.

[0078] For example, consider the microservice call chain: Edge Order Microservice → Local Communication Proxy → Cloud Message Queue Cluster → Cloud Payment Microservice → Cloud Message Queue Cluster → Edge Order Microservice → Critical SLO Node. The root cause localization system directly detects abnormal microservice nodes, such as critical SLO nodes experiencing severe timeouts with an exception window of [10:00:00, 10:05:00]. The critical SLO node is an abnormal downstream node, and the upstream nodes corresponding to this abnormal downstream node include the Edge Order Microservice, Local Communication Proxy, Cloud Message Queue Cluster, Cloud Payment Microservice, Cloud Message Queue Cluster, and Edge Order Microservice.

[0079] In one specific implementation, the root cause localization system can use the downstream abnormal node as the sink and traverse the service dependency graph of the microservice system in reverse. It can take all upstream nodes within k layers upstream of the downstream abnormal node as the target upstream node to obtain the target call path. The target upstream path is the path from the target upstream node to the downstream abnormal node. k is a positive integer, such as k=6.

[0080] In one specific implementation, each call edge in the service dependency graph is bound to a cross-domain jittering hybrid kernel. This kernel is used to quantify the latency impact of upstream node anomalies on downstream abnormal nodes. In this embodiment, the cross-domain jittering hybrid kernel is obtained by convolving the latency propagation kernel of the upstream node with the latency distribution models of the first, second, and third links, respectively, then weighting and fusing them according to the time proportion of each link, and finally pruning them using a cumulative energy threshold. Here, the first link is a link within the cloud data center, the second link is a link between edge nodes, and the third link is a cross-domain link between the cloud data center and edge nodes. The cross-domain jitter hybrid kernel fully captures the latency propagation patterns of different links by convolving the latency propagation kernel of the upstream node with the latency distribution models of the three types of links one by one. Then, through weighted fusion based on the link time proportion, the final hybrid kernel can truly reflect the latency impact weight of the entire path in the microservice call chain from cloud to edge (i.e., cloud data center end and edge node end). This solves the problem that traditional single kernel functions cannot adapt to heterogeneous links and improves the quantitative accuracy of anomaly propagation. The propagation kernel after convolution and weighted fusion contains a large number of redundant low-frequency components. The hybrid kernel is optimized by pruning the cumulative energy threshold: the energy distribution of the hybrid kernel is calculated, and high-frequency components with a cumulative energy proportion exceeding the threshold (e.g., 95%) are retained, while low-frequency redundant components are removed, which can reduce computational complexity. The cross-domain jitter hybrid core is bound one-to-one with each call edge of the service dependency graph, which can transform the service dependency graph into an intelligent topology graph with anomaly propagation quantification features. This enables the upstream node's anomaly contribution to be inferred by the hybrid core of the call edge when a downstream node experiences latency anomalies. Combined with the anomaly window consistency condition, this allows for accurate tracing from the downstream anomaly node to the upstream root cause node.

[0081] For ease of explanation later, all call paths from upstream node u to downstream abnormal node v are represented as follows:

[0082]

[0083] S20, obtain the path propagation delay for each call path.

[0084] The path propagation delay of a call path refers to the maximum time required for abnormal information to be transmitted along the call path to the downstream abnormal node. In this application, the abnormal windows of the upstream and downstream abnormal nodes are the same, and the longest propagation delay can be used as the time boundary threshold of the abnormal window. For example, the start time of the downstream abnormal node is greater than or equal to the sum of the start time of the upstream node and the path propagation delay, which can provide a rigid time constraint for node screening.

[0085] For example, Appendix Figure 2 A flowchart illustrating a method for determining the path propagation delay of a call path, provided in an embodiment of this application, is shown. The method includes the following steps:

[0086] S201: Obtain the edge-level quantile delay of each calling edge in the calling path. The edge-level quantile delay of the calling edge indicates the maximum propagation delay from the starting node to the ending node of the calling edge.

[0087] S202: Determine the path propagation delay of the call path based on the edge-level quantile delay of each call edge.

[0088] Specifically, for the same call path, for example The path propagation delay of the call path is the sum of the edge-level quantile delays of each edge in the call path.

[0089] In one example, the root cause localization system can determine the link attributes of each call edge; then, by applying a pre-set confidence level to the latency distribution model corresponding to the link attributes, it obtains the edge-level quantile latency of the call edge. The link attributes of the call edge include the first link (referred to as cloud), the second link (referred to as cloud), and so on. The first link is one of the following: the first link is a link within the cloud data center, the second link is a link between edge nodes, and the third link is a cross-domain link between the cloud data center and edge nodes.

[0090] For example, the confidence level for call path P is The edge-level quantile delay of edge e is called. As shown in formula (1):

[0091] (1)

[0092] in, It can be adjusted as needed, for example . For quantile calculations, `type` indicates the link attribute and the time-series distribution model of the link. After sorting by probability, select the one with the highest cumulative probability. The latency value. This method, compared to a fixed latency threshold, can adapt to the latency fluctuation characteristics of the link. For example, if the latency distribution model of a cross-type link... If 95% of the latency is ≤100ms, then the edge-level latency is 100ms. This means that there is a 95% chance that the propagation latency of the cross-domain call edge e will not exceed 100ms.

[0093] It should be noted that the link attribute of the calling edge is... .

[0094] Then the path propagation delay of path P. As shown in formula (2):

[0095] (2)

[0096] It's understandable that the more network domains a path traverses, the more the long-tail characteristic accumulates. The long-tail characteristic refers to the phenomenon where, in cloud, edge, and cross-domain links, some requests experience significantly higher latency than the average link latency due to factors such as network congestion, route hopping, and fluctuations in edge node resources. Although these slow requests account for a small percentage, they are not negligible, resulting in a trailing shape in the high-latency range of the latency distribution curve. Therefore, the path percentile latency provided in this application's embodiments more accurately reflects the propagation characteristics of cross-domain paths, and this approach can improve the accuracy of causal reachability determination.

[0097] In this embodiment, the link latency distribution model refers to the link latency probability density distribution model, which is used to quantify the latency fluctuation pattern and long-tail characteristics of the link. In this embodiment, the root cause localization system can obtain latency samples of the first link, the second link, and the third link from the distributed tracing log. Through probability distribution fitting, the latency samples of the first link, the second link, and the third link are fitted respectively to obtain the latency distribution models of the first link, the second link, and the third link.

[0098] Furthermore, to address the fitting accuracy issue caused by variations in the number of link delay samples, the root cause localization system can also determine the link delay distribution model based on whether the number of link delay samples is greater than or equal to a preset threshold. If so, a Gaussian mixture model is applied to the link delay samples to obtain the link delay distribution model, thus adapting to the multi-peak characteristics of link delay. If not, a unimodal Gamma distribution is used to fit the link delay samples to obtain the link delay distribution model, avoiding overfitting of the Gaussian mixture model due to insufficient samples. The preset threshold can be adjusted as needed, for example, a preset threshold of 500 samples.

[0099] Specifically, the root cause localization system can determine whether the number of delay samples in the first link is greater than or equal to a preset threshold. If so, it performs a Gaussian mixture model fitting on the delay samples of the first link to obtain the delay distribution model of the first link; otherwise, it uses a unimodal Gamma distribution to fit the delay samples of the first link to obtain the delay distribution model of the first link. Similarly, it can determine whether the number of delay samples in the second link is greater than or equal to a preset threshold. If so, it performs a Gaussian mixture model fitting on the delay samples of the second link to obtain the delay distribution model of the second link; otherwise, it uses a unimodal Gamma distribution to fit the delay samples of the second link to obtain the delay distribution model of the second link. Finally, it can determine whether the number of delay samples in the third link is greater than or equal to a preset threshold. If so, it performs a Gaussian mixture model fitting on the delay samples of the third link to obtain the delay distribution model of the third link; otherwise, it uses a unimodal Gamma distribution to fit the delay samples of the third link to obtain the delay distribution model of the third link.

[0100] It should be noted that the Gaussian mixture model provided in this application embodiment can be a 2-3 component Gaussian mixture model, or other Gaussian models, and this application embodiment is not limited thereto.

[0101] S30: Determine the root cause of the exception based on the path propagation delay of each call path.

[0102] In one specific implementation, the root cause localization system can determine the optical cone front time corresponding to the path propagation delay of each call path; the optical cone front time of a call path indicates the sum of the path propagation delay of that call path and the start time of the exception window. For example, the exception window is [ The path propagation delay of path P is Then call the optical cone front time of path P. As shown in formula (3):

[0103] (3)

[0104] This indicates the start time of the upstream node within the anomaly window, i.e. An anomaly is triggered at any given time, up to the latest time it affects downstream anomalous nodes. The root cause localization system can determine the candidate region of the optical cone based on the optical cone leading-edge time of each calling path. The optical cones in the candidate region are upstream nodes whose optical cone leading-edge time is less than or equal to the termination time of the anomaly window, and the root cause of the anomaly is determined based on the candidate region. This embodiment of the application removes upstream nodes corresponding to temporally impossible paths, which can reduce the impact of upstream nodes corresponding to temporally impossible paths on root cause localization. This method helps improve the accuracy of anomaly root cause localization. For example, the optical cone leading-edge time of path P... Since this path is a temporally impossible path, the path P is removed from the candidate domain of the optical cone.

[0105] In another implementation, the root cause localization system can determine the initial optical cone candidate region based on the optical cone leading-edge time of each call path. The root cause localization system also determines the mean delay based on the path propagation delay of each call path. and latency standards The trailing baseline value is determined based on the mean and standard deviation of the delay. The trailing baseline value is correlated with the mean and standard deviation of the delay; for example, the trailing baseline value is... Extreme upstream nodes are removed from the initial optical cone candidate domain to obtain the target optical cone candidate domain; the optical cones in the target optical cone candidate domain are upstream nodes whose path propagation delay to the called path is greater than the tail reference value. In this embodiment, extreme upstream nodes can be removed from the optical cone candidate domain, that is, upstream nodes whose path propagation delay to the downstream abnormal node deviates significantly from the normal delay distribution of most paths. This method can avoid the influence of the node on subsequent root cause localization and further improve the localization accuracy.

[0106] In another example, if an upstream node has multiple call paths, the target call path from the upstream node to the downstream abnormal node is obtained; the target call path indicates the call path with the shortest path propagation delay among the multiple call paths of the upstream node; based on the path propagation delay of each target call path, the average delay and delay standard are determined; for each upstream node, if the path propagation delay of the target call path corresponding to the upstream node is greater than the tailing benchmark value, the upstream node is determined to be an extreme upstream node; extreme upstream nodes are removed from the initial optical cone candidate domain to obtain the target optical cone candidate domain.

[0107] For example, for an upstream node u, there are multiple call paths to reach the downstream abnormal node v. In this embodiment of the application, only the path corresponding to the shortest path propagation delay is retained, denoted as... As shown in formula (4):

[0108] (4)

[0109] In this way, each upstream node retains only one representative path. This approach can avoid multiple paths from the same upstream node participating in subsequent calculations, reduce redundancy, and also help reduce the complexity of constructing the subsequent kernel dictionary.

[0110] In summary, this application embodiment clarifies the temporal reachability of upstream nodes to downstream abnormal nodes by quantitatively calculating edge-level quantization delay and path propagation delay, reducing inverse causality from the physical propagation level. Therefore, compared with using sliding window correlation or threshold alarms, this abnormal root cause localization method can directly lock the most relevant indicators by using Pearson correlation coefficient, Granger causality, or rule base, which can reduce the probability of misjudging downstream symptoms as root causes and improve the accuracy of abnormal root cause localization.

[0111] The following, in conjunction with specific embodiments, illustrates... Figure 3 ~Appendix Figure 5 This application introduces a method for determining the root cause of anomalies, based on embodiments of the present application. (Appendix) Figure 3 A flowchart illustrating a method for a health baseline model and actual residuals of nodes, provided in an embodiment of this application. (Attached) Figure 4 This application provides a flowchart of a method for obtaining a cross-domain jitter hybrid core, as illustrated in the embodiments of this application. Figure 5 This document provides a flowchart of a method for locating the root cause of anomalies based on cross-domain jitter hybrid kernels, as illustrated in an embodiment of this application. It should be noted that in this embodiment, the target time is time t, and the health baseline model is a sliding baseline model, updated over time, as an example. Furthermore, to improve positioning accuracy, the residuals are normalized to unify the processing units.

[0112] like Figure 3 As shown, the method includes steps S310 to S340.

[0113] S310: The root cause localization system obtains the health range of a node.

[0114] In this embodiment, the health interval of a node includes the values ​​of multiple indicators, each of which has a value less than or equal to a preset SLO upper limit. Furthermore, the health interval excludes indicator values ​​corresponding to fault windows. In this embodiment, the root cause localization system uses the monitoring logs from the nearest D days to time t as the original dataset. It filters out indicator values ​​in the original dataset that exceed the preset SLO upper limit and removes fault windows registered by the monitoring and maintenance platform. The remaining continuous data segments constitute the health interval of the node.

[0115] It should be noted that D is a number greater than 0, for example, D is 10. The value of D can be adjusted as needed. The preset SLO upper limit is a pre-set upper limit value, which can be adjusted as needed. For example, the SLO upper limit for latency-related indicators is 50ms, and the SLO upper limit for error-related indicators can be set to 1%.

[0116] S320: Baseline model fitting.

[0117] If the values ​​in the health interval are periodic and continuous, an autoregressive model can be used to fit the values ​​in the health interval to obtain the health baseline model for that node. If the base value of the values ​​in the health interval is less than or equal to a preset base value threshold and is a discrete value, a Kalman filter can be used to fit the values ​​in the health interval to obtain the health baseline model for that node.

[0118] For example, metrics such as request processing latency and throughput exhibit periodicity and continuity. An autoregressive model can be used to fit the values ​​within the healthy interval to obtain the health baseline model for that node. Similarly, metrics such as misalignment rate or the number of failed API calls have values ​​within the healthy interval that are less than or equal to a preset threshold and are discrete. A Kalman filter model can be used to fit the values ​​within the healthy interval to obtain the health baseline model for that node.

[0119] In practical implementation, the autoregressive model can be an autoregressive model with a seasonal term, such as SARIMA, and the Kalman filter can be a Beta-Binomial Kalman filter. The preset threshold number is a positive integer and can be adjusted as needed; for example, the preset threshold number can be 20.

[0120] This application's embodiments employ an autoregressive model with a seasonal term for fitting periodic and continuous interval values, accurately capturing periodic trends and seasonal fluctuations. Therefore, the constructed health benchmark model accurately reflects the true changing patterns of the indicators. For interval values ​​with small and discrete base values, Kalman filtering is used for fitting, which helps solve the problem of fitting distortion to low-base data and improves the accuracy of characterizing the sparse fluctuation patterns of the indicators.

[0121] In one example, the root cause localization system specifically outputs the parameter vector and prediction error variance corresponding to the health baseline model for each node. The parameter vector is the core representation factor of the health baseline model, containing all the key parameters of the model and directly determining its shape and prediction accuracy. The values ​​of the parameter vector are obtained by training on historical operational data under the node's health state and are crucial for adapting the model to the node's health characteristics. The prediction error variance is a validity metric for the health baseline model, reflecting the degree of deviation between the model's predicted values ​​and the actual observed values ​​under the node's health state. It measures the model's fitting accuracy to the node's health characteristics and provides a threshold basis for anomaly detection.

[0122] S330: Root cause localization model sliding reestimation.

[0123] The root cause localization system re-estimates the health status by taking the most recent new health segment of a preset fixed duration at the first interval. This forms a sliding health benchmark model. The first duration can be adjusted as needed, for example, to 30 minutes, and the fixed duration can also be adjusted as needed, for example, to 24 hours.

[0124] In this embodiment of the application, information corresponding to the health baseline model of each node can be stored. For example, the node... The following information is stored in a data warehouse for real-time forecasting during runtime:

[0125] Where v is the identifier of node v, used to uniquely identify the node. For baseline fitting model, The baseline fitting time, Let v be the parameter vector of node v. Let V be the prediction error variance for node v. Thus, during operation, based on the node and its monitoring metrics, the corresponding fitting model and parameter vectors are retrieved from the data warehouse, and the parameter vectors are input into the model to obtain the healthy baseline model.

[0126] S340, the root cause localization system determines the actual residuals of nodes based on the node's health baseline model.

[0127] In one specific application, after the microservice system enters the runtime phase, sampling is performed at fixed intervals. The system continuously receives real-time monitoring values ​​from each node. At time t, the root cause localization system provides the expected value of the node based on its health baseline model. For example, based on the health baseline model of node v, it provides the expected value of node v. Root cause localization systems use the difference between a node's actual observed value and its expected value as the actual residual at time t. For example, the actual observed value of node v... With this expected value Subtracting the two yields the actual residual at time t. As in formula (5):

[0128] (5)

[0129] Prediction error variance Standardize the actual residual at time t to obtain the standardized residual. As in formula (6):

[0130] (6)

[0131] This ensures that the residuals of different nodes and indicators are on a uniform scale, facilitating subsequent threshold comparisons. Furthermore, for ease of use, the root cause localization system will also include <v, t, , > Store the residual cube to form an anomaly sequence that is updated over time.

[0132] Furthermore, in order to automatically identify the real abnormal segments in the continuous monitoring stream, a dual detection strategy is applied to the standardized residuals, which specifically includes the following: a point-level anomaly detection strategy and a window-level aggregation strategy.

[0133] Specifically, the point-level anomaly detection strategy involves quantifying transient anomalies by utilizing the degree of deviation from the normal baseline. Specifically, control limits are set. .when When the sampling time is t, it is considered that node v has a transient anomaly. For numbers greater than 0, adjustments can be made as needed, for example... The value is 3.

[0134] For example, suppose the health baseline model of an e-commerce edge caching service has been trained, and set... =3, the real-time acquisition delay during operation is 210ms; the expected delay of the healthy baseline model is 180ms, the variance of the prediction error during the healthy period is 100, the residual is 30ms, and the standardized residual is 3. Since the standardized residual does not exceed 3, this indicator is considered to be within the normal range.

[0135] Considering that sporadic noise may cause single-point exceedances, this application embodiment filters sporadic noise by exceeding the threshold in continuous intervals. Specifically, it targets continuous intervals whose length exceeds a preset length (e.g., a preset length of 3 sampling periods). , Continue to exceed or its average value exceeds In this interval The window has been identified as abnormal.

[0136] Detected anomaly windows are added to the event stream. Each event is recorded in the event stream with the following information: node, start time of the anomaly window, end time of the anomaly window, monitoring index type, and maximum actual error. These events provide temporal boundaries for subsequent light cone candidate domains and limit the observation range that needs to be interpreted in sparse inversion, achieving a crucial transition from raw monitoring streams to structured anomaly information.

[0137] Appendix Figure 4 This application provides a flowchart of a method for obtaining a cross-domain jitter hybrid core, as shown in the embodiments. Figure 4 As shown, this section includes S410~S450:

[0138] S410: The root cause localization system extracts the initial time delay effect kernel from the original residual signal.

[0139] The time-delay impact kernel is a function that quantifies the propagation of an upstream node anomaly over time and its influence on the response of downstream anomalous nodes. In this embodiment, the time-delay impact kernel is used to describe the effect residual's influence on the effect residual after a unit impulse input of the cause residual, at different lag times. The time-delay impact kernel is used to capture the decay law of lag time and impact amplitude. Here, lag time refers to the time difference between the occurrence of an upstream node anomaly and the formation of an anomalous impact at the downstream anomalous node after a certain time delay.

[0140] For ease of description, we will use the upstream node u as the cause node and the downstream abnormal node v as the result node as an example. The standardized residual is obtained by standardizing the actual residual (called the causal residual) of the upstream node u. The standardized residual is obtained by standardizing the actual residual (effect residual) of the downstream abnormal nodes. See [link to documentation] for details on how to obtain it. Figure 3 As shown.

[0141] In this embodiment of the application, the root cause localization system for and First, perform a first-order difference and normalize the result to zero mean and unit variance to obtain the first target sequence. .in, and It is a zero-mean, approximately stationary sequence. This ensures that subsequent cross-correlation analysis focuses on short-term fluctuations rather than long-term trends.

[0142] Among them, the upstream and downstream abnormal nodes share a common abnormal time window. .

[0143] S420, the root cause localization system determines the cross-correlation curve.

[0144] Specifically, the root cause localization system can obtain the cross-correlation curves corresponding to the lag residuals and the causal residuals; the lag residual is the lag time of the effect residual. The resulting residual. For example, The lagged residual is .exist The cross-correlation curves are calculated as shown in formulas (7) to (8):

[0145] (7)

[0146] N (8)

[0147] L0 represents the effective time range of the actual residual. Lag for the second target sequence The corresponding residuals. The cross-correlation curve shows the correlation strength of "cause-lead-effect" at different time lags. Specifically, the cross-correlation curve... The dependent variable is the cross-correlation value between the lagged residuals and the causal residuals, and the independent variable of the cross-correlation curve is... .

[0148] S430, the root cause localization system extracts the main peak of the cross-correlation function and performs kernel normalization to obtain the preliminary time delay influence kernel.

[0149] Root cause localization system in The region is used to obtain the baseline lag time at which the global maximum peak of the cross-correlation curve is located. ;by Centered on, the cross-correlation curves at The initial time delay impact kernel is obtained by truncating the interval and normalizing it according to the peak value. As shown in formula (9):

[0150] (9)

[0151] in, The target lag time corresponding to the first decrease of the cross-correlation value in the cross-correlation curve to the target correlation value. For example, the lag time corresponding to when the cross-correlation value first drops to 10% of its peak. Normalization ensures that the numerical scale of the kernel is comparable; truncation ensures that its support length only covers the period of significant influence.

[0152] Furthermore, if the global maximum peak occurs at a negative lag time (i.e., If the cross-correlation curve shows no significant peak, it is concluded that the two sequences lack unidirectional causal indications, and kernel estimation of the signal pair is stopped. Thus, a preliminary time-delay-influence kernel is obtained. and the corresponding benchmark lag time Target lag time And peak amplitude value.

[0153] S440, the root cause localization system uses regularized deconvolution to process the initial delay impact kernel to obtain the refined delay impact kernel.

[0154] For ease of explanation, the call path below refers to the call path from upstream node v to downstream abnormal node v, and the initial latency impact core of this call path is... Regularized deconvolution affects the initial time delay kernel The noise reduction and optimization process will be explained. In the embodiments of this application, a first target sequence is used. Second target sequence This will be used as a benchmark for explanation.

[0155] The specific processing method includes the following steps:

[0156] Step A1: For and Perform convolution matrix conversion.

[0157] In the error window In the middle, let the support length be... .right and Perform convolution matrix processing. In this embodiment of the application, the first target sequence is... Constructing Toeplitz convolution matrices ,in ); for the second target sequence Constructing the Toeplitz convolution matrix And extract the first line to get y.

[0158] For example, the first target sequence The corresponding X matrix is ​​as follows:

[0159]

[0160] Second target sequence The corresponding y-matrix is ​​as follows:

[0161]

[0162] in, .

[0163] in It is the initial delay that needs to be corrected that affects the kernel. This represents observation noise. After matrixing, the convolution operation involving sliding multiplication and summation can be transformed into ordinary matrix multiplication for subsequent processing.

[0164] Step A2: Least squares with smoothing regularization

[0165] We introduce a first-order difference matrix and a smoothed least squares objective function. The first-order difference matrix D is:

[0166]

[0167] The objective function for smooth least squares is:

[0168]

[0169] in, These are fixed parameters that can be adjusted as needed, for example... The value is 0.01. The first term requires convolution. Try to reproduce The second term suppresses the jagged edges between adjacent kernel coefficients. We put... Normalized As the starting point for optimization, the solution is obtained through the conjugate gradient iteration method.

[0170] Step A3: Post-processing

[0171] Root cause localization system Scanning from the peak to the right: If the h value is lower than the target value for f consecutive sampling points, it is considered that the effective energy has decayed. The tail is truncated at this position to obtain the refined time delay influence kernel. f is a positive integer, for example, f=3. The target value is related to the peak value, for example, target value = 15% × peak value.

[0172] Furthermore, the root cause localization system, upon obtaining... First normalize by the highest point, so that Then, scanning from the peak to the right: if the kernel value is 5% lower than the peak value for three consecutive sampling points, it is considered that the effective energy has decayed, and the tail is truncated at that position to obtain the final length. . To refine the impact of latency on the core The effective time range, also known as the support length.

[0173] Root cause localization system calculates nuclear energy As in formula (10):

[0174] (10)

[0175] The kernel is affected by the refined time delay. The details are as follows:

[0176]

[0177] Among them, the refinement of latency affects the core. It has been smoothed, denoised, and energy calibrated.

[0178] S450, Root Cause Localization System acquires cross-domain jitter hybrid core.

[0179] Refined nucleus This only represents the average propagation shape; in a cloud-edge collaborative hybrid architecture, the call path will traverse three network segments: cloud-internal, edge-internal, and cloud-edge, each with different jitter. By... The kernel is convolved with three types of delay distributions, then linearly fused according to the link proportion, and the tail is pruned using a cumulative energy threshold (e.g., 95% energy threshold) to obtain a cross-domain jitter hybrid kernel. .

[0180] First, we will introduce the latency distribution model of the link.

[0181] Root cause localization systems can extract three types of latency samples from distributed tracing logs. , , .in, This is a delay sample for the first link. This is a delay sample for the second link. This is a time delay sample for the third link. The number of time delay samples in each category is determined. For time delay samples with a sample size greater than or equal to a preset threshold, a 2–3 component Gaussian mixture fitting is used. For time delay samples with a sample size less than the preset threshold, a unimodal Gamma fitting is used. The resulting link time delay distribution model, i.e., the density function, is derived. As shown in formulas (11) to (12):

[0182] (11)

[0183] (12)

[0184] In this embodiment of the application, the root cause localization system will Convolution is performed with each of the three types of delay distributions, and the kernels of the three types of domains are obtained as shown in formula (13):

[0185] (13)

[0186] Next, the root cause localization system uses tracing logs to calculate the time percentage of the three types of links in the complete call path. ),in, This represents the percentage of time corresponding to the first link. This represents the second time segment corresponding to the second link. This represents the third time segment corresponding to the third link. For example... =0.3:0.3:0.4.

[0187] Cross-domain jitter hybrid core As in formula (14):

[0188] (14)

[0189] Then Uniform scaling enables .

[0190] Furthermore, the root cause localization system can also calculate the cumulative energy curve. As in formula (15):

[0191] (15)

[0192] Determine the minimum make and will The tail of the core is set to zero; the final support length for .in The sampling interval is denoted as .

[0193] Furthermore, in the embodiments of this application, the root cause localization system can also detect cross-domain jitter hybrid kernels. The specific storage format is as follows: .

[0194] In this embodiment of the application, the cross-domain jitter hybrid core The cross-correlation peak shape is preserved, and a long tail adaptively covers 95% of the propagation energy. In the embodiments of this application, a cross-domain dithering hybrid kernel is used. The path propagation delay can be correlated with the propagation model determined by the candidate domain of the light cone. This approach avoids missing the effects of slow cross-domain propagation and also controls the scale of subsequent computations.

[0195] Appendix Figure 5 This document provides a flowchart of a method for locating the root cause of anomalies based on a cross-domain jitter hybrid kernel, as illustrated in an embodiment of this application. Figure 5 As shown, this step includes S510~S540.

[0196] This application embodiment can also determine the candidate domain of the optical cone where runtime anomalies occur based on a cross-domain jitter hybrid model. A convolutional dictionary is constructed within the candidate domain. Sparse inversion is used to process the convolutional dictionary to obtain the root cause nodes of the anomalies. By compressing the candidate range from all upstream nodes to the local feasible region through the optical cone candidate domain, and then using sparse inversion to process the transitive kernel dictionary determined based on the optical cone candidate domain, the root cause nodes of the anomalies are obtained. This method avoids the candidate explosion problem caused by full graph traversal and improves search speed. Furthermore, by retaining upstream nodes that can influence the anomaly nodes within the quantile delay, the search space is compressed from the source, reducing inverse causal nodes and improving the accuracy of anomaly root cause localization.

[0197] S510, the root cause localization system determines the candidate optical cone domain corresponding to the downstream abnormal node.

[0198] Nodes in the optical cone candidate domain are upstream nodes whose shortest path quantile delay to downstream anomalous nodes does not exceed the span of the anomalous window. Upstream nodes in the optical cone candidate domain are considered to be within the range where they can influence downstream anomalous nodes.

[0199] This application provides a method for obtaining a candidate region of light cones. This method calculates the quantile propagation delay from an upstream node to an abnormal downstream node and then prunes it according to an adaptive threshold. The following is a detailed description in conjunction with the appendix. Figure 6 Explanation. (Attached) Figure 6A flowchart of a method for obtaining a candidate region of light cones is provided in this application embodiment. The method includes the following:

[0200] S6100, obtain the quantization propagation delay from any upstream node to the abnormal downstream node and the corresponding optical cone front time.

[0201] In the application embodiment, the quantile propagation delay from the upstream node to the abnormal downstream node refers to the maximum time required for abnormal information to travel from the upstream node to the downstream abnormal node under a preset confidence level (e.g., 95% confidence level). If the quantile propagation delay does not exceed the abnormal window span, it indicates that the abnormality of the upstream node is sufficient to affect the downstream abnormal node. If the quantile propagation delay exceeds the abnormal window span, it indicates that the upstream node is unreachable in terms of timing and does not belong to the optical cone candidate domain.

[0202] The embodiments of this application obtain the quantization propagation delay through the following steps:

[0203] Step B1: Enumerate topology paths.

[0204] Obtain the service dependency graph from the microservice registry or distributed tracing system of the microservice system. In this context, V represents a node, signifying an independent functional unit within the microservice system. This can be a service node in a cloud data center, a service instance on an edge node, or a key component such as middleware (e.g., RPC services or message queues) and databases. E represents an edge, signifying the call relationship or data transfer path between nodes.

[0205] Using the downstream abnormal node v as the sink, traverse the service dependency graph in reverse. By filtering out all upstream nodes u within the upstream k layers, we obtain all directed paths (referred to as topological paths) that do not contain loops and have a length less than or equal to k:

[0206] Where k is a positive integer, for example, k=6, and e is the calling edge.

[0207] Step B2: Extract edge-level quantile delay.

[0208] For each call edge e in the topology path, based on its link attributes, the delay quantiles at a specified confidence level are extracted from the pre-constructed delay distribution and used as the edge-level quantile delay for that call edge. For example, the delay distribution is... Confidence level is Then the edge-level quantile delay of the called edge is .

[0209] in, The probability of meeting the latency target can be adjusted as needed, for example... . For quantile calculation, for delayed distribution After sorting by probability, select the one with the highest cumulative probability. The latency value. This method, compared to a fixed latency threshold, can adapt to the latency fluctuation characteristics of the link.

[0210] In this embodiment, the pre-constructed latency distribution consists of three types of baseline latency distributions used by the cross-domain jitter hybrid core, including the baseline latency distribution for cross-type links, the baseline latency distribution for cloud-type links, and... The baseline delay distribution for the type of link. For example, the baseline delay distribution for a cross-type link. If 95% of the latency is ≤100ms, then the edge-level latency is 100ms. This means that there is a 95% chance that the propagation latency of the cross-domain call edge e will not exceed 100ms.

[0211] It should be noted that the link attribute of the calling edge is... .

[0212] Step B3: Accumulate the edge-level quantile delays to obtain the quantile propagation delays.

[0213] Since the propagation delays of each calling edge e within the same path p in the topological path are independent, the quantile propagation delay of path p is the sum of the edge-level quantile delays, as shown in formula (16):

[0214] (16)

[0215] in, Let P be the propagation delay of the path P.

[0216] It's understandable that the more network domains a path traverses, the more the long-tail characteristic accumulates. The long-tail characteristic refers to the fact that in cloud, edge, and cross-domain links, some requests experience significantly higher latency than the average link latency due to factors such as network congestion, route jumps, and fluctuations in edge node resources. Although the proportion of these "slow requests" is low, it is not negligible, resulting in a "tail" shape in the high-latency range of the latency distribution curve. Therefore, the path percentile latency provided in this application's embodiment more accurately reflects the propagation characteristics of cross-domain paths, which can improve the accuracy of causal reachability determination.

[0217] When an upstream node u triggers an anomaly, the latest time from which the anomaly affects the downstream anomalous node v is called the optical cone front time from upstream node u to downstream anomalous node v. For example, if path p is from upstream node u to downstream anomalous node v, and the starting point of the anomaly window is time t0, the optical cone front time along path p is:

[0218] Let t0 be the light cone leading edge time of path p. This time represents the latest time from when the upstream node u triggers the anomaly at time t0 to when it affects the downstream anomalous node v.

[0219] Furthermore, in this embodiment, each enumerated path can record its corresponding upstream node, path, path quantile propagation delay, and optical cone front time to obtain a candidate list. For example, the enumerated path is the path from upstream node u to downstream abnormal node v, and the corresponding quantile propagation delay is... The time of the light cone's leading edge is The triplet is obtained as Add the triple to the candidate list.

[0220] It is understandable that the candidate list includes each enumerated path, as well as the upstream node, path, path quantile propagation delay, and optical cone front time corresponding to each enumerated path.

[0221] S6110 uses adaptive pruning with quantile thresholds to obtain the candidate domain of the light cone.

[0222] For the upstream paths in the topology, candidate optical cone regions are obtained by eliminating temporally impossible paths, merging redundant paths, and removing extreme noise paths. The specific acquisition method includes the following steps:

[0223] Step C1: Remove upstream nodes corresponding to impossible time-series paths.

[0224] Specifically, if the optical cone leading edge time of a certain path is later than the termination time t1 of the abnormal window, for example, the optical cone leading edge time of path P... The path is a temporally impossible path, so the path P is removed from the candidate list.

[0225] Step C2: Merge the shortest paths.

[0226] Specifically, for the same upstream node u, there may be multiple paths to reach the downstream abnormal node v. In this application, only the path corresponding to the shortest quantile delay is retained in the embodiment, denoted as:

[0227]

[0228] This represents the shortest quantile delay corresponding to the upstream node u.

[0229] In this way, each candidate node retains only one representative path. This approach can avoid multiple paths from the same upstream node participating in subsequent calculations, reduce redundancy, and also help reduce the complexity of constructing the subsequent kernel dictionary.

[0230] Step C3: Remove extreme trailing nodes.

[0231] An extreme tail node refers to an upstream node whose quantile propagation delay to the downstream abnormal node v significantly deviates from the normal delay distribution of most paths, and belongs to a low-probability propagation delay anomaly. In the embodiments of this application, the quantile propagation delay from the upstream node u to the downstream abnormal node v is greater than or equal to the target threshold, and the upstream node u is an extreme tail node.

[0232] The target threshold and the average quantile propagation delay of other candidate nodes after excluding upstream node u and standard deviation There is a positive correlation, for example, the target threshold is If the propagation delay of the upstream node u is... satisfy:

[0233]

[0234] If the upstream node is considered an extreme trailing node, it will be removed from the candidate list. Removing extreme trailing nodes from the candidate list can prevent them from affecting subsequent root cause localization and further improve localization accuracy.

[0235] Step C4: Obtain the candidate region of the light cone.

[0236] The candidate list after removing upstream nodes corresponding to impossible temporal paths and extreme trailing nodes constitutes the candidate region of the light cone. .

[0237] It is understandable that this optical cone candidate domain reduces the solution complexity of candidate root cause anomaly localization while avoiding the missed detection of potential root causes that are time-reachable.

[0238] S520: Construct a convolution dictionary and solve for the target activation vector using sparse inversion.

[0239] The target activation vector includes only the upstream nodes essential for reconstructing the residual signals of downstream anomalous nodes, along with the weakest anomalous activation intensity of each upstream node. It can be understood that the target activation vector minimizes the number of upstream nodes, and by utilizing the weakest anomalous activation intensity of each upstream node, the anomalous image of a single upstream node can be avoided. Using the target activation vector, the residual signals of downstream anomalous nodes can be reconstructed to the greatest extent possible through propagation kernel convolution stacking.

[0240] This application provides a method for obtaining a target activation vector, including the following steps:

[0241] Step D1: Align the propagation kernel.

[0242] Cross-domain jitter hybrid kernel of any upstream node u This describes the change in the intensity of the unit anomalous excitation at node u over time τ. As a shock response model without a time reference, it is unclear how long it takes for the excitation to reach the downstream anomalous node v after it originates from the upstream node u.

[0243] The path propagation delay from upstream node u to downstream anomalous node v in the candidate region of the optical cone is less than or equal to the span of the anomalous window. Without alignment, the cross-domain jittering hybrid kernel... The peak value may occur earlier than the path propagation delay from upstream node u to downstream abnormal node v, or later than the path propagation delay from upstream node u to downstream abnormal node v, i.e., exceeding the allowed time of the optical cone, which leads to misjudgment of the excitation time in subsequent sparse inversion and destroys causal consistency.

[0244] In view of this, the embodiments of this application first align cross-domain jitter hybrid cores. Time is used to avoid a mismatch between the propagation kernel time and the actual propagation delay. The specific alignment method is as follows:

[0245] Cross-domain jitter hybrid kernel for upstream node u in the candidate domain of the optical cone The overall propagation delay of its shortest path is shifted to the right. 1 sampling point, to obtain the alignment propagation kernel As shown in formula (17):

[0246] (17)

[0247] This is equivalent to delaying the abnormal excitation effect of the upstream node u. Then it takes effect again. For example, Cross-domain jitter hybrid core Peak at =5ms, and the peak appears at 15ms after alignment. That is, the upstream node u triggers the anomaly at t=0 and is most affected at t=15ms, which is consistent with the definition of light cone.

[0248] Step D2: Obtain multiple shift cores triggered by each upstream node at different times within the abnormal window.

[0249] This application embodiment solves the problem that the propagation kernel cannot cover multiple time-lapse stimuli by expanding the alignment propagation kernel of a single upstream node u into multiple shift kernels triggered at different times within the abnormal window.

[0250] If the alignment propagation kernel of upstream node u The core length is The abnormal window is By sliding along the abnormal window, you can obtain... The dictionary atoms are shown in formulas (18) to (19):

[0251] (18)

[0252] (19)

[0253] in, For the sampling interval, each dictionary atom represents the upstream node u at time. The timing response is triggered only once. This method ensures that the effective effect of each shift core falls entirely within the anomaly window and does not exceed the window boundaries.

[0254] Step D3: Construct the convolution dictionary.

[0255] To obtain the upstream node u, at time... When an anomaly is triggered, the superimposed responses can best fit the residual of the downstream anomaly node v, thus matching the upstream node u at time t. shift nuclei Flatten into column vectors This transforms the time-series propagation problem into a linear algebra problem that can be solved efficiently.

[0256] Stack the column vectors of the same upstream node u in ascending time order, and concatenate them vertically between different upstream nodes to obtain the convolution dictionary D as follows:

[0257]

[0258] in, As an observation dimension, , It is understandable that each column in the convolutional dictionary D... Represents node u at time t. The theoretical response of a unit shock to the residual at the downstream anomalous node v.

[0259] Furthermore, the convolution dictionary D and its columns can be indexed to... The mapping is then used. Then, the convolution dictionary D and the causal residual are used... Forming linear equations:

[0260]

[0261] The excitation coefficient is used to locate the smallest and most reliable root cause node and trigger time of the anomaly. In the embodiments of this application, it can be obtained through sparse regularization optimization. The value of .

[0262] In the embodiments of this application, the target residual refers to the standardized residual sequence of downstream abnormal nodes within the abnormal window, which is the abnormal signal after normal fluctuations have been removed.

[0263] Step D4: Obtain the target incentive vector based on the sparse incentive optimization model.

[0264] The following uses the abnormal window as an example. The sampling interval for the indicator detection values ​​is [missing information]. The residual effect is , Let's take an example to illustrate. From the linear equation composed of the convolution dictionary D and the effect residual r, we obtain the sparsest and non-negative activation coefficients. This minimizes the convolution reconstruction error. The specific method for obtaining this is as follows:

[0265] The target activation vector includes only the upstream nodes essential for reconstructing the residual signals of downstream anomalous nodes, along with the weakest anomalous activation intensity of each upstream node. It can be understood that the target activation vector minimizes the number of upstream nodes, and by utilizing the weakest anomalous activation intensity of each upstream node, the anomalous image of a single upstream node can be avoided. The target activation vector can be superimposed with a propagation kernel convolution to maximize the reconstruction of the residual effect of downstream anomalous nodes. In this embodiment, each element of the target activation vector is greater than or equal to 0, and the vector strength value of the residual vector satisfying the condition that the reconstructed residual and the effect residual are minimized; the reconstructed residual is the product of the target activation vector and the convolution dictionary.

[0266] In one example, the upstream nodes are multiple upstream nodes within the optical cone candidate domain of the downstream anomalous nodes. Nodes within the optical cone candidate domain are upstream nodes whose path propagation delay to the downstream anomalous node does not exceed the span of the anomalous window. The root cause localization system can utilize a pre-constructed loss function, combined with the effect residual and convolution dictionary, to determine the target activation vector. The target activation vector is a global activation vector, including sub-activation vectors of multiple upstream nodes. The loss function includes a reconstruction error term, an L1 coefficient regularization term, and a node-level group sparse regularization term. The reconstruction error term is the squared L2 norm of the difference vector between the effect residual and the residual to be reconstructed. The residual to be reconstructed is the product of the global activation vector to be solved and the convolution dictionary. The L1 sparse regularization term is the product of the L1 norm of the global activation vector to be solved and the first weight. The node-level group sparse regularization term is the product of the sum of the L2 norms of the sub-activation vectors of multiple upstream nodes and the second weight.

[0267] For example, if the effect residual is The pre-defined functions are as follows:

[0268]

[0269] Among them, the reconstruction error term Used to make the selected excitation superimposed fit as closely as possible to the observed residual. Sparse regularization At the column (dictionary atom) level, coefficient sparsity is encouraged, prompting only a small number of dictionary atoms to participate in the reconstruction. First weight. Node group sparsity regularization. :in Treat all excitation coefficients of the same upstream node u as a group; if the entire group To reduce to zero means to remove the node entirely – further compressing the root cause set at the node level. This is the second weight. Non-negativity constraint. This indicates that abnormal excitation can only be amplified in the positive direction, and there is no reverse cancellation.

[0270] For each upstream node u, based on the target activation vector Obtain the overall incentive intensity and the moment when the strongest incentive occurs. The specific method for obtaining this information is as follows:

[0271]

[0272]

[0273] in, The overall excitation intensity representing node u can be considered as a confidence score; This represents the moment when the strongest incentive occurs at that node.

[0274] Sort the overall incentive intensity according to its intensity, for example, from low to high or from high to low, to obtain a root cause priority list. For example, the root cause priority list sorted from high to low is as follows:

[0275]

[0276] Based on the root cause priority list, the ranking of root cause nodes and their corresponding incentive times and intensities can be obtained, thus yielding the target incentive vector. Therefore, the root cause localization system completes the entire chain of inference, from the abnormal residual to which node and when the current fault is most likely to be triggered.

[0277] S530 verifies the target excitation vector through replay verification.

[0278] To ensure that the sparse inversion results have sufficient explanatory power for the effect residuals, embodiments of this application also include an exception window. Three quantifiable metrics are calculated: reconstruction error, peak alignment, and energy coverage. Based on these metrics, the sparse inversion results are assessed to determine whether the interpretation is adequate.

[0279] In practical implementation, the reconstruction error, peak alignment, and energy coverage corresponding to the target excitation vector can be determined. If the reconstruction error is greater than or equal to the first threshold, the peak alignment is greater than or equal to the second threshold, and the energy coverage is greater than or equal to the preset third threshold, the target excitation vector is considered to have passed verification. If any indicator fails to meet the standard, the restrictions can be relaxed step by step, for example, by expanding the light cone quantile, lowering the regularization weight, and re-executing the sparse excitation optimization model to obtain the target excitation vector until it passes or reaches the maximum iteration. The embodiments of this application can use comprehensive indicators such as reconstruction error, peak alignment, and energy coverage for evaluation, which can maintain lower error, higher peak synchronization, and more complete energy restoration when reconstructing the residual waveform of the target node. This makes the characterization of abnormal patterns more realistic and the root cause localization results more accurate.

[0280] S540 outputs root cause ranking and confidence level.

[0281] For optical cone candidate domain any upstream node in to comprehensively incentivize its intensity Linear scaling to the 0–1 interval yields the confidence score.

[0282]

[0283] If there is only one node, then set =1. This represents the minimum combined excitation intensity of multiple upstream nodes in the candidate domain of the optical cone. It represents the maximum value of the combined excitation intensity of multiple upstream nodes in the candidate domain of the light cone.

[0284] according to Sort the root causes from highest to lowest confidence score, and output the root cause nodes corresponding to the top m confidence scores, along with their rankings and the confidence scores for each node. m is a positive integer, for example, m=5. Therefore, based on the output root cause rankings and confidence scores, the root cause of the anomaly can be located.

[0285] In summary, the embodiments of this application can be evaluated using comprehensive indicators such as reconstruction error, peak alignment, and energy coverage. When reconstructing the residual waveform of the target node, it can maintain low error, high peak synchronization, and relatively complete energy restoration, resulting in more realistic characterization of abnormal patterns and more accurate root cause localization. The cross-domain jitter hybrid core integrates the multimodal latency distribution of the three links—intra-cloud, intra-edge, and cloud-edge interconnection—into the propagation model, avoiding long-tail loss caused by a single Gaussian assumption. This design effectively covers complex network jitter scenarios, reduces the risk of mis-pruning or miscalculation of long-tail paths, and improves robustness in cloud-edge collaborative environments. The entire inference link relies only on FFT and linear algebra operations, without relying on GPUs or deep learning inference frameworks. In typical deployment environments, a complete root cause analysis can be completed within seconds or even sub-seconds, meeting the timeliness requirements of online operation and maintenance while significantly reducing hardware and operation and maintenance investment.

[0286] Regarding the appendix Figure 1 ~Appendix Figure 6 In addition to the method for determining the root cause of anomalies shown in the present application, this application also provides an apparatus for determining the root cause of anomalies.

[0287] Appendix Figure 7 This is a schematic diagram of an anomaly root cause determination device provided in an embodiment of this application. The device 700 includes:

[0288] The path determination unit 701 is used to determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has an abnormality; the abnormal window of the upstream node and the downstream abnormal node is the same.

[0289] The delay determination unit 702 is configured to perform the following for each of the call paths: obtain the edge-level quantile delay of each call edge in the call path; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the start node to the end node of the call edge; determine the path propagation delay of the call path based on the edge-level quantile delay of each call edge; the path propagation delay of the call path indicates the longest time required for the abnormal information to be transmitted along the call path to the downstream abnormal node;

[0290] The root cause determination unit 703 is used to determine the root cause of the anomaly based on the path propagation delay of each of the call paths.

[0291] Optionally, determining the root cause of the exception based on the path propagation delay of each of the call paths includes:

[0292] Determine the optical cone leading edge time corresponding to the path propagation delay of each of the aforementioned call paths; the optical cone leading edge time of the call path indicates the sum of the path propagation delay of that call path and the start time of the anomaly window;

[0293] Based on the leading edge time of the optical cone for each of the calling paths, a candidate region of optical cones is determined; the optical cones in the candidate region are the upstream nodes whose leading edge time is less than or equal to the termination time of the abnormal window.

[0294] The root cause of the anomaly is determined based on the candidate region of the light cone.

[0295] Optionally, if the candidate optical cone region is an initial candidate optical cone region, after determining the candidate optical cone region based on the optical cone leading edge time of each calling path, the method further includes:

[0296] Based on the path propagation delay of each of the aforementioned call paths, determine the average delay and the delay standard;

[0297] A trailing baseline value is determined based on the mean delay and the delay standard; the trailing baseline value is correlated with the mean delay and the standard deviation.

[0298] Extreme upstream nodes are removed from the initial candidate light cone domain to obtain the target candidate light cone domain; the light cones in the target candidate light cone domain are upstream nodes whose path propagation delay of the calling path is greater than the tail reference value.

[0299] The step of determining the root cause of the anomaly based on the candidate optical cone region includes:

[0300] The root cause of the anomaly is determined based on the target light cone candidate domain.

[0301] Optionally, the method further includes:

[0302] If an upstream node has at least one of the aforementioned call paths, obtain the target call path from the upstream node to the downstream abnormal node; the target call path indicates the call path with the shortest path propagation delay among at least one of the aforementioned call paths of the upstream node;

[0303] Based on the path propagation delay of each of the aforementioned call paths, determine the average delay and delay standard, including:

[0304] The mean delay and the delay standard are determined based on the path propagation delay of each target call path;

[0305] The step of removing extreme upstream nodes from the initial optical cone candidate domain to obtain the target optical cone candidate domain includes:

[0306] For each upstream node, if the path propagation delay of the target call path corresponding to that upstream node is greater than the tailing benchmark value, the upstream node is determined to be the extreme upstream node.

[0307] The target optical cone candidate domain is obtained by removing the extreme upstream node from the initial optical cone candidate domain.

[0308] Optionally, the trailing reference value is linearly positively correlated with the sum of the mean delay and three times the standard deviation.

[0309] Optionally, determining all target call paths for downstream abnormal nodes in the microservice system includes:

[0310] Using the downstream abnormal node as the sink, the service dependency graph of the microservice system is traversed in reverse. All upstream nodes within k layers upstream of the downstream abnormal node are taken as target upstream nodes to obtain the target call path. The target upstream path is the path from the target upstream node to the downstream abnormal node. k is a positive integer.

[0311] Optionally, each call edge in the service dependency graph is bound to a cross-domain jitter hybrid kernel. The cross-domain jitter hybrid kernel is obtained by convolving the latency propagation kernel of the upstream node with the latency distribution models of the first link, the second link, and the third link, respectively, weighting and fusing them according to the link time proportion, and then pruning them by a cumulative energy threshold. The first link is a link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

[0312] Optionally, obtaining the edge-level quantile delay of each call edge in the target call path includes:

[0313] For each call edge, determine its link attribute; take a preset signal level on the delay distribution model corresponding to the link attribute of the call edge to obtain the edge-level quantile delay of the call edge;

[0314] The link attributes of the calling edge include one of the first link, the second link, and the third link; the first link is a link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

[0315] Alternatively, the latency distribution model of the link can be obtained in the following ways:

[0316] From the distributed tracing logs, obtain the latency samples of the first link, the second link, and the third link;

[0317] Determine whether the number of delay samples of the first link is greater than or equal to a preset threshold. If yes, fit the delay samples of the first link with a Gaussian mixture model to obtain the delay distribution model of the first link. If no, fit the delay samples of the first link with a unimodal Gamma distribution to obtain the delay distribution model of the first link.

[0318] Determine whether the number of delay samples of the second link is greater than or equal to a preset threshold. If yes, fit the delay samples of the second link with a Gaussian mixture model to obtain the delay distribution model of the second link. If no, fit the delay samples of the second link with a unimodal Gamma distribution to obtain the delay distribution model of the second link.

[0319] Determine whether the number of delay samples of the third link is greater than or equal to a preset threshold. If yes, perform Gaussian mixture model fitting on the delay samples of the third link to obtain the delay distribution model of the third link. If no, use a unimodal Gamma distribution to fit the delay samples of the third link to obtain the delay distribution model of the third link.

[0320] This application provides an anomaly root cause determination device. This device can clarify the temporal reachability of upstream nodes to downstream anomaly nodes through quantitative calculation of edge-level quantification delay and path propagation delay, reducing inverse causality at the physical propagation level. Therefore, compared to sliding window correlation or threshold alarms, using Pearson correlation coefficient, Granger causality, or rule bases to directly lock the most relevant indicators, this anomaly root cause localization method can reduce the probability of misjudging downstream symptoms as root causes and improve the accuracy of anomaly root cause localization.

[0321] According to the method provided in the embodiments of this application, this application also provides a chip system, which includes one or more processors for calling and executing instructions stored in memory, thereby causing the method described in the embodiments of this application to be executed. The chip system may be composed of chips or may include chips and other discrete devices.

[0322] The chip system may include input circuits or interfaces for transmitting information or data, and output circuits or interfaces for receiving information or data.

[0323] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0324] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to execute the various steps or processes executed by the network device or terminal device in any of the foregoing method embodiments.

[0325] The computer-readable storage medium may be the aforementioned volatile memory or non-volatile memory, or it may include both volatile memory and non-volatile memory.

[0326] In the embodiments of this application, the terms and English abbreviations are exemplary examples given for ease of description and should not be construed as limiting the application in any way. This application does not preclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

[0327] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated.

[0328] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

Claims

1. A method for determining the root cause of anomalies, characterized in that, Applied to a microservice system, wherein the microservice system has a cloud-edge collaborative hybrid architecture, the cloud-edge collaborative hybrid architecture includes a cloud data center and edge nodes, and the method includes: Determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has experienced an exception; the exception window of the upstream node and the downstream abnormal node is the same; For each of the aforementioned call paths, the following steps are performed: The edge-level quantile delay of each call edge in the call path is obtained; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the starting node to the ending node of the call edge; based on the edge-level quantile delay of each call edge, the path propagation delay of the call path is determined; the path propagation delay of the call path indicates the maximum time required for the exception information to propagate along the call path to the downstream exception node. The root cause of the exception is determined based on the path propagation delay of each of the aforementioned call paths.

2. The method according to claim 1, characterized in that, The step of determining the root cause of the exception based on the path propagation delay of each of the call paths includes: Determine the optical cone leading edge time corresponding to the path propagation delay of each of the aforementioned call paths; the optical cone leading edge time of the call path indicates the sum of the path propagation delay of that call path and the start time of the anomaly window; Based on the leading edge time of the optical cone for each of the calling paths, a candidate region of optical cones is determined; the optical cones in the candidate region are the upstream nodes whose leading edge time is less than or equal to the termination time of the abnormal window. The root cause of the anomaly is determined based on the candidate region of the light cone.

3. The method according to claim 2, characterized in that, If the candidate optical cone region is the initial candidate optical cone region, after determining the candidate optical cone region based on the optical cone leading edge time of each calling path, the method further includes: Based on the path propagation delay of each of the aforementioned call paths, determine the mean delay and the standard deviation of the delay; A trailing baseline value is determined based on the mean delay and the standard deviation of the delay; the trailing baseline value is correlated with the mean delay and the standard deviation of the delay. Extreme upstream nodes are removed from the initial candidate light cone domain to obtain the target candidate light cone domain; the light cones in the target candidate light cone domain are upstream nodes whose path propagation delay of the calling path is greater than the tail reference value. The step of determining the root cause of the anomaly based on the candidate optical cone region includes: The root cause of the anomaly is determined based on the target light cone candidate domain.

4. The method according to claim 3, characterized in that, The method further includes: If an upstream node has at least one of the aforementioned call paths, obtain the target call path from the upstream node to the downstream abnormal node; the target call path indicates the call path with the shortest path propagation delay among at least one of the aforementioned call paths of the upstream node; The step of determining the mean delay and standard deviation of delay based on the path propagation delay of each of the call paths includes: The mean delay and standard deviation of the delay are determined based on the path propagation delay of each target call path; The step of removing extreme upstream nodes from the initial optical cone candidate domain to obtain the target optical cone candidate domain includes: For each upstream node, if the path propagation delay of the target call path corresponding to that upstream node is greater than the tailing benchmark value, the upstream node is determined to be the extreme upstream node. The target optical cone candidate domain is obtained by removing the extreme upstream node from the initial optical cone candidate domain.

5. The method according to claim 3, characterized in that, The trailing baseline value is linearly positively correlated with the sum of the mean delay and three times the standard deviation of the delay.

6. The method according to claim 1, characterized in that, Determining all target call paths for downstream abnormal nodes in the microservice system includes: Using the downstream abnormal node as the sink, the service dependency graph of the microservice system is traversed in reverse. All upstream nodes within k layers upstream of the downstream abnormal node are taken as target upstream nodes to obtain the target call path. The target upstream path is the path from the target upstream node to the downstream abnormal node. k is a positive integer.

7. The method according to claim 6, characterized in that, Each call edge of the service dependency graph is bound to a cross-domain jitter hybrid kernel. The cross-domain jitter hybrid kernel is a propagation kernel obtained by convolving the latency propagation kernel of the upstream node with the latency distribution models of the first link, the second link, and the third link respectively, weighting and fusing them according to the link time proportion, and then pruning them by the cumulative energy threshold. The first link is an internal link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

8. The method according to claim 1, characterized in that, The step of obtaining the edge-level quantile delay of each call edge in the target call path includes: For each call edge, determine its link attribute; take a preset signal level on the delay distribution model corresponding to the link attribute of the call edge to obtain the edge-level quantile delay of the call edge; The link attributes of the calling edge include one of the first link, the second link, and the third link; the first link is a link within the cloud data center, the second link is a link between the edge nodes, and the third link is a cross-domain link between the cloud data center and the edge nodes.

9. The method according to claim 8, characterized in that, The method further includes: From the distributed tracing logs, obtain the latency samples of the first link, the second link, and the third link; Determine whether the number of delay samples of the first link is greater than or equal to a preset threshold. If yes, fit the delay samples of the first link with a Gaussian mixture model to obtain the delay distribution model of the first link. If no, fit the delay samples of the first link with a unimodal Gamma distribution to obtain the delay distribution model of the first link. Determine whether the number of delay samples of the second link is greater than or equal to a preset threshold. If yes, fit the delay samples of the second link with a Gaussian mixture model to obtain the delay distribution model of the second link. If no, fit the delay samples of the second link with a unimodal Gamma distribution to obtain the delay distribution model of the second link. Determine whether the number of delay samples of the third link is greater than or equal to a preset threshold. If yes, perform Gaussian mixture model fitting on the delay samples of the third link to obtain the delay distribution model of the third link. If no, use a unimodal Gamma distribution to fit the delay samples of the third link to obtain the delay distribution model of the third link.

10. An apparatus for determining the root cause of an anomaly, characterized in that, Applied to a microservice system, wherein the microservice system has a cloud-edge collaborative hybrid architecture, the cloud-edge collaborative hybrid architecture includes a cloud data center and edge nodes, and the device includes: A path determination unit is used to determine all call paths of downstream abnormal nodes in the microservice system; the call path indicates the call path from the upstream node of the downstream abnormal node to the downstream abnormal node; the downstream abnormal node is the microservice node in the microservice system that has experienced an exception; the exception window of the upstream node and the downstream abnormal node is the same. The delay determination unit is configured to perform the following for each of the call paths: obtain the edge-level quantile delay of each call edge in the call path; the edge-level quantile delay of the call edge indicates the maximum propagation delay from the start node to the end node of the call edge; determine the path propagation delay of the call path based on the edge-level quantile delay of each call edge; the path propagation delay of the call path indicates the maximum time required for the exception information to be transmitted along the call path to the downstream exception node; The root cause determination unit is used to determine the root cause of the anomaly based on the path propagation delay of each of the call paths.

Images