Cloud service fault root cause positioning method based on dynamic baseline modeling

Abstract

The invention relates to the technical field of fault root cause positioning, in particular to a cloud service fault root cause positioning method based on dynamic baseline modeling, and the method comprises the steps: collecting multi-source indexes in real time, and carrying out the time-space alignment; constructing a heterogeneous propagation graph containing service instances and calling behavior nodes; calculating a request amount and an error rate operator, and realizing dynamic weight distribution by using an information entropy algorithm; generating a business index confidence interval by using a dynamic baseline algorithm, and positioning a damaged starting node; and constructing a transition probability matrix containing a reverse edge and a self-loop edge, executing Markov chain iterative walk, and outputting a root cause performance index. According to the method, logic disassembly of a fine-grained calling relation is realized through a heterogeneous propagation graph, a self-adaptive weight adjustment mechanism for service fluctuation is established by using an information entropy algorithm, and the method can adapt to instantaneous topology migration in a cloud native environment in cooperation with a reverse edge and self-loop edge design and a random walk technology, so that the service performance of the system is improved. And locking fault root causes and outputting key performance indexes.

Description

Technical Field

[0001] This invention relates to the field of root cause analysis technology, specifically a cloud service fault root cause analysis method based on dynamic baseline modeling. Background Technology

[0002] With the development of cloud computing technology, enterprise applications are accelerating their evolution towards microservices, containerization, and cloud-native architectures. In cloud-native environments, application systems consist of hundreds or thousands of highly decoupled microservices, running on dynamically scalable containers and underlying physical resources. This high degree of flexibility and complex dependencies bring unprecedented challenges to fault detection and root cause localization in distributed systems.

[0003] In cloud-native architectures, while the high-level structure of microservices and the dynamic scheduling of underlying resources improve deployment efficiency, they also significantly increase the complexity of fault localization. Current localization methods have significant limitations in dealing with anomaly monitoring: First, the deep nested call relationships between microservices cause even small performance deviations in underlying components (such as Pods or host machines) to have a dramatic ripple effect, leading to the rapid spread of faults along the call chain and triggering a large-scale alarm storm; second, traditional localization algorithms often struggle to effectively distinguish between the initial root cause and affected victim nodes in high-dimensional observation data, especially when the anomaly characteristics of victim nodes are more significant than those of the root cause node, making misjudgments highly likely; third, the frequent container scheduling and load balancing strategies unique to cloud-native environments cause the routing relationships between services to change dynamically in real time, and existing tracing methods, unable to capture such instantaneous topology shifts in real time, often lock onto incorrect propagation paths. To address this, a cloud service fault root cause localization method based on dynamic baseline modeling is proposed. Summary of the Invention

[0004] The purpose of this invention is to provide a cloud service fault root cause localization method based on dynamic baseline modeling to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A cloud service failure root cause localization method based on dynamic baseline modeling includes: Real-time collection of business metrics, performance metrics, and link tracing data streams, and spatiotemporal alignment processing based on a unified timestamp; Based on the link tracing data stream, a heterogeneous propagation graph is constructed, which includes cloud service instance nodes and call behavior nodes located between cloud service instance nodes; The request volume operator and interface error rate operator between adjacent nodes are extracted from the business metrics and link tracing data stream, and normalized to obtain dynamic edge weights; the confidence interval of the business metric stream is calculated using the dynamic baseline algorithm; when the business metric exceeds the confidence interval, the business metric is abnormally associated with the damaged starting node in the heterogeneous propagation graph according to the link tracing data stream. In the heterogeneous propagation graph, reverse edges and self-loop edges are constructed for each node. A transition probability matrix is ​​constructed based on the dynamic edges, reverse edges, and self-loop edges. Iterative walk calculations are performed starting from the damaged initiating node to generate the fault root cause service node and associated root cause performance indicators.

[0006] Preferably, the business metrics include the total number of requests within the time window, average response latency, service success rate, and custom transaction processing status codes at the business level. The performance metrics include CPU utilization, memory usage, disk I / O throughput, network bandwidth consumption of the host on which the cloud service is located, as well as JVM stack information, garbage collection frequency and time during cloud service operation; The link tracing data stream contains distributed call logs with TraceID and SpanID.

[0007] Preferably, the specific process of constructing the heterogeneous propagation graph is to parse the upstream and downstream relationships in the link tracing data stream, map each type of independent cloud service instance to a cloud service instance node, and synchronously load the performance indicators as static attributes of the cloud service instance node. Between two cloud service instance nodes with a calling relationship, at least one calling behavior node is inserted according to the interface path and operation type in the link tracing data stream; the calling behavior node is used to carry the request volume, error rate and time consumption distribution data of a specific interface; Based on the unified timestamp, a directed connection is established between the cloud service instance node and the corresponding call behavior node.

[0008] Preferably, the request volume operator calculation process involves: counting the number of times a specific call behavior is triggered between two adjacent cloud service instance nodes within the current sampling period from the link tracing data stream, and recording it as the real-time request volume; obtaining the average number of requests for this call behavior within the historical baseline window from the business metric stream, and recording it as the baseline request volume; and obtaining the request volume operator based on the real-time request volume and the baseline request volume. The interface error rate operator calculation process involves calculating the ratio of the number of failed calls between adjacent nodes to the total number of requests within the current sampling period based on the status code markers in the link tracing data stream, thereby obtaining the real-time error rate; and then obtaining the interface error rate operator based on the real-time error rate. Dynamic edge weights are obtained by using the request volume operator and the interface error rate operator, and by performing normalization processing.

[0009] Preferably, the specific calculation process of the dynamic edge weight is to calculate the contribution feature values ​​of the request volume operator and the interface error rate operator in the current sampling window, respectively; Based on the contribution feature value, the real-time weight allocation ratio of the request volume operator and the interface error rate operator is calculated using the information entropy algorithm; the request volume operator, the interface error rate operator and their respective real-time weight allocation ratios are weighted and summed to obtain the original association score of each edge. For any source node in the heterogeneous propagation graph, obtain the original association scores of all outgoing edges, perform proportion normalization processing, and convert the scores into probability distribution values ​​as the dynamic edge weights.

[0010] Preferably, the dynamic baseline algorithm obtains the historical sequence of business indicators within a preset sliding window before the current time point, calculates the probability density distribution, and identifies the central trend characteristics and the degree of fluctuation dispersion; the degree of fluctuation dispersion serves as the corresponding boundary offset, and the upper and lower boundary points are automatically determined in combination with the central trend characteristics to generate a confidence interval; Monitor the degree to which real-time business metrics deviate from the confidence interval. When a business metric exceeds the confidence interval, retrieve the entry call chain associated with the abnormal timestamp through the link tracing data stream, and determine the cloud service instance node at the forefront of the call chain in the logical topology as the damaged starting node.

[0011] Preferably, in the heterogeneous propagation graph, a reverse edge and a self-loop edge are established for each node; the weight of the reverse edge is mapped to the weight of the corresponding forward edge; the weight of the self-loop edge is the maximum value of the weights of all incoming edges of the node; For each node, the weights of the forward, reverse, and self-loop edges are normalized as a whole to construct a transition probability matrix; with the damaged starting node as the initial probability vector, a Markov chain iterative walk is performed in combination with the transition probability matrix to output the root cause performance index associated with the fault root cause service node.

[0012] Preferably, the specific process of performing the Markov chain iterative walk in conjunction with the transition probability matrix is ​​as follows: The initial probability is injected starting from the damaged starting node, and the probability is distributed to adjacent nodes along the forward, reverse and self-loop edges according to the transition probability matrix; the change of the probability distribution of nodes in the whole graph is monitored during the iteration process, and the walk stops when the number of iterations reaches the iteration limit; Extract the cloud service instance node with the highest probability value in the stationary distribution as the root cause service node of the fault, and output the root cause performance index associated with the root cause service node in descending order according to the degree of deviation of the performance index from the dynamic baseline.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a heterogeneous propagation graph containing cloud service instance nodes and call behavior nodes based on the link tracing data flow, achieving logical decomposition of fine-grained call relationships between microservices. Compared to traditional service-level single-layer topologies, this heterogeneous modeling approach can independently carry specific interface paths, operation types, and time-consuming distribution data on call behavior nodes. When underlying resources experience performance deviations and trigger chain alarms, it can logically distinguish which specific type of call behavior induced the business anomaly, thereby clearly defining the impact boundary of the fault in complex nested call relationships.

[0014] 2. This invention establishes an adaptive monitoring system for business fluctuations by combining a dynamic baseline algorithm with a real-time weight allocation mechanism based on information entropy. It utilizes non-parametric statistical methods to perform quantile regression analysis on historical business indicators, and the automatically generated confidence intervals can be dynamically adjusted with the time window, thereby more objectively identifying abnormal deviations in business indicators. In conjunction with the information entropy algorithm, it quantifies the contribution of the request volume operator and the error rate operator in real time, which can automatically adjust the correlation scores of each edge, effectively solving the problem of location interference caused by the significant abnormal characteristics of the victim node during fault propagation.

[0015] 3. This invention introduces reverse edges and self-loop edges into a heterogeneous propagation graph and combines iterative walk technology using Markov chains to construct a complete fault tracing calculation model. The weight mapping of reverse edges and the correlation design of the weights of the incoming edges of self-loop edges ensure that the random walk process can both trace back upstream along the call chain to find the root cause and retain its own performance anomaly characteristics through node self-loops, avoiding the probability from being lost too quickly in the topology. This stable distribution calculation method based on the transition probability matrix can adapt to the instantaneous topology shift caused by frequent container scheduling in cloud-native environments and output a sequence of performance indicators that are highly correlated with the root cause of the fault. Attached Figure Description

[0016] Figure 1 A flowchart illustrating a cloud service fault root cause localization method based on dynamic baseline modeling. Figure 2 A schematic diagram illustrating the process of constructing a heterogeneous propagation graph; Figure 3 This is a schematic diagram of the iterative walk and root cause localization process of a Markov chain. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Please see Figure 1 This invention provides a cloud service fault root cause localization method based on dynamic baseline modeling, the technical solution of which is as follows: A cloud service failure root cause localization method based on dynamic baseline modeling includes: Real-time collection of business metrics, performance metrics, and link tracing data streams, and spatiotemporal alignment processing based on a unified timestamp; Based on the link tracing data stream, a heterogeneous propagation graph is constructed, which includes cloud service instance nodes and call behavior nodes located between cloud service instance nodes; The request volume operator and interface error rate operator between adjacent nodes are extracted from the business metrics and link tracing data stream, and normalized to obtain dynamic edge weights; the confidence interval of the business metric stream is calculated using the dynamic baseline algorithm; when the business metric exceeds the confidence interval, the business metric is abnormally associated with the damaged starting node in the heterogeneous propagation graph according to the link tracing data stream. In the heterogeneous propagation graph, reverse edges and self-loop edges are constructed for each node. A transition probability matrix is ​​constructed based on the dynamic edges, reverse edges, and self-loop edges. Iterative walk calculations are performed starting from the damaged initiating node to generate the fault root cause service node and associated root cause performance indicators.

[0019] The business metrics include the total number of requests within the time window, average response latency, service success rate, and custom transaction processing status codes at the business level. The performance metrics include CPU utilization, memory usage, disk I / O throughput, network bandwidth consumption of the host on which the cloud service is located, as well as JVM stack information, garbage collection frequency and time during cloud service operation; The link tracing data stream contains distributed call logs with TraceID and SpanID.

[0020] Specifically, the process of acquiring business metrics involves embedding tracking points in the cloud service gateway or application framework layer to capture traffic data in real time and perform aggregation calculations within a preset time window. For the total number of requests, all HTTP requests received within the window are counted. For the average response latency, the arithmetic mean of the time difference between each request entering the cloud service system and receiving a response is recorded. For the service success rate, the percentage of successful requests is calculated based on HTTP status codes (e.g., 2xx indicates success). Furthermore, by parsing application logs, custom transaction processing status codes at the business layer (e.g., payment status codes, inventory lock codes, etc.) are extracted and converted into discrete time-series data. The performance metric acquisition process involves deploying an Agent program on each host or container instance running the cloud service. This Agent periodically polls hardware resource status using the underlying operating system's system call interface (in this embodiment, the / proc file system of Linux). The Agent extracts real-time CPU utilization (including user-mode and kernel-mode ratios), memory usage (including resident memory and virtual memory), disk I / O throughput (bytes read and written per second), and network bandwidth consumption (inflow and outflow bit rates). Simultaneously, for Java Virtual Machine environments, the Agent dynamically obtains JVM heap information through Java Management Extensions interfaces or tools (such as jstat), including heap memory usage across generations, garbage collection trigger frequency, and pause times for single garbage collection. These fine-grained underlying metrics are then mapped to the corresponding cloud service instance nodes. The process of acquiring the link tracing data stream involves using distributed tracing technology to transparently transmit and record tracing metadata during service calls. When an external request enters the system, a globally unique TraceID is generated, which runs through all microservice nodes traversed by the request. Between each pair of upstream and downstream service calls, a unique SpanID is generated to identify the start and end points, duration, and parent-child relationship of the call link. These distributed call logs with TraceID and SpanID are output as streaming data, containing the call depth, direction, and real-time status code of the interface. The spatiotemporal alignment process aligns the unified timestamp with the aforementioned business and performance metrics, thereby completely reconstructing the flow path of the request in the heterogeneous propagation graph within the logical topology.

[0021] By collecting multi-source heterogeneous data from all dimensions, a complete observation chain has been established from underlying hardware resources and middleware runtime to upstream business logic. By aligning link tracing data with TraceID and SpanID with business and performance metrics under a unified timestamp, fragmented monitoring data can be transformed into a logically related global view, effectively solving the problem of difficulty in reconstructing fault propagation paths caused by missing monitoring dimensions or data silos in cloud-native environments.

[0022] See Figure 2 The specific process of constructing the heterogeneous propagation graph is as follows: parsing the upstream and downstream relationships in the link tracing data stream, identifying the running entities that can be parsed by TraceID / SpanID and mapped by performance metrics within the same time window as cloud service instance nodes; each cloud service instance node is associated with at least its service identifier and instance identifier, and CPU utilization, memory usage, disk I / O throughput, network bandwidth consumption, and runtime JVM-related metrics are loaded as attributes of the node, and the performance metrics are loaded synchronously as static attributes of the cloud service instance node; Between two cloud service instance nodes with a calling relationship, at least one calling behavior node is inserted according to the interface path and operation type in the link tracing data stream; the calling behavior node is used to carry the request volume, error rate and time consumption distribution data of a specific interface; Based on the unified timestamp, a directed connection is established between the cloud service instance node and the corresponding call behavior node.

[0023] Specifically, the process of parsing upstream and downstream relationships and mapping instance nodes involves reading distributed call logs with TraceID and SpanID from the link tracing data stream, identifying the client and server identifiers in each Span, and reconstructing the topological connections between microservices. For each type of cloud service instance with independent functions (such as order service and inventory service), it is mapped to a cloud service instance node in the heterogeneous propagation graph. Simultaneously, real-time collected performance metrics (such as CPU, memory, disk I / O, etc.) are synchronously loaded as static attributes of the node, forming a physical node entity with resource status awareness. The specific implementation process of inserting the call behavior node is as follows: between two adjacent cloud service instance nodes with a call relationship, the interface path and operation type in the link tracing data are further parsed; for different interfaces provided by the same service instance, at least one call behavior node is inserted between the instance nodes to decouple the logical differences at the interface level. The call behavior node is a logical relay entity built between adjacent cloud service instance nodes. Its specific content includes the interface path, operation type, and specific protocol metadata parsed from the link tracing data stream. As the core data carrier, it carries in real time the request volume, error rate, and response time distribution data of the specific interface within the sampling period. Through this heterogeneous node design, abstract service calls can be decoupled into fine-grained business behaviors. Thus, in the event of a failure, it can not only record the flow path of the request in the topology, but also characterize the abnormal behavior of each interface at the logical level. The specific implementation process for establishing a directed connection is as follows: based on the unified timestamp, cloud service instance nodes are associated with corresponding call behavior nodes in the logical topology; for the request initiator, the source service instance node points to the call behavior node it triggered; for the request receiver, the call behavior node points to the target service instance node. By constructing a heterogeneous topology system of "service instance - call behavior - service instance", the decoupling and binding of physical resource indicators and logical business indicators are realized. This modeling method can not only perceive the static attribute changes of the underlying hardware, but also refine the abnormal behavior at the interface level through call behavior nodes. With the directed connection established by the unified timestamp, it can rigorously restore the spatiotemporal evolution logic of the fault in the complex distributed environment, and provide a structured topological foundation for locking the root cause.

[0024] The request volume operator calculation process involves: counting the number of times a specific call behavior is triggered between two adjacent cloud service instance nodes within the current sampling period from the link tracing data stream, which is recorded as the real-time request volume; obtaining the average number of requests for this call behavior within the historical baseline window from the business metric stream, which is recorded as the baseline request volume; and obtaining the request volume operator based on the real-time request volume and the baseline request volume. The interface error rate operator calculation process is as follows: based on the status code markers in the link tracing data stream, the ratio of the number of failed calls between adjacent nodes to the total number of requests within the current sampling period is calculated to obtain the real-time error rate; the interface error rate operator is obtained based on the real-time error rate.

[0025] Based on the request volume operator and the interface error rate operator, and after normalization processing, dynamic edge weights are obtained. Specifically, the calculation process of the request volume operator is as follows: From the link tracing data stream, all Span records within the current sampling period are selected and normalized according to the instance identifiers of the caller and receiver; for specific call behaviors defined between two adjacent cloud service instance nodes in the heterogeneous propagation graph, the total number of times the behavior is triggered is counted in real time and recorded as the real-time request volume; the system retrieves the average number of requests for this call behavior within the historical baseline window from the business indicator stream and records it as the baseline request volume; by calculating the absolute value of the difference between the real-time request volume and the baseline request volume and dividing it by the baseline request volume for normalization, a request volume operator reflecting traffic fluctuations is obtained. The specific calculation process of the interface error rate operator is as follows: parsing the status code marker carried by the Span in the link tracing data stream to identify requests marked as failed; within the current sampling period, for a specific call link between adjacent nodes, the ratio of the number of failed calls to the total number of requests in that link is calculated to obtain the real-time error rate; the obtained real-time error rate is directly used as the interface error rate operator to characterize the degree of impairment of the call behavior at the logical level. By calculating the relative change rate between real-time request volume and historical baseline request volume, it is possible to effectively capture abnormal fluctuations caused by traffic surges or drops. At the same time, the error rate of the interface is calculated in real time by using the status codes in the distributed call log and transforming them into operators that characterize the degree of logical damage. This dual-operator design logic can dynamically evaluate the edge weights in the heterogeneous propagation graph from both the dimensions of traffic load and logical correctness.

[0026] The specific calculation process of the dynamic edge weight is to calculate the contribution feature values ​​of the request volume operator and the interface error rate operator in the current sampling window, respectively. Based on the contribution feature value, the real-time weight allocation ratio of the request volume operator and the interface error rate operator is calculated using the information entropy algorithm; the request volume operator, the interface error rate operator and their respective real-time weight allocation ratios are weighted and summed to obtain the original association score of each edge. For any source node in the heterogeneous propagation graph, obtain the original association scores of all outgoing edges, perform proportion normalization processing, and convert the scores into probability distribution values ​​as the dynamic edge weights.

[0027] Extract the original values ​​of the request volume operator and interface error rate operator corresponding to all calling edges within the current sampling window, and perform range standardization to convert all edge request volume fluctuations and error rate levels into dimensionless values ​​between 0 and 1. These values ​​are the contribution feature values. The information entropy algorithm is used to evaluate the information content of the data distribution of edges in the entire graph. The proportion of the contribution feature value of each edge in the total of similar indicators in the entire graph is calculated. The information entropy values ​​of the request volume indicator and the interface error rate indicator are calculated, and the information utility value (i.e., 1 minus the information entropy value) is further calculated. If a certain type of indicator is more unevenly distributed and fluctuates more drastically within the current window, its information utility value is higher. The specific process of calculating the information entropy values ​​of the request volume indicator and the interface error rate indicator is as follows: For each edge in the heterogeneous propagation graph, the proportion of its standardized operator feature value in the total of similar indicators in the entire graph is calculated. Then, these proportions are logarithmically calculated and accumulated using the information entropy formula, and multiplied by an adjustment coefficient (the adjustment coefficient is a normalization coefficient related to the number of edges) to ensure that the entropy value is between 0 and 1. Finally, the information utility value is obtained by subtracting the information entropy value from 1. If a certain type of indicator is more unevenly distributed and fluctuates more drastically within the current sampling window, its information entropy value is smaller, and the corresponding information utility value and real-time weight allocation ratio are higher, thereby achieving automatic focusing on key abnormal indicators.

[0028] For each edge, a weighted summation calculation is performed. Specifically, the standardized request volume contribution feature value of the edge is multiplied by the corresponding request volume weight, the standardized interface error rate contribution feature value is multiplied by the corresponding interface error rate weight, and finally the two products are added together. In other words, the features of the two dimensions of traffic fluctuation and logical damage are integrated into a comprehensive score, namely the original correlation score.

[0029] For each source node (including cloud service instance nodes and call behavior nodes) in the heterogeneous propagation graph, obtain the original association score of all outgoing edges emitted by it; perform proportion normalization processing, that is, divide the original association score of a single outgoing edge by the sum of the scores of all outgoing edges emitted by that node, and transform the score into a probability distribution value with a sum of one, which serves as the final dynamic edge weight. This set of weights constitutes the core parameters of the transition probability matrix, which determines the proportion of the failure probability flowing between nodes during subsequent iterative walks.

[0030] This invention achieves dynamic adaptive weight allocation through the information entropy algorithm, which can automatically focus on key anomaly indicators based on the fluctuation intensity of real-time monitoring data. This effectively solves the problem of inaccurate positioning caused by manually setting fixed weights in different fault scenarios (such as traffic surge or logic error faults), ensuring that dynamic edge weights can objectively and truthfully reflect the propagation law of faults in heterogeneous topologies.

[0031] The confidence interval of the business indicator stream is calculated using a dynamic baseline algorithm; when the business indicator exceeds the confidence interval, the business indicator is abnormally associated with the damaged starting node in the heterogeneous propagation graph according to the link tracing data stream. The dynamic baseline algorithm obtains the historical sequence of business indicators within a preset sliding window before the current time point, calculates the probability density distribution, and identifies the central trend characteristics and the degree of fluctuation dispersion. The degree of fluctuation dispersion serves as the corresponding boundary offset, and the upper and lower boundary points are automatically determined in combination with the central trend characteristics to generate a confidence interval. Monitor the degree to which real-time business metrics deviate from the dynamic confidence interval. When a business metric exceeds the confidence interval, retrieve the entry call chain associated with the abnormal timestamp through the link tracing data stream, and determine the cloud service instance node at the forefront of the call chain in the logical topology as the damaged starting node.

[0032] Specifically, the historical sequence of business indicators within a preset sliding window prior to the current sampling time point is obtained, and the preset sliding window is set to the previous 24 hours according to the periodic characteristics of the business. The historical sequence of the business indicators is preprocessed to remove obvious outlier noise points. The probability density distribution of the indicators within the sliding window is calculated using the kernel density estimation method. Based on the probability density distribution, the mean and standard deviation of the historical sequence of the business indicators are calculated. The mean is the central trend feature of the historical sequence of the business indicators, and the standard deviation is the degree of fluctuation and dispersion of the historical sequence of the business indicators. The degree of fluctuation dispersion is used as the corresponding boundary offset, and the upper and lower boundary points are automatically determined in combination with the central trend characteristics to generate a confidence interval; Specifically, the current fluctuation dispersion value (i.e., the standard deviation value) is obtained and directly used as the length unit of the boundary expansion. When the business indicator fluctuates drastically, the standard deviation increases, and the boundary offset increases by an equal amount; when the business indicator is stable, the standard deviation decreases, and the boundary offset decreases by an equal amount. This mapping method ensures that the offset is completely consistent with the physical dimensions and numerical values ​​of the historical fluctuation amplitude.

[0033] Using the central trend feature as the center point, the value of the central trend feature is added to the quantile offset, and the resulting value is automatically determined as the upper boundary point; at the same time, the value of the central trend feature is subtracted from the boundary offset, and the resulting value is automatically determined as the lower boundary point. In this way, the upper and lower boundaries are always numerically symmetrically distributed relative to the reference central axis; a confidence interval is generated based on the upper and lower boundary points.

[0034] The real-time business indicator data stream is compared point by point with the dynamic confidence interval generated at the current sampling time to determine whether the observed value of the business indicator falls within the numerical range defined by the upper and lower boundary points. Once the business indicator exceeds the upper limit of the dynamic confidence interval or falls below its lower limit, it is determined that a deviation has occurred. The abnormal alarm process is immediately triggered at the moment the deviation is determined, and the timestamp of the abnormal trigger is recorded. When a business metric deviates from the confidence interval, the distributed tracing database is accessed immediately. The abnormal timestamp is used as a key index to retrieve all tracing data streams generated at that time. All tracing serial numbers contained in these data streams are extracted, and based on the completeness of the serial numbers, all distributed call chains corresponding to that abnormal moment are aggregated. The parent-child relationship and call order between each call segment in the call chain are analyzed. The original call request corresponding to the root node at the top of the call chain is identified, and the entry call chain is determined accordingly. By analyzing the distributed service identifier corresponding to the request, the first path reached after the abnormal traffic enters the cloud service system can be reconstructed. By traversing the hierarchical structure of the call chain, the cloud service instance at the very front of the logical topology that directly receives external requests is identified. In a specific cloud service architecture implementation, this node is typically the first processing instance corresponding to the load balancer backend or API gateway. This cloud service instance node at the very top of the call chain is identified as the initial node of the damage. Abnormal business metrics are bound in real time to the identified starting node of the damage using service identifiers.

[0035] By using kernel density estimation to probabilistically model 24-hour historical indicators, and using dynamic mapping of mean and standard deviation to adaptively generate confidence intervals, the detection threshold can be automatically adjusted according to the periodic pattern of business traffic, effectively filtering business spikes while maintaining high sensitivity. At the same time, through deep coupling of timestamps and link tracing flow, instantaneous source tracing from macroscopic anomalies to the starting node of microscopic damage is achieved.

[0036] In the heterogeneous propagation graph, reverse edges and self-loop edges are constructed for each node. A transition probability matrix is ​​constructed based on the dynamic edge weights, reverse edge weights, and self-loop edge weights. Iterative walk calculations are performed starting from the damaged initiation node, and root cause performance indicators associated with the fault root cause service node are output. In the heterogeneous propagation graph, a reverse edge and a self-loop edge are established for each node; the weight of the reverse edge is proportionally mapped to the weight of the corresponding forward edge; the weight of the self-loop edge is the maximum value of the weights of all incoming edges of that node; For each node, the weights of its forward, reverse, and self-loop edges are normalized as a whole to construct a transition probability matrix; using the damaged starting node as the initial probability vector, a Markov chain iterative walk is performed in combination with the transition probability matrix to output the root cause performance index associated with the fault root cause service node.

[0037] Specifically, after constructing the basic heterogeneous propagation graph, a structured completion operation is performed on each node in the graph. These nodes include cloud service instance nodes representing physical entities and call behavior nodes representing logical interactions. For each original call edge (i.e., forward edge) in the graph, a reverse edge with the completely opposite direction is established between the same two nodes to carry the backtracking probability during fault tracing. The initial weight of the reverse edge is set to be proportionally mapped to the weight of the corresponding forward edge. In this embodiment, an equal-value mapping method is used, that is, the weight of the forward edge comes from the dynamic edge weight calculated by the information entropy algorithm in the previous steps. When establishing the reverse edge, the value of the dynamic edge weight is read and directly assigned to the corresponding reverse edge. At the same time, a self-loop edge is established for each node in the graph, that is, the starting point and ending point are connections of the node itself. The weight calculation logic of the self-loop edge is as follows: traverse all the current incoming edges (including forward and reverse incoming edges) of the node in real time, extract the maximum value among these incoming edge weights, and assign the maximum value to the self-loop edge to ensure that the node can retain the corresponding probability share according to its own degree of damage during iteration. See Figure 3 For all connections originating from each node, a global normalization process is performed. Taking any node as the center, the weights of all forward edges, reverse edges, and self-loop edges emanating from it are accumulated to obtain the total weight of that node. The independent weight of each outgoing edge is divided by this total weight, thereby converting all edge weights into probability distribution values ​​between 0 and 1, with the sum of the probabilities of outgoing edges from a single node being 1. This probability pointing relationship between all nodes in the entire graph is integrated into a two-dimensional square matrix to construct a transition probability matrix that describes the flow law of fault impact. Starting from the damaged initiating node determined by the dynamic baseline algorithm, a one-dimensional probability vector is initialized, with its dimension equal to the total number of nodes in the graph. The component corresponding to the damaged initiating node is assigned a value of 1.0, while the components of all other nodes in the graph are assigned a value of 0. That is, the damaged initiating node is used as the initial probability vector. Matrix multiplication is performed using the initial probability vector and the transition probability matrix to start the Markov chain iterative walk. In each iteration, the current probability value of a node is distributed and transferred to adjacent nodes along forward edges, reverse edges, and self-loop edges according to the proportions defined by the transition probability matrix. The specific process of performing Markov chain iterative walks using the transition probability matrix is ​​as follows: The initial probability is injected starting from the damaged starting node, and the probability is distributed to adjacent nodes along the forward, reverse and self-loop edges according to the transition probability matrix; during the iteration process, the change of the probability distribution of nodes in the whole graph is monitored, and the walk stops when the probability distribution of adjacent iteration cycles reaches the iteration limit; Extract the cloud service instance node with the highest probability value in the stationary distribution as the suspected root cause, and output the root cause performance index associated with the fault root cause service node in descending order according to the degree of deviation of the performance index from the dynamic baseline.

[0038] During iterative execution, the changes in the probability distribution vector of all nodes in the graph are continuously monitored. After each iteration, the difference between the probability distribution vector of the current period and the previous period is calculated. When the change in probability distribution between two adjacent iterations reaches a convergence state, the walk calculation stops. At this point, the probability values ​​of all nodes in the graph reach equilibrium, forming a stable distribution, where the probability value represents the likelihood that the node is the source of the fault. The convergence state is determined by calculating the distance between the vectors of two adjacent iterations, using Euclidean distance as the metric. When this distance value is less than a preset convergence threshold for three consecutive iterations (set to a threshold in this embodiment), the convergence is achieved. When the iteration count exceeds the preset maximum value (1000 times in this embodiment) and convergence is reached, the process stops. If convergence is not achieved after the number of iterations exceeds the preset maximum value, the process is forcibly stopped and the result of the last iteration is taken. From the final stable distribution results, the cloud service instance node with the highest probability value is selected and identified as the suspected root cause service node. For this suspected root cause node, real-time data of its associated performance indicators are retrieved, and the deviation of these indicators from their historical dynamic baseline is calculated. The indicators are then sorted in descending order according to the magnitude of the deviation, and the root cause performance indicator sequence associated with the faulty root cause service node is output. The deviation is calculated by extracting the real-time observation value of the suspected root cause node at the abnormal moment, as well as the corresponding dynamic baseline central trend value and standard deviation. The deviation is defined as the absolute value of the difference between the real-time value and the central trend value divided by the standard deviation. By introducing reverse edges and self-loop edges into the heterogeneous propagation graph and combining them with Markov chain iterative walk technology, a complete fault tracing calculation model was constructed. The weight mapping of reverse edges and the weight association design of self-loop edges ensure that the random walk process can both trace back upstream along the call chain to find the root cause and retain its own performance anomaly characteristics through node self-loops, avoiding the probability from being lost too quickly in the topology. This stable distribution calculation method based on the transition probability matrix can adapt to the instantaneous topology shift caused by frequent container scheduling in cloud-native environments, thereby effectively eliminating the abnormal interference of victim nodes in complex nested call relationships and outputting a sequence of performance indicators that are highly correlated with the root cause of the fault.

[0039] This application constructs a heterogeneous propagation graph containing cloud service instance nodes and call behavior nodes based on the link tracing data flow, realizing the logical decomposition of fine-grained call relationships between microservices. This enables logical differentiation of specific call behaviors that induce business anomalies and clearly defines the boundaries of fault impact. Simultaneously, it utilizes a dynamic baseline algorithm to generate confidence intervals that dynamically adjust with the time window, and combines information entropy to allocate the weights of request volume and error rate operators in real time, establishing an adaptive monitoring system for business fluctuations. This effectively solves the problem of location interference caused by the significant abnormal characteristics of victim nodes. Furthermore, by introducing reverse edges and self-loop edges into the heterogeneous propagation graph and combining iterative Markov chain walking technology, a complete fault tracing calculation model is constructed. This model can adapt to the instantaneous topology shifts caused by frequent container scheduling in cloud-native environments, ensuring accurate output of performance indicator sequences highly correlated with the root cause of the fault in complex nested call relationships.

[0040] Example 2: This implementation applies a cloud service fault root cause localization method based on dynamic baseline modeling to a smart warehousing and logistics robot system; During peak sorting periods in the automated warehousing system, data is collected in real time with a sampling period of 5 seconds. Business metrics show that the average response latency of the robot path planning interface surged from 50ms to 1200ms. Performance metrics show that the CPU utilization of the edge computing node is 85%, and the disk I / O throughput is 120MB / s. The link tracing data stream records robot task scheduling logs with TraceID. Based on a unified timestamp, the above discrete metrics are spatiotemporally aligned to ensure that the fault propagation path is restored within the same logical window. The real-time request volume of the / get_path interface within the current period is counted as 2000 from the link tracing data stream, and the historical baseline request volume is 1500 from the business metrics stream. The calculated request volume operator is 0.33. Simultaneously, the real-time error rate is calculated to be 0.15 based on status code markings, which is used as the interface error rate operator. Using the information entropy algorithm, the error rate metric is found to contribute more to the current fluctuations, and is assigned a weight of 0.7, while the request volume is assigned a weight of 0.3. After weighted summation and proportion normalization, the dynamic edge weights of each edge are obtained. The system retrieves historical business metrics from the past 24 hours, calculates probability density distribution using kernel density estimation, and identifies a mean latency of 55ms (central trend characteristic) and a standard deviation of 10ms (fluctuation dispersion). Based on the standard deviation, it automatically determines upper and lower quantiles, generating a confidence interval of [35ms, 75ms]. When the real-time value exceeds 1200ms and falls outside this interval, the system retrieves the associated entry call chain and identifies the damaged starting node as the task receiving service at the very front of the logical topology. In the heterogeneous propagation graph, reverse edges and self-loop edges are constructed for each node. The weights of reverse edges and forward dynamic edges are set to be equally mapped; the weights of self-loop edges are taken as the maximum value of the weights of all incoming edges of the node (the maximum weight of the incoming edge is 0.45, so the weight of the self-loop edge is 0.45), ensuring that the node can retain its own performance anomaly characteristics; for each node, the weights of its forward, reverse, and self-loop edges are normalized as a whole to construct a complete transition probability matrix; Using the damaged starting node as the initial probability vector, a Markov chain iterative walk is performed in conjunction with the transition probability matrix. During the iteration process, the changes in the probability distribution of nodes across the entire graph are monitored. When the Euclidean distance between the probability distributions of adjacent cycles reaches the convergence threshold... Stop walking; extract the node with the highest probability value in the stationary distribution, "path planning service", as the suspected root cause, and sort it in descending order according to the degree of deviation of the performance index from the dynamic baseline (disk I / O deviation of 6 times the standard deviation). Output the root cause performance index associated with the fault root cause service node. The root cause performance index is selected from CPU utilization, memory utilization, disk I / O throughput, network bandwidth consumption and runtime JVM related indicators, and sorted and output according to the degree of deviation from the dynamic baseline.

[0041] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cloud service fault root cause localization method based on dynamic baseline modeling, characterized in that, include: Real-time collection of business metrics, performance metrics, and link tracing data streams, and spatiotemporal alignment processing based on a unified timestamp; Based on the link tracing data stream, a heterogeneous propagation graph is constructed, which includes cloud service instance nodes and call behavior nodes located between cloud service instance nodes; The request volume operator and interface error rate operator between adjacent nodes are extracted from the business metrics and link tracing data stream, and normalized to obtain dynamic edge weights; the confidence interval of the business metric stream is calculated using the dynamic baseline algorithm. When a business metric exceeds the confidence interval, the business metric is abnormally associated with the damaged starting node in the heterogeneous propagation graph based on the link tracing data stream. In the heterogeneous propagation graph, reverse edges and self-loop edges are constructed for each node. A transition probability matrix is ​​constructed based on the dynamic edges, reverse edges, and self-loop edges. Iterative walk calculations are performed starting from the damaged initiating node to generate the fault root cause service node and associated root cause performance indicators. 2.The cloud service fault root cause localization method based on dynamic baseline modeling of claim 1, wherein, The business metrics include the total number of requests within the time window, average response latency, service success rate, and custom transaction processing status codes at the business level. The performance metrics include CPU utilization, memory usage, disk I / O throughput, network bandwidth consumption of the host on which the cloud service is located, as well as JVM stack information, garbage collection frequency and time during cloud service operation; The link tracing data stream contains distributed call logs with TraceID and SpanID. 3.The cloud service fault root cause localization method based on dynamic baseline modeling of claim 1, wherein, The specific process of constructing the heterogeneous propagation graph is to parse the upstream and downstream relationships in the link tracing data stream, map each type of independent cloud service instance to a cloud service instance node, and synchronously load the performance indicators as static attributes of the cloud service instance node. Between two cloud service instance nodes with a calling relationship, at least one calling behavior node is inserted according to the interface path and operation type in the link tracing data stream; the calling behavior node is used to carry the request volume, error rate and time consumption distribution data of a specific interface; Based on the unified timestamp, a directed connection is established between the cloud service instance node and the corresponding call behavior node.

4. The cloud service fault root cause localization method based on dynamic baseline modeling according to claim 1, characterized in that, The request volume operator calculation process involves counting the number of times a specific call behavior is triggered between two adjacent cloud service instance nodes within the current sampling period from the link tracing data stream, which is recorded as the real-time request volume; and obtaining the average number of requests for this call behavior within the historical baseline window from the business metric stream, which is recorded as the baseline request volume. A request volume operator is obtained based on the real-time request volume and the baseline request volume; The interface error rate operator calculation process is based on the status code markers in the link tracing data stream, and calculates the ratio of the number of failed calls between adjacent nodes to the total number of requests within the current sampling period to obtain the real-time error rate; The interface error rate operator is obtained based on the real-time error rate. Dynamic edge weights are obtained by using the request volume operator and the interface error rate operator, and by performing normalization processing.

5. The cloud service fault root cause localization method based on dynamic baseline modeling according to claim 4, characterized in that, The specific calculation process of the dynamic edge weight is to calculate the contribution feature values ​​of the request volume operator and the interface error rate operator in the current sampling window, respectively. Based on the contribution feature value, the real-time weight allocation ratio of the request volume operator and the interface error rate operator is calculated using the information entropy algorithm; the request volume operator, the interface error rate operator and their respective real-time weight allocation ratios are weighted and summed to obtain the original association score of each edge. For any source node in the heterogeneous propagation graph, obtain the original association scores of all outgoing edges, perform proportion normalization processing, and convert the scores into probability distribution values ​​as the dynamic edge weights.

6. The cloud service fault root cause localization method based on dynamic baseline modeling according to claim 1, characterized in that, The dynamic baseline algorithm obtains the historical sequence of business indicators within a preset sliding window before the current time point, calculates the probability density distribution, and identifies the central trend characteristics and the degree of fluctuation dispersion. The degree of fluctuation dispersion is used as the corresponding boundary offset, and the upper and lower boundary points are automatically determined in combination with the central trend characteristics to generate a confidence interval; Monitor the degree to which real-time business metrics deviate from the confidence interval. When a business metric exceeds the confidence interval, retrieve the entry call chain associated with the abnormal timestamp through the link tracing data stream, and determine the cloud service instance node at the forefront of the entry call chain in the logical topology as the damaged starting node.

7. The cloud service fault root cause localization method based on dynamic baseline modeling according to claim 1, characterized in that, In the heterogeneous propagation graph, a reverse edge and a self-loop edge are established for each node; the weight of the reverse edge is mapped to the weight of the corresponding forward edge; the weight of the self-loop edge is the maximum value of the weights of all incoming edges of the node; For each node, the weights of the forward, reverse, and self-loop edges are normalized as a whole to construct a transition probability matrix; with the damaged starting node as the initial probability vector, a Markov chain iterative walk is performed in combination with the transition probability matrix to output the root cause performance index associated with the fault root cause service node.

8. The cloud service fault root cause localization method based on dynamic baseline modeling according to claim 7, characterized in that, The specific process of performing Markov chain iterative walks using the transition probability matrix is ​​as follows: The initial probability is injected starting from the damaged starting node, and the probability is distributed to adjacent nodes along the forward, reverse and self-loop edges according to the transition probability matrix; the change of the probability distribution of nodes in the whole graph is monitored during the iteration process, and the walk stops when the number of iterations reaches the iteration limit; Extract the cloud service instance node with the highest probability value in the stationary distribution as the root cause service node of the fault, and output the root cause performance index associated with the root cause service node in descending order according to the degree of deviation of the performance index from the dynamic baseline.

Images