Cloud native system fault root cause positioning method and device

Abstract

The invention provides a cloud native system fault root cause positioning method and device, and the method comprises the steps: determining the abnormal performance index data of each instance and a server based on the micro-service instance of each target micro-service and the performance index abnormal score of the server, so as to construct the nodes corresponding to each micro-service instance and the server, according to a fault propagation direction between the abnormal performance index data, constructing an edge between the nodes to obtain an index-level cause and effect graph; normalizing the performance index anomaly score to obtain a target anomaly score of each node, and constructing a transition probability matrix; and obtaining the access frequency of each node in the causal graph by using a random walk algorithm so as to obtain fault root cause positioning result data of the cloud native system. According to the invention, the automation degree, efficiency and accuracy of fault root cause positioning of the cloud native system can be effectively improved, the efficiency and reliability of fault early warning and recovery of the cloud native system can be effectively improved, and the operation stability of the cloud native system can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method and device for locating the root cause of a cloud native system failure. Background Art

[0002] Cloud-native systems are application architectures designed specifically for cloud environments, aiming to leverage the core advantages of cloud computing, such as elasticity, scalability, and automation. In cloud-native systems, resource entities are typically divided into three tiers: server nodes, services, and container groups (pods), corresponding to infrastructure servers, microservices, and microservice instances, respectively. To ensure the stable operation of cloud-native systems, periodic root cause identification is required.

[0003] Currently, existing methods for locating the root cause of faults in cloud-native systems use fault-based system-level event graphs or causal graphs to analyze and rank potential fault causes. However, while this approach is not limited by the number of microservice instances, due to the large number of resource entities in cloud-native systems, the causal graphs constructed directly using these methods often contain a large number of nodes unrelated to the fault. This results in inefficient causal graph construction, and the presence of unrelated nodes in the causal graph can also reduce the accuracy of root cause location. More importantly, existing methods fail to fully model the fault propagation path between instances and servers when constructing causal graphs to represent fault propagation, a flaw that further limits the accuracy of root cause location. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a method and apparatus for locating the root cause of a cloud-native system fault to eliminate or improve one or more defects in the prior art.

[0005] One aspect of the present application provides a method for locating the root cause of a cloud native system failure, including:

[0006] Based on the performance indicator anomaly scores corresponding to each microservice instance contained in each target microservice in the target time period and the server to which each microservice instance belongs, determine the abnormal performance indicator data corresponding to each microservice instance and each server to construct nodes corresponding to each microservice instance and each server, and construct edges between each node according to the fault propagation direction between each abnormal performance indicator data to obtain a corresponding indicator-level causal graph; wherein the target microservice includes the abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservices corresponding to the abnormal microservice;

[0007] The performance indicator anomaly scores corresponding to each of the abnormal performance indicator data are normalized to obtain the target anomaly scores corresponding to each of the nodes, and a transition probability matrix representing the transition probability between each of the nodes is constructed based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the root cause location result data of the cloud native system in the target time period.

[0008] In some embodiments of the present application, the abnormal performance indicator scores corresponding to each microservice instance included in each target microservice and each server to which each microservice instance belongs are determined based on the target time period, and the abnormal performance indicator data corresponding to each microservice instance and each server includes:

[0009] Determine, based on the performance indicator data of each microservice instance included in each target microservice within the target time period, the performance indicator anomaly score of the performance indicator data corresponding to each microservice instance;

[0010] and, determining, based on the performance indicator data of the servers to which the respective microservice instances belong within the target time period, the performance indicator anomaly scores of the performance indicator data corresponding to the respective servers;

[0011] Among the performance indicator data corresponding to each of the microservice instances and each of the servers, determining the performance indicator data having a performance indicator anomaly score greater than a first threshold as abnormal performance indicator data for the target time period;

[0012] The server corresponding to the abnormal performance indicator data is determined as the current abnormal server. If the abnormal server includes a microservice instance for which the performance indicator abnormality score has not been determined, the performance indicator abnormality score of the performance indicator data corresponding to the microservice instance is determined based on the performance indicator data of the microservice instance within the target time period. If the performance indicator abnormality score corresponding to the microservice instance is greater than the first threshold, the performance indicator data corresponding to the microservice instance is determined as the abnormal performance indicator data for the target time period.

[0013] In some embodiments of the present application, constructing edges between the nodes according to the fault propagation direction between the abnormal performance indicator data includes:

[0014] For the abnormal performance indicator data corresponding to the nodes corresponding to each of the microservice instances, determine the fault propagation direction between each type of abnormal performance indicator data according to preset expert indication data, so as to construct edges between the nodes corresponding to each of the microservice instances; wherein the types of abnormal performance indicator data include: CPU utilization, memory utilization, disk utilization, and latency;

[0015] For each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship, determine the fault propagation direction between each type of abnormal performance indicator data corresponding to each of the different microservice instances that are used to represent the existence of a call relationship based on the Granger causality test algorithm, so as to construct edges between each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship;

[0016] For each node representing the server and the microservice instance, respectively, determining the fault propagation direction between each type of abnormal performance indicator data corresponding to each server and the microservice instance based on the Granger causality test algorithm, so as to construct edges between each node representing the server and the microservice instance;

[0017] Wherein, one of the nodes is used to represent the name corresponding to one of the microservice instances or one of the servers, the abnormal performance indicator data, and the target abnormality score; wherein, the abnormal performance indicator data contains the value of the abnormal performance indicator and the indicator name used to indicate the type of the abnormal performance indicator;

[0018] If the similarity between the target anomaly scores corresponding to multiple microservice instances corresponding to the microservice is greater than a similarity threshold, it is predicted that the microservice has a fault and the fault is propagated to the microservice instance corresponding to the microservice.

[0019] In some embodiments of the present application, constructing a transition probability matrix for representing transition probabilities between the nodes according to the target anomaly scores corresponding to the nodes includes:

[0020] A transition probability matrix is ​​defined, wherein each element in the transition probability matrix is ​​used to represent a transition probability between two different nodes;

[0021] Determine, according to the target anomaly scores corresponding to the two nodes respectively corresponding to each element in the transition probability matrix, a transition probability of forward transfer between the two nodes respectively corresponding to each element, and determine, based on the transition probability of forward transfer between the two nodes respectively corresponding to each element, a transition probability of backward transfer and a transition probability of stay between the two nodes respectively corresponding to each element, so as to complete the assignment of the transition probability matrix;

[0022] The assigned transfer probability matrix is ​​normalized.

[0023] In some embodiments of the present application, obtaining the access frequency corresponding to each of the nodes in the indicator-level causal graph using a random walk algorithm based on the transition probability matrix to obtain the root cause location result data of the cloud native system in the target time period includes:

[0024] Starting from the node with the highest target anomaly score in the indicator-level causal graph, based on the normalized transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph;

[0025] The nodes are sorted in descending order of access frequency, and the obtained sorting result data is used as the root cause location result data of the cloud native system in the target time period.

[0026] In some embodiments of the present application, before determining the abnormal performance indicator data corresponding to each of the microservice instances and each of the servers, the method further includes:

[0027] Extracting a fusion feature vector corresponding to each microservice based on the performance indicator data of each microservice in the cloud native system within the target time period, and the log data and link tracking data of each microservice instance corresponding to each microservice within the target time period;

[0028] Inputting the fused feature vectors corresponding to each microservice into a graph autoencoder respectively, so that the graph autoencoder outputs the reconstruction error scores corresponding to each microservice respectively;

[0029] Determine whether any reconstruction error score corresponding to each of the microservices has a reconstruction error score greater than a second threshold. If so, determine the microservice with the reconstruction error score greater than the second threshold as an abnormal microservice within the target time period, and use the abnormal microservice and the associated microservice corresponding to the abnormal microservice as the target microservice.

[0030] In some embodiments of the present application, extracting a fusion feature vector corresponding to each microservice in the cloud native system according to the performance indicator data of each microservice within the target time period, and the log data and link tracking data of each microservice instance corresponding to each microservice within the target time period, includes:

[0031] Inputting the performance indicator data of each of the microservices in the cloud native system within the target time period into a one-dimensional dilated causal convolutional layer, so that the one-dimensional dilated causal convolutional layer outputs an indicator feature vector corresponding to each of the microservices;

[0032] Based on the log parsing algorithm, the log data of each microservice instance corresponding to each microservice within the target time period is parsed, and the key events and keywords corresponding to each microservice instance are determined; the parsed log data corresponding to each microservice instance are respectively input into the fully connected layer, so that the fully connected layer outputs the occurrence frequency of the key events and keywords corresponding to each microservice instance; the occurrence frequencies corresponding to each microservice instance belonging to the same microservice are aggregated to obtain the instance log aggregate feature vector corresponding to each microservice;

[0033] Furthermore, the link tracking data of each microservice instance is input into a one-dimensional dilated causal convolutional layer, so that the one-dimensional dilated causal convolutional layer outputs a link tracking feature vector corresponding to each microservice instance; and the link tracking feature vectors corresponding to each microservice instance belonging to the same microservice are aggregated to obtain an instance tracking aggregate feature vector corresponding to each microservice.

[0034] Feature fusion is performed on the indicator feature vector, the instance log aggregate feature vector, and the instance tracking aggregate feature vector corresponding to each of the microservices to obtain a fused feature vector corresponding to each of the microservices.

[0035] A second aspect of the present application provides a device for locating the root cause of a cloud-native system fault, including:

[0036] A causal graph construction module is used to determine the abnormal performance indicator data corresponding to each microservice instance and each server in the target microservice within the target time period, based on the performance indicator anomaly scores corresponding to each microservice instance and each server to which each microservice instance belongs, to construct nodes corresponding to each microservice instance and each server, and to construct edges between each node according to the fault propagation direction between each abnormal performance indicator data, to obtain a corresponding indicator-level causal graph; wherein the target microservice includes an abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservices corresponding to the abnormal microservice;

[0037] The fault root cause location module is used to normalize the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data to obtain the target anomaly scores corresponding to each of the nodes, and construct a transition probability matrix for representing the transition probability between each of the nodes based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the fault root cause location result data of the cloud native system in the target time period.

[0038] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the cloud-native system fault root cause location method when executing the computer program.

[0039] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for locating the root cause of a cloud-native system failure.

[0040] The fifth aspect of the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method for locating the root cause of a cloud-native system failure.

[0041] The cloud native system fault root cause location method provided by the present application is based on the performance indicator abnormality scores corresponding to each microservice instance contained in each target microservice in the target time period and the server to which each microservice instance belongs, to determine the abnormal performance indicator data corresponding to each microservice instance and each server to construct the nodes corresponding to each microservice instance and each server, and to construct the edges between each node according to the fault propagation direction between each abnormal performance indicator data to obtain the corresponding indicator-level causal graph; wherein, the target microservice includes the abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservice corresponding to the abnormal microservice; the performance indicator abnormality scores corresponding to each of the abnormal performance indicator data are normalized to obtain the nodes corresponding to each of the nodes , and construct a transition probability matrix for representing the transition probability between each of the nodes according to the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the root cause location result data of the cloud native system in the target time period, thereby realizing real-time monitoring of cloud native system operation anomalies and accurate location of the root cause of the fault, which can effectively solve the three key technical bottlenecks existing in the existing technology: difficulty in adapting to the dynamic changes of microservice instances, insufficient modeling of cloud native system resource entities, and inefficient modeling, etc., and can effectively improve the automation level, efficiency and accuracy of cloud native system fault root cause location, thereby effectively improving the efficiency and reliability of cloud native system fault warning and recovery, and improving the operational stability of the cloud native system.

[0042] Additional advantages, purposes, and features of the present application will be described in part in the following description and will become apparent to those skilled in the art upon study of the following or may be learned from practice of the present application. The purposes and other advantages of the present application may be achieved and obtained by the structures specifically pointed out in the specification and drawings.

[0043] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present application are not limited to the above specific description, and the above and other purposes that can be achieved by the present application will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are intended to provide a further understanding of the present application, constitute a part of the present application, and do not constitute a limitation of the present application. The components in the drawings are not drawn to scale, but are only for the purpose of illustrating the principles of the present application. In order to facilitate the illustration and description of some parts of the present application, the corresponding parts in the drawings may be enlarged, that is, they may become larger than other components in the exemplary device actually manufactured according to the present application. In the drawings:

[0045] Figure 1 This is a first flow chart of a method for locating the root cause of a cloud-native system failure in one embodiment of the present application.

[0046] Figure 2 This is a second flow chart of the method for locating the root cause of a cloud-native system failure in one embodiment of the present application.

[0047] Figure 3 This is a third flow chart of the method for locating the root cause of a cloud-native system failure in one embodiment of the present application.

[0048] Figure 4 This is a fourth flow chart of the method for locating the root cause of a cloud-native system failure in one embodiment of the present application.

[0049] Figure 5 Schematic diagram of the structure of a cloud native system fault root cause location device in one embodiment of the present application.

[0050] Figure 6 This is a technical framework diagram of a method for locating the root cause of a cloud-native system failure in an example of this application.

[0051] Figure 7 This is a schematic diagram of a root cause location algorithm in an example of this application.

[0052] Figure 8 This is a schematic diagram illustrating the causal relationship between internal indicators of a microservice instance guided by expert knowledge in an example of this application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail in conjunction with the embodiments and drawings. Here, the illustrative embodiments of this application and their descriptions are used to explain this application, but are not intended to limit this application.

[0054] It should also be noted here that in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show structures and / or processing steps that are closely related to the scheme according to the present application, while other details that are not closely related to the present application are omitted.

[0055] It should be emphasized that the term "include / comprises" when used herein refers to the existence of features, elements, steps or components, but does not exclude the existence or addition of one or more other features, elements, steps or components.

[0056] It should also be noted that, unless otherwise specified, the term "connection" herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0057] Hereinafter, embodiments of the present application will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0058] With the rapid development of cloud computing, microservices architecture has become a core component of cloud-native systems. By splitting applications into multiple independent service units, microservices architecture significantly improves system scalability and flexibility. However, the rise of this architecture also brings new challenges: systems generate massive amounts of operational monitoring data daily, and the mixing of abnormal and normal data makes system fault diagnosis difficult. System fault diagnosis primarily involves anomaly detection and fault root cause location. Due to the complex call dependencies between microservices and the dynamic deployment relationship between microservice instances and the underlying servers, faults often propagate and diffuse across components at different levels (servers, microservices, and microservice instances), greatly increasing the complexity of fault diagnosis. Existing research methods in this field primarily rely on multi-dimensional data collected by monitoring tools, including metrics, traces, and logs, for fault diagnosis. Given that both system anomaly detection and fault root cause location are based on the same monitoring data system, current research trends are moving towards end-to-end integrated anomaly detection and root cause location to optimize data processing, avoid redundant computation, and improve system operation and maintenance efficiency.

[0059] From a methodological perspective, existing research can be roughly divided into two categories: deep learning-based methods and traditional machine learning methods. Within the field of deep learning, there is an increasingly prominent trend in using graph neural networks (GNNs) to model the relationships between system resource entities. Taking HolisticRCA as an example, this method uses a graph attention network (GAT) to comprehensively analyze system resource entities, thereby achieving accurate fault location. However, this type of deep learning method has a significant limitation: the model training and inference process requires the input of fixed graph-structured data, thus assuming that the number of system resource entities remains unchanged. This assumption may be valid at the server and microservice level, but it is difficult to maintain at the microservice instance level, as the number of microservice instances changes dynamically with workload fluctuations.

[0060] To address the limitations of these deep learning methods, some researchers have proposed constructing system-level event graphs or causal graphs to analyze and prioritize potential fault causes. This approach is not limited by the number of microservice instances. However, in cloud-native systems, due to the large number of resource entities, causal graphs constructed directly using these methods often contain a large number of nodes unrelated to the fault. This results in inefficient causal graph construction, and the presence of unrelated nodes in the graph can also reduce the accuracy of fault root cause location. More importantly, existing methods fail to fully model the fault propagation paths between instances and servers when constructing causal graphs to represent fault propagation, a flaw that further limits the effectiveness of fault root cause location. Therefore, comprehensive fault diagnosis in cloud-native systems is a key technical issue that urgently needs to be addressed in this field.

[0061] Based on this, in order to solve the problems existing in the existing methods of changing the eye features of characters in videos, such as requiring high shooting skills, time-consuming post-processing and easily affecting the viewer experience, the embodiments of the present application respectively provide a cloud-native system fault root cause location method, a cloud-native system fault root cause location device for executing the cloud-native system fault root cause location method, a physical device and a computer-readable storage medium, which can greatly improve the efficiency of film and television shooting, make the film and television presentation more natural, and can greatly reduce the cost of film and television production and shorten the production cycle time.

[0062] The details are described in detail through the following examples.

[0063] Based on this, the embodiment of the present application provides a cloud native system fault root cause location method that can be implemented by a cloud native system fault root cause location device, see Figure 1 The method for locating the root cause of a cloud native system failure specifically includes the following:

[0064] Step 100: Based on the performance indicator anomaly scores corresponding to each microservice instance contained in each target microservice in the target time period and the server to which each microservice instance belongs, determine the abnormal performance indicator data corresponding to each microservice instance and each server to construct a node corresponding to each microservice instance and each server, and construct edges between each node according to the fault propagation direction between each abnormal performance indicator data to obtain a corresponding indicator-level causal graph; wherein, the target microservice includes the abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservice corresponding to the abnormal microservice.

[0065] It is understandable that the cloud native system consists of three types of resource entities: servers, microservices, and microservice instances. Since faults can occur in any resource entity and can be propagated in a cascading manner, the embodiments of the present application are based on the perspective of overall fault propagation to quickly discover and report cloud native system faults and their root causes. The cloud native system includes an application consisting of multiple microservices and a service cluster consisting of multiple servers. Each microservice has several microservice instances deployed in the service cluster, and the specific number is determined by the workload requirements. Each microservice instance independently generates log data, link tracking data, and performance indicator (metric) data (i.e., instance indicator data), each microservice independently generates its performance indicator data (i.e., microservice indicator data), and each server h independently generates its performance indicator data (i.e., server indicator data).

[0066] In one or more embodiments of the present application, the target time period is the time period for locating the root cause of the fault in the cloud native system, which can be the time period closest to the current time point, and the time period can be an observation window (i.e., a time window) of length T.

[0067] The target microservice refers to the abnormal microservice (i.e., the microservice that may have a fault, also referred to as the faulty microservice) pre-detected in the cloud native system within the target time period and the associated microservices corresponding to the abnormal microservice. The associated microservices can refer to all microservices that have a call relationship with the abnormal microservice, which ensures that if the root cause is located in the call chain related to the fault, it is still within the scope of analysis of this application.

[0068] In the target time period, the embodiment of the present application can first detect whether a microservice anomaly exists. Once an anomaly is detected, step 100 is executed to construct an indicator-level causal graph G = (V, E) for the abnormal performance indicator data corresponding to each of the microservice instances and each of the servers in the target time period.

[0069] In one or more embodiments of the present application, an indicator-level causal graph may be referred to as a causal graph. In this causal graph, nodes (i.e., vertices) represent corresponding microservice instances or servers, forming a finite, non-empty vertex set V, and edges represent the direction of fault propagation, forming an edge set E. The goal of this application is to recommend the k most likely abnormal performance indicator data (i.e., entities) for a fault by analyzing the causal graph.

[0070] Among them, one of the nodes is used to represent the name corresponding to one of the microservice instances or one of the servers, the abnormal performance indicator data, and the target abnormality score; wherein, the abnormal performance indicator data contains the value of the abnormal performance indicator and the indicator name used to indicate the type of the abnormal performance indicator.

[0071] Specifically, the node is a structure, the name attribute of the node is the name corresponding to the microservice instance or the server, the value attribute of the node is the value of the abnormal performance indicator contained in the abnormal performance indicator data, the value name (value_name) attribute of the node is the indicator name contained in the abnormal performance indicator data for indicating the type of the abnormal performance indicator, and the score attribute of the node is the corresponding target anomaly score. Therefore, the final positioning result can not only know which microservice instance or server is faulty, but also know which performance indicator corresponding to the faulty microservice instance or server is abnormal.

[0072] Among them, the performance indicator anomaly score refers to a score used to indicate the degree of abnormality of the performance indicator data, and the performance indicator data refers to an indicator value used to indicate the operating performance of the microservice instance or server corresponding to the performance indicator data within the target time period; the type of the performance indicator data can be set according to the actual application scenario, and specifically CPU utilization, memory utilization, disk utilization and latency, etc. can be used; abnormal performance indicator data refers to performance indicator data whose performance indicator anomaly score exceeds a preset threshold (such as the second threshold mentioned in the subsequent embodiments); similarly, the type division of the abnormal performance indicator data can be the same as or less than the type division of the performance indicator data.

[0073] Step 200: Normalize the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data to obtain the target anomaly scores corresponding to each of the nodes, and construct a transition probability matrix for representing the transition probability between each of the nodes based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, use a random walk algorithm to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the root cause location result data of the cloud native system in the target time period.

[0074] In step 200, the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data are normalized to obtain the target anomaly scores corresponding to each of the nodes, which is to initialize the nodes in the indicator-level causal graph and provide the random walk algorithm with a probability transfer matrix of the anomaly detection results obtained in integration step 100, which can effectively improve the effectiveness and reliability of the probability transfer matrix setting, and thus improve the accuracy of the fault root cause location results.

[0075] It is understandable that the nodes may be sorted in descending order of access frequency, and the obtained sorting result data may be output as the root cause location result data of the cloud native system in the target time period.

[0076] From the above description, it can be seen that the cloud native system fault root cause location method provided in the embodiment of the present application can effectively improve the automation, efficiency and accuracy of cloud native system fault root cause location, thereby effectively improving the efficiency and reliability of cloud native system fault warning and recovery, and improving the operational stability of the cloud native system.

[0077] In order to further improve the effectiveness and reliability of causal skeleton extraction in cloud native system fault root cause location, in a cloud native system fault root cause location method provided in an embodiment of the present application, see Figure 2 Step 100 of the cloud native system fault root cause location method specifically includes the following:

[0078] Step 110: Determine the performance indicator anomaly score of the performance indicator data corresponding to each microservice instance according to the performance indicator data of each microservice instance included in each target microservice within the target time period.

[0079] In step 110, the average value of the performance indicator data corresponding to the microservice instance in the target time period, as well as the average value and variance of the microservice instance in each historical time period and the target time period, where each historical time period is earlier than the target time period, can be calculated. The difference between the average value of the performance indicator data corresponding to the microservice instance in the target time period and the average value of the microservice instance in each historical time period and the target time period is then divided by the variance to obtain a corresponding ratio, and the absolute value of the ratio is used as the performance indicator anomaly score of the performance indicator data corresponding to the microservice instance.

[0080] And, step 120: determining the performance indicator anomaly scores of the performance indicator data corresponding to each of the servers according to the performance indicator data of each of the microservice instances within the target time period.

[0081] In step 120, the average value of the performance indicator data corresponding to the server in the target time period, as well as the average value and variance of the server in each historical time period and the target time period, can be calculated, where each historical time period is earlier than the target time period. The difference between the average value of the performance indicator data corresponding to the server in the target time period and the average value of the server in each historical time period and the target time period is then divided by the average value of the server in each historical time period and the target time period to obtain a corresponding ratio, and the absolute value of the ratio is used as the performance indicator anomaly score of the performance indicator data corresponding to the server.

[0082] Step 130: Among the performance indicator data corresponding to each of the microservice instances and each of the servers, determine the performance indicator data having a performance indicator anomaly score greater than a first threshold as abnormal performance indicator data for the target time period.

[0083] The first threshold value may be set to a value of 1, indicating that when the performance indicator anomaly score exceeds 1 standard deviation of normal data, the performance indicator data corresponding to the performance indicator anomaly score is determined to be abnormal.

[0084] Step 140: Determine the server corresponding to the abnormal performance indicator data as the current abnormal server. If the abnormal server contains a microservice instance whose performance indicator abnormality score has not been determined, determine the performance indicator abnormality score of the performance indicator data corresponding to the microservice instance based on the performance indicator data of the microservice instance within the target time period. If the performance indicator abnormality score corresponding to the microservice instance is greater than the first threshold, determine the performance indicator data corresponding to the microservice instance as the abnormal performance indicator data for the target time period.

[0085] Among them, the causal skeleton refers to a simplified causal graph consisting of abnormal entities and their association relationships.

[0086] It is understandable that step 140 may determine the performance indicator anomaly score of the performance indicator data corresponding to the microservice instance in the same manner as step 110 .

[0087] In order to further improve the effectiveness and reliability of causal graph learning in locating the root cause of cloud native system faults, in a cloud native system fault root cause locating method provided in an embodiment of the present application, see Figure 2 Step 100 of the cloud native system fault root cause location method further specifically includes the following:

[0088] Step 150: constructing nodes corresponding to each microservice instance and each server according to the number of abnormal performance indicators corresponding to each microservice instance and each server.

[0089] Step 160: For the abnormal performance indicator data corresponding to the nodes corresponding to each of the microservice instances, determine the fault propagation direction between each type of abnormal performance indicator data according to preset expert indication data to construct edges between the nodes corresponding to each of the microservice instances; wherein the types of abnormal performance indicator data include: CPU utilization, memory utilization, disk utilization and latency.

[0090] Step 170: For each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship, determine the fault propagation direction between each type of abnormal performance indicator data corresponding to each of the different microservice instances that are used to represent the existence of a call relationship based on the Granger causality test algorithm, so as to construct edges between each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship.

[0091] Step 180: For each node representing each of the servers and microservice instances, determine the fault propagation direction between each type of abnormal performance indicator data corresponding to each of the servers and microservice instances based on the Granger causality test algorithm, so as to construct edges between each node representing each of the servers and microservice instances. A node represents the name, abnormal performance indicator data, and target abnormality score corresponding to a microservice instance or a server; the abnormal performance indicator data contains the value of the abnormal performance indicator and the indicator name indicating the type of the abnormal performance indicator; and if the similarity between the target abnormality scores corresponding to multiple microservice instances corresponding to the microservice is greater than a similarity threshold, a fault is predicted for the microservice and the fault is propagated to the microservice instance corresponding to the microservice.

[0092] Among them, the Granger causality test algorithm can be called the Granger causal inference algorithm. It is a statistical hypothesis testing method based on time series data, which is used to analyze the causal relationship between variables. The core idea of ​​this method is: if the historical information of a variable X can significantly improve the future prediction accuracy of another variable Y, then X is considered to be the Granger cause of Y.

[0093] In order to further improve the effectiveness and reliability of constructing a transition probability matrix in locating the root cause of cloud native system faults, in a cloud native system fault root cause locating method provided in an embodiment of the present application, see Figure 2 Step 200 of the cloud native system fault root cause location method specifically includes the following:

[0094] Step 210: normalize the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data to obtain target anomaly scores corresponding to each of the nodes.

[0095] Step 220: defining a transition probability matrix, wherein each element in the transition probability matrix is ​​used to represent a transition probability between two different nodes;

[0096] Step 230: determining a forward transfer probability between the two nodes corresponding to each element in the transfer probability matrix based on the target anomaly scores corresponding to the two nodes, and determining a backward transfer probability and a stay transfer probability between the two nodes corresponding to each element based on the forward transfer probability between the two nodes corresponding to each element, thereby completing the assignment of the transfer probability matrix.

[0097] Step 240: normalize the assigned transfer probability matrix.

[0098] Specifically, a forward step refers to moving from the effect indicator node to the cause indicator node. A backward step refers to moving from the cause indicator node to the effect indicator node to prevent the algorithm from being trapped in a node with low correlation with the abnormal node. Random walk allows movement from the cause node to the effect node. A self step refers to staying at the current node. If the algorithm reaches the indicator node and the probability of transitioning to an adjacent node is low, this indicator may be the root cause. In this case, it should stay at this indicator node.

[0099] In order to further improve the effectiveness and reliability of root cause location in cloud native system fault root cause location, in a cloud native system fault root cause location method provided in an embodiment of the present application, see Figure 2 Step 200 of the cloud native system fault root cause location method further specifically includes the following:

[0100] Step 250: Starting from the node with the highest target anomaly score in the indicator-level causal graph, based on the normalized transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph.

[0101] Step 260: Sort the nodes in descending order of access frequency, and use the obtained sorting result data as the root cause location result data of the cloud native system in the target time period.

[0102] Specifically, a random walk is a mathematical model based on random steps, used to describe the process of random movement in a certain space (such as one-dimensional, two-dimensional, or higher-dimensional space). Its core concept is that the direction and length of each step are random, usually determined by a probability distribution. Each step depends only on the current state and is independent of the previous historical path (Markov property).

[0103] In order to further improve the effectiveness and reliability of triggering root cause location in cloud native system fault root cause location, in a cloud native system fault root cause location method provided in an embodiment of the present application, see Figure 3 The cloud native system fault root cause location method further includes the following contents before step 100:

[0104] Step 010: Based on the performance indicator data of each microservice in the cloud native system within the target time period, and the log data and link tracking data of each microservice instance corresponding to each microservice within the target time period, extract the fusion feature vector corresponding to each microservice.

[0105] Step 020: Input the fused feature vectors corresponding to each microservice into the graph autoencoder respectively, so that the graph autoencoder outputs the reconstruction error scores corresponding to each microservice respectively.

[0106] Specifically, the graph autoencoder GAE (which can be simply referred to as the GAE model) can be trained based on historical normal data. The historical normal data can include the performance indicator data of each of the microservices in the cloud native system in each historical time period, as well as the log data and link tracking data of each of the microservice instances corresponding to each of the microservices in each historical time period. In the online stage, the GAE model analyzes the monitoring data at specified time intervals (i.e., each time period) and generates a reconstruction error score for each microservice. If any reconstruction error score exceeds a predefined threshold, the root cause location process will be triggered.

[0107] To reduce the reliance on labeled data, this application adopts unsupervised learning and chooses the Graph Autoencoder (GAE) because it is able to process graph data by combining Graph Neural Networks (GNNs) with autoencoders. GAE consists of two main components: an encoder and a decoder. This application uses the Graph Attention Network (GAT) to design these two components, leveraging its multi-head attention mechanism to capture node representations from multiple perspectives. The encoder extracts microservice features and dependencies, embedding them into dense latent vectors. The decoder is a mirror image of the encoder and is used to reconstruct the original features. The model is trained to minimize reconstruction error, ensuring that the output features closely match the input features. Unlike most anomaly detection models that operate at the system level, GAE is able to detect anomalies at the microservice level.

[0108] Because systems are typically stable, the GAE model is trained on historical normal data to ensure robustness. During testing, faulty data causes an increase in the reconstruction error score, which reflects the degree of deviation from the normal pattern. If this reconstruction error score exceeds a predefined threshold, the root cause identification process is triggered. The GAE model generates a ranked dictionary of normalized reconstruction error scores for all microservices. Microservices with reconstruction error scores exceeding a certain threshold are marked as anomalies, providing an initial estimate of the potential root cause of the failure.

[0109] Step 030: Determine whether any of the reconstruction error scores corresponding to the microservices has a reconstruction error score greater than a second threshold. If so, determine the microservice with the reconstruction error score greater than the second threshold as an abnormal microservice within the target time period, and use the abnormal microservice and its associated microservice as the target microservice.

[0110] If it is determined that the reconstruction error scores corresponding to the current microservices are all less than or equal to the reconstruction error score of the second threshold, it is determined that there are no abnormal microservices in the target time period, and the process can return to step 010 to continue execution in the next time period.

[0111] In order to further improve the effectiveness and reliability of extracting fusion feature vectors in the process of triggering root cause location in cloud native system fault root cause location, in a cloud native system fault root cause location method provided in an embodiment of the present application, see Figure 4 Step 010 in the cloud native system fault root cause location method specifically includes the following:

[0112] Step 011: Input the performance indicator data of each microservice in the cloud native system within the target time period into a one-dimensional dilated causal convolution layer, so that the one-dimensional dilated causal convolution layer outputs the indicator feature vector corresponding to each microservice.

[0113] Using one-dimensional dilated causal convolution (DCC) layers is an effective way to extract features from time series data.

[0114] Step 012: Based on the log parsing algorithm, the log data of each microservice instance corresponding to each microservice within the target time period is parsed, and the key events and keywords corresponding to each microservice instance are determined; and the parsed log data corresponding to each microservice instance are respectively input into the fully connected layer, so that the fully connected layer outputs the occurrence frequency of the key events and keywords corresponding to each microservice instance; the occurrence frequencies corresponding to each microservice instance belonging to the same microservice are aggregated to obtain the instance log aggregate feature vector corresponding to each microservice.

[0115] The log parsing algorithm may adopt the advanced log parsing algorithm Drain.

[0116] And, step 013: inputting the link tracking data of each of the microservice instances into a one-dimensional dilated causal convolutional layer respectively, so that the one-dimensional dilated causal convolutional layer respectively outputs the link tracking feature vector corresponding to each of the microservice instances; and aggregating the link tracking feature vectors corresponding to each of the microservice instances belonging to the same microservice to obtain the instance tracking aggregated feature vector corresponding to each of the microservices.

[0117] Step 014: performing feature fusion on the indicator feature vector, the instance log aggregate feature vector, and the instance tracking aggregate feature vector corresponding to each of the microservices to obtain a fused feature vector corresponding to each of the microservices.

[0118] From the software level, the present application also provides a cloud native system fault root cause location device for executing all or part of the cloud native system fault root cause location method, see Figure 5 The cloud native system fault root cause location device specifically includes the following contents:

[0119] The causal graph construction module 10 is used to determine the abnormal performance indicator data corresponding to each microservice instance and each server based on the performance indicator abnormality scores corresponding to each microservice instance contained in each target microservice in the target time period and the server to which each microservice instance belongs, so as to construct the nodes corresponding to each microservice instance and each server, and to construct the edges between each node according to the fault propagation direction between each abnormal performance indicator data to obtain the corresponding indicator-level causal graph; wherein, the target microservice includes the abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservices corresponding to the abnormal microservice.

[0120] The fault root cause location module 20 is used to normalize the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data to obtain the target anomaly scores corresponding to each of the nodes, and construct a transition probability matrix for representing the transition probability between each of the nodes based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the fault root cause location result data of the cloud native system in the target time period.

[0121] The embodiment of the cloud native system fault root cause locating device provided in this application can be specifically used to execute the processing flow of the embodiment of the cloud native system fault root cause locating method in the above embodiment. Its functions will not be repeated here, and reference can be made to the detailed description of the above embodiment of the cloud native system fault root cause locating method.

[0122] The portion of the cloud native system fault root cause location device that locates the cloud native system fault root cause can be completed in the server or client device. The specific selection can be based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of cloud native system fault root cause location.

[0123] The client device may include a communication module (i.e., a communication unit) that can establish a communication connection with a remote server to implement data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0124] The server and the client device may communicate using any suitable network protocol, including network protocols that have not yet been developed as of the filing date of this application. Examples of such network protocols include TCP / IP, UDP / IP, HTTP, and HTTPS. Furthermore, examples of such network protocols include RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols, which are used on top of the aforementioned protocols.

[0125] From the above description, it can be seen that the cloud native system fault root cause locating device provided in the embodiment of the present application can effectively improve the degree of automation, efficiency and accuracy of cloud native system fault root cause locating, thereby effectively improving the efficiency and reliability of cloud native system fault warning and recovery, and can improve the operational stability of the cloud native system.

[0126] To further illustrate the above-mentioned embodiments of the present application, the present application also provides a specific example of a method for locating the root cause of cloud-native system faults, which is mainly used in the field of fault diagnosis of cloud-native systems to achieve real-time monitoring of abnormal cloud-native system operations and accurate location of the root cause of faults. In response to three key technical bottlenecks in the existing technology: difficulty in adapting to the dynamic changes of microservice instances, insufficient modeling of cloud-native system resource entities, and inefficient modeling, a comprehensive method for locating the root cause of cloud-native system faults has been specially designed and developed to provide engineers with effective diagnostic advice for subsequent rapid troubleshooting.

[0127] This application uses the example of overall fault propagation to deeply study the problem of fault root cause location in cloud native systems and proposes a cross-level fault root cause location framework (Cross-level Failure Root Cause Localization, CFRCL). First, similar to most existing work, in order to avoid redundant processing of monitoring data, this application uses the example of system anomaly detection. The proposed framework aggregates microservice instance monitoring data to the microservice level to effectively cope with the dynamic changes of microservice instances. On this basis, this application uses the example of establishing a connection between instance-level and microservice-level faults: if multiple instances of a microservice show similar anomaly scores, this application uses the example of inferring that the microservice has failed and propagating the fault (Fault Propagation) to its instances. In order to quickly construct an efficient fault propagation causal relationship graph, this application uses the example of anomaly detection results and combined with the deployment relationship of microservice instances to construct a causal skeleton including microservice instances and server nodes, thereby significantly narrowing the scope of potential root causes. Subsequently, this application uses the Granger causal reasoning algorithm to derive the causal relationship between nodes from the causal skeleton. Finally, this application proposes a method for initializing a causal graph and, based on this graph, generating a transition probability matrix for use in implementing a random walk algorithm to locate the root cause of a fault. Furthermore, white-box analysis based on causal graphs and random walks can improve the confidence of maintenance personnel in the results compared to black-box methods based on deep learning.

[0128] This application provides an example of a comprehensive method for locating the root cause of cloud native system faults, which is mainly used in the field of fault diagnosis of cloud native systems. It can realize rapid anomaly detection and root cause localization of cloud native systems, improve monitoring accuracy and risk identification capabilities, reduce potential cost losses, and improve the efficiency of engineers in quickly repairing current faults. Compared with existing fault diagnosis technologies, this application example can respond to dynamic changes in microservice instances in a timely manner, quickly model possible propagation paths of faults, locate the root cause, and accurately identify risks and provide timely warnings. It has the characteristics of comprehensive monitoring, scientific evaluation, and timely warnings, providing strong guarantees for the safe and reliable operation of cloud native systems.

[0129] Cloud-native systems are composed of three types of resource entities: servers, microservices, and microservice instances. Because failures can occur in any resource entity and propagate in a cascading manner, this application requires a customized cloud-native system fault root cause location model based on the overall fault propagation perspective to quickly discover and report failures and their root causes. The process of locating the root cause of a failure in a cloud-native system can be described as follows:

[0130] Assume that the system includes an application consisting of N microservices and a service cluster consisting of H servers. Each microservice has several microservice instances deployed in the service cluster, and the specific number is determined by the workload requirements. Each microservice instance i independently generates log data L i , link trace data T i and instance metric data M i , each microservice I independently generates microservice metric data M I , each server h independently generates server metric data M h .

[0131] In the observation window of length T, this application first uses the microservice-level aggregation feature Detect whether microservice anomalies exist, including Represents the indicator feature vector corresponding to the microservice, Represents the instance tracking aggregate feature vector corresponding to the microservice; Represents the instance log aggregation feature vector corresponding to the microservice. agg() represents the aggregation of corresponding features from all instances of the same microservice. Once an anomaly is detected, this application example aims to i The abnormal values ​​in (i.e., the abnormal performance indicator data corresponding to the microservice instance) and the server indicator data M h The set of abnormal values ​​in (i.e., abnormal performance indicator data corresponding to the server) constructs an indicator-level causal graph G = (V, E). In this indicator-level causal graph, the nodes represent different abnormal performance indicator data, and the edges represent the direction of fault propagation. The goal of the example in this application is to recommend the k most likely root cause entities for the fault by analyzing the indicator-level causal graph, where k is a positive integer. In the example in this application, the microservice instance can be referred to as an instance.

[0132] In this application example, the algorithms used in the cloud native system fault root cause location method include:

[0133] Algorithm 1: Granger Causal Inference Algorithm

[0134] The Granger causal inference algorithm is a statistical hypothesis testing method based on time series data, used to analyze causal relationships between variables. The core idea of ​​this method is: if the historical information of a variable X can significantly improve the future prediction accuracy of another variable Y, then X is considered to be the Granger cause of Y. Specifically, the definition of Granger causality is based on the following assumptions:

[0135] Time series stationarity: The time series being analyzed must be stationary, meaning its statistical properties (such as mean and variance) do not change over time. If the series is non-stationary, it is usually necessary to perform differencing or transformation to make it stationary.

[0136] Causal directionality: Granger causality is a statistical causal relationship, not an actual physical causal relationship. It can only reflect the predictive relationship between variables, but cannot prove the true causal mechanism.

[0137] The specific steps of the Granger causal inference algorithm are as follows:

[0138] (1) Get two time series X t and Y t , where t represents the time point;

[0139] (2) Test the stationarity of the time series (e.g., using the ADF test). If the series is non-stationary, perform a difference or logarithmic transformation.

[0140] (3) Construct two regression models to test whether X is the Granger cause of Y:

[0141] Restricted Model:

[0142]

[0143] The model uses only the historical values ​​of Y to predict the current value of Y; t-i Represents the historical value of Y.

[0144] Unrestricted Model:

[0145]

[0146] This model uses the historical values ​​of both Y and X to predict the current value of Y. Among them, p and q are the lag orders of Y and X, respectively, which are usually determined by the AIC (Akaike Information Criterion) or BIC (Bayesian Information Criterion) criteria; α is a constant term, β i and γ j is the regression coefficient, ∈ t is the error term.

[0147] (4) Use the F test to compare the fitting effect of the restricted model and the unrestricted model:

[0148] Null hypothesis H0: X is not the Granger cause of Y (i.e. all γ j =0);

[0149] Alternative hypothesis H1: X is the Granger cause of Y (i.e., there is at least one γ j ≠0);

[0150] Calculate the statistic F:

[0151]

[0152] RSS R is the residual sum of squares of the restricted model, RSS UR is the residual sum of squares for the unrestricted model.

[0153] (5) Based on the F distribution table, determine whether to reject the null hypothesis. If the F statistic is greater than the critical value, then reject H0 and conclude that X is the Granger cause of Y.

[0154] (6) Bidirectional test: In order to fully analyze the relationship between variables, it is also necessary to test whether Y is the Granger cause of X. That is, swap the roles of X and Y and repeat the above steps.

[0155] Algorithm 2: Random Walk Algorithm

[0156] A random walk is a mathematical model based on random steps, used to describe the process of random movement in a certain space (such as one-dimensional, two-dimensional, or higher-dimensional space). Its core idea is that the direction and step length of each step are random, usually determined by a probability distribution. Each step depends only on the current state and is independent of the previous historical path (Markov property). Take a one-dimensional random walk as an example:

[0157] Let the initial position be x0, and the movement of each step be Δx, which takes the value of +1 or -1 (fixed step size);

[0158] The position of step t is: x t =x t-1 +Δx;

[0159] After n steps, the position is:

[0160] i represents any step among n steps.

[0161] See also Figure 6 The cloud native system fault root cause location method provided by this application consists of four key steps, where Indicator 1 and Indicator 2 refer to different abnormal performance indicators respectively; the details are as follows:

[0162] Step 1: Microservice anomaly detection

[0163] During the microservice anomaly detection phase, graph autoencoders (GAE) are trained based on historical normal data. During the online phase, the GAE model analyzes monitoring data at specified intervals and generates a reconstruction error score for each microservice. If any reconstruction error score exceeds a predefined threshold, root cause identification is triggered.

[0164] Specifically, this application exemplifies the use of anomaly detection at the microservice level based on the understanding of failures, where failures are defined as the inability of a microservice to achieve its intended functionality. This is consistent with actual production needs, where the root cause location phase is triggered by a microservice violating its service level objective (SLO), rather than by a failure of a single instance. For the link tracing data collected by microservice instances, this application exemplifies the use of a one-dimensional dilated causal convolution (DCC) layer, which is an effective method for extracting features from time series data. Specifically, within the prediction window T, this application exemplifies the extraction of link tracing features from the delay in the link tracing data: This application exemplifies the extraction of link tracing features for each microservice instance i, denoted as For the log data collected by the microservice instance, this application uses the advanced log parsing algorithm Drain to parse the log template. In order to standardize the log feature dimensions of all instances, this application identifies the key log events and keywords of each instance, and then uses a fully connected layer to represent their occurrence frequency as a feature vector, which is represented as Next, this application takes the example of aggregating the link tracking and log features of all instances from the same microservice I, which are recorded as and For the indicator data collected by microservices, such as response rate, success rate, etc., this application uses the DCC layer to extract features and record them as Finally, this application uses an example to connect these three types of features into a fusion feature vector F I (T) is used as the input of the GAE model, namely:

[0165] In order to reduce the dependence on labeled data, this application uses unsupervised learning as an example and selects the Graph Autoencoder (GAE) because it is able to process graph data by combining graph neural networks (GNN) with autoencoders. GAE consists of two main components: an encoder and a decoder. This application uses the Graph Attention Network (GAT) as an example to design these two components, using its multi-head attention mechanism to capture node representations from multiple perspectives. The encoder extracts microservice features and dependencies and embeds them into dense latent vectors. The decoder is a mirror structure of the encoder and is used to reconstruct the original features. The model is trained to minimize reconstruction errors and ensure that the output features closely match the input features. Unlike most anomaly detection models that operate at the system level, the GAE model is able to detect anomalies at the microservice level.

[0166] Because the system is generally stable, the GAE model is trained on historical normal data to ensure robustness. During testing, faulty data causes the reconstruction error score to increase, reflecting the degree of deviation from the normal pattern. If this reconstruction error score exceeds a predefined threshold, the root cause identification process is triggered. The GAE model generates a ranked dictionary Dict_1 of standardized reconstruction error scores for all microservices, represented as Dict_1 = {serv1:sco1,serv2:sco2,...}. Microservices with reconstruction error scores exceeding a certain threshold are marked as anomalies, providing a preliminary estimate of the potential root cause of the fault. serv1 and serv2 represent different microservices; sco1 and sco2 represent different reconstruction error scores.

[0167] Step 2: Causal Skeleton Extraction

[0168] The first step in root cause location in the cross-level fault root cause location framework CFRCL is causal skeleton extraction ( Figure 7 In lines 2-20 of the causal skeleton extraction phase, all microservices that maintain call relationships with the anomalous microservice and the server node where the anomalous microservice instance resides are identified. Furthermore, the server hosting the anomalous microservice instance is analyzed to examine the anomaly. Constructing the causal skeleton (including the identified instances, servers, and their anomaly indicators) is intended to narrow down the potential root causes and marks the first key difference from other causal inference-based approaches.

[0169] Previous research has shown that the root cause of a fault may exhibit a lower degree of abnormality than the affected entity, and therefore the root cause may not be ranked high in the GAE model output. To address this issue, this application example first identifies microservices from Dict_1 whose reconstruction error scores exceed a predefined threshold. Next, this application example identifies the microservices from the triggered prediction window ( Figure 7Extract all microservices that have call relationships with these abnormal microservices from the link tracking data in lines 2-7 of the . This ensures that if the root cause is located in the call chain related to the fault, it is still within the scope of analysis of this application instance. Figure 7 In the example, G represents the causal skeleton, G.node represents the entity node (instance / server), G.node.value represents the indicator, G.edge represents the causal edge, and G.edge.weight represents the transition probability; server_list represents the abnormal server list; v i and v j Represents two entity nodes connected by a causal edge, namely G.node; Represents a metric of an instance of a microservice; represents the j-th metric of server i.

[0170] Additionally, the server needs to be included in the analysis ( Figure 7 To include the root cause while deriving the most concise causal skeleton possible, this application example analyzes server metric data to determine whether the server hosting the abnormal instance also exhibits abnormalities.

[0171] Specifically, this application example uses the data within the prediction window where historical root cause location was not triggered as normal data to calculate the anomaly of each metric for each server:

[0172]

[0173] Among them, E(metr i,j (T)) represents the average value of the jth indicator of the i-th entity within the prediction window T, and Indicates metr i,j The average value and variance of historical normal data, abs() means taking the absolute value operation. ′ >1: If the server indicator abnormality exceeds 1 standard deviation of the normal data, the indicator is considered abnormal.

[0174] For cloud providers of cloud native systems, the deployment location information of microservice instances is easy to obtain. This application example directly evaluates the servers hosting abnormal instances, and if any of their indicators show abnormalities, this application example will include them in the causal framework. This application example will use sco ′ Get the ranking dictionary of abnormal server indicators, i.e. This ensures that the causal skeleton includes the potential root cause server. In addition, the application example also analyzes other instances ( Figure 719-19 in the metric), these instances may use the server as a "springboard" to propagate failures. Specifically, for example, the metric adopts the same process as described in equation (1) to calculate the anomaly, that is, The indicator anomaly score for a microservice instance is defined as:

[0175]

[0176] Among them, J represents microservice serv i The number of all instances of , sco_i comes from Dict_1. Then, this application application instance constructs a ranking dictionary of instance indicators based on the instance scores ( Figure 7 Line 20 in ), i.e. Ultimately, the identified instances and servers and their corresponding metrics together form the final causal skeleton.

[0177] Step 3: Causal graph learning

[0178] During the causal graph learning phase, this application example uses the Granger causal inference algorithm and expert knowledge to mine the causal relationships within the skeleton, thereby generating a causal graph. By focusing only on entities related to the fault, this approach improves efficiency compared to analyzing the entire system.

[0179] At this stage, the application example of this application only focuses on the indicators of entities in the causal skeleton G ( Figure 7 This targeted approach greatly improves the efficiency of constructing the final causal graph. This application example observes four categories of performance indicators: CPU utilization (M C ), memory utilization (M M ), disk utilization (M D ) and delay (M L Note that latency metrics are unique to microservice instances and represent not only network-related failures but also the combined impact of all resource metrics. By analyzing the causal relationships between latency metrics, this application example can determine the direction of fault propagation between instances. The metrics in Dict_2 and Dict_3 are mapped to the corresponding nodes in the causal skeleton to construct an undirected causal graph.

[0180] For the causal relationship within an instance, this application example uses expert knowledge based on previous research to directly obtain the causal relationship between the resource indicators of each microservice instance, such as Figure 8 In order to determine the causal relationship between instances, this application focuses on instances with call relationships and uses the causal reasoning algorithm Granger to determine the causal relationship between instances.

[0181] Finally, to capture the causal relationship between instances and servers, this application applies instances to analyze the fault propagation reflected in their corresponding resource metrics. Granger testing is again used to identify the causal relationship between abnormal server metrics and their related instance metrics.

[0182] Step 4: Locate the root cause of the fault

[0183] Finally, during the root cause location phase, this application requires initializing the causal graph node scores to construct a random walk transition probability matrix. Starting from the node with the highest anomaly score, CFRCL applies a random walk algorithm to generate a ranked list of potential root causes.

[0184] To identify the root cause, this application example implements a random walk algorithm, a statistical scoring mechanism that simulates fault propagation through random trajectories. Before applying the algorithm to the causal graph, this application example initializes the node scores to construct the required transition probability matrix ( Figure 7 Specifically, we apply the example to uniformly normalize the scores in Dict_2 and Dict_3 and use the updated score, called the anomaly score (AS), to initialize the scores of nodes in the causal graph, marking the second key difference from other causal inference-based methods.

[0185] Next, the application example of this application defines an initial transition probability matrix P, where each element represents the possibility of transitioning between two indicator nodes. Given two indicator time series i and j, represented as node v in the causal graph i and v j , where there is a node v due to the index j To the result indicator node v i The edge of v i to v j The transition probability is calculated as follows:

[0186] Forward step (transfer from the result indicator node to the cause indicator node):

[0187] This application application example defines v i Transfer to v j Probability of:

[0188]

[0189] Among them, correlation() means calculating the Pearson correlation coefficient of two variables, and α∈[0,1] controls v i and v j The contribution weight of the correlation coefficient to the transition probability, K represents all the dependent indicators of indicator i.

[0190] Backward step (transfer from the cause indicator node to the result indicator node):

[0191] To prevent the algorithm from being trapped in nodes with low correlation with the abnormal node, the random walk is allowed to move from the cause node to the result node with the transition probability:

[0192] P j,i =β·P i,j Formula (4)

[0193] Where β∈[0,1] represents the probability of executing the Backward step, which is a hyperparameter and is taken as 0.2 in this paper.

[0194] Self step (stay at the current node):

[0195] If the algorithm reaches the indicator node and the probability of transitioning to the adjacent node is low, then this indicator may be the root cause. Then the walker should stay at this indicator node, and the probability of staying there is:

[0196] P i,i =max [0,P i,i -max k∈K P i,k ] Formula (5)

[0197] Finally, this application example obtains by normalizing each row of P

[0198] The random walk algorithm starts from the node with the highest anomaly score and moves to the node according to the final probability transition matrix. Random walk n steps, record the visit frequency of each node. Then, these frequencies will be sorted in descending order to generate the root cause location results.

[0199] That is to say, the application example of this application conducts a detailed analysis of the fault propagation between resource entities at different levels, models the fault propagation between microservice instances and between microservice instances and servers as a causal graph model, and provides a probability transfer matrix for the random walk algorithm through a novel causal graph initialization method to achieve a practical and efficient solution for cross-level fault root cause location.

[0200] The causal skeleton extraction method is one of the key technologies in the application example of this application. This method integrates the instances and related servers obtained in the anomaly detection phase into the causal graph learning phase, significantly narrowing the scope of potential root causes and thus improving the efficiency of root cause analysis.

[0201] The node initialization method in the causal graph is another key technology in the application example of this application. This method integrates the anomaly detection results, which helps to build a more accurate transition probability matrix for the random walk algorithm, thereby improving the accuracy of root cause location.

[0202] The present application also provides an electronic device that may include a processor, a memory, a receiver, and a transmitter. The processor is configured to execute the cloud-native system fault root cause location method described in the above embodiment. The processor and the memory may be connected via a bus or other means, with bus connection being used as an example. The receiver may be connected to the processor and the memory via a wired or wireless means.

[0203] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0204] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer executable programs, and modules, such as the program instructions / modules corresponding to the cloud-native system fault root cause location method in the embodiments of this application. The processor executes the non-transitory software programs, instructions, and modules stored in memory to perform various processor functions and data processing, thereby implementing the cloud-native system fault root cause location method in the aforementioned method embodiments.

[0205] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory may optionally include a memory remotely located relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0206] The one or more modules are stored in the memory, and when executed by the processor, perform the cloud native system fault root cause locating method in the embodiment.

[0207] In some embodiments of the present application, the user equipment may include a processor, a memory and a transceiver unit, and the transceiver unit may include a receiver and a transmitter. The processor, memory, receiver and transmitter may be connected through a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0208] As an implementation method, the functions of the receiver and transmitter in this application can be considered to be implemented through a transceiver circuit or a dedicated transceiver chip, and the processor can be considered to be implemented through a dedicated processing chip, a processing circuit or a general-purpose chip.

[0209] As another implementation method, it is possible to use a general-purpose computer to implement the server provided in the embodiments of the present application. That is, the program code for implementing the functions of the processor, receiver, and transmitter is stored in a memory, and the general-purpose processor implements the functions of the processor, receiver, and transmitter by executing the code in the memory.

[0210] The present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the aforementioned cloud-native system fault root cause location method. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, floppy disk, hard disk, removable storage disk, CD-ROM, or any other form of storage medium known in the art.

[0211] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the aforementioned cloud native system fault root cause locating method.

[0212] It should be understood by those skilled in the art that the various exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is specifically performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of this application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0213] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0214] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0215] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art will appreciate that various modifications and variations of the present embodiment are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for locating the root cause of a cloud native system failure, characterized in that: include: Based on the performance indicator anomaly scores corresponding to each microservice instance contained in each target microservice in the target time period and the server to which each microservice instance belongs, determine the abnormal performance indicator data corresponding to each microservice instance and each server to construct nodes corresponding to each microservice instance and each server, and construct edges between each node according to the fault propagation direction between each abnormal performance indicator data to obtain a corresponding indicator-level causal graph; wherein the target microservice includes the abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservices corresponding to the abnormal microservice; The performance indicator anomaly scores corresponding to each of the abnormal performance indicator data are normalized to obtain the target anomaly scores corresponding to each of the nodes, and a transition probability matrix representing the transition probability between each of the nodes is constructed based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the root cause location result data of the cloud native system in the target time period.

2. The cloud native system fault root cause location method according to claim 1, characterized in that: The abnormal performance indicator scores corresponding to each microservice instance and each server to which each microservice instance belongs, respectively, included in each target microservice in the target time period, and determining abnormal performance indicator data corresponding to each microservice instance and each server, including: Determine, based on the performance indicator data of each microservice instance included in each target microservice within the target time period, the performance indicator anomaly score of the performance indicator data corresponding to each microservice instance; and, determining, based on the performance indicator data of the servers to which the respective microservice instances belong within the target time period, the performance indicator anomaly scores of the performance indicator data corresponding to the respective servers; Among the performance indicator data corresponding to each of the microservice instances and each of the servers, determining the performance indicator data having a performance indicator anomaly score greater than a first threshold as abnormal performance indicator data for the target time period; The server corresponding to the abnormal performance indicator data is determined as the current abnormal server. If the abnormal server includes a microservice instance for which the performance indicator abnormality score has not been determined, the performance indicator abnormality score of the performance indicator data corresponding to the microservice instance is determined based on the performance indicator data of the microservice instance within the target time period. If the performance indicator abnormality score corresponding to the microservice instance is greater than the first threshold, the performance indicator data corresponding to the microservice instance is determined as the abnormal performance indicator data for the target time period.

3. The method for locating the root cause of a cloud native system failure according to claim 1, wherein: The constructing edges between the nodes according to the fault propagation directions between the abnormal performance indicator data includes: For the abnormal performance indicator data corresponding to the nodes corresponding to each of the microservice instances, determine the fault propagation direction between each type of abnormal performance indicator data according to preset expert indication data, so as to construct edges between the nodes corresponding to each of the microservice instances; wherein the types of abnormal performance indicator data include: CPU utilization, memory utilization, disk utilization, and latency; For each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship, determine the fault propagation direction between each type of abnormal performance indicator data corresponding to each of the different microservice instances that are used to represent the existence of a call relationship based on the Granger causality test algorithm, so as to construct edges between each node corresponding to each of the different microservice instances that are used to represent the existence of a call relationship; For each node representing the server and the microservice instance, respectively, determining the fault propagation direction between each type of abnormal performance indicator data corresponding to each server and the microservice instance based on the Granger causality test algorithm, so as to construct edges between each node representing the server and the microservice instance; Wherein, one of the nodes is used to represent the name corresponding to one of the microservice instances or one of the servers, the abnormal performance indicator data, and the target abnormality score; wherein, the abnormal performance indicator data contains the value of the abnormal performance indicator and the indicator name used to indicate the type of the abnormal performance indicator; If the similarity between the target anomaly scores corresponding to multiple microservice instances corresponding to the microservice is greater than a similarity threshold, it is predicted that the microservice has a fault and the fault is propagated to the microservice instance corresponding to the microservice.

4. The method for locating the root cause of a cloud native system failure according to claim 1, wherein: The constructing a transition probability matrix for representing transition probabilities between the nodes according to the target anomaly scores corresponding to the nodes includes: A transition probability matrix is ​​defined, wherein each element in the transition probability matrix is ​​used to represent a transition probability between two different nodes; Determine, according to the target anomaly scores corresponding to the two nodes respectively corresponding to each element in the transition probability matrix, a transition probability of forward transfer between the two nodes respectively corresponding to each element, and determine, based on the transition probability of forward transfer between the two nodes respectively corresponding to each element, a transition probability of backward transfer and a transition probability of stay between the two nodes respectively corresponding to each element, so as to complete the assignment of the transition probability matrix; The assigned transfer probability matrix is ​​normalized.

5. The method for locating the root cause of a cloud native system failure according to claim 4, wherein: The step of obtaining the access frequency corresponding to each of the nodes in the indicator-level causal graph using a random walk algorithm based on the transition probability matrix to obtain the root cause location result data of the cloud native system in the target time period includes: Starting from the node with the highest target anomaly score in the indicator-level causal graph, based on the normalized transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph; The nodes are sorted in descending order of access frequency, and the obtained sorting result data is used as the root cause location result data of the cloud native system in the target time period.

6. The method for locating the root cause of a cloud native system fault according to any one of claims 1 to 5, characterized in that: Before determining the abnormal performance indicator data corresponding to each of the microservice instances and each of the servers, the method further includes: Extracting a fusion feature vector corresponding to each microservice based on the performance indicator data of each microservice in the cloud native system within the target time period, and the log data and link tracking data of each microservice instance corresponding to each microservice within the target time period; Inputting the fused feature vectors corresponding to each microservice into a graph autoencoder respectively, so that the graph autoencoder outputs the reconstruction error scores corresponding to each microservice respectively; Determine whether any reconstruction error score corresponding to each of the microservices has a reconstruction error score greater than a second threshold. If so, determine the microservice with the reconstruction error score greater than the second threshold as an abnormal microservice within the target time period, and use the abnormal microservice and the associated microservice corresponding to the abnormal microservice as the target microservice.

7. The cloud native system fault root cause location method according to claim 6, characterized in that: Extracting the fused feature vector corresponding to each microservice based on the performance indicator data of each microservice in the cloud native system within the target time period, and the log data and link tracking data of each microservice instance corresponding to each microservice within the target time period, includes: Inputting the performance indicator data of each of the microservices in the cloud native system within the target time period into a one-dimensional dilated causal convolutional layer, so that the one-dimensional dilated causal convolutional layer outputs an indicator feature vector corresponding to each of the microservices; Based on the log parsing algorithm, the log data of each microservice instance corresponding to each microservice within the target time period is parsed, and the key events and keywords corresponding to each microservice instance are determined; the parsed log data corresponding to each microservice instance are respectively input into the fully connected layer, so that the fully connected layer outputs the occurrence frequency of the key events and keywords corresponding to each microservice instance; the occurrence frequencies corresponding to each microservice instance belonging to the same microservice are aggregated to obtain the instance log aggregate feature vector corresponding to each microservice; Furthermore, the link tracking data of each microservice instance is input into a one-dimensional dilated causal convolutional layer, so that the one-dimensional dilated causal convolutional layer outputs a link tracking feature vector corresponding to each microservice instance; and the link tracking feature vectors corresponding to each microservice instance belonging to the same microservice are aggregated to obtain an instance tracking aggregate feature vector corresponding to each microservice. Feature fusion is performed on the indicator feature vector, the instance log aggregate feature vector, and the instance tracking aggregate feature vector corresponding to each of the microservices to obtain a fused feature vector corresponding to each of the microservices.

8. A device for locating the root cause of a cloud native system failure, characterized in that: include: A causal graph construction module is used to determine the abnormal performance indicator data corresponding to each microservice instance and each server in the target microservice within the target time period, based on the performance indicator anomaly scores corresponding to each microservice instance and each server to which each microservice instance belongs, to construct nodes corresponding to each microservice instance and each server, and to construct edges between each node according to the fault propagation direction between each abnormal performance indicator data, to obtain a corresponding indicator-level causal graph; wherein the target microservice includes an abnormal microservice within the target time period pre-detected in the cloud native system and the associated microservices corresponding to the abnormal microservice; The fault root cause location module is used to normalize the performance indicator anomaly scores corresponding to each of the abnormal performance indicator data to obtain the target anomaly scores corresponding to each of the nodes, and construct a transition probability matrix for representing the transition probability between each of the nodes based on the target anomaly scores corresponding to each of the nodes; based on the transition probability matrix, a random walk algorithm is used to obtain the access frequency corresponding to each of the nodes in the indicator-level causal graph to obtain the fault root cause location result data of the cloud native system in the target time period.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for locating the root cause of a cloud-native system fault according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for locating the root cause of a cloud-native system fault as described in any one of claims 1 to 7 is implemented.

Images