A method for locating the root cause of failures in microservice applications

Abstract

The present invention provides a method for locating the root cause of a fault of a microservice application program, comprising: step 1, collecting various indicator data information from a monitoring system at regular intervals by calling an interface; step 2, checking the validity of the acquired sKPI time series data, if the data is invalid, discarding the data at the time point or reacquiring the data; if the data is valid, entering step 3; step 3, using an anomaly detector EXPoSE based on a kernel function to detect the sKPI anomaly of ESB online, and obtaining the occurrence time T_fault of the fault; step 4, calculating the wasserstein distance of the call time consumption between network element nodes, and extracting the mutation characteristics of mKPI; step 5, using a random forest association model to predict the root cause node and the root cause mKPI.

Description

Technical Field

[0001] The present invention relates to the technical field of fault root cause locating, and in particular to a fault root cause locating method for a microservice application program. Background Art

[0002] In order to ensure the reliability of the microservice system, the monitoring system regularly collects two types of indicators. Service-level indicators sKPI, such as call duration; machine-level performance indicators mKPI, such as CPU usage, network send queue and other indicators. These monitoring indicators help engineers diagnose performance and locate the root cause of the failure when the system fails. It is difficult to locate the root cause of the failure in real scenarios. Due to the large number of microservice system modules, there are complex interactions between the components of the system, and the microservice system has many different reasons for failure, which can be divided into external failures and internal failures. The former includes hardware failures, network crashes, resource overload, etc., and the latter mainly refers to software failures.

[0003] See also Figure 1 , the figure shows the traditional method for locating the root cause of microservice failures, including the following steps: 1. Determine the fault time T_fault by testing a single sKPI sequence through 3-sigma (data is outside μ-3σ, μ+3σ, σ represents the standard deviation, and μ represents the mean); 2. Determine whether it is a database (db) error near T_fault, if so, output the db fault network element and fault indicator; 3. Determine the call type (callType), if it is a remote call, output the operating system (OS) type fault network element and fault indicator of serial number 1-10; 4. Determine whether it is a CPU type fault of the container, if so, output the docker type fault, otherwise output the OS type network element and fault indicator of serial number 11-16. Existing root cause location methods are mainly based on the experience of operation and maintenance personnel and threshold methods. There are also some clustering deep learning methods based on historical data, which are time-consuming and cannot adapt to system changes. Some methods can only locate the root cause of coarse-grained network elements. In traditional methods, the mutual influence of machine indicators mKPI under the network element node will lead to too high false positive rate (FPrate) in the judgment of the fault root cause KPI. Summary of the invention

[0004] Purpose of the invention: The purpose of the invention is to provide a solution for automatic fault detection and root cause location for microservice systems, and to provide reasonable explanation for the solution. The invention detects in real time, improves the detection speed, and has a finer detection granularity.

[0005] The technical problem to be solved by the present invention can be achieved by adopting the following technical solutions:

[0006] A method for locating the root cause of a fault in a microservice application (the microservice architecture has been adopted by many Internet companies. It consists of many independent applications that are distributed on multiple machines and communicate through the network. A user operation may involve multiple microservices deployed on different network nodes, resulting in complex call relationships), including the following steps:

[0007] Step 1: collect various indicator data information from the monitoring system every time X (generally 1 minute) by calling the interface. The different indicator data information includes the overall service level indicator sKPI, the time-consuming data M between network element nodes, and the time-consuming data M between network element nodes. i，j (r), and machine-level performance indicator data mKPI;

[0008] Step 2: Use hive to aggregate and count the overall service-level indicator sKPI of the statistical system, such as the average service delay within 1 minute, the average number of calls, and other time series data, and perform real-time validity checks on the data at the current time t. If the data is invalid, the data at the current time t is discarded or the data is retrieved again to obtain the value vector x of the overall service-level indicator sKPI at the time t. t ; If the data is detected as valid, save the data x t , go to step 3;

[0009] Step 3: Use the kernel function-based anomaly detector EXPoSE to detect all service-level indicators sKPI anomalies of the system bus online and obtain the occurrence time T of the fault. fault ;

[0010] Step 4: Calculate the wasserstein distance of the call time between network element nodes and extract the mutation characteristics of the machine performance level (mKPI); the higher the distance, the more obvious the change in the state of the node before and after the failure, so it is more likely to be a failure. This step can preliminarily determine the faulty node. The mutation characteristics include the mutation point, mutation gradient, mutation time, false alarm coefficient and other characteristics of each time series. Generally, the earlier the mutation occurs, the more likely it is to be the root cause sequence.

[0011] Step 5: Use the random forest association model to predict the root cause node and root cause mKPI. The machine-level performance indicator data mKPI is a sequence, and the root cause mKPI refers to the sequence where the fault occurs in all machine-level performance indicator data mKPIs.

[0012] Step 3 includes:

[0013] The anomaly score s is calculated by the following formula t :

[0014]

[0015] It is x t Perform feature mapping, all mappings are based on Gaussian kernel function (rbf sampler).

[0016] μ[P t ] is the kernel mean mapping, which embeds the probability distribution of normal data into the reproducing kernel Hilbert space, Use the empirical distribution to approximate the mean.

[0017] According to the obtained anomaly score, a threshold θ is set. When the anomaly scores at time t and t+1 are both greater than the threshold θ, time t is judged as the time when the fault occurred. Of course, this specific threshold can be set according to the business situation. In this case, 0.1 is used because the anomaly score is basically less than 0.01 under other normal circumstances.

[0018] In step 4, the bulldozer distance of the network element node is calculated using the following method:

[0019] Each time the fault is predicted to occur, T fault Then, start calculating:

[0020] Set t = T fault , set the window w, for network element nodes i and j:

[0021] Set for all x

[0022] Among them, x represents the time consumption data M between network element nodes in the time window before the failure time t i，j (r), x is a random variable; r is the start time of the time window before the failure time t; Represents the distribution of random variables of network element node time consumption; Represents the indicator function of the random variable x;

[0023]

[0024] Indicates the distribution of node time consumption in the time window after the failure;

[0025] Calculate the bulldozer distance s between network element nodes i and j i，j (T fault ):

[0026]

[0027] Where Π(P, Q) represents the set of all possible joint distributions, and the joint distribution satisfies the marginal distributions P and Q; for each possible joint distribution r, sample (x, y), calculate the distance and find the expected value inf represents the lower limit of this expected value in all possible joint distributions; i∈N, j∈N, N is the full set of network element nodes, and i and j can be the same. x represents the distribution from The sampling time of the network element node is obtained from The sampling time of the network element node is as follows:

[0028] In step 4, the extraction of mutation features of machine performance level features specifically includes: setting a mutation time window [T fault -w1,T fault +w2];

[0029] w1 is the length of the backward time window, and w2 is the length of the forward time window. This time window is the part of the window where mKPI is most likely to affect the change of sKPI;

[0030] set up x represents the value of machine-level performance indicator data mKPI, x * represents the normalized value, is the mean within the mutation time window, std(x) is the standard deviation of x;

[0031] At each time t, set Represents the sequence gradient value at time t; Represents the normalized value at the next moment;

[0032] Then follow these steps:

[0033] Step 4-1: Set T s =1,T s is the time when the mutation occurred;

[0034] Step 4-2: Traverse t=w2-1:w1 (indicates time t from w1 to w2), if Setting T s =t, exit the loop;

[0035] After the cycle of step 4-3 is completed, the earliest time T that exceeds three times the standard deviation gradient is obtained. s With gradient is the gradient at the moment when the mutation occurs.

[0036] Step 5 includes: training the random forest association model, using the integrated learning method to automatically learn the judgment rules of the fault node and the fault root cause mKPI; when judging the root cause in real time, inputting the mutation characteristics of the bulldozer distance characteristics and the machine performance level characteristics between the network element nodes into the random forest association model to predict the fault root cause node and the root cause.

[0037] In step 5, the fault root cause node H1(x) is predicted according to the following formula:

[0038]

[0039] Where x is the time consumption distribution characteristic of all nodes, h i (x) represents each trained decision tree when calling the node time-consuming distribution feature;

[0040] In step 5, the root cause H2(y) is predicted according to the following formula:

[0041]

[0042] Among them, y is the mutation feature calculated by all machine-level performance indicators mKPI and the previous possible root cause node H1(x), h i (y) represents each trained decision tree when calling other features (such as standard deviation and gradient value) except for the node time consumption distribution feature. Because the type of the fault root cause node H1(x) will affect the judgment of the root cause mKPI, for example, the failure of the database DB node will be reflected in the performance indicators of the DB.

[0043] In a preferred embodiment of the present invention, the collected index data information includes the average call time of the system bus service, all call time data of 33 nodes, 51 host (OS) indicators, 46 database (DB) indicators, and 9 container (Docker) indicators. They mainly include CPU usage, number of network requests, etc. The present invention can be used to process hundreds of indicators in the entire complex system, and can associate various parts of the system for real-time intelligent analysis.

[0044] Due to the adoption of the above technical solution, the beneficial effects of the present invention are: the method realizes the real-time detection of microservice system failures and can quickly locate the root cause with low time complexity. The bulldozer distance (wasserstein distance) is introduced to quantify the abnormality of the faulty node and preliminarily find the faulty network element node. At the same time, the method locates the root cause indicator by extracting the mutation characteristics of the machine-level performance indicators of each node, completing the challenge of reducing the false positive rate and improving the accuracy. Compared with the traditional method, this method is very effective, with an F1 value of up to 0.833, far exceeding the traditional method of 0.506. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0046] Figure 1 This is a flowchart of the traditional method for locating the root cause of a fault in microservices.

[0047] Figure 2It is a flow chart of the method for locating the root cause of a fault of a microservice according to the present invention.

[0048] Figure 3 Schematic diagram of three model architectures based on random forest. DETAILED DESCRIPTION

[0049] The present invention provides a method for locating the root cause of a fault in a microservice application, comprising the following steps:

[0050] Step 1: Collect different indicator data information from the monitoring system every X time interval (generally 1 minute) by calling the interface aggregation calculation method; the data is mainly service-level indicator data (sKPI), average service delay per minute, average number of calls, average response time, and average number of successful calls. Machine-level performance indicator data (mKPI) has a lot of data, involving host indicators (OS), database indicators (DB), and container indicators (Docker).

[0051] Step 2: Obtain the overall service level indicator (sKPI) through the aggregated statistical system bus (ESB), perform real-time validity check on the data at the current time t to see if the data format is (t, value). If the data is invalid, remove the data at the current time t or re-acquire the data; if the data is valid, proceed to step 3.

[0052] Step 3: Use the kernel function-based anomaly detector EXPoSE to detect all service-level indicators sKPI anomalies of the system bus online and score them. When the score exceeds the threshold for two consecutive moments (the threshold is selected as 0.1 in the experiment, because the score of the detector for stable normal data is very small, with an average score of 0.01), the fault occurrence time T is obtained. fault ;

[0053] Step 4: Calculate the bulldozer distance (wasserstein) of the network element node based on the time consumption distribution and extract the mutation characteristics of the machine performance level (mKPI); the higher the distance, the more obvious the state change of the node before and after the failure, so it is more likely to be a failure. This step can preliminarily derive the faulty node. The mutation characteristics include the mutation point, mutation gradient, mutation time, false alarm coefficient and other characteristics of each time series. Generally, the earlier the mutation occurs, the more likely it is to be the root cause sequence.

[0054] Step 5: Use the random forest association model to predict the root cause nodes and root cause mKPIs.

[0055] Step 3 includes: let x t Represents the value vector of all service-level indicators sKPI at time t;

[0056] The anomaly score s is calculated by the following formulat :

[0057]

[0058] It is x t Perform feature mapping, Approximated by RBFSampler;

[0059] μ[P t ] is the kernel mean mapping, which embeds the probability distribution of normal data into the reproducing kernel Hilbert space, Use the empirical distribution to approximate the mean. Set the relevant parameters of the RBFSampler kernel mean mapping to gamma = 0.4, dimension = 2000, decay = 0.01.

[0060] According to the obtained anomaly score, a threshold θ is set. When the anomaly scores at time t and t+1 are both greater than the threshold θ, time t is judged as the time when the fault occurred. Of course, this specific threshold can be set according to the business situation. In this case, 0.1 is used because the anomaly score is basically less than 0.01 under other normal circumstances.

[0061] In step 4, the bulldozer distance of the network element node is calculated using the following method:

[0062] Each time the fault is predicted to occur, T fault Then, start calculating:

[0063] Set t = T fault , set the window w, for network element nodes i and j:

[0064] Set for all x

[0065] Where x represents the time M consumed by the network element node in the time window before the failure time t i，j (r) This random variable; r is the start time of the time window before the failure time t; Represents the distribution of random variables of network element node time consumption; Represents the indicator function of the random variable x;

[0066] Set for all x

[0067] Indicates the distribution of node time consumption in the time window after the failure;

[0068] Calculate the bulldozer distance s between network element nodes i and j i，j (T fault ):

[0069]

[0070] Where Π(P, Q) represents the set of all possible joint distributions, and the joint distribution satisfies the marginal distributions P and Q; for each possible joint distribution r, sample (x, y), calculate the distance and find the expected value inf represents the lower limit of this expected value in all possible joint distributions; i∈N, j∈N, N is the complete set of network element nodes, and i and j can be the same.

[0071] In step 4, the extraction of mutation features of machine performance level features specifically includes: setting a mutation time window [T fault -w1,T fault +w2];

[0072] w1 is the length of the backward time window, w2 is the length of the forward time window, and this time window is the window part where mKPI is most likely to affect the change of sKPI; the values ​​of w1 and w2 in the present invention are +-1min, and in another embodiment, 1min is taken here, and 10min is also acceptable.

[0073] set up x represents the value of the sequence mKPI, x * represents the normalized value, is the mean within the mutation time window, std(x) is the standard deviation of x;

[0074] At each time t, set Represents the sequence gradient value at time t; Indicates the normalized value at the next moment.

[0075] Then follow these steps:

[0076] Step 4-1: Set T s =1,T s is the time when the mutation occurred;

[0077] Step 4-2: When traversing t = w2-1: w1 (indicates w1 from to w2), if Setting T s =t, exit the loop;

[0078] After the cycle of step 4-3 is completed, the earliest time T that exceeds three times the standard deviation gradient is obtained. s With gradient is the gradient at the moment when the mutation occurs.

[0079] Step 5 includes: training the random forest association model, using the ensemble learning method to automatically learn the judgment rules of the fault node and the fault root cause mKPI; when judging the root cause in real time, all features, namely the bulldozer distance features between network element nodes obtained above and the mutation features of each mKPI, are input into the random forest association model to predict the fault root cause node and the root cause mKPI respectively. The final parameters of the random forest model are: the number of random forest trees n estimators = 600, the maximum depth limit of the tree max depth = 6, and the features selected by sampling each tree each time and max faatures = 5.

[0080] Example

[0081] The present invention proposes a real-time root cause location method, MicroRootLocator. The anomaly detector first detects the anomaly of sKPI (service-level key performance indicator) online and obtains the start time of the fault. Then the algorithm novelly calculates the wasserstein distance of the node running time change and extracts the mutation features of mKPI (machine-level key performance indicator combinations) based on the fault time. Finally, all features are input into the random forest association model for fine-grained root cause prediction.

[0082] See also Figure 2 , the figure shows a method for locating the root cause of a fault in a microservice application, which includes the following steps:

[0083] Step S10, collect different indicator data information from the monitoring system at each time period by calling the interface. The collected data includes the average call time of the sKPI system bus service, all call time data of 33 nodes (i.e. sKPI), and mKPI includes 51 host performance indicators, 46 database performance indicators, and 9 container performance indicators.

[0084] Step S20, verify the validity of the acquired sKPI time series data; if the data is invalid, discard the data at that time point or re-acquire the data; if the data is valid, proceed to step S30.

[0085] Step S30 uses the kernel function-based anomaly detector EXPoSE to detect sKPI anomalies online and obtain the start time T_fault of the fault. EXPoSE (EXPected Similarity Estimation) is an unsupervised anomaly detector for real-time detection of sKPI anomalies. This method quickly calculates the anomaly score for the current point of the time series in a fixed time and fixed memory. It is based on the kernel function and has no prior assumptions about the data distribution. From a practical application point of view, the detector is more effective than manual judgment or rule judgment and has interpretability.

[0086] Step S40, calculating the wasserstein distance of the network element node and extracting the mutation feature of the mKPI.

[0087] Abnormal score of call time between nodes. In a microservice system, a user service request involves multiple microservices. At the same time, many users initiate a large number of service requests, bringing a large amount of tracking data. In a high-concurrency scenario, the time consumption of nodes related to the fault changes greatly. Therefore, the focus is on the change in call time consumption to quantify the degree of abnormality of the network element node when the fault occurs. The tracking consumption time between the caller node i and the server node j at time t is represented by M_(i,j)(t), and when the fault occurs at T_fault, the M_(i,j)(t) of the faulty node and its related nodes always increases. The state change before and after the fault is very important for fault diagnosis. This distribution change is quantitatively represented by the abnormal score s_(i,j)(T_fault). The wasserstein distance in the statistical distance is introduced to calculate the abnormal score s_(i,j)(T_fault). The wasserstein distance measures the deviation from one probability distribution to another probability distribution. In the scenario of the present invention, it is the difference between the empirical distributions of the samples collected before and after the fault. This step can preliminarily analyze the root cause node set of the fault for reference by engineers.

[0088] Then this method mainly analyzes three types of mKPIs: docker, OS, and database mKPIs. The mKPI data contains various information about the machine's operating status, including CPU, memory, network interface, file system, process and other indicators. Analyzing these mKPIs can find the cause of the machine failure. All containers deployed on the same host are prone to mutual influence due to network data communication, so their mKPIs are also related. Once the root cause mKPI fails, it will affect other related mKPIs. Therefore, when locating the root cause, it is necessary to exclude many false positive mKPIs, that is, to avoid false alarms.

[0089] There are two main modes of mKPI: stable mode and trend mode. For the stable mode, the mKPI mainly oscillates around a certain central value. For the trend mode, the mKPI grows steadily throughout the day, and the trend can be considered linear. For abnormal mKPI, the abnormal mode is mainly a sudden and rapid change in value. But in turn, a sudden and rapid change in the value of an indicator does not necessarily mean that the mKPI is the root cause of the abnormality, sometimes it is even a normal change. 3 main features are extracted from the mKPI time series: change start time (T_s), change gradient (C_g), and false alarm coefficient (FC). These three features will help discover abnormal mKPIs, infer dependencies, and avoid false alarms.

[0090] Step S50: The random forest association model predicts the root cause node and the root cause mKPI.

[0091] In a preferred embodiment of the present invention, the cause of the fault is found through two interrelated models, while reducing the spatial complexity of the root cause indicator search, which realizes the function of efficient dual-layer root cause location.

[0092] The specific actual data calculation example is shown in Table 1 below:

[0093] Table 1

[0094]

[0095]

[0096] According to the data in Table 1, because random forest can learn failure modes through historical failure characteristics, random forest will predict that a failure occurs when docker1 calls docker2. The specific machine-level indicator is DB.

[0097] Random Forest: Although node anomaly scores and mKPI features can rank and analyze the root causes, this method builds a model to utilize historical fault information. The ensemble learning method random forest is introduced. Random forest classifies samples based on the majority vote of all trees, which is proven to be more accurate and stable. The association model uses node anomaly scores and mKPI features to build a multi-classification model to locate the root cause. Figure 3 Three model architectures based on random forest are shown. These three solutions correspond to different fault location requirements and adopt the best probability association model. Due to the correlation between nodes and mKPI indicators, this solution can reduce the search space of mKPI.

[0098] The present invention provides a method for locating the root cause of a fault in a microservice application. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A method for locating the root cause of a fault in a microservice application, characterized in that: The steps include: Step 1: collect various indicator data information from the monitoring system at intervals of X by calling the interface. The various indicator data information includes the overall service level indicator sKPI, the time-consuming data M between network element nodes, and the time-consuming data M between network element nodes. i,j (r), and various machine-level performance indicator data mKPI; Step 2: Use hive to aggregate the overall service-level indicator sKPI of the statistical system and perform real-time validity check on the data at time t. If the data is invalid, remove the data at time t or re-acquire the data to obtain the value vector x of the overall service-level indicator sKPI at time t. t ; If the data is detected as valid, save the data x t , go to step 3; Step 3: Use the kernel-based anomaly detector EXPoSE to detect the anomaly scores of all service-level indicators sKPI of the system bus online and obtain the occurrence time T of the fault. fault ; Step 4, calculate the bulldozer distance between network element nodes, and extract the mutation characteristics of the machine performance level characteristics; Step 5: Use the random forest association model to predict the fault root cause node and root cause mKPI; In step 4, the bulldozer distance of the network element node is calculated using the following method: Each time the fault is predicted to occur, T fault Then, start calculating: Set t = T fault , set the window w, for network element nodes i and j: Set for all x Among them, x represents the time consumption data M between network element nodes in the time window before the failure time t i,j (r), x is a random variable; r is the start time of the time window before the failure time t; Represents the distribution of random variables of network element node time consumption; Represents the indicator function of the random variable x; Set for all x Indicates the distribution of node time consumption in the time window after the failure; Calculate the bulldozer distance s between network element nodes i and j i,j (T fault ): Among them, ∏(P,Q) represents the set of all possible joint distributions, and the joint distribution satisfies the marginal distributions P and Q; for each possible joint distribution r, sample (x,y) to calculate the distance and obtain the expected value inf represents the lower limit of this expected value in all possible joint distributions; i∈N, j∈N, N is the full set of network element nodes, and i and j can be the same; x represents the distribution from The sampled network element node call time, y is from The time consumption of the sampled network element node call; In step 4, extracting the mutation characteristics of the machine performance level characteristics specifically includes: Set the mutation time window [T fault -w1,T fault +w2]; w1 is the backward time window length, w2 is the forward time window length; set up x represents the value of machine-level performance indicator data mKPI, x * represents the normalized value, is the mean within the mutation time window, std(x) is the standard deviation of x; At each time t, set represents the sequence gradient value at time t, Represents the normalized value at the next moment; Then follow these steps: Step 4-1: Set T s =1,T s is the time when the mutation occurred; Step 4-2: Traverse t=w2-1:w1, if Setting T s =t, exit the loop; After the cycle of step 4-3 is completed, the earliest time T that exceeds three times the standard deviation gradient is obtained. s With gradient is the gradient at the moment when the mutation occurs.

2. The method according to claim 1, characterized in that Step 3 includes: The anomaly score s is calculated by the following formula t : is x t Perform feature mapping, all mappings are based on Gaussian kernel functions; μ[P t ] is the kernel mean mapping, 3. The method according to claim 2, characterized in that In step 3, a threshold θ is set according to the obtained abnormality score. When the abnormality scores at time t and time t+1 are both greater than the threshold θ, time t is determined to be the time when the fault occurs.

4. The method according to claim 3, characterized in that Step 5 includes: training the random forest association model, using the integrated learning method to automatically learn the judgment rules of the fault node and the fault root cause mKPI; when judging the root cause in real time, inputting the mutation characteristics of the bulldozer distance characteristics and the machine performance level characteristics between the network element nodes into the random forest association model to predict the fault root cause node and the root cause.

5. The method according to claim 4, characterized in that In step 5, the fault root cause node H1(x) is predicted according to the following formula: Where x is the time consumption distribution characteristic of all nodes, h i (x) represents each trained decision tree when calling the node time-consuming distribution feature.

6. The method according to claim 5, characterized in that In step 5, the root cause mKPIH2(y) is predicted according to the following formula: Where y is the mutation feature of the first n possible fault root cause nodes H1(x) and the machine performance level feature, h i (y) represents each trained decision tree when calling the mutation feature of the machine performance level feature.

Images