Log and index fused hierarchical micro-service fault root cause positioning method

Abstract

The invention relates to a fault root cause positioning method in a micro-service system, and solves the problem of fault positioning caused by complex logs and index data in a large-scale micro-service system. According to the method, error information in log data is analyzed and embedded into time sequence index data, so that multi-source data fusion is realized, and the fault detection sensitivity is enhanced. The method comprises the following steps: firstly, identifying a global fault time point through a multivariable Bayesian online change point detection method by adopting a hierarchical positioning mechanism, and generating a coarse-grained service-level root cause list; secondly, carrying out fine-grained analysis on the abnormal service through a univariate Bayesian online change point detection method, and generating an index-level root cause list; by means of step-by-step accurate positioning from the service level to the index level, the speed and accuracy of fault positioning can be remarkably improved, complex fault scenes in the micro-service system can be effectively handled, and high availability and stability of the micro-service system are guaranteed.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent operation and maintenance and network fault detection, and specifically relates to a hierarchical microservice fault root cause locating method by integrating logs and indicators. Background Art

[0002] Modern software systems are becoming increasingly complex. Microservice architecture has become a popular development paradigm for large-scale cloud-native systems due to its flexibility and scalability. Microservice architecture splits large software systems into multiple loosely coupled microservice components. Due to the complex dependencies between microservices, the occurrence of a microservice failure may cause a series of chain reactions. The anomaly will quickly spread in the microservice system and affect other dependent services, generating a large amount of abnormal operation and maintenance monitoring data and alarms, which will affect the operation and maintenance engineers to quickly and accurately locate the root cause of the failure.

[0003] The root cause location method of microservice system based on causal inference is a commonly used root cause location method. However, in real scenarios, large-scale microservice systems will generate a large amount of monitoring data. The causal inference root cause location method based on neural networks and traditional PC algorithms and their variants often only focuses on indicator data, requiring a long training and reasoning time, and the accuracy of root cause location on different data sets varies, and it is impossible to quickly and accurately locate the root cause of the fault. Therefore, developing a method that can quickly, accurately and efficiently locate the root cause of microservice failures is of great significance to improving the reliability and operation and maintenance efficiency of the system. Summary of the invention

[0004] The purpose of the present invention is to provide a hierarchical microservice fault root cause location method that integrates logs and indicators. By parsing log data into indicator data, using the Bayesian change point detection method, identifying the location where the data distribution changes significantly, capturing the abnormal change time points in the microservice delay indicator data, separating normal and abnormal data, and combining hypothesis testing with univariate change point detection methods, hierarchical processing is performed to obtain a coarse and fine granularity root cause list, thereby improving the speed and accuracy of root cause location, and solving the root cause location problem caused by complex indicator data in large-scale microservice systems.

[0005] In order to solve the above technical problems, the present invention provides the following technical solution: a method for locating the root cause of a layered microservice fault by change point detection, comprising the following steps:

[0006] Step 1: Through log parsing, convert the error and failure information of each service log metadata into time series data and embed it into the indicator data of the microservice.

[0007] Step 2: Preprocess the microservice indicator data, including the load and latency of each service, remove constant value indicators, use interpolation to fill in missing data, and normalize the latency-related data of each service to provide high-quality data for subsequent change point detection.

[0008] Step 3: Use the multivariate Bayesian online change point detection method to detect the mutation points of the delay data of all services and obtain the failure time points of the microservice system, thereby dividing the indicator data into normal data and abnormal data.

[0009] Step 4: Use statistical methods to learn the data distribution in the normal time period, calculate the cumulative data deviation score of each indicator in the abnormal time period compared with the data distribution in the normal time period, sort the indicators in descending order according to the deviation score and group them, obtain a coarse-grained, service-level root cause list, and complete the root cause location at the service level.

[0010] Step 5: Use the univariate Bayesian online change point detection method for the coarse-grained root cause services to obtain the abnormal time of each indicator under the service, sort them in ascending order according to time, obtain the fine-grained indicator-level root cause list of the service, and complete the root cause location at the indicator level.

[0011] Optionally, in step 2, the Min-Max normalization method is used to normalize the latency indicator data, and the formula is as follows:

[0012]

[0013] where x′ i is the latency indicator data collected by the i-th service, min(x (l，i) ) is the minimum value of the delay indicator data collected by the i-th service, max(x(l,i)) is the maximum value of the delay indicator data collected by the i-th service, and x′ i It is the standardized delay indicator data collected by the i-th service.

[0014] Optionally, the execution process of step 3 includes the following steps:

[0015] Step 3.1: Model the run length of the time-delay data using the multivariate Bayesian online change point detection method, calculate the posterior distribution of the current run length, and obtain the most likely run length at each time step in the time series data. The calculation formula for the posterior distribution of the current run length is as follows:

[0016]

[0017] Among them, r t is the run length at time t, i.e., the number of consecutive data points since the last change point, x 1：tis all the observed data from time 1 to time t, p(r t |x 1：t ) is given data x 1：t In the case of t The posterior probability, p(x t |r t ) is given a run length r t In the case of t The likelihood, p(r t |r t-1 ) is the transition probability of the run length, which is usually determined by the risk function, p(r t-1 |x 1：t-1 ) is given data x 1：t-1 In the case of t-1 The posterior probability, p(x t |x 1：t-1 ) is a normalization constant that ensures the sum of probabilities is 1.

[0018] Step 3.2: Record the mutation point of the downstream length at each time step of the delay data, select the first point as the failure time point of the microservice system, and divide all indicator data into normal data and abnormal data according to the time point.

[0019] Optionally, the multivariate Bayesian online change point detection method in step 3 uses multivariate T distribution modeling to calculate the likelihood of the observed data given the run length.

[0020] Optionally, the execution process of step 4 includes the following steps:

[0021] Step 4.1: Learn the median and interquartile range distribution of all normal indicator data, and for abnormal indicator data points, calculate their deviation scores relative to the normal data distribution. The calculation formula for the deviation score of each data point is as follows:

[0022]

[0023] in, is the deviation score of the jth data point of the ith metric at time t, is the jth time series data point of the ith indicator at time t, med is the median of the normal data, and IQR is the interquartile range of the normal data.

[0024] Step 4.2: Accumulate the deviation scores of all data points in the abnormal data by indicator to obtain the deviation score of each indicator abnormal data from the normal data distribution. Group and accumulate the deviation scores of each indicator sorted in descending order by service with a fixed-size increasing window, and sort in descending order to obtain a coarse-grained service-level root cause list. The coarse-grained service-level root cause list can be expressed as:

[0025] L coarse ={s k ∣s k ∈services,k=1,2,…,K}

[0026] Among them, L coarse is a coarse-grained service-level root cause list, s k is the kth service name, service is the set of service names, and K is the total number of service root cause lists.

[0027] Optionally, step 5 performs a univariate Bayesian online change point detection method on the indicator data of the first few coarse-grained service-level root causes, selects the first mutation point of the indicator time series data as the abnormal time point of the indicator, and sorts them in ascending order by the abnormal time point to obtain a fine-grained indicator-level root cause list. The fine-grained indicator-level root cause list can be expressed as:

[0028] L fine ={sm m ∣sm m ∈metrics,m=1,2,…,M}

[0029] Among them, L fine is a fine-grained indicator-level root cause list, sm m is the mth metric name, metrics is the set of metric names, and M is the total number of metric root cause lists.

[0030] The present invention provides a hierarchical microservice fault root cause location method by integrating logs and indicators. Aiming at the multi-source heterogeneous characteristics of complex indicator data and log data in large-scale microservice systems, a fault location method combining coarse-grained service-level analysis and fine-grained indicator-level analysis is proposed. When dealing with fault location in microservice systems, traditional methods are difficult to cope with the dynamic changes of indicator data and the complex dependencies between services. In addition, the fusion and anomaly detection process of multi-source data is easily affected by noise interference and high false alarm rate. The present invention adopts a hierarchical analysis method combining multivariate Bayesian online change point detection and univariate Bayesian online change point detection. First, the global abnormal changes of service-level delay data are captured by multivariate change point detection to identify the fault time point in the system; secondly, the specific indicators of abnormal services are deeply analyzed by univariate change point detection to locate the fine-grained fault root cause. The hierarchical analysis mechanism introduces statistical distribution learning and deviation score calculation, and realizes gradual and accurate positioning from service level to indicator level through normalization processing and group sorting. Through this method, the present invention significantly improves the speed and accuracy of fault root cause location, can adapt to dynamic and complex fault scenarios in large-scale microservice systems, and ensures the high availability and stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to facilitate a clearer understanding of the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings involved in the embodiments or related technical descriptions. These drawings are only used to assist in explaining the technical solutions of the present invention and are limited to showing some embodiments of the present invention and should not be regarded as limiting the present invention.

[0032] Figure 1 It is a flow chart of a method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators according to a specific embodiment of the present invention.

[0033] Figure 2 It is a flowchart of a method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with specific implementation cases. The following implementation cases will help those skilled in the art to further understand the present invention and are not intended to limit the present invention.

[0035] The present invention provides a method for locating the root cause of a hierarchical microservice failure by integrating logs and indicators. The following are the steps of the method:

[0036] S1: Through log parsing, the error and failure information of each service log metadata is converted into time series data and embedded into the indicator data of the microservice.

[0037] S2: Preprocess the microservice indicator data, including the load and latency of each service, remove constant value indicators, use interpolation to fill in missing data, and normalize the latency-related data of each service to provide high-quality data for subsequent change point detection.

[0038] S3: Use the multivariate Bayesian online change point detection method to detect the mutation points of the delay data of all services and obtain the failure time point of the microservice system, thereby dividing the indicator data into normal data and abnormal data.

[0039] S4: Use statistical methods to learn the data distribution in the normal time period, calculate the cumulative data deviation score of each indicator in the abnormal time period compared with the data distribution in the normal time period, sort the indicators in descending order according to the deviation score and group them, and obtain a coarse-grained, service-level root cause list.

[0040] S5: Use the univariate Bayesian online change point detection method for the coarse-grained root cause services respectively to obtain the abnormal time of each indicator under the service, sort them in ascending order according to time, and obtain the root cause list of the fine-grained indicator level of the service.

[0041] The specific implementation steps and processes are as follows: Figure 2As shown, the following is further described in conjunction with specific embodiments and execution steps:

[0042] The execution process of step S1 can be divided into the following steps:

[0043] (1.1) Parse the log data. First, parse the log data in the microservice system to extract key information such as errors, failures, and timeout events. This information usually exists in unstructured or semi-structured forms, such as text logs, error codes, etc. Use log parsing tools to convert log data into structured time series data.

[0044] (1.2) Data embedding. Associate the parsed log data with the indicator data of the microservices. Embed the parsed and transformed log time series data into the indicator data of each service to form a fusion of multi-source data. In this way, the abnormal information in the log data can be directly reflected in the indicator data, providing richer information for subsequent change point detection and root cause location.

[0045] The execution process of step S2 can be divided into the following steps:

[0046] (2.1) Data cleaning. Clean the indicator data of microservices and remove constant value indicators and invalid data. Constant value indicators are not helpful for fault location, so they need to be removed. In addition, for missing data, interpolation method is used to fill in to ensure data continuity and integrity.

[0047] (2.2) Normalization. Normalize the latency-related data of microservices to eliminate the scale differences between different indicators. Use the Min-Max normalization method to map the latency data to the [0, 1] interval. The normalization formula is as follows:

[0048]

[0049] where x′ i is the latency indicator data collected by the i-th service, min(x (l，i) ) is the minimum value of the delay indicator data collected by the i-th service, max(x(l,i)) is the maximum value of the delay indicator data collected by the i-th service, and x′ i It is the standardized delay indicator data collected by the i-th service.

[0050] The execution process of step S3 can be divided into the following steps:

[0051] (3.1) Multivariate Bayesian change point detection. The multivariate Bayesian online change point detection method is used to model the latency data of the microservice system. The number of times a point appears consecutively in the same distribution is used to determine whether the data has changed significantly. This method identifies mutation points in the time series by calculating the posterior distribution of the current run length. The run length represents the number of consecutive data points since the last change point. The calculation formula for the posterior distribution is as follows:

[0052]

[0053] Among them, r t is the run length at time t, i.e., the number of consecutive data points since the last change point, x 1：t is all the observed data from time 1 to time t, p(r t |x 1：t ) is given data x 1：t In the case of t The posterior probability, p(x t |r t ) is based on the multivariate T distribution, given a run length r t In the case of t The likelihood, p(r t |r t-1 ) is the transition probability of the run length, which is usually determined by the risk function, p(r t-1 |x 1：t-1 ) is given data x 1：t-1 In the case of t-1 The posterior probability, p(x t |x 1：t-1 ) is a normalization constant that ensures the sum of probabilities is 1.

[0054] The calculation formula of multivariate T distribution is as follows:

[0055]

[0056] Among them, x t is the observed data at time t, μ is the mean vector, indicating the central position of the data, Σ is the covariance matrix, indicating the dispersion of the data, ν is the degree of freedom, controlling the thickness of the tail of the distribution. d is the dimension of the data, and Γ(·) is the Gamma function, which is used to normalize the distribution.

[0057] (3.2) Fault time point identification. The multivariate Bayesian online change point detection method is used to record the mutation point of the downstream length at each time step in the delay data. The first mutation point is selected as the fault time point of the microservice system, and all indicator data are divided into normal data and abnormal data according to this time point.

[0058] The execution process of step S4 can be divided into the following steps:

[0059] (4.1) Normal data distribution learning and abnormal data deviation score calculation. Learn the median med and interquartile range IQR distribution of all indicator data in the normal time period. The median and interquartile range can better reflect the central tendency and dispersion of the data. For each data point in the abnormal time period, calculate its deviation score relative to the normal data distribution. The calculation formula of the deviation score is as follows:

[0060]

[0061] in, is the deviation score of the jth data point of the ith metric at time t, is the jth time series data point of the ith indicator at time t, med is the median of the normal data, and IQR is the interquartile range of the normal data.

[0062] (4.2) Coarse-grained root cause list generation. The deviation scores of all data points in the abnormal data are accumulated by indicator to obtain the deviation score of the abnormal data of each indicator to the normal data distribution. Specifically, the deviation score of the ith indicator is Its cumulative deviation score S i The calculation formula is as follows:

[0063]

[0064] Among them, S i is the cumulative deviation score, T is the total length of the abnormal time period, and N i is the number of data points of the ith indicator in time t, is the deviation score of the jth data point of the ith indicator at time t.

[0065] Each indicator deviation score sorted in descending order is grouped and accumulated by service using a fixed-size increasing window. The window size is set to 5 in this embodiment, and a coarse-grained service-level root cause list is obtained by sorting in descending order:

[0066] L coarse ={s k ∣s k ∈services,k=1,2,…,K}

[0067] Among them, L coarse is a coarse-grained service-level root cause list, s k is the kth service name, service is the set of service names, K is the total number of service root cause lists, and the list is sorted by the cumulative deviation score S of the service sSort in descending order to ensure that the service with the highest score is at the front. In this way, the service with the largest deviation can be quickly located, and a coarse-grained service-level root cause list can be generated, providing a clear direction for subsequent fine-grained analysis.

[0068] The execution process of step S5 can be divided into the following steps:

[0069] (5.1) Univariate Bayesian online change point detection. Univariate Bayesian online change point detection is performed on the indicator data of the first two coarse-grained service-level root causes. This method identifies the mutation points in the indicator time series by calculating the posterior distribution of the run length of each indicator. The first mutation point of the indicator time series data is selected as the abnormal time point of the indicator.

[0070] (5.2) Generation of fine-grained root cause list. Sort the indicators in ascending order according to the abnormal time point to obtain a fine-grained indicator-level root cause list:

[0071] L fine ={sm m ∣sm m ∈metrics,m=1,2,…,M}

[0072] Among them, L fine is a fine-grained indicator-level root cause list, sm m is the mth indicator name, metrics is the indicator name set, M is the total number of indicator root cause lists, and the list is sorted in ascending order by indicator abnormal time point, ensuring that the earliest abnormal indicator is ranked first. In this way, the abnormal time point of a specific indicator can be accurately located, helping operation and maintenance personnel to quickly locate the root cause of the fault.

[0073] Compared with the prior art solutions, the present invention has the following beneficial effects:

[0074] The present invention proposes a method for locating the root cause of faults in large-scale microservice systems through multi-source data fusion and hierarchical positioning mechanism. By parsing the error information in the log data and embedding it into the time series indicator data, multi-source data fusion is achieved, and the sensitivity of fault detection is enhanced. Secondly, a hierarchical positioning mechanism is adopted. The global fault time point is first identified through the multivariate Bayesian online change point detection method to generate a coarse-grained service-level root cause list; then the abnormal service is fine-grained analyzed through the univariate Bayesian online change point detection method to generate an indicator-level root cause list. Through the gradual and precise positioning from the service level to the indicator level, the present invention significantly improves the speed and accuracy of fault location, can effectively deal with complex fault scenarios in microservice systems, and ensure the high availability and stability of the system.

[0075] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for locating the root cause of a hierarchical microservice failure by integrating logs and indicators, characterized in that: The following steps are involved: Step 1: Through log parsing, convert the error and failure information of each service log metadata into time series data and embed it into the indicator data of the microservice. Step 2: Preprocess the microservice indicator data, including the load and latency of each service, remove constant value indicators, use interpolation to fill in missing data, and normalize the latency-related data of each service to provide high-quality data for subsequent change point detection. Step 3: Use the multivariate Bayesian online change point detection method to detect the mutation points of the delay data of all services and obtain the failure time points of the microservice system, thereby dividing the indicator data into normal data and abnormal data. Step 4: Use statistical methods to learn the data distribution in the normal time period, calculate the cumulative data deviation score of each indicator in the abnormal time period compared with the data distribution in the normal time period, sort the indicators in descending order according to the deviation score and group them to obtain a coarse-grained, service-level root cause list. Step 5: Use the univariate Bayesian online change point detection method for the coarse-grained root cause services to obtain the abnormal time of each indicator under the service, sort them in ascending order according to time, and obtain the root cause list of the fine-grained indicator level of the service.

2. The method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators as claimed in claim 1, characterized in that: The execution process of step 1 includes the following operations: Step 1.1: Parse the log data, extract key information such as errors, failures, and timeout events, and convert the log data into structured time series data using log parsing tools. Step 1.2: Associate the parsed log data with the indicator data of the microservices, and embed the parsed and transformed log time series data into the indicator data of each service to form a fusion of multi-source data.

3. The method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators as claimed in claim 1, characterized in that: The execution process of step 2 includes the following operations: Step 2.1: Clean the indicator data of microservices, remove constant value indicators and invalid data, and use interpolation to fill in missing data to ensure data continuity and integrity. Step 2.2: Normalize the latency data of the microservices and use the Min-Max normalization method to map the latency data to the [0,1] interval. The normalization formula is as follows: where x′ i is the latency indicator data collected by the i-th service, max(x (l,i) ) is the minimum value of the delay indicator data collected by the i-th service, max(x (l,i) ) is the maximum value of the delay indicator data collected by the i-th service, x′ i It is the normalized delay indicator data collected by the i-th service.

4. The method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators as claimed in claim 1, characterized in that: The execution process of step 3 includes the following operations: Step 3.1: Use the multivariate Bayesian online change point detection method to model the delay data of the microservice system, calculate the posterior distribution of the current run length, and identify the mutation points in the time series. The calculation formula of the posterior distribution is as follows: Among them, r t is the run length at time t, x 1:t is all the observed data from time 1 to time t, p(r t |x 1:t ) is given data x 1:t In the case of t The posterior probability, p(x t |r t ) is given a run length r t In the case of t The likelihood, p(r t |r t-1 ) is the transition probability of the run length, p(r t-1 |x 1:t-1 ) is given data x 1:t-1 In the case of t-1 The posterior probability, p(x t |x 1:t-1 ) is the normalization constant. Step 3.2: Use multivariate Bayesian online change point detection to identify the mutation points of the delay data, select the first mutation point as the fault time point, and divide the indicator data into normal and abnormal parts.

5. The method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators as claimed in claim 1, characterized in that: The execution process of step 4 includes the following operations: Step 4.1: Learn the median med and interquartile range IQR distribution of all indicator data in the normal time period, and for each data point in the abnormal time period, calculate its deviation score relative to the normal data distribution. The calculation formula for the deviation score is as follows: in, is the deviation score of the jth data point of the ith metric at time t, is the jth time series data point of the ith indicator at time t, med is the median of the normal data, and IQR is the interquartile range of the normal data. Step 4.2: Accumulate the deviation scores of abnormal data by indicator and calculate the cumulative deviation score of each indicator. By grouping by fixed window and accumulating by service, sorting in descending order, a coarse-grained service-level root cause list is generated to complete the root cause location at the service level. The coarse-grained service-level root cause list can be expressed as: L coarse =(s k |s k ∈services,k=1,2,...,K} Among them, L coarse is a coarse-grained service-level root cause list, s k is the kth service name, service is the set of service names, and K is the total number of service root cause lists.

6. The method for locating the root cause of a hierarchical microservice fault by integrating logs and indicators as claimed in claim 1, characterized in that: The execution process of step 5 includes the following operations: Step 5.1: Perform univariate Bayesian online change point detection on the indicator data of the first few service-level root causes, and select the first mutation point of each indicator time series as its abnormal time point. Step 5.2: Sort the indicators in ascending order according to the abnormal time point to obtain a fine-grained indicator-level root cause list and complete the root cause location at the indicator level. The fine-grained indicator-level root cause list can be expressed as: L fine =(sm m |sm m ∈metrics,m=1,2,...,M} Among them, L fine is a fine-grained indicator-level root cause list, sm m is the mth metric name, metrics is the set of metric names, and M is the total number of metric root cause lists.

Images