A microservice transient fault locating method based on multi-modal anomaly detection
By employing multimodal anomaly detection and fault responsibility scoring mechanisms, the challenge of locating transient faults in microservice systems has been solved, enabling efficient and accurate root cause analysis and reducing operational difficulty and costs.
Patent Information
- Application Number
- CN202310469236.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-04-27
- Publication Date
- 2026-02-10
- Estimated Expiration
- 2043-04-27
AI Technical Summary
Existing technologies struggle to effectively locate transient failures in microservice systems, especially given the complexity of microservice calls and the propagation of faults, leading to high operational difficulty and costs.
A multimodal anomaly detection method is adopted. By collecting multimodal data from the microservice system, a multidimensional detection model is constructed. Combined with a fault responsibility scoring mechanism, the root cause of instantaneous faults is automatically located.
It improves the accuracy and efficiency of fault location in microservice systems, reduces operation and maintenance costs, and enhances system stability.
Smart Images
Figure CN116489003B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer, and in particular to a microservice transient fault locating method based on multi-modal anomaly detection. BACKGROUND
[0002] With the development of Internet applications, the traditional software architecture has been difficult to meet the flexible and variable business requirements. The microservice architecture splits a single web application into several independently deployable microservices, which cooperate with each other through a lightweight communication protocol, and becomes one of the current mainstream architectures. However, the complex interaction between microservices, the dynamic nature in time and space, and the long call process increase the difficulty of microservice system operation and maintenance. When a microservice has a transient fault, it is more complex than a traditional software fault, and it is difficult to find the root cause of the fault. Moreover, due to the propagation of the fault, when an abnormality occurs in a microservice, it may also cause abnormalities in the microservices directly or indirectly related to it, especially in the face of transient faults, which are unique in their suddenness and irregularity, and it means that more manpower and resources are needed for the operation and maintenance personnel.
[0003] Currently, there are few technical solutions for microservice transient faults in the work of microservice fault. Most of them monitor the state of microservices based on microservice performance indicator data, but due to the complex calls between microservices and the influence of fault propagation, it is difficult to find the actual microservice fault point with this method. In addition, the method of monitoring the microservice call process and locating the fault according to the microservice call relationship diagram cannot find the performance influencing factors in the microservice system, and with the continuous update and expansion of the microservice system, the cost overhead is rising, and ultimately an ideal result cannot be obtained. Therefore, how to locate the transient fault in the microservice system and reduce the cost and time is a challenge. SUMMARY
[0004] To solve the above-mentioned problems, the present application provides a microservice transient fault locating method based on multi-modal anomaly detection.
[0005] The present application collects multi-modal data in the microservice system to carry out multi-dimensional detection of microservice transient faults, determines the abnormal microservice that has a transient fault in combination with the transient recurrence characteristics of microservice transient faults, and further locates the microservice system transient fault according to the microservice call sequence and fault responsibility scoring scheme.
[0006] To achieve the above purpose, the present application adopts the following technical scheme:
[0007] The specific steps of the method of the present application are as follows:
[0008] Step 1. The performance data of the container where the microservice in the microservice system is located, the performance data of each microservice in the call chain and the log data are preprocessed, including missing value processing, normalization processing and data alignment processing.
[0009] Step 2. The template matrix is extracted from the preprocessed log data using the drainage algorithm, and the feature matrix is extracted from the performance data. The multi-modal serialized data matrix is constructed according to the template matrix, the feature matrix and the time sequence, which is used as the input data of the multi-modal prediction model.
[0010] Step 3. The trained multi-modal prediction model based on the converter is used to predict the microservice system, and the dynamic abnormal threshold is set for abnormal detection. The microservice with the most abnormal times in the time window is selected as the transient fault node.
[0011] Step 4. The microservice call chain information in the user request period containing the transient fault node is collected through the distributed link tracking tool, and the local microservice link topology graph is constructed.
[0012] Step 5. Based on the local microservice link topology graph, the dynamic fault responsibility score of the microservice is calculated, and the microservice node of the transient fault root cause is judged.
[0013] Compared with the prior art, the present application has the following advantages:
[0014] 1. An unsupervised microservice anomaly detection scheme designed for a container-based microservice environment is provided, which is independent of the application. The current conventional microservice anomaly detection method is often a supervised method that needs to label the data in advance.
[0015] 2. The proposed microservice transient fault determination method based on multi-modal time series is more in line with the actual microservice system production scene. The fusion of multi-modal data provides the model with the ability to predict multiple results that cannot be revealed by a single data source.
[0016] 3. The proposed fault responsibility score is a new mechanism to screen the microservice nodes that cause transient faults in the microservice system. By analyzing from multiple angles and providing the microservice node set that is most likely to be the source of transient fault occurrence, the accuracy of microservice transient fault root cause positioning is improved, the difficulty of fault troubleshooting for operation and maintenance personnel is reduced, and the stability of the microservice system is improved. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 The flowchart of the microservice transient fault positioning method based on multi-modal anomaly detection in the specific embodiment of the method of the present application.
[0018] Figure 2A schematic diagram for constructing a serialized feature matrix based on preprocessed multi-modal data samples in a specific embodiment of the method of the application.
[0019] Figure 3 A schematic diagram of a multi-modal prediction model structure based on a transformer in a specific embodiment of the method of the application.
[0020] Figure 4 A schematic diagram for generating a microservice call topology based on microservice call chain information in a specific embodiment of the method of the application. DETAILED DESCRIPTION
[0021] In order to make the technical solutions of the application clearer, further detailed description will be made below in combination with the drawings, but the implementation and protection of the application are not limited thereto.
[0022] The main technical idea of the application: In view of the characteristics of transient faults and multiple monitoring data sources in the current microservice system, the application proposes a new microservice transient fault determination method based on multi-modal time series. This method fuses heterogeneous data from different data sources and establishes a multi-modal prediction model based on a transformer. This model can capture the time and internal relationship between multi-modal data, and at the same time set a dynamic abnormal threshold to more accurately detect the abnormal state of the microservice system, which is the premise of microservice transient fault localization. Fault diagnosis is an important link to maintain the stability of the microservice system. Due to the characteristics of the microservice architecture, there is indirect propagation of faults between services, and an effective dynamic positioning mechanism is needed to determine the root microservice node that caused the transient fault to further determine the root cause. The application proposes a new transient fault positioning scheme based on fault responsibility scoring. This scheme uses the aforementioned multi-modal prediction model to capture the time point and specific fault service of the microservice transient fault, combines the microservice call chain information within the time window, automatically scores the fault responsibility of the microservice, and finally sorts the fault responsibility score of the service to help the operation and maintenance personnel find the root cause of the problem.
[0023] The application will be further described below in combination with embodiments.
[0024] As shown in Figure 1 the present embodiment includes the following steps:
[0025] Step 1. Preprocess the performance data of the microservice in the microservice system, the performance data of each microservice in the call chain and the log data, including missing value processing, normalization processing and data alignment processing.
[0026] The performance data of the container in the microservice system refers to the performance monitoring using a distributed link tracking tool, including the CPU usage, memory usage and network transmission speed of the microservice runtime container;
[0027] The performance data of each microservice in the call chain refers to the response time of each microservice node to the request each time the distributed request is called;
[0028] The log data refers to the log records collected by using the distributed link tracking tool;
[0029] The missing value processing refers to using linear interpolation to fill in the missing values in the performance data of the container and the response time data of each microservice in the microservice system.
[0030] The data normalization processing refers to normalizing the performance data of the container and the response time of each microservice after the missing value processing to a smaller range, so that they are distributed in [0, 1].
[0031] The data alignment refers to aligning the log data with other data according to the timestamp, and processing the different generation rates between different data.
[0032] Step 2. Use the drainage algorithm to extract the template matrix from the preprocessed log data, and extract the feature matrix from the performance data. Construct a multi-modal serialized data matrix according to the template matrix, the feature matrix and the time sequence, as the input data of the converter prediction model.
[0033] Figure 2 The figure for constructing the serialized feature matrix according to the preprocessed data samples in this embodiment is shown. For log data, use the drainage algorithm to extract log templates from the log data, a total of D templates are extracted and numbered, and each template is represented as a D-dimensional vector. The elements in the vector can only be 1 or 0. When a log data belongs to the i-th template, the i-th element of the corresponding vector is 1, and the rest are 0. Then, each log data is grouped according to the time sequence to form a (N, D) template matrix. For the performance data of the container and the response time data of each microservice to the user request, they are converted into a (N, K) feature matrix according to the time sequence, where K is the number of features of the performance data of the container and the response time data of each microservice, and N is the number of time points collected, i.e. the number of rows of the matrix. The template matrix, the feature matrix and the time sequence are spliced to obtain a multi-modal serialized matrix with a shape of (N, D+K+1), which is used as the input of the multi-modal prediction model.
[0034] Step 3. Use the trained converter-based multi-modal prediction model to predict the microservice system, set a dynamic anomaly threshold for anomaly detection, and select the microservice with the most number of anomalies in the time window as the transient fault node.
[0035] The structure diagram of the converter-based multi-modal prediction model is as shown in Figure 3 , which includes an input layer, an encoder, a decoder and an output layer.
[0036] The time window refers to the time interval for anomaly detection and statistics within a certain time range. The size of the time window is determined according to actual requirements and system characteristics. The range of this time window is the average time for the microservice system to respond to a user request once.
[0037] The instantaneous failure node refers to the microservice node in the microservice system that exhibits transient and reproducible anomalies.
[0038] The dynamic threshold refers to dynamically determining the response time threshold of each microservice at a certain time point. Based on the prediction result, it is judged whether the microservice at this time point is an instantaneous failure node. If it is greater than the dynamic threshold, it is considered an instantaneous failure node; otherwise, it is a normal node, and the anomaly detection determination result is obtained.
[0039] Use correct input data to establish a transformer-based multi-modal prediction model for predicting and outputting the performance data, microservice response time, and log records of the microservice system at the next time point.
[0040] Use the threshold selection method POT (peaks-over-thresholds) based on extreme value theory to dynamically determine the response time threshold of each microservice at a certain time point. Its core idea is to use the generalized Pareto distribution (Generalized Pareto Distribution, GPD) with parameters to fit the tail of the flow data. The distribution of each microservice response time data is as follows:
[0041]
[0042] where m is the initial threshold of the outlier, x is any value in the microservice response time data {x1, x2, x3,..., x n}, X is the microservice response time data set, γ and β are the shape parameters of the GPD, and c is m - x min . The initial threshold is set as a low quantile (e.g., 5%) according to historical experience, and the parameters γ and β are estimated by maximum likelihood estimation (Maximum likelihoodestimation, MLE).
[0043] Thus, the final threshold M is obtained from the following formula:
[0044]
[0045] where are the GPD parameters obtained by maximum likelihood estimation, q is the expected probability of X < m, that is, the low quantile 5%, n is the sample size of the microservice response time data within the time window, N t is the number of peaks, that is, the number of X t and Xt >m.
[0046] Step 4. Collect microservice call chain information within the user request cycle, including nodes with transient failures, using a distributed tracing tool, and construct a local microservice link topology graph.
[0047] Figure 4 In this embodiment, a local microservice link topology graph containing transient failure nodes is constructed based on the microservice call chain information. Specifically, the call chain collected by the distributed link tracing tool represents the basic information of the interaction between microservices called within a single user request. Each row represents a call between microservices, including the calling end, the called end, the start time, and the end time. A local microservice link topology graph is constructed, where nodes represent microservices and edges represent the call relationships between microservices.
[0048] Step 5. Based on the local microservice link topology, calculate the dynamic fault responsibility score of the microservice and determine the microservice node of the instantaneous fault root cause.
[0049] The fault responsibility score is used to measure the degree of fault responsibility correlation of each microservice node in the local service link topology. In this embodiment, the fault responsibility score includes anomaly correlation score, performance correlation score and causal correlation score.
[0050] In one embodiment, the anomaly relevance score is used to measure the amount of anomaly information contained in each microservice. Based on the local service link topology graph, the microservice that is closer to the instantaneous failure node contains more anomaly-related information, and therefore has a higher anomaly relevance score. The formula for calculating the anomaly relevance score is as follows:
[0051]
[0052] Among them EScore i d represents the anomaly relevance score of microservice i. i→f represents the distance from microservice i to transient failure node f, and c represents the constant value of the anomaly correlation score of transient failure node f, with a value of 1.
[0053] In one embodiment, the performance-related score refers to the degree to which each microservice affects the performance of its container. In the local microservice link topology, the longer the activity time of a microservice, the greater its proportion, and the higher the possibility of affecting the performance of the container where the microservice resides, thus resulting in a higher performance-related score.
[0054] Furthermore, the microservice activity time refers to a given time t. If t falls between the start and end times of the microservice, but not between the start and end times of any sub-microservice, then the microservice node is considered active during that time. The performance-related score is calculated using the following formula:
[0055]
[0056] PScore i Let represent the performance-related score of microservice i, u represent the start and end times of microservices in the local microservice link topology graph scanned linearly in ascending order, and divide the user request cycle into u discrete segments, each constrained by a start and end time, t. j a represents the time length of the j-th segment. j I(i,j) represents the number of microservices active within the j-th segment. I(i,j) is an indicator function that has a value of 1 when microservice i is active within the j-th segment period, and a value of 0 otherwise.
[0057] In one embodiment, the causal correlation score is used to measure the degree of causal influence of each microservice on the transient failure node. Since there is influence between microservices, the higher the causal relationship with the transient failure node, the more likely it is to be the root microservice causing the anomaly, and thus the higher the causal correlation score. When the microservice response time is expressed as a time series, it is stationary. The Granger causality test can be used to verify whether there is predictive value between the microservice and the transient failure node. The formula for calculating the causal correlation score is as follows:
[0058] CScore i =1-P i
[0059] Among them, CScore i P represents the causal correlation score of microservice i. i The p-value is the value obtained when testing the hypothesis that microservice i has predictive value for transiently failed node f. The smaller the p-value, the stronger the evidence to reject the null hypothesis, and therefore the larger the causal correlation score.
[0060] P i R represents i(t) For R f(t) Predictive hypothesis (i.e., R) i(t) It is R f(t) Granger's cause, R i(t) Represents the response time of microservice i at time t, and similarly, R... f(t) The significance level of P) i The calculation method depends on the model and method used for testing. In this embodiment, a vector autoregression (VAR) model is established, and R is calculated. i(t) and R f(t) As a variable, and with the lag order k determined, the F-distribution is used for testing:
[0061]
[0062] Where n is the number of samples, F i It follows an F-distribution with degrees of freedom k and n-2k, and RSS. i Refers to R i(t) The squared residuals obtained from individual regressions, RSS is a measure of R i(t) and R f(t) Simultaneously, the sum of squared residuals obtained during regression can be used to obtain the corresponding P-value based on the F-distribution table.
[0063] This embodiment addresses the aforementioned EScore. i PScore i CScore i After standardization, the final fault responsibility score for each microservice is as follows:
[0064] RScore i =w1*EScore i +w2*PScore i +w3*CScore i
[0065] RScore i This represents the fault responsibility score for service i, where w1, w2, and w3 are weights and are preset.
[0066] In a local microservice link topology graph, the critical path refers to the path with the longest response time among the microservice nodes traversed from the starting point of a request entering the microservice system to the ending point of a response leaving the microservice system. In this application, the microservices with the z highest fault responsibility scores on the critical path are first selected as set B1, where e is the lowest fault responsibility score in B1. Microservices on non-critical paths whose execution times overlap with set B1 and whose fault responsibility scores are greater than e are selected. B = B1∪B2 is calculated, and finally, the set B of microservice nodes most likely to be the source of transient failures is obtained. The microservice with the highest fault responsibility score in set B is the microservice of the root cause of the transient failure.
Claims
1. A method for locating transient faults in microservices based on multimodal anomaly detection, characterized in that, The method includes the following steps: Step 1. Preprocess the performance data of the container where the microservice resides, the performance data of each microservice in the call chain, and the log data in the microservice system, including missing value handling, normalization, and data alignment. Step 2. Use the drainage algorithm to extract the template matrix from the preprocessed log data, extract the feature matrix from the performance data, and construct a multimodal serialized data matrix based on the template matrix, feature matrix, and time series data, which will serve as the input data for the multimodal prediction model. Step 3. Use the trained multimodal prediction model based on converters to predict the microservice system, set a dynamic anomaly threshold for anomaly detection, and mark the microservice with the most anomalies within the time window as the instantaneous failure node. The dynamic anomaly threshold The calculation is as follows: in, The GPD parameters are obtained from the maximum likelihood estimation. yes The expected probability, This is a dataset of microservice response times. The initial threshold for outliers is set to the low quantile. , It is the sample size of microservice response time data within the time window. It is the number of peaks, that is Quantity, ; Step 4. Collect microservice call chain information within the user request cycle, including nodes with transient failures, using a distributed tracing tool, and construct a local microservice link topology graph; Step 5. Based on the local microservice link topology graph, calculate the dynamic fault responsibility score of the microservices and determine the microservice node of the root cause of the instantaneous fault; The fault responsibility score is used to measure the degree of fault responsibility correlation among various microservice nodes in the local service link topology. The fault responsibility score specifically includes anomaly correlation score, performance correlation score, and causal correlation score; the final fault responsibility score is obtained by weighted summation of these three types of scores. Preset Select the critical path The microservice with the highest fault liability score is a set. Choice and set On non-critical paths with overlapping execution times and whose fault responsibility scores are greater than The microservice with the lowest fault liability score is a set Take the union of the two. , where set The microservice with the highest fault responsibility score is the root cause node of the transient fault.
2. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, In step 1, the performance data of the container where the microservice resides in the microservice system includes the CPU utilization, memory utilization, and network transmission rate of the container during microservice runtime. The performance data of each microservice in the call chain refers to the response time of each microservice node to the request during each distributed request call.
3. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, In step 3, the time window refers to the time interval for anomaly detection and statistics within a certain time range; The dynamic anomaly threshold refers to the dynamic determination of the response time threshold of each microservice at a certain point in time. The threshold selection method POT based on extreme value theory is used to dynamically determine the response time threshold of each microservice at a certain point in time.
4. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, In step 4, the basic information of the call chain includes the calling end, the called end, the start time and the end time, and a local microservice link topology graph is constructed, where nodes represent microservices and edges represent the call relationships between microservices.
5. A microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, The anomaly correlation score is used to measure the amount of anomaly information contained in each microservice, and its calculation formula is as follows: in microservices Abnormal correlation score, microservices To abnormal microservices distance, Indicates abnormal microservice The constant value of the abnormality correlation score is 1.
6. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, The performance-related score refers to the degree to which each microservice affects the performance of its container, and its calculation formula is as follows: in microservices Performance-related scores This indicates a linear scan of the start and end times of all microservices in the local microservice link topology diagram in ascending order, dividing the user request cycle into... Each segment is a discrete segment, defined by a start time and an end time. Indicates the first The length of the segment Indicates the first The number of microservices active within the segment. It is an indicator function, when the microservice The value is 1 when the activity is in the j-th period, and 0 otherwise.
7. A microservice instantaneous fault location method based on multimodal anomaly detection according to claim 1, characterized in that, The causal correlation score is obtained by verifying the existence of predictive value between different microservices and abnormal microservices using the Granger causality test, and the causal influence of each microservice on the abnormal microservice is calculated using the following formula: in microservices The causal correlation score It is a test of microservices abnormal microservices The value obtained when there is a predictive value in the assumption.