Microservice transient fault locating method based on multimodal abnormality detection

WO2024221833A9PCT designated stage expired Publication Date: 2024-12-05HANGZHOU DIANZI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/CN2023/130662
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-04-27
Filing Date
2023-11-09
Publication Date
2024-12-05

AI Technical Summary

Technical Problem

Transient faults in microservice systems are difficult to locate, and traditional monitoring methods cannot effectively discover the root cause of the fault, resulting in increased operation and maintenance costs and time costs.

Method used

Adopt a multi-modal anomaly detection method, collect multi-modal data from the microservice system for pre-processing, build a multi-modal serialized data matrix, use a converter-based multi-modal prediction model for anomaly detection, and combine it with fault responsibility scoring Plan to locate the root cause of transient faults in microservice nodes.

Benefits of technology

It improves the accuracy of microservice system fault location, reduces the difficulty of operation and maintenance, reduces cost and time overhead, and improves system stability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023130662_05122024_PF_FP_ABST
    Figure CN2023130662_05122024_PF_FP_ABST
Patent Text Reader

Abstract

Disclosed in the present invention is a microservice transient fault locating method based on multimodal abnormality detection. The present invention comprises: first, collecting multimodal data in a microservice system to carry out multidimensional detection on a microservice transient fault; capturing time and internal relationships among the multimodal data; setting a dynamic abnormality threshold for abnormality detection, and, according to the characteristic of transient reproduction of the microservice transient fault, determining an abnormal microservice in which a transient fault occurs; further building a local microservice link topological graph according to a microservice calling sequence; and finally, obtaining transient fault source microservice nodes by means of a dynamic fault responsibility scoring method, and determining a fault root source microservice. According to the present invention, by analyzing from multiple perspectives and providing a microservice node set most likely to be a transient fault source, the accuracy of microservice transient fault root cause locating is improved, the troubleshooting difficulty of operations and maintenance personnel is reduced, and the stability of a microservice system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

A microservice transient fault location method based on multimodal anomaly detection Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a microservice transient fault location method based on multimodal anomaly detection. Background Art

[0002] With the growth of internet applications, traditional software architectures are no longer able to meet the ever-changing demands of flexible business needs. Microservices, which split a single web application into several independently deployable microservices that collaborate via lightweight communication protocols, have become a mainstream architecture. However, the complex interactions between microservices, their dynamic nature across time and space, and the lengthy call process make microservice systems more challenging to operate and maintain. Transient failures in microservices are more complex than those in traditional software, and the root cause is difficult to identify. Furthermore, due to the propagation of failures, an anomaly in a microservice can potentially cause anomalies in directly or indirectly related microservices. This, coupled with the sudden and irregular nature of transient failures, demands significant human and material resources from operations and maintenance personnel.

[0003] Currently, in the work of microservice failure, there are few technical solutions for microservice transient failures. Most of them monitor the microservice status based on microservice performance indicator data. However, due to the complex calls between microservices and the influence of fault propagation, such methods are difficult to find the actual microservice failure point. In addition, the method of monitoring the microservice call process and locating the fault based on the microservice call relationship diagram cannot discover the performance influencing factors in the microservice system, and as the microservice system is continuously updated and expanded, the cost overhead continues to rise, and ultimately the ideal result cannot be obtained. Therefore, how to locate transient failures in the microservice system and reduce overhead and time cost is a challenge. Contents of the invention

[0004] To solve the above problems, the present invention provides a microservice transient fault location method based on multimodal anomaly detection.

[0005] The present invention collects multimodal data in the microservice system to conduct multi-dimensional detection of microservice transient faults. Combined with the characteristics of instantaneous recurrence of microservice transient faults, the abnormal microservice with transient faults is determined. The transient faults of the microservice system are further located based on the microservice call sequence and the fault responsibility scoring scheme.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] The specific steps of the inventive method are:

[0008] Step 1. Preprocess the performance data of the container where the microservice is located in the microservice system, the performance data of each microservice in the call chain, and the log data, including missing value processing, normalization processing, and data alignment processing.

[0009] Step 2. Use the drainage algorithm to extract the template matrix from the preprocessed log data, and extract the feature matrix from the performance data. Construct a multimodal serialized data matrix based on the template matrix, feature matrix and time series as the input data of the multimodal prediction model.

[0010] Step 3. Use the trained converter-based multimodal prediction model to predict the microservice system, set a dynamic anomaly threshold for anomaly detection, and select the microservice with the most anomalies within the time window and mark it as a transient fault node.

[0011] Step 4. Use distributed link tracing tools to collect microservice call chain information within the user request cycle including the transient fault node, and build a local microservice link topology map.

[0012] Step 5. Based on the local microservice link topology, calculate the dynamic fault responsibility score of the microservice and determine the microservice node that is the root cause of the transient fault.

[0013] Compared with the prior art, the present invention has the following advantages:

[0014] 1. This paper provides an application-independent, unsupervised microservice anomaly detection solution designed for container-based microservice environments. Currently, conventional microservice anomaly detection methods are often supervised methods that require pre-labeling of data.

[0015] 2. The proposed microservice transient fault judgment method based on multimodal time series is more in line with the actual microservice system production scenario. The fusion of multimodal data provides the model with the ability to predict multiple results that cannot be revealed by a single data source.

[0016] 3. The proposed fault responsibility scoring is a new mechanism to screen the microservice nodes that cause transient faults in the microservice system. By analyzing from multiple angles and providing the set of microservice nodes that are most likely to be the source of transient faults, it improves the accuracy of locating the root cause of microservice transient faults, reduces the difficulty of troubleshooting for operation and maintenance personnel, and improves the stability of the microservice system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a flow chart of a microservice transient fault location method based on multimodal anomaly detection in a specific embodiment of the method of the present invention.

[0018] FIG2 is a schematic diagram of constructing a serialized feature matrix based on preprocessed multimodal data samples in a specific embodiment of the method of the present invention.

[0019] FIG3 is a schematic diagram of the structure of a multimodal prediction model based on a converter in a specific embodiment of the method of the present invention.

[0020] FIG4 is a schematic diagram of generating a microservice call topology based on microservice call chain information in a specific embodiment of the method of the present invention. DETAILED DESCRIPTION

[0021] In order to make the technical solution of the present invention clearer, further detailed description is given below with reference to the accompanying drawings, but the implementation and protection of the present invention are not limited thereto.

[0022] The main technical concept of the present invention is: In response to the characteristics of transient faults and multiple monitoring data sources in current microservice systems, the present invention proposes a new microservice transient fault determination method based on multimodal time series. This method integrates heterogeneous data from different data sources and establishes a multimodal prediction model based on transformers. This model can capture the time and internal relationships between multimodal data. At the same time, it sets dynamic abnormality thresholds to more accurately detect abnormal states of microservice systems, which is the prerequisite for locating microservice transient faults. Fault diagnosis is an important link in maintaining the stability of microservice systems. Due to the characteristics of the microservice architecture itself, there is indirect propagation of faults between services. An effective dynamic positioning mechanism is needed to determine the root microservice node where the transient fault occurs to further determine the root cause. The present invention thus proposes a new transient fault location scheme based on fault responsibility scoring. This scheme uses the aforementioned multimodal prediction model to capture the time point and specific fault service of the microservice transient fault, and combines the microservice call chain information within the time window to automatically score the fault responsibility of the microservice. Finally, the fault responsibility scores of the services are ranked to help operation and maintenance personnel find the root cause of the problem.

[0023] The present invention will be further described below with reference to the following embodiments.

[0024] As shown in FIG1 , this embodiment includes the following steps:

[0025] Step 1. Preprocess the performance data of the container where the microservice is located in the microservice system, the performance data of each microservice in the call chain, and the log data, including missing value processing, normalization processing, and data alignment processing.

[0026] The performance data of the container in the microservice system refers to performance monitoring using distributed link tracing tools, including the CPU usage, memory usage, and network transmission speed of the container when the microservice is running;

[0027] The performance data of each microservice in the call chain refers to the response time of each microservice node to the request during each distributed request call;

[0028] The log data refers to log records collected using distributed link tracking tools;

[0029] Missing value processing refers to using linear interpolation to fill missing values ​​in a row or column of the performance data of the container and the response time data of each microservice in the microservice system;

[0030] The data normalization process refers to normalizing the container performance data and the corresponding time of each microservice after missing value processing to a smaller range so that they are distributed in [0, 1].

[0031] The data alignment refers to aligning log data with other data according to timestamps, and processing different generation rates of different data.

[0032] Step 2. Use the drainage algorithm to extract the template matrix from the preprocessed log data, and extract the feature matrix from the performance data. Construct a multimodal serialized data matrix based on the template matrix, feature matrix and time series as the input data of the converter prediction model.

[0033] Figure 2 is a schematic diagram of constructing a serialized feature matrix based on the preprocessed data samples in this embodiment. For log data, a drainage algorithm is used to extract log templates from the log data. A total of D templates are extracted and numbered. Each template is represented as a D-dimensional vector. The elements in the vector can only be 1 or 0. When a log data belongs to the i-th template, the i-th element of the corresponding vector is 1, and the remaining elements are 0; then each log data is organized into a (N, D) template matrix in chronological order. The performance data of the container and the response time data of each microservice to the user request are converted into a (N, K) feature matrix in chronological order, where K is the number of features of the performance data of the container and the response time data of each microservice, and N is the number of time points collected, that is, the number of rows in the matrix. The template matrix, feature matrix and time series are spliced ​​to obtain a multimodal serialization matrix with a shape of (N, D+K+1), which is used as the input of the multimodal prediction model.

[0034] Step 3. Use the trained converter-based multimodal prediction model to predict the microservice system, set a dynamic anomaly threshold for anomaly detection, and select the microservice with the most anomalies within the time window and mark it as a transient fault node.

[0035] The schematic diagram of the structure of the converter-based multimodal prediction model is shown in FIG3 , which includes an input layer, an encoder, a decoder, and an output layer.

[0036] The time window refers to the time interval for anomaly detection and statistics within a certain time range. The size of the time window is determined according to actual requirements and system characteristics. The range of this time window is the average time for the microservice system to respond to a user request once.

[0037] The instantaneous failure node refers to the microservice node that experiences transient and recurring anomalies in the microservice system.

[0038] The dynamic threshold refers to dynamically determining the response time threshold of each microservice at a certain time point. Based on the prediction result, it is judged whether the microservice at this time point is an instantaneous failure node. If it is greater than the dynamic threshold, it is considered an instantaneous failure node; otherwise, it is a normal node, and the anomaly detection determination result is obtained.

[0039] Build a transformer-based multimodal prediction model using correct input data to predict and output the performance data, microservice response time, and log records of the microservice system at the next time point.

[0040] Use the threshold selection method POT (peaks-over-thresholds) based on extreme value theory to dynamically determine the response time threshold of each microservice at a certain time point. Its core idea is to use the generalized Pareto distribution (Generalized Pareto Distribution, GPD) with parameters to fit the tail of the flow data. The distribution of each microservice response time data is as follows:

[0041] where m is the initial threshold of the outlier, x is any value in the microservice response time data {x1, x2, x3,..., x n}, X is the microservice response time data set, γ and β are the shape parameters of the GPD, and c is m - x min . The initial threshold is set as a low quantile (e.g., 5%) according to historical experience, and the parameters γ and β are estimated by maximum likelihood estimation (Maximum likelihood estimation, MLE).

[0042] Thus, the final threshold M is obtained by the following formula:

[0043] where are the GPD parameters obtained by maximum likelihood estimation, q is the expected probability of X < m, that is, the low quantile 5%, n is the sample size of the microservice response time data within the time window, N t is the number of peaks, that is, the number of X t , and X t > m.

[0044] Step 4. Use distributed link tracing tools to collect microservice call chain information within the user request cycle including the transient fault node, and build a local microservice link topology map.

[0045] Figure 4 shows a local microservice link topology diagram including a transient fault node constructed based on the microservice call chain information in this embodiment. Specifically, the call chain collected by the distributed link tracing tool represents the basic information of the interaction between the microservices called in a user request. Each row represents a call between microservices, including the calling end, the called end, the start time, and the end time. A local microservice link topology diagram is constructed, in which nodes represent microservices and edges represent the call relationships between microservices.

[0046] Step 5. Based on the local microservice link topology, calculate the dynamic fault responsibility score of the microservice and determine the microservice node that is the root cause of the transient fault.

[0047] The fault responsibility score is used to measure the degree of relevance of the fault responsibility of each microservice node in the local service link topology diagram. The fault responsibility score in this embodiment includes an abnormality relevance score, a performance relevance score, and a causal relevance score.

[0048] In one embodiment, the anomaly relevance score is used to measure the amount of anomaly information contained in each microservice. Based on the local service link topology, the closer the microservice is to the transient fault node, the more anomaly-related information it contains, and the higher the anomaly relevance score. The calculation formula for the anomaly relevance score is as follows:

[0049] EScore i represents the anomaly correlation score of microservice i, d i→f represents the distance from microservice i to the transient fault node f, and c represents the constant value of the abnormal correlation score of the transient fault node f, which is 1.

[0050] In one embodiment, the performance-related score refers to the degree of impact of each microservice on the performance of its container. In the local microservice link topology diagram, the longer the microservice activity time, the greater the proportion, and the higher the possibility of affecting the performance of the container where the microservice is located, and the higher the performance-related score;

[0051] Furthermore, the microservice activity time refers to a given time t. If t is between the start time and end time of the microservice, but not between the start time and end time of any sub-microservice, the microservice node is considered to be active during this time. The performance-related score is calculated as follows:

[0052] Among them, PScore irepresents the performance-related score of microservice i, u represents the start and end time of microservices in the linear scan of all local microservice link topology graphs in increasing order, and divides the user request cycle into u discrete segments, each segment is limited by the start and end time, t j represents the length of time of the jth segment, a j represents the number of active microservices in segment j, I ( i,j ) is an indicator function whose value is 1 when microservice i is active in the jth period and 0 otherwise.

[0053] In one embodiment, the causal correlation score is used to measure the degree of causal influence of each microservice on the transient fault node. Since there is an influence between microservices, the higher the causal relationship with the transient fault node, the more likely it is that the microservice is the root cause of the microservice anomaly, and the higher the causal correlation score. When the microservice response time is expressed as a time series, it has stationarity. The Granger Causality Test can be used to verify whether there is predictive value between the microservice and the transient fault node. The calculation formula of the causal correlation score is as follows: CScore i =1-P i

[0054] Among them, CScore i represents the causal relevance score of microservice i, P i It is the value obtained when testing the hypothesis that microservice i has predictive value for transient fault node f. The smaller the P value, the stronger the evidence for rejecting the null hypothesis, and therefore the larger the causal correlation score.

[0055] P i Indicates R i(t) R f(t) Hypotheses with predictive value (i.e., R i(t) It is R f(t) Granger cause, R i(t Represents the response time of microservice i at time t. Similarly, R f(t) ) significance level, P i The calculation method of depends on the model and method used for the test. In this embodiment, a vector autoregression model (VAR) is established to calculate R i(t) and R f(t) As a variable, and determine the lag order k, use F distribution to test:

[0056] Where n is the number of samples, F i Obeying F distribution, its degrees of freedom are k and n-2k, RSS i Point to R i(t) The residual square obtained when performing a separate regression is RSS.i(t) and R f(t) At the same time, the residual sum of squares obtained during regression can be used to obtain the corresponding P value based on the F distribution table.

[0057] This embodiment compares the above EScore i 、PScore i 、CScore i After standardization, the final fault responsibility score of each microservice is: RScore i =w1*EScore i +w2*PScore i +w3*CScore i

[0058] RScore i represents the fault responsibility score of service i, w1, w2, and w3 are weights and are pre-set.

[0059] The critical path in the local microservice link topology diagram refers to the path with the longest response time among the microservice nodes from the starting point where the request enters the microservice system to the end point where the response leaves the microservice system. In this application, the z microservices with the highest fault responsibility scores on the critical path are first selected as set B1, e is the lowest fault responsibility score in B1, and microservices on the non-critical path with overlapping execution time of set B1 and whose fault responsibility score is greater than e are selected. B = B1 ∪ B2 is calculated, and finally the set B of microservice nodes most likely to be the source of transient faults is obtained. The microservice with the highest fault responsibility score in set B is the microservice at the root of the transient fault.

Claims

1. A microservice instantaneous fault location method based on multimodal anomaly detection, characterized in that: The method comprises the following steps: Step 1: Preprocess the performance data of the container where the microservice is located in the microservice system, the performance data of each microservice in the call chain, and the log data, including missing value processing, normalization processing, and data alignment processing; Step 2. Use the drainage algorithm to extract the template matrix from the preprocessed log data, extract the feature matrix from the performance data, and construct a multimodal serialized data matrix based on the template matrix, feature matrix and time series as the input data of the multimodal prediction model; Step 3. Use the trained converter-based multimodal prediction model to predict the microservice system, set a dynamic anomaly threshold for anomaly detection, and select the microservice with the most anomalies in the time window as the instantaneous fault node; Step 4. Use the distributed link tracking tool to collect the microservice call chain information within the user request cycle including the transient fault node, and build a local microservice link topology map; Step 5. Based on the local microservice link topology, calculate the dynamic fault responsibility score of the microservice and determine the microservice node that is the root cause of the instantaneous fault.

2. According to claim 1, a microservice instantaneous fault location method based on multimodal anomaly detection is characterized in that: In step 1, the performance data of the container where the microservice is located in the microservice system includes the CPU usage, memory usage and network transmission rate of the container when the microservice is running; The performance data of each microservice in the call chain refers to the response time of each microservice node to the request each time a distributed request is called.

3. According to the method for locating microservice instantaneous faults based on multimodal anomaly detection according to claim 1, it is characterized in that: In step 3, the time window refers to the time interval for performing anomaly detection and statistics within a certain time range; The dynamic threshold refers to dynamically determining the response time threshold of each microservice at a certain point in time, and dynamically determining the response time threshold of each microservice at a certain point in time using a threshold selection method POT based on extreme value theory.

4. According to the method for locating microservice transient faults based on multimodal anomaly detection according to claim 1, it is characterized in that: In step 4, the basic information of the call chain includes the calling end, the called end, the start time and the end time, and a local microservice link topology diagram is constructed, in which the nodes represent microservices and the edges represent the call relationships between microservices.

5. According to claim 1, a microservice instantaneous fault location method based on multimodal anomaly detection is characterized in that: In step 5, the fault responsibility score is used to measure the fault responsibility correlation degree of each microservice node in the local service link topology diagram. The fault responsibility score specifically includes an abnormality correlation score, a performance correlation score, and a causal correlation score; The weighted sum of these three scores yields the final fault responsibility score; Set z in advance, select the z microservices with the highest fault responsibility scores on the critical path as set B1, select the microservices on the non-critical path that overlap with the execution time of set B1 and whose fault responsibility scores are greater than the lowest fault responsibility score of B1 as set B2, and take the union of the two as set B, where the microservice with the highest fault responsibility score in set B is the root node of the instantaneous fault.

6. According to claim 5, a microservice instantaneous fault location method based on multimodal anomaly detection is characterized in that: The anomaly correlation score is used to measure the amount of anomaly information contained in each microservice, and its calculation formula is: EScore i represents the anomaly correlation score of microservice i, d i→f represents the distance from microservice i to abnormal microservice f, and c represents the constant value of the abnormal correlation score of abnormal microservice f, which is 1.

7. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 5 is characterized in that: The performance-related score refers to the degree of impact of each microservice on its container performance, and its calculation formula is: PScore i represents the performance-related score of microservice i, u represents the start and end time of microservices in the local microservice link topology graph in increasing order, and divides the user request cycle into u discrete segments, each segment is limited by the start and end time, t j represents the length of the jth segment, a j It represents the number of microservices active in the jth period. I(i,j) is an indicator function whose value is 1 when microservice i is active in the jth period and 0 otherwise.

8. The microservice instantaneous fault location method based on multimodal anomaly detection according to claim 5 is characterized in that: The causal correlation score verifies whether there is a predictive value between different microservices and abnormal microservices by using the Granger causality test method, and obtains the causal influence of each microservice on the abnormal microservice. The calculation formula is: CScore i =1-P i CScore i represents the causal relevance score of microservice i, P i It is the value obtained when testing the hypothesis that microservice i has predictive value for abnormal microservice f.