A microservice anomaly diagnosis method based on attribute relationship graph

Abstract

The present invention relates to a microservice anomaly diagnosis method based on an attribute relationship graph, so as to solve the problems that the existing algorithm is not robust and the diagnostic effect is poor after an anomaly occurs in a microservice system. First, the anomaly is detected in real time according to the service call trajectory information collected by the monitoring proxy service; after the anomaly is found, a microservice call topology relationship graph is established based on the call and deployment information at the time point of the anomaly occurrence to characterize the real-time anomaly propagation relationship of the microservice; then, comprehensive monitoring information for a period of time before and after the anomaly occurs is obtained, and the personalized weight attributes of the nodes and edges on the graph are calculated using a custom formula to establish a microservice attribute relationship graph; the nodes on the graph are evaluated based on the PageRank algorithm, and the most likely abnormal root cause node is inferred. The present invention realizes real-time detection of microservice anomalies, automatically establishes an attribute relationship graph, and intelligently infers the degree of abnormality of service nodes to realize abnormal diagnosis of microservices.

Description

Technical Field

[0001] The invention relates to an abnormality diagnosis method for a microservice software system and belongs to the technical field of software. Background Art

[0002] Monolithic architecture and SOA software architecture are commonly used by software companies. After more than ten years of development, software systems have become extremely complex, with low scalability and maintainability, and enterprises are burdened with heavy technical debt. Today's Internet competition is fierce, user needs and market environment are constantly changing rapidly. When facing today's Internet applications, the scalability and flexibility of traditional software architecture are obviously insufficient, while the design, development, testing and operation and maintenance costs have increased significantly. Therefore, the concept of microservices was proposed. Microservices is a software architecture form that treats a single application as a set of software service suites. Each service runs in an independent process and communicates with each other through lightweight protocols. The characteristics of microservice architecture are very suitable for agile development and continuous integration, solving the pain points of traditional software architecture and gaining extensive attention and research from academia and industry.

[0003] When software systems are microserviced, while maintaining and being flexible, the dependencies between services become more complex, increasing the probability of failures and the losses caused by failures. For example, in a high-traffic website, once a service component is delayed, all application resources may be exhausted, causing the so-called avalanche effect, which may paralyze the entire system in severe cases. Therefore, effective system monitoring and rapid fault location are key technologies to ensure the reliability and performance of microservices.

[0004] There are several main types of work on microservice fault diagnosis: (1) Diagnostic methods based on metric monitoring. This method mainly collects system operation indicators, such as CPU, memory, network, etc., to reflect the current status of the application and the operation trend over a period of time. If a metric exceeds the preset threshold, it means that there is a problem with the system and an alarm is triggered. Then, the administrator uses the monitoring data as a basis and combines his own experience to solve the problem; (2) Log-based monitoring and analysis methods. Logs clearly record the operation status of the system, are easy to persist, and can be easily searched. They are usually an effective means to identify the cause of the fault and support more business goals; (3) Diagnostic methods based on distributed call trace information. According to the call trace, a service call topology map is established, and a search algorithm is used to infer the root cause.

[0005] Among them, the fault diagnosis method based on metrics and log monitoring is simple to implement, but it cannot reflect the overall status of the system and cannot track business flows. The level of fault location is usually service components. In complex microservice interaction relationships, administrators will spend a lot of time finding and locating problems. The algorithm based on distributed call traces establishes an incomplete topology map because there are server nodes that only have deployment relationships but no call relationships. In addition, only trajectory information is used, the algorithm has poor robustness and inaccurate diagnostic effects. Summary of the invention

[0006] The technology of the present invention solves the problem: Overcoming the shortcomings of the prior art, providing an efficient anomaly diagnosis system for microservices, automatically establishing a topology map by parsing call trajectory information, reflecting the overall state of the system in real time, and improving the scalability of the system. The personalized weight is calculated by mixing and weighting the three anomaly scores to improve the robustness of the algorithm and make the diagnosis more accurate.

[0007] Through transparent call monitoring of services, real-time processing of call trajectory information and online anomaly detection, the call relationship is automatically parsed according to the detection results, combined with the deployment relationship, and a variety of monitoring indicators including the call failure ratio are used to establish a hybrid weight attribute relationship graph, so that service nodes with only deployment relationships can also be reflected in the graph, thereby improving the scalability of the system. By inferring the attribute graph, the root cause of service-level anomalies can be located.

[0008] The technical solution implemented in the present invention is implemented in the following steps:

[0009] Step 1: Real-time anomaly detection of data streams. Obtain call trajectory information based on the monitoring proxy service, process the call trajectory information, and perform anomaly detection on the processed time series through an online clustering algorithm to determine the time point when the anomaly occurred.

[0010] The call trace information mainly includes:

[0011] (callType, startTime, elapsedTime, success, traceId, id, pid, cmdb_id, serviceName), where callType represents the call type, startTime represents the call start time, elapsedTime represents the call time, success represents whether it is successful, traceId represents the call id of a complete request, id represents the current call id, pid represents the parent node id, serviceName represents the network element id to which the service belongs, and serviceName represents the service name. The specific steps are as follows:

[0012] Step 101: Obtain all call trace data of a single request by calling the ID, and obtain the execution time of the current node by subtracting the time consumed by the child node from the time consumed by the parent node;

[0013] Step 102: For the call time between two microservice subnodes, a 30-second time window is used, and the median of the call time in the window is taken as a single data value of a set of time series. This method is used to obtain a column of call time series;

[0014] Step 103: Perform online real-time anomaly detection on each acquired call time series through the BIRCH online clustering algorithm to identify the time point when the anomaly occurs;

[0015] Step 2: Establish a real-time topology map. For the time point of the exception occurrence identified in step 1, analyze the call trajectory data when the exception occurs, identify the parent service node and child service node involved in the current period call, and build a real-time topology map including servers, containers, and databases in combination with the service deployment information.

[0016] Step 201: In step 1, the call trace time sequence is in the form of a key-value pair of "parent node-child node: call time". The call time sequence is parsed, and the parent node is the out-degree node of the edge in the graph, and the child node is the in-degree node of the edge in the graph. Read the deployment configuration file. For the deployment relationship, the node where the service container is located is the out-degree node of the edge in the graph, and the deployment server node is the in-degree node of the edge in the graph.

[0017] Step 201, for the time point of the exception occurrence specified in step 1, parse the call trajectory time series, which is in the form of a key-value pair of "parent node-child node: call time", split the parent and child service nodes, read the configuration file, obtain the deployment server information corresponding to the parent and child service nodes, and obtain all the nodes for constructing the topology map.

[0018] Step 202: For all topology graph nodes obtained in step 201, if there is a call relationship between the nodes, the parent node is the out-degree node of the edge in the graph, the child node is the in-degree node of the edge in the graph, and the direction of the edge is from the parent node to the child node; if there is a deployment relationship between the nodes, the node where the service is located is the out-degree node of the edge in the graph, the deployment server node is the in-degree node of the edge in the graph, and the direction of the edge is from the service node to the deployment server.

[0019] Step 3: Calculate personalized weights and establish an attribute relationship diagram. After the abnormality occurrence time point is determined in step 1, obtain monitoring data for a period of time before and after the abnormality occurs, calculate personalized weights according to the formula, and establish an attribute relationship diagram.

[0020] Step 301, topology map node weight, construct the feature vector N = (node_on-off, node_sess-connect, node_network, node_cpu, node_memory) of the node on the topology map, where node_on-off represents the database monitoring switch, node_sess-connect represents the maximum number of database connections, node_network represents network traffic, node_cpu represents the container or server CPU usage, node_memory represents the container or memory usage, and at the same time construct the weight calculation formula of the relevant indicators: where λ i Represents the similarity of the indicator feature vectors before and after the anomaly occurs, The cosine similarity calculation is performed by taking the mean of the feature vectors 5 minutes before the anomaly occurs and the mean within 5 minutes after the anomaly occurs. The greater the similarity, the lower the degree of anomaly. i Represents the attenuation coefficient, which is used to control when an anomaly is released. Sometimes, it will be reflected in the indicators of multiple service nodes, such as network failure. Here, a decay sequence parameter is defined to control the anomaly score to be higher the earlier the anomaly occurs.

[0021] Step 302: Weight of the topology node connection edge: Use the mixed weight of the service call time after the exception occurs, the resource utilization information of the container or server to which the service belongs, and the call failure rate within a period of time to establish a weight calculation formula: W ij =c1S t +c2S m +c3S f ,in:

[0022] ①S t represents the call latency anomaly score, S t =max(-logP x ), P x is a kernel density estimation function. The sum density function Px of the call time series is obtained through normal data, and then the call time of the abnormal time period is obtained through the kernel density estimation function to obtain the delayed abnormal score value;

[0023] ②S m Represents the server resource utilization indicator, The specific parameter meaning and calculation method are the same as step 301;

[0024] ③S f Represents the failure call anomaly score. The calculation formula of the failure call anomaly score is defined as follows, where n represents the number of failures and m represents the total number of calls;

[0025]

[0026] For steps ①②③, the coefficients c1c2c3 are used to control the degree of influence of indicators on different types of nodes such as server nodes, container nodes, and database nodes. For only deployed servers, the resource utilization indicator feature is more important, and for database nodes, the call failure ratio feature is more important.

[0027] Step 4: Diagnose the root cause of the anomaly. For the attribute relationship graph established in step 3, use the PageRank algorithm to evaluate the degree of anomaly of each service on the attribute graph, give a ranked list of the most likely root cause nodes, and diagnose the root cause of the anomaly.

[0028] Principle of the present invention: In view of the real-time requirements of the operation and maintenance system, an unsupervised online clustering method is used to diagnose data flow anomalies. When an anomaly occurs, due to the multi-language, multi-node, and dynamic characteristics of the microservice system, the parent-child node relationship between the parsed call traces is used to clarify the call relationship of the current service execution, and combined with the deployment information of the service, a real-time system topology diagram is established, which is more conducive to accurately describing the propagation of system anomalies. At the same time, the abnormal characteristics will be reflected in multiple monitoring indicators. The present invention proposes a mixed weighted method to evaluate the degree of abnormality on the attribute graph, establish an attribute relationship graph, and finally use the PageRank algorithm to score the degree of abnormality of the service and find out the service node that caused the abnormality.

[0029] Compared with the prior art, the present invention has the following advantages. First, a new anomaly evaluation indicator is proposed: the failure ratio anomaly characteristic value, which makes the feature description of the anomaly richer and more accurate. Secondly, a hybrid weighted calculation method is used to evaluate the anomaly scores between adjacent nodes on the topology map. Compared with the original method of using only the call time single indicator to calculate the anomaly score, on the one hand, servers with only deployment relationships but no direct interface calls can be reflected on the topology map, making the method of diagnosing anomalies based on the topology map more intelligent; on the other hand, the hybrid weighted method evaluates multiple indicators, making the anomaly diagnosis more accurate and more robust. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an overall flow chart of a microservice anomaly diagnosis method based on an attribute relationship graph of the present invention;

[0031] Figure 2 This is the usage environment of the example method of the present invention. DETAILED DESCRIPTION

[0032] The present invention is described in detail below with reference to specific implementation examples and accompanying drawings.

[0033] like Figure 2As shown, as the use environment of the implementation example method of the present invention, the target microservice application is Sock-Shop, Kubernetes is used as the basic operating environment, and the service instances are deployed on the pod, where each of the 10 core services has an instance, the MongoDB service has three instances, and MySQL has one instance. An agent is deployed on each pod to monitor service call information and metric changes within the service. The load generator simulates user requests to generate load; the fault injector injects faults into the system through preset scripts to test the diagnostic effect of the fault diagnosis system; the fault diagnosis system performs fault diagnosis based on the collected data. The method proposed by the present invention is implemented in the fault diagnosis system.

[0034] like Figure 1 As shown, the method flow of the embodiment of the present invention is as follows:

[0035] Step 1: Collect the call trace information between each sub-node in the microservice through the deployed monitoring agent. The call trace information mainly includes (callType, startTime, elapsedTime, success, traceId, id,

[0036] pid, cmdb_id, serviceName), where callType represents the call type, startTime represents the call start time, elapsedTime represents the call time, success represents whether it is successful, traceId represents the call id of a complete request, id represents the current call id, pid represents the parent node id, serviceName represents the network element id to which the service belongs, and serviceName represents the service name. The call trace information is processed by subtracting the call time of all child nodes from the parent node call time, and a 30-second sliding window is used to take the median to reduce data noise. The processed time series is used through an online clustering algorithm to perform real-time anomaly detection on the data stream to determine the time point when the anomaly occurs;

[0037] Step 2: Based on the time point of the anomaly detection in step 1, the call trajectory information is parsed and combined with the deployment information of the microservice to determine the call relationship between the sub-service nodes in the microservice and the deployment relationship between the sub-service nodes and the server. The nodes on the topology map are used to represent the microservice sub-nodes, and the call relationship or deployment relationship is used to represent the edges between the nodes on the map, and the real-time topology map of the microservice application system is automatically constructed.

[0038] Step 3: Based on the time point of anomaly detection in step 1, obtain the detection data within a period of time before and after the anomaly occurs, query the monitoring data 5 minutes before and after the anomaly occurs, including the resource utilization information of the server or container, the service call trajectory information and the number of call failures, first calculate the weight attributes of the nodes on the topology map, and construct the feature vector N = (node_on-off, node_sess-connect, node_network, node_cpu, node_memory) where node_on-off represents the database monitoring switch, node_sess-connect represents the maximum number of database connections, node_network represents network traffic, node_cpu represents the container or server CPU usage, node_memory represents the container or memory usage, and calculate the similarity of the indicator feature vectors before and after the anomaly occurs. where r i represents the feature vector of the ith data point after the anomaly occurs, The feature vector mean of the five data points five minutes before the anomaly occurs is taken as the cosine similarity between the feature vector mean five minutes before the anomaly occurs and the mean within five minutes after the anomaly occurs, and the indicator weight is calculated according to the formula: The weight attribute of each node is calculated, where T i Represents the attenuation coefficient of the ith data point after the anomaly occurs, and n represents the number of data points. It is determined according to the frequency of data acquisition and the number of vectors after processing. Here, [0.95, 0.85, 0.75, 0.65, 0.55] is used. Then calculate the weight of the topological graph. The weight formula W ij =c1S t +c2S m +c3S f , S t represents the abnormality score of the abnormal trajectory, S m Represents the resource utilization anomaly score of the container or server, S f Represents the abnormal score of the failed call ratio. The parameters c1c2c3 represent the weight coefficients, which are determined according to the type of node. For general servers, containers and database nodes, c1=c2=c3=0.33; for deployment server nodes, since there is no direct service call, C1=1, C2=C3=0.

[0039] Step 4: For the attribute relationship graph established in step 3, use the PageRank algorithm to locate abnormal service nodes. In the initial stage, the abnormal weight value of the service node is used as the initial PR value of the service, P = [P0, P1,, ..., P n ]T is a column vector composed of the PR initial values ​​of multiple services. Calculate the PR value of each service, where q is the damping coefficient, usually 0.85, I(p j ) points to p j The collection of microservice subnodes, O(p j ) is p j The set of microservice subnodes pointed to, P k (p i ) is the kth iteration service p i After multiple iterations, when P k (p i ) satisfies |P k -P k-1 |<δ, that is, when P k (p i ) converges, the iteration ends. The abnormality degree of the services is sorted according to the abnormality score of the services, and the service with the highest score is considered to be the most likely to cause abnormalities.

[0040] In summary, the present invention detects service anomalies in real time by detecting service call trajectory data streams, and constructs a service topology map based on the service call relationship and service deployment information during the abnormal period. The mixed weighted weights of the nodes and edges of the topology map are calculated based on the call trajectory information of service monitoring, resource utilization information, and failure ratio within a period of time, and an attribute relationship map of abnormal service propagation is established. Finally, the most likely abnormal root cause node is inferred using the PageRank algorithm. The present invention realizes real-time detection of microservice anomalies, automatically establishes an attribute relationship map, and intelligently infers the abnormal degree of service nodes to realize abnormal diagnosis of microservices.

Claims

1. A microservice anomaly diagnosis method based on attribute relationship graph, characterized in that: The steps include: Step 1: Obtain call trajectory information based on the monitoring proxy service, process the call trajectory information, and use the processed time series through an online clustering algorithm to perform real-time anomaly detection on the data stream to determine the time point when the anomaly occurred; Step 2: Based on the abnormality detection time point of step 1, by parsing the call trajectory information and combining the deployment information of the microservice, determine the call relationship between each sub-service node in the microservice and the deployment relationship between the sub-service node and the server, use the nodes on the topology map to represent the microservice sub-nodes, use the call relationship or deployment relationship to represent the edge between the nodes on the map, and automatically build a real-time topology map of the microservice application system; Step 3: Based on the abnormality detection time point in step 1, obtain the monitoring data for a period of time before and after the abnormality occurs, calculate the personalized abnormality weight, and establish an attribute relationship diagram; Step 4: For the attribute relationship graph established in step 3, use the PageRank algorithm to evaluate the abnormality of each microservice sub-node on the attribute relationship graph, obtain the most likely root cause node ranking list, and diagnose the abnormal root cause; In step 3, the specific calculation steps are as follows: Step 301, calculate the node weights of the topology map, construct the feature vector N=(node_on-off, node_sess-connect, node_network, node_cpu, node_memory) of the nodes on the topology map, where node_on-off represents the database monitoring switch, node_sess-connect represents the maximum number of database connections, node_network represents network traffic, node_cpu represents the container or server CPU usage, node_memory represents the container or server memory usage, and customize the relevant indicator weight calculation formula: ,in Represents the similarity of the indicator feature vectors before and after the anomaly occurs, represents the attenuation coefficient, and the formula calculates the abnormality degree attribute of the node; Step 302: Calculate the weight of the topology map node connection edge, calculate the personalized weight by using the service call time before and after the exception occurs, the resource utilization information of the container or server to which the service belongs, and the call failure rate within a period of time. The weight calculation formula is established: = + + , where the specific calculation method is as follows: Represents the call latency anomaly score, calculated as , is the kernel density estimation function; Represents the failure call anomaly score, calculated as follows: n represents the number of failures, and m represents the total number of calls; 。 2. According to the microservice anomaly diagnosis method based on the attribute relationship graph of claim 1, it is characterized by: In the step 1, when processing the call trace information, a 30-second time window is used for the call time between two microservice sub-nodes, and the median of the call time within the window is taken as a single data value of a set of time series. By taking the median, the noise in the data is reduced and the data quality is improved.

3. According to the microservice anomaly diagnosis method based on the attribute relationship graph of claim 1, it is characterized by: The topology diagram of step 2 is established as follows: Step 201: for the time point of the exception occurrence specified in step 1, parse the call trace time sequence, which is in the form of a key-value pair of "parent node-child node: call time", split the parent and child service nodes, read the configuration file, obtain the deployment server information corresponding to the parent and child service nodes, and obtain all the nodes for constructing the topology map; Step 202: For all topology graph nodes obtained in step 201, if there is a call relationship between the nodes, the parent node is the out-degree node of the edge in the graph, the child node is the in-degree node of the edge in the graph, and the direction of the edge is from the parent node to the child node; if there is a deployment relationship between the nodes, the node where the service is located is the out-degree node of the edge in the graph, the deployment server node is the in-degree node of the edge in the graph, and the direction of the edge is from the service node to the deployment server.

Images