Micro-service anomaly positioning method based on spectral analysis and error reconstruction

Abstract

The invention discloses a micro-service abnormity positioning method based on spectral analysis and error reconstruction, and the method comprises the steps: 1, injecting an agent into each micro-service instance, and achieving the collection of the index information and link information of the micro-service instance; the agent comprises at least two modules, namely a link information acquisition module and an index information acquisition module, and can be realized by integrating existing open source components; step 2, performing aggregation calculation by using the collected link data, and constructing a service calling link; step 3, data preprocessing is carried out on the collected index data, and then training of an auto-encoder is carried out; in the anomaly detection stage, coding reconstruction is carried out on the current index data of the system through a variational auto-encoder, the coding reconstruction is compared with a set threshold value, and whether the micro-service instance is abnormal or not is detected; step 4, based on the obtained calling link and the reconstruction error, calculating scores of abnormal nodes from two perspectives of index abnormal scores and calling link abnormal scores by using spectral analysis and a Pearson's correlation function, and explaining weights of the scores by using a proper weight; and step 5, constructing a transition probability matrix based on the abnormal score of the node, and sorting abnormal examples or middleware by using a random walk algorithm for many times.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computers, and particularly relates to a microservice exception positioning method based on spectrum analysis and error reconstruction. BACKGROUND

[0002] Microservice exception positioning is a process of collecting various index information, link information and log information of the running state of a microservice information system, and obtaining an abnormal node or an abnormal instance through a series of analyses. The main purpose of this technology is to quickly and accurately locate system exceptions and performance bottlenecks, and to take certain measures to realize exception repair and performance bottleneck elimination.

[0003] With the continuous expansion of the scale of microservice information systems, some microservice systems include hundreds or even thousands of instances, which brings about reliability and availability problems. Any exception of service quality may propagate along the service call chain, be affected by a small number of root cause nodes to associated nodes, cause system cascading failures, and eventually lead to business-level reliability and availability problems. Therefore, when a service fails, it is necessary to quickly and accurately locate the root cause of the service failure. However, there are a large number of available monitoring indicators for microservice systems. If all these indicators are used for performance problem diagnosis, it will cause a large overhead and the service needs to be frequently updated to meet the needs of customers. This highly dynamic environment exacerbates the difficulty of root cause positioning.

[0004] Some existing methods separately process exception detection and exception positioning, resulting in high complexity of the used models or algorithms and difficulty in combination. Some of them use complex machine learning or deep learning models, which require huge computing resources and training data and are difficult to apply in actual work; the other part adopts detection models suitable for traditional monolithic architecture services, which cannot accurately reflect the state of microservice systems.

[0005] Therefore, this paper proposes an exception positioning method suitable for large-scale microservices. This method uses microservice agents for index collection and service management; uses several performance indicators reflecting service quality for exception detection, which can reduce the performance overhead of exception detection; uses an algorithm based on a random walk strategy to sort abnormal nodes, and locates abnormal microservices or middleware based on the sorting results, thereby supporting the subsequent development of operation and maintenance work. SUMMARY

[0006] The purpose of the application is to use machine learning technology and spectrum analysis method, combined with a weighted random walk method, to achieve accurate microservice exception detection and positioning with less computing resources and training data. Compared with the current mainstream exception positioning methods, this method consumes less computing resources while achieving or even exceeding the performance of the mainstream methods.

[0007] To achieve the above object, the technical scheme provided by the present application is as follows:

[0008] Step 1: Collecting the index information and link information of the microservice instance by injecting an agent into the microservice instance;

[0009] Step 2: Using the collected link data to perform aggregation calculation to build a service call link;

[0010] Step 3: Encoding and reconstructing the collected index data by using a variational autoencoder to detect whether the microservice instance is abnormal;

[0011] Step 4: Based on the obtained call link and reconstruction error, using spectral analysis and Pearson correlation function to calculate the score of the abnormal node;

[0012] Step 5: Using a random walk algorithm to sort the abnormal instances or middleware to calculate the abnormal occurrence location.

[0013] Further, the microservice agent needs to be able to capture all network traffic between microservices, including request volume, response time, error rate, and other indicators. For example, it can record the time consumption of each request from issuance to response, so as to clearly understand the performance of the service.

[0014] Further, the microservice agent can track the propagation path of the request in the microservice architecture by assigning a unique tracking identifier to each request. In addition, the microservice agent should also display and persistently store the service-related indicators, such as CPU usage, memory usage, network throughput, etc. in the form of intuitive charts. This allows the operation and maintenance personnel to easily understand the running status and resource usage of the service.

[0015] Further, the specific steps of using the collected link data to perform aggregation calculation include:

[0016] Step 21: Using the collected microservice call link data through a stream computing engine to obtain a plurality of service call links;

[0017] Step 21: Persistently storing the generated call link and displaying it through a visualization software.

[0018] Further, the reconstruction error calculated by the autoencoder includes:

[0019] Step 31: Formatting the index information data, removing dirty data, and filling missing data by using adjacent data to calculate the average;

[0020] Step 32: Inputting the data within a certain time window into the variational autoencoder for training to obtain a converged autoencoder model.

[0021] Step 33: The real-time collected data is reconstructed by the autoencoder to calculate the reconstruction error, and whether the microservice is abnormal is determined based on the reconstruction error.

[0022] In the autoencoder training phase, the response time monitoring sequence of the microservice system during normal operation is used as training data to train the autoencoder model. After multiple rounds of training, the converged autoencoder model learns the characteristics of the normal response time sequence data. Therefore, the reconstruction value of the autoencoder model for the response time monitoring value of the microservice during normal operation will be close to the monitoring value, and the corresponding reconstruction error will be small and fluctuate within a stable range.

[0023] Due to the complexity of the microservice information system, it is difficult to accurately describe the running state of different instances and middleware in the microservice information system using single modal data. The present application considers the computational overhead and result accuracy and uses an index abnormal score and call trajectory abnormal score method. The index abnormal score uses the Pearson correlation function to measure the correlation between the detection index on the microservice node and the reconstruction error. In addition, the microservice call link information is used, and the frequency spectrum analysis method is used to consider the call success rate of the related links of the microservice to obtain the abnormal score of the call trajectory. The specific steps are as follows:

[0024] Step 41: Use the Pearson function to measure the correlation between the current state and the reconstruction error to obtain the abnormal score e1;

[0025] Step 42: Use the frequency spectrum analysis method to measure the abnormality of the call trajectory to obtain the abnormal score e2;

[0026] Step 43: Consider e1 and e2 comprehensively to measure the abnormal score E of a service instance or middleware with a suitable weight;

[0027] Further, the random walk algorithm is used to realize the probability size sorting of the abnormal root cause. In a graph structure, there is a connection relationship between nodes, and the random walk starts from a starting node and moves to adjacent nodes according to a certain probability, realizing the probability sorting of nodes in the graph walk. The main steps of implementing the random walk algorithm in the present application are as follows:

[0028] Step 51: Use the generated service topology and abnormal score to define the transition probability matrix in the random walk algorithm;

[0029] Step 52: Use the transition probability matrix and service topology to perform random walk several times to obtain the abnormal node sorting;

[0030] Step 53: The obtained abnormal node is positioned as the starting node of the next random walk.

[0031] Advantages:

[0032] (1) The application provides a microservice abnormality positioning method based on spectrum analysis and error reconstruction. The method uses a microservice abnormality detection method based on an autoencoder to learn the correlation between microservice index states and response time fluctuations, and to detect microservice abnormality states in real time.

[0033] (2) The microservice abnormality positioning method provided by the application captures the calling relationship between microservices through a microservice agent, and then constructs a service calling relationship graph to simulate the propagation path, and updates the abnormal weight of the nodes in the service calling relationship graph in combination with the reconstructed error generated by the abnormality detection.

[0034] (3) The microservice abnormality positioning method provided by the application is based on the abnormal weight relationship and combines a spectrum-based analysis method to propose a random walk algorithm, which iterates the probability of a walker staying at each node in the service relationship graph, and then sorts the root causes of the abnormal service nodes. BRIEF DESCRIPTION OF DRAWINGS

[0035] The drawings used in the application are described as follows.

[0036] Figure 1 A microservice abnormality positioning method flowchart is shown.

[0037] Figure 2 A microservice abnormality positioning method flowchart is shown. DETAILED DESCRIPTION

[0038] In order to more clearly illustrate the technical solutions of the application, the application will be described in detail based on the drawings. In addition, the description here is only used to explain the application, and all other embodiments obtained by those skilled in the art without creative labor achievements belong to the scope of protection of the application.

[0039] The application discloses a microservice exception positioning method based on spectrum analysis and error reconstruction, and the implementation steps of the method comprise the following steps: step 1: by injecting an agent into each microservice instance, the microservice instance index information and link information are collected. The agent comprises at least two modules, namely a link information collection module and an index information collection module, and the agent can be realized by integrating existing open source components; step 2: the collected link data is used for convergence calculation to construct a service call link; step 3: the collected index data is preprocessed, and then the training of an autoencoder is performed; in the abnormality detection stage, the current index data of the system is encoded and reconstructed by using a variational autoencoder, and is compared with a set threshold value to detect whether the microservice instance is abnormal; step 4: based on the obtained call link and reconstruction error, spectrum analysis and a Pearson correlation function are used to calculate the scores of abnormal nodes from two aspects of index abnormality scores and call link abnormality scores, and a suitable weight is used to explain the weights of the abnormal nodes; and step 5: based on the node abnormality scores, a transition probability matrix is constructed, and a random walk algorithm is used multiple times to sort abnormal instances or middleware.

[0040] Figure 1 The microservice exception positioning method of the application is shown.

[0041] Figure 2 The microservice exception positioning method of the application is shown. The identification process specifically comprises the following steps:

[0042] Step 1: an agent is injected into each microservice, and the agent must comprise at least the following two functional modules: an index information collection module and a link call information collection module. The agent can realize the index information collection of the instance by combining an open source monitoring component prometheus, simultaneously track the link call by intercepting a request and injecting a unique identifier, and forward the request sent by the agent to the agent corresponding to the target service. After the index information and the link information are collected, the collected information is transmitted to a remote server through an http\gRPC protocol, so that subsequent processing or formatted storage is realized;

[0043] Step 2: after the service link information is collected, first, data cleaning is performed to filter irrelevant and error data, and then the distributed request links are aggregated together according to the request ID and the tracking ID; then each service is regarded as a node, the call relationship between services is extracted from the service link data, and two service corresponding nodes are connected by using a weighted directed edge, the direction of the directed edge represents the service call direction, and the weight of the directed edge can represent the call frequency, success rate, request response time or other parameter information.

[0044] Step 3: After collecting the indicator information, first filter the indicator information to remove error data and dirty data; then normalize the data, which can increase the convergence speed; then extract the weight information through an attention layer to learn the contribution of different indicators in system anomaly detection; after extracting the feature information, pass the feature information through a variational autoencoder. For example, for microservice v e ∈V, the average response time series data of duration time window w collected at time t As the input of the autoencoder, through the encoding layer is mapped to the latent feature representation

[0045]

[0046] where g is the activation function, is the weight matrix between the input layer and the hidden layer, and b is the bias vector of the input layer. Then through the decoder, the latent feature representation is reconstructed to the average response time series data of microservice module v e

[0047]

[0048] where is the weight matrix of the hidden layer and the output layer, and c is the bias vector of the hidden layer. Then calculate the reconstruction mean square error between and

[0049]

[0050] After multiple rounds of training, the variational autoencoder can learn the characteristics of the normal response time series data. By calculating the mean μ and standard deviation σ of the reconstruction error at this time, the anomaly detection threshold α of microservice module v e is determined = μ + 3σ. In the real-time detection process of microservice v e , if the reconstruction error is found, it is considered that the microservice v e has an anomaly.

[0051] Step 4: In the calculation of node anomaly score stage, we need to calculate the indicator anomaly score and link anomaly score. First, we define the indicator anomaly score, and in this paper, we use the Pearson correlation function to measure the correlation between the detection indicator X e on microservice v e and the reconstruction error, which can be expressed as:

[0052] ​​

[0053] Secondly, we use spectral analysis to measure the abnormal degree of the invocation trajectory, that is, the greater the abnormal score of the invocation trajectory connecting more abnormal requests and fewer normal requests. For each microservice, the invocation trajectory passes through multiple microservices, and the attribute graph construction respectively counts the number of abnormal and normal invocation trajectories of each service wherein, represents the number of abnormal invocation trajectories containing the microservice, represents the number of normal invocation trajectories containing the microservice, represents the number of abnormal invocation trajectories not containing the microservice. Then, the abnormal invocation trajectory score of the node is calculated using the Ochiai formula, which is defined as follows:

[0054]

[0055] Considering the abnormal score, E e1 needs to be normalized to obtain E' e1 , so the abnormal score of the node can be defined as follows:

[0056] E(v e )=αE e1 +βE e2

[0057] Step 5: After obtaining the abnormal score and service topology graph, the present application uses a random walk algorithm to sort the abnormal node probability. The random walk algorithm is based on the probability of each node accessing other nodes, so the transition probability matrix U needs to be defined first. Wherein u ij represents the probability of random walking from node v i to node v j , and the calculation method is as follows:

[0058]

[0059] wherein, v i →v j represents that there is an execution trajectory from v i to v j , and linkOut(v j ) represents the sum of abnormal weights of microservice nodes called by microservice v j . For all nodes, initialize PR score as PR0(v i )=1 / n, connect all node PR scores into a vector R0, and update node PR score in each round of random walk success:

[0060] R m =dU·R n-1 +(1-d)R0

[0061] wherein R m represents the PR score vector of all nodes after m iterations, d is a coefficient with a value range of (0, 1), and is usually set to 0.85. After continuous iteration, the PR score tends to converge, at which time the higher the PR score, the greater the abnormal possibility of the corresponding microservice. Finally, a ranking list of abnormal root microservices is output according to the PR score from high to low.

[0062] The disclosed microservice abnormal positioning method based on spectrum analysis and reconstruction error uses a microservice agent to collect indexes and manage services; uses a plurality of performance indexes reflecting service quality to detect abnormalities, which can reduce the performance overhead of abnormal detection; uses an algorithm based on a random walk strategy to sort abnormal nodes, and locates abnormal microservices or middleware based on the sorting result, thereby providing strong support for subsequent operation and maintenance work.

[0063] The above description is only a description of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application is included in the protection scope of the present application.

Claims

1. A microservice anomaly localization method based on spectrum analysis and error reconstruction, characterized in that, Comprise the following steps: Step 1: realize the collection of microservice instance index information and link information by injecting an agent into the microservice instance; Step 2: use the collected link data for aggregation calculation to build a service call link; Step 3: use the collected index data to realize encoding reconstruction through a variational autoencoder to detect whether the microservice instance is abnormal; Step 4: based on the obtained call link and reconstruction error, use spectral analysis and Pearson correlation function to calculate the score of the abnormal node; Step 5: use a random walk algorithm to sort the abnormal instances or middleware to calculate the abnormal occurrence position.

2. The microservice abnormal positioning method based on spectrum analysis and error reconstruction according to claim 1, characterized in that, The agent in step 1 at least includes an index information collection module and a link information collection module. The collected information is transmitted to a remote server through HTTP\gRPC protocol.

3. The method of claim 1, wherein, The specific steps of using the collected link data for aggregation calculation include: Step 21: use the collected microservice call link data to obtain a plurality of service call links through a stream computing engine; Step 21: persistently store the generated call link and display it through a visualization software.

4. The method of claim 1, wherein, The autoencoder calculates the reconstruction error, which includes: Step 31: format the index information data, remove dirty data, and fill in missing data by using adjacent data to calculate the average; Step 32: input the data in a certain time window into the variational autoencoder for training to obtain a converged autoencoder model. Step 33: use the autoencoder to calculate the reconstruction error of the real-time collected data, and determine whether the microservice is abnormal based on the reconstruction error.

5. The method of claim 1, wherein, The present application comprehensively considers the calculation overhead and result accuracy, and uses a method of index abnormal score and call trajectory abnormal score. The index abnormal score uses Pearson correlation function to measure the correlation between the detection index on the microservice node and the reconstruction error. In addition, the microservice call link information is used, and the spectral analysis method is used to consider the call success rate of the related link of the microservice to obtain the abnormal score of the call trajectory. The specific steps are as follows: Step 41: use the Pearson function to measure the correlation between the current state and the reconstruction error to obtain an abnormal score e1; Step 42: use the spectral analysis method to measure the abnormality of the call trajectory to obtain an abnormal score e2; Step 43: comprehensively consider e1 and e2 to measure the abnormal score E of a service instance or middleware with a suitable weight.

6. The method of claim 1, wherein, The random walk algorithm is used to realize the probability size sorting of the abnormal root cause. In a graph structure, there is a connection relationship between nodes, and the random walk starts from a starting node, moves to adjacent nodes according to a certain probability, and realizes node probability sorting in the graph walk. The main steps of implementing the random walk algorithm in the present application are as follows: Step 51: use the generated service topology and abnormal score to define the transition probability matrix in the random walk algorithm; Step 52: use the transition probability matrix and service topology to perform a plurality of random walks to obtain an abnormal node sorting; Step 53: use the obtained abnormal node to locate the starting node of the next random walk.

Images