Micro-service call chain anomaly detection method based on diffusion model

Abstract

The invention provides a micro-service call chain anomaly detection method based on a diffusion model, and the method comprises the following steps: S1, obtaining KPIs data and call chain data, and carrying out the preprocessing of the KPIs data and the call chain data; s2, performing time feature extraction on the preprocessed KPIs data based on LSTM (Long Short Term Memory); s3, fusing the KPIs time characteristic data and the preprocessed calling chain data to construct a micro-service calling relation graph; s4, performing feature extraction on the micro-service calling relation graph based on GCN to obtain a latent variable of the micro-service calling relation graph; s5, reconstructing the micro-service calling relation graph based on a diffusion model and a latent variable, and outputting a trained model; s6, determining an abnormal score threshold according to a reconstruction error output by the trained model; and S7, determining an abnormal call chain based on an abnormal score threshold. According to the method and the device, the problems of relatively low anomaly detection accuracy and low call chain reconstruction precision caused by neglecting the micro-service deployment environment resource consumption anomaly in the existing anomaly detection are solved.

Description

Technical Field

[0001] This invention relates to the field of microservice application system technology, and more particularly to a method for detecting anomalies in microservice call chains based on a diffusion model. Background Technology

[0002] In microservice application systems, monitoring the microservices system can collect KPIs data from the container where the microservices reside and call chain data generated by user requests. KPIs data, such as CPU utilization and memory utilization, can intuitively reflect the system status, while call chain data can accurately reflect the call paths of services involved in the microservice system to respond to user requests. Due to their distributed deployment and dynamic scaling characteristics, complex microservice architectures are prone to rapid propagation of service failures through inter-service dependencies, potentially triggering a cascading failure effect. This can not only cause irreparable and huge losses to the enterprise but also severely impact the user experience. Anomaly detection of the call chain is an effective way to ensure reliable system operation. However, the call chain only records the interaction information between services within the system, while KPIs data records external performance information. Therefore, combining KPIs data can compensate for the lack of external performance information.

[0003] Currently, mainstream microservice call chain anomaly detection methods include statistical methods, traditional machine learning methods, and deep learning methods. Statistical call chain anomaly detection methods typically detect anomalies based on the distribution characteristics of call chain data. It models the normal behavior of the call chain and then identifies abnormal deviations, usually using descriptive statistical analysis data such as mean and standard deviation for comparison. Traditional machine learning-based call chain anomaly detection methods have gradually replaced statistical methods due to their flexibility in handling various problems. This method typically preprocesses and extracts features from the call chain data before subsequent model training and anomaly detection. Deep learning-based call chain anomaly detection methods include response time-based and call structure-based methods. Response time-based methods judge the user's response time; if the response time is outside the normal response time range of the microservice's normal operation, it is considered abnormal. However, due to the complexity of microservice systems and the numerous call relationships between services, this method is difficult to comprehensively and accurately reflect the system's normal behavior patterns. Call structure-based methods record the microservices involved in the execution of user requests to reflect the call relationships between microservices.

[0004] In recent years, graph neural networks (GNNs) have demonstrated excellent performance in extracting graph structural features, and the rapid development of generative models has led to their increasing application in various fields. Many studies have proposed using a combination of GNNs and generative models for anomaly detection in call chains. For example, VGAE (Variational Graph Autoencoder) uses graph convolutional networks to extract structural features of the call chain graph and reconstructs the call chain structure graph based on the VAE framework, using the reconstruction error to determine anomalies in the call chain. However, because the adjacency matrix of the call chain structure graph is sparse, the quality of samples generated by the model during call chain reconstruction is poor, resulting in low detection accuracy.

[0005] In summary, most methods for anomaly detection in microservice systems based on call chain data still have the following problems: (1) When performing anomaly detection on the call chain, the monitoring information of the microservice's environment is not fully utilized, making it impossible for the model to capture more and finer-grained relevant information, resulting in low accuracy of anomaly detection. (2) When using current mainstream generative models such as VAE and GAN (Generative Adversarial Network) to reconstruct the call chain, only a limited number of samples with little difference and low quality can be generated, which leads to low accuracy of call chain reconstruction and thus low accuracy of anomaly detection. Summary of the Invention

[0006] In view of this, the purpose of this invention is to propose a microservice call chain anomaly detection method based on a diffusion model, so as to solve the problems of low anomaly detection accuracy and low call chain reconstruction accuracy caused by ignoring abnormal resource consumption in the microservice deployment environment in existing anomaly detection methods.

[0007] The technical means employed in this invention are as follows: A method for detecting anomalies in microservice call chains based on a diffusion model includes the following steps: S1. Obtain KPIs data and call chain data from the microservice system, and preprocess the KPIs data and call chain data; S2. Extract time features from the preprocessed KPIs data based on LSTM; S3. Construct a microservice call relationship graph by integrating KPI time feature data and preprocessed call chain data; S4. Based on GCN, feature extraction is performed on the microservice call relationship graph to obtain the latent variables of the microservice call relationship graph; S5. Reconstruct the microservice call relationship graph based on the diffusion model and latent variables, and output the trained model; S6. Determine the anomaly score threshold based on the reconstruction error output by the trained model. S7. Determine the abnormal call chain based on the abnormal score threshold.

[0008] Furthermore, S1 specifically includes the following steps: The KPIs are constructed using D dimensions of KPIs that reflect the resource consumption of the microservice deployment environment, and are represented as follows: T represents the total time step, and at any time step t, we have ; Use the min-max normalization method to modify the size of all KPI data to the range [0,1), as shown below: (1) in It is the minimum value of x. It is the maximum value of x. It is the minimum constant; The call chain data is generated by a distributed tracing system. A call chain includes several pieces of information, including: cmdb_id, traceid, spanid, and parentspan. cmdb_id identifies the object name of the environment in which the call chain data resides; traceid uniquely identifies a call chain generated by a single user request; parentspan identifies the initiator of the microservice call; and spanid identifies the responder of the microservice call. Each call chain is extracted using traceid, and the call relationships within each chain are obtained using parentspan and spanid. The response time of each node in the call chain is calculated by subtracting the timestamp of the client receiving the server's response from the timestamp of the client sending the request to the server (cs).

[0009] Furthermore, S3 specifically includes the following steps: Construct a graph structure for the call chain; using spanid and parentspan, identify the caller and callee for each microservice call in a call chain, and construct an adjacency matrix for the nodes of the microservice call relationship graph based on the caller and callee information. The response time is used as a feature of each microservice node. Simultaneously, based on the environment object information and call timestamp of each microservice node in the call chain, the KPIs time features under the corresponding timestamp and environment object are used as feature attributes of the microservice node, collectively forming the original feature matrix of the graph. .

[0010] Furthermore, S4 specifically includes the following steps: Construct an undirected graph of microservice call relationships, use a GCN of depth L to extract features from the graph, and then extract its original feature matrix. Encoding as latent variables ; in the The layer encodes the input features according to the following formula: (2) The final latent variables can be obtained using the following formula. : (3) in Indicates the first The node feature matrix of the layer; This represents the adjacency matrix with self-loops added, i.e. , Let be the adjacency matrix of the graph. It is the identity matrix; express The degree matrix; Indicates the first The learnable weight matrix of the layer; Indicates the activation function; Multiplication represents a linear transformation, used for feature extraction; This is the approximate mean of the posterior probability. The standard deviation of the approximate posterior probability. This represents Gaussian noise.

[0011] Furthermore, S5 specifically includes the following steps: S51. Diffusion process; latent variables Noise is gradually added until a completely random noise sample is obtained; S52, Denoising process: A noise predictor is used to predict the noise level contained in the original latent variable at each time step. The predicted noise is subtracted from the noisy latent variable to generate a latent variable that is close to the original latent variable. Finally, a decoder is used to decode and generate data samples.

[0012] Furthermore, S51 specifically includes the following steps: For a latent variable Define a latent variable distribution To describe time At the time Differences from the original latent variables; Initially, Gaussian noise is gradually added. until a completely random noise sample is obtained. , and The relationship between them is as follows: (4) in, Indicates the weight of the current sample; The weight representing the noise; Obtain a completely random noise sample The formula is as follows: (5) in, , Indicates adding to Sampling noise.

[0013] Furthermore, S52 specifically includes the following steps: Define a conditional distribution Used to describe how to extract from noisy samples Obtain denoised samples From completely random noise samples ( Start by gradually removing prediction noise. Until a sample with completely removed noise is obtained. Then, the decoder is used to decode and generate the reconstructed microservice call relationship diagram. ; The loss function is It is necessary to train a system that enables the generation of... The parameter with the highest probability The transformation process, which employs the variational lower bound of the objective log-likelihood, is as follows: (6) in, Indicates KL divergence; express sequence, express sequence; Convert the KL divergence to a logarithm. The summation of the terms in the form of the first term is given by the following formula: (7) in As a variational lower bound, the variational lower bound is derived and simplified to obtain the following formula: (8) Among them, the first item There are no learnable parameters; its KL divergence is a fixed value and can be ignored. The second term is a KL divergence, consisting of two normally distributed parts, the first part being... The mean If the variance follows a normal distribution, the mean formula is as follows: (9) The variance formula is as follows: (10) in ; from Subtracting the predicted random noise from the result completes the denoising process.

[0014] Furthermore, S6 specifically includes the following steps: A threshold is determined based on the reconstruction error of the training data. High percentiles are selected to capture outliers with large reconstruction errors. The IQR method is used to determine the threshold. First, the reconstruction error in all training datasets is calculated. S As shown below: (11) in, For noise sampled from a Gaussian distribution, Indicates a noise predictor; Then, based on the reconstruction error of the training set, first specify the first quartile Q1 and the third quartile Q3, and then calculate the interquartile range (IQR) according to the following formula: (12) Set the abnormal threshold using The formula is as follows: (13) Use the IQR method to specify that Q1 and Q3 adjust the anomaly thresholds based on the false positive rate and the false negative rate.

[0015] Furthermore, S7 specifically includes the following steps: The microservice call relationship graph is input into the trained model, which can calculate its anomaly score. Then, the anomaly score is compared with a pre-set anomaly score threshold. If the anomaly score is greater than the threshold, it is marked as anomaly data; otherwise, it is marked as normal data.

[0016] Compared with the prior art, the present invention has the following advantages: Using call chains to detect system anomalies in distributed microservice systems is a widely recognized and effective method. However, relying solely on call chains ignores the abnormal resource consumption issues inherent in the deployment environment of the microservice system. This not only leads to the incomplete utilization of data generated in the system but also results in a decline in anomaly detection performance. Therefore, this invention proposes a method that combines KPI data and call chain data to jointly detect anomalies in microservice systems, thereby fully utilizing external data information of the microservice system. Current call chain anomaly detection methods based on refactoring mostly face the problems of limited sample types and low accuracy, resulting in poor refactoring results. Therefore, this invention utilizes the diffusion model's ability to generate more diverse and higher-quality samples to replace the traditional generative model for refactoring, thereby improving refactoring accuracy and achieving better anomaly detection results. Experiments demonstrate that this invention achieves better call chain anomaly detection results and has significant advantages compared to other mainstream call chain anomaly detection methods. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of the method of the present invention.

[0019] Figure 2 This is a microservice call relationship diagram based on the integration of KPIs and call chains in this invention.

[0020] Figure 3 This invention relates to a microservice call relationship graph reconstruction model based on a diffusion model. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] like Figure 1 As shown, this invention provides a microservice call chain anomaly detection method based on a diffusion model, characterized by the following steps: S1. Obtain KPIs data and call chain data, and preprocess the KPIs data and call chain data; To assist in anomaly detection in microservice systems using call chain data and improve detection accuracy, this invention uses D-dimensional KPIs reflecting resource consumption in the microservice deployment environment, including CPU-related metrics (such as CPU utilization), memory-related metrics (such as used memory size), and network-related metrics (such as network transmission speed), to construct a KPI time series, represented as follows: T represents the total time step, and at any time step t, we have Since there are differences in magnitude between different indicators, it is necessary to normalize these data. This invention uses the min-max normalization method to modify the size of all KPI data to the range [0,1), as shown in formula (1).

[0024] (1) in It is the minimum value of x. It is the maximum value of x, and It is a local constant, the purpose of which is to prevent the denominator from being zero.

[0025] Call chain data is generated by a distributed tracing system, helping operations personnel track every microservice call. A call chain includes multiple pieces of information (such as cmdb_id, traceid, spanid, parentspan, etc.). cmdb_id identifies the object name of the environment in which the call chain data resides; traceid uniquely identifies a call chain generated by a user request; spanid and parentspan identify the caller and callee in a microservice call; parentspan identifies the initiator of the microservice call; and spanid identifies the responder. The original call chain dataset stores data for single calls, but because traceid is a globally unique value, each call chain can be extracted using traceid, and the call relationships within each chain can be obtained using parentspan and spanid. A service call involves four processes: the client sends a request to the server (cs); the server receives the client's request (sr); the server responds to the request and sends it back to the client (ss); and the client receives the server's response (cr). The response time of a node in the call chain can be calculated by subtracting the timestamp of cr from the timestamp of cs, which is the time from when the client sends a request to when it receives a response.

[0026] S2. Extract time features from the preprocessed KPIs data based on LSTM; During the training phase, LSTM (Long Short-Term Memory) is first used to capture the temporal features of the KPI data. As a type of temporal recurrent neural network, LSTM uses a gating mechanism, employing a forget gate to help the network forget irrelevant information and input and output gates to help the network retain useful long-term dependencies. Therefore, this invention uses LSTM to capture the temporal features in the KPI data.

[0027] S3. Construct a microservice call relationship graph by integrating KPI time feature data and preprocessed call chain data; First, construct the graph structure of the call chain. A call chain contains many microservice calls. The caller and callee for each call can be identified using `spanid` and `parentspan`. Based on this information, construct the adjacency matrix of the nodes in the microservice call relationship graph. The response time is used as a feature of each microservice node. Simultaneously, based on the environment object information and call timestamp of each microservice node in the call chain, the KPIs time features under the corresponding timestamp and environment object are used as feature attributes of the microservice node, collectively forming the original feature matrix of the graph. The process of constructing a microservice call relationship graph by integrating KPI information and call chain data is as follows: Figure 2 As shown.

[0028] S4. Based on GCN, feature extraction is performed on the microservice call relationship graph to obtain the latent variables of the microservice call relationship graph; Inspired by the achievements of CNN (Convolutional Neural Network) in computer vision, GCN extends traditional convolution operations to graph-structured data. It can learn global and local information of microservice call relationship graphs, capturing node and structural information. Furthermore, because GCN uses graph convolution to extract information from graphs, it is more suitable for computation on undirected graphs. Therefore, this invention constructs an undirected graph of microservice call relationships, uses a GCN of depth L to extract features from the graph, and then extracts its original feature matrix. Encoding as latent variables In the first The layer encodes the input features according to formula (2), and uses formula (3) to obtain the final latent variables. .

[0029] (2) (3) in Indicates the first Layer node feature matrix (input layer) (The original feature matrix); This represents the adjacency matrix with self-loops added, i.e. ( Let be the adjacency matrix of the graph. (where the matrix is ​​an identity matrix), allowing nodes to utilize their own information; express degree matrix ( ); Indicates the first The learnable weight matrix of the layer; Indicates the activation function; Multiplication represents a linear transformation, used for feature extraction. and The outputs of the graph convolution represent the mean and standard deviation of the approximate posterior probability, respectively. This represents Gaussian noise.

[0030] S5. Reconstruct the microservice call relationship graph based on the diffusion model and latent variables, and output the trained model; This invention proposes a microservice call relationship graph reconstruction model based on a diffusion model. This model uses two stages to complete the reconstruction of the microservice call relationship graph. The first stage is a diffusion process, which modifies the latent variables obtained in step 4 by encoding the microservice call relationship graph using GCN. The process involves gradually adding noise until a completely random noise sample is obtained. The second stage is denoising, which essentially uses a noise predictor to predict the amount of noise contained in the original latent variable at each time step. Then, subtracting the predicted noise from the noisy latent variable generates a latent variable that is very close to the original one. Finally, a decoder is used to decode and generate a data sample, thus completing the reconstruction of the microservice call graph. The microservice call graph reconstruction model based on the diffusion model is as follows: Figure 3 As shown.

[0031] S51, Diffusion process; For a latent variable Define a latent variable distribution Used to describe time At the time The difference. From the original latent variables Initially, Gaussian noise is gradually added. until a completely random noise sample is obtained. . and The relationship between them is shown in formula (4).

[0032] (4) in, It is a set of pre-determined parameters that represent the weights of the current sample. The weight of the noise is t, the larger t is. The smaller the value, the greater the added noise becomes as the time step progresses. This is because for a latent variable that has no added noise... It can be easily changed by adding a little noise, but as time goes on, the noise increases, and more noise needs to be added to change it.

[0033] The goal of the diffusion process is to obtain a completely random noise sample. However, the solution is very inefficient by iteratively solving the problem. Since the noise added in each step is independent and follows a normal distribution, formula (4) can be derived and simplified to obtain formula (5).

[0034] (5) in, , Indicates adding to The sampling noise at each time step is... Both can use the original latent variables It is obtained by calculation with known parameters.

[0035] S52, Denoising and Reconstruction Process; Define a conditional distribution Used to describe how to extract from noisy samples Obtain denoised samples This stage starts from completely random noise samples. ( Start by gradually removing prediction noise. Until a sample with completely removed noise is obtained. Then, the decoder is used to generate the refactored microservice call relationship graph. .

[0036] The loss function in this stage is To minimize it, we need to make it so that Maximum, that is, training a value that enables generation The parameter with the highest probability From stage 1, we know that... Depends on each previous time step ,so The probability is difficult to calculate. This problem is optimized by calculating the variational lower bound of the objective log-likelihood, and the transformation process is shown in Equation (6).

[0037] (6) in, Let KL represent the KL divergence, which measures the similarity between two distributions. Since the KL divergence is non-negative, the formula must hold true. express sequence, express Sequence. Convert KL divergence to logarithm. If the form is summed with the preceding term, then formula (6) can be simplified to formula (7).

[0038] (7) in This is the variational lower bound. Now the problem is to optimize the variational lower bound. The variational lower bound is derived and simplified to obtain the result shown in formula (8).

[0039] (8) Among them, the first item There are no learnable parameters; its KL divergence is a fixed value and can be ignored. The second term is also a KL divergence, consisting of two normally distributed parts, the first part being... The mean The variance follows a normal distribution. The mean and variance are the values ​​that need to be solved. The calculation methods for the mean and variance can be derived as shown in formula (9) and formula (10).

[0040] (9) (10) in Therefore, it is related to Both are fixed values. From formula (9), we can see that we only need to... Subtracting the predicted random noise from the result completes the denoising process.

[0041] S6. Determine the anomaly score threshold based on the reconstruction error output by the trained model. After training the diffusion model, a threshold is determined based on the reconstruction error of the training data. Since the reconstruction error of normal samples is usually small and the reconstruction error of abnormal samples is large, a high percentile can be selected to capture outliers with large reconstruction errors. In order to further reduce the false alarm rate, this invention uses the IQR method to determine the threshold. First, the reconstruction error in all training datasets needs to be calculated. The reconstruction error S of the microservice call relationship graph reconstruction model based on the diffusion model proposed in this invention is calculated as shown in formula (11).

[0042] (11) in, For noise sampled from a Gaussian distribution, This indicates a noise predictor.

[0043] Then, based on the reconstruction error of the training set, first specify the first quartile Q1 and the third quartile Q3, and then calculate the interquartile range IQR according to formula (12).

[0044] (12) Set the abnormal threshold using The calculation is shown in formula (13).

[0045] (13) Using the IQR method, Q1 and Q3 can be used to adjust the anomaly thresholds based on the false alarm rate and the false negative rate. When normal data far outnumbers abnormal data, the false alarm rate can be reduced, thereby improving the accuracy of anomaly detection.

[0046] S7. Determine the abnormal call chain based on the abnormal score threshold.

[0047] The microservice call relationship graph is input into the pre-trained model, which can calculate its anomaly score. Then, the anomaly score is compared with a pre-set anomaly score threshold. If the anomaly score is greater than the pre-set anomaly score threshold, it is marked as abnormal data; otherwise, it is marked as normal data.

[0048] Example The effectiveness of the method of this invention is verified from two aspects. Firstly, because the diffusion model used in this invention is a traditional generative model, the method of this invention is experimentally compared with that using a traditional VAE model. Secondly, three popular call chain anomaly detection methods are selected for experimental comparison with the method of this invention. Both are evaluated using three widely used evaluation metrics.

[0049] All models used in this invention are implemented using PyTorch, and the model and the comparison model are trained on an RTX3050 GPU.

[0050] (1) Dataset and Experiment Evaluation Metrics The dataset used in this invention comes from the public dataset released in the 2022 AIOps Challenge. This dataset is derived from a simulated e-commerce system based on a microservice architecture. The system sets up 10 microservices, each deployed on 4 different pods.

[0051] The evaluation metrics used in the comparative experiments were precision, recall, and F1 score. These three metrics are commonly used to evaluate the performance of classification models in binary or multi-class classification tasks. Their calculations are shown in equations (14), (15), and (16), respectively.

[0052] (14) (15) (16) Precision represents the proportion of samples predicted as positive that are actually positive; recall represents the proportion of samples that are actually positive that are correctly predicted as positive; the F1 score is the harmonic mean of precision and recall, used to comprehensively reflect the model's accuracy and robustness. TP represents true positives, FP represents false positives, and FN represents false negatives.

[0053] (2) Comparative experiment The effectiveness of the method of the present invention will be verified from two aspects below.

[0054] 1) The method of this invention was compared experimentally with a VAE-based call chain anomaly detection method to verify whether using a diffusion model instead of a traditional generative model can improve the accuracy of call chain anomaly detection. The experimental results are shown in Table 1.

[0055] Table 1. Experimental Comparison of the Invention Method and Traditional Generative Modeling Methods

[0056] As shown in Table 1, the experimental results demonstrate that the method of this invention achieves precision, recall, and F1 score all exceeding 94% for call chain anomaly detection. In contrast, the traditional generative model method, while achieving slightly lower precision, exhibits significantly lower recall and F1 score. This indicates that the traditional generative model generates a limited set of data samples for call chain reconstruction, resulting in a narrow discrimination boundary and an inability to correctly identify diverse call chains. Consequently, a large number of normal call chains are misclassified as anomalous, leading to a decline in detection performance. These experimental results confirm the effectiveness of using the diffusion model for call chain anomaly detection.

[0057] 2) The call chain anomaly detection model of this invention is compared experimentally with three mainstream call chain anomaly detection models. The following three call chain anomaly detection methods are selected as comparison methods.

[0058] TraceAnomaly: Trains a deep Bayesian network-based model based on the service call sequence of a normal call chain.

[0059] TraceVAE: It uses a graph variational autoencoder-based model to perform unsupervised anomaly detection on call chains.

[0060] GTrace: It uses a Tree-LSTM-based encoder and decoder model to detect structural anomalies in the call chain; and a graph neural network-based encoder and decoder to detect temporal anomalies in the call chain.

[0061] Table 2. Experimental comparison of the call chain anomaly detection method of this invention with three other methods.

[0062] The experimental results are shown in Table 2. The method of this invention is named DTrace. As can be seen from the table, the precision and F1 score of this invention are the highest compared to the other three methods, while the recall rate is only lower than that of the TraceAnomaly method. This is because TraceAnomaly can perform anomaly detection on the entire microservice, proposing a unified approach to simultaneously detect response time anomalies and call chain structure anomalies, but it does not consider the correlation between these two anomalies. It may not simultaneously determine anomalies based on response time and call chain structure features, causing the model to classify normal data as anomalies, resulting in low precision but high recall. TraceVAE and GTrace can both be seen as anomaly detection based on improved VAE models, but their reconstruction capabilities are not as good as diffusion models, leading to relatively poor results. Overall, the anomaly detection performance of the method of this invention is worse than the other three methods.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for detecting anomalies in microservice call chains based on a diffusion model, characterized in that, Includes the following steps: S1. Obtain KPIs data and call chain data from the microservice system, and preprocess the KPIs data and call chain data; S2. Extract time features from the preprocessed KPIs data based on LSTM; S3. Construct a microservice call relationship graph by integrating KPI time feature data and preprocessed call chain data; S4. Based on GCN, feature extraction is performed on the microservice call relationship graph to obtain the latent variables of the microservice call relationship graph; S5. Reconstruct the microservice call relationship graph based on the diffusion model and latent variables, and output the trained model; S6. Determine the anomaly score threshold based on the reconstruction error output by the trained model. S7. Determine the abnormal call chain based on the abnormal score threshold.

2. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S1 specifically includes the following steps: The KPIs are constructed using D dimensions of KPIs that reflect the resource consumption of the microservice deployment environment, and are represented as follows: T represents the total time step, and at any time step t, we have ; Use the min-max normalization method to modify the size of all KPI data to the range [0,1), as shown below: （1） in It is the minimum value of x. It is the maximum value of x. It is the minimum constant; The call chain data is generated by a distributed tracing system. A call chain includes multiple pieces of information as follows: cmdb_id, traceid, spanid, and parentspan; cmdb_id is used to identify the object name of the environment in which the call chain data is located, traceid is used to uniquely identify a call chain generated by a user request, parentspan identifies the initiator of this microservice call, and spanid identifies the responder of this microservice call. Each call chain is extracted using traceid, and the call relationship in each call chain is obtained using parentspan and spanid. The response time of each node in the call chain is calculated by subtracting the timestamp of the client receiving the server's response cr from the timestamp of the client sending the request to the server cs.

3. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S3 specifically includes the following steps: Construct a graph structure for the call chain; using spanid and parentspan, identify the caller and callee for each microservice call in a call chain, and construct an adjacency matrix for the nodes of the microservice call relationship graph based on the caller and callee information. ; Response time is used as a feature of each microservice node. Simultaneously, based on the environment object information and call timestamp of each microservice node in the call chain, the KPIs time features under the corresponding timestamp and environment object are used as feature attributes of the microservice node, collectively forming the original feature matrix of the graph. .

4. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S4 specifically includes the following steps: Construct an undirected graph of microservice call relationships, use a GCN of depth L to extract features from the graph, and then extract its original feature matrix. Encoding as latent variables ; in the The layer encodes the input features according to the following formula: （2） The final latent variables can be obtained using the following formula. : （3） in Indicates the first The node feature matrix of the layer; This represents the adjacency matrix with self-loops added, i.e. , Let be the adjacency matrix of the graph. It is the identity matrix; express The degree matrix; Indicates the first The learnable weight matrix of the layer; Indicates the activation function; Multiplication represents a linear transformation, used for feature extraction; This is the approximate mean of the posterior probability. The standard deviation of the approximate posterior probability. This represents Gaussian noise.

5. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S5 specifically includes the following steps: S51. Diffusion process; latent variables Noise is gradually added until a completely random noise sample is obtained; S52, Denoising process: A noise predictor is used to predict the noise level contained in the original latent variable at each time step. The predicted noise is subtracted from the noisy latent variable to generate a latent variable that is close to the original latent variable. Finally, a decoder is used to decode and generate data samples.

6. The microservice call chain anomaly detection method based on diffusion model according to claim 5, characterized in that, S51 specifically includes the following steps: For a latent variable Define a latent variable distribution To describe time At the time Differences from the original latent variables; Initially, Gaussian noise is gradually added. until a completely random noise sample is obtained. , and The relationship between them is as follows: （4） in, Indicates the weight of the current sample; The weight representing the noise; Obtain a completely random noise sample The formula is as follows: （5） in, , Indicates adding to Sampling noise.

7. The microservice call chain anomaly detection method based on diffusion model according to claim 5, characterized in that, S52 specifically includes the following steps: Define a conditional distribution Used to describe how to extract from noisy samples Obtain denoised samples ; From completely random noise samples ( Begin by gradually removing prediction noise. Until a sample with completely removed noise is obtained. Then, the decoder is used to decode and generate the reconstructed microservice call relationship diagram. ; The loss function is It is necessary to train a system that can generate... The parameter with the highest probability The transformation process, which employs the variational lower bound of the objective log-likelihood, is as follows: （6） in, Indicates KL divergence; express sequence, express sequence; Convert the KL divergence to a logarithm. The summation of the terms in the form of the first term is given by the following formula: （7） in As a variational lower bound, the variational lower bound is derived and simplified to obtain the following formula: （8） Among them, the first item There are no learnable parameters; its KL divergence is a fixed value and can be ignored. The second term is a KL divergence, consisting of two normally distributed parts, the first part being... The mean If the variance follows a normal distribution, the mean formula is as follows: （9） The variance formula is as follows: （10） in ; from Subtracting the predicted random noise from the result completes the denoising process.

8. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S6 specifically includes the following steps: A threshold is determined based on the reconstruction error of the training data. High percentiles are selected to capture outliers with large reconstruction errors. The IQR method is used to determine the threshold. First, the reconstruction error in all training datasets is calculated. S As shown below: （11） in, For noise sampled from a Gaussian distribution, Indicates a noise predictor; Then, based on the reconstruction error of the training set, first specify the first quartile Q1 and the third quartile Q3, and then calculate the interquartile range (IQR) according to the following formula: （12） Set the abnormal threshold using The formula is as follows: （13） Use the IQR method to specify that Q1 and Q3 adjust the anomaly thresholds based on the false positive rate and the false negative rate.

9. The microservice call chain anomaly detection method based on diffusion model according to claim 1, characterized in that, S7 specifically includes the following steps: The microservice call relationship graph is input into the trained model, which can calculate its anomaly score. Then, the anomaly score is compared with a pre-set anomaly score threshold. If the anomaly score is greater than the threshold, it is marked as anomaly data; otherwise, it is marked as normal data.

Images