Gated graph neural network (GGNN) and Deep SVDD fusion-based micro-service call graph anomaly false positive detection method

Abstract

The invention belongs to the field of big data analysis and processing in a micro-service environment, and relates to a false positive detection method for micro-service call graph abnormity based on fusion of a gated graph neural network (GGNN) and a Deep SVDD. The method comprises the following steps: obtaining a micro-service calling graph through a data set formed by link data of a micro-service log, and preprocessing the micro-service calling graph to obtain a model training data set; constructing an abnormal false positive detection model based on the micro-service calling graph; training the model to obtain normal / abnormal learning parameters of a gated graph neural network (GGNN), and respectively obtaining normal / abnormal Deep SVDD hyperspheres; and inputting to-be-detected suspected abnormal micro-service link data into the model by using the trained false positive detection model, and calculating the abnormal measurement of the micro-service call graph to judge the normal / abnormal state of the overall graph structure of the current micro-service call graph. According to the method, the problems of false positive and high false alarm rate of a current micro-service detection algorithm in an actual production environment can be solved.

Description

Technical Field

[0001] The present invention belongs to the field of big data analysis and processing in a microservice environment, and relates to a false positive detection method for microservice call graph anomalies based on the fusion of a gated graph neural network (GGNN) and Deep SVDD. Background Art

[0002] On January 11, 2022, IDC (Internet Data Center) released the top ten predictions for China's future network in 2022, of which three predictions involved cloud networks. The predictions mentioned that cloud network software, cloud management and commercial scale will grow rapidly in the next three years. Microservices, as one of the basic ideas of cloud networks, will inevitably usher in a stage of rapid development. However, as microservice architecture gradually becomes one of the mainstream architectures adopted by Internet applications, the necessity of anomaly detection for microservice software has become increasingly prominent. How to effectively implement anomaly detection for microservice software is one of the urgent issues to be solved to ensure the performance and reliability of microservice software and improve the economic and social benefits of products.

[0003] In a distributed system, the service call graph of microservices reflects the dependencies between services. Its structure and node characteristics have an important impact on the overall health of the system. Existing microservice anomaly detection methods find it difficult to effectively utilize the structural information and node characteristics of the call graph, thus limiting the effectiveness of anomaly detection.

[0004] In addition, existing microservice anomaly detection methods all have a common assumption that applications have similar performance, record similar events, or perform similar interactions in a similar order. However, because the actual behavior of multiple microservice applications also depends on the conditions under which they are executed, and these conditions are usually dynamically changing, this leads to the current microservice detection algorithms mostly having high false positive and false alarm rates in actual production environments. Summary of the invention

[0005] In view of this, the purpose of the present invention is to provide a false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD, which can solve the problem that the false positive and false alarm rates of current microservice detection algorithms are mostly high in actual production environments by capturing the topological information and complex dependencies between nodes in the normal service call graph and the abnormal service call graph of microservices respectively, generating normal / abnormal hyperspheres and using them to detect false positives.

[0006] In order to achieve the above object, the present invention provides the following technical solutions:

[0007] A false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD, the method comprising the following steps:

[0008] S1: Obtain the microservice call graph through the data set composed of link data of the microservice log, obtain the characteristics of each service node through the microservice call graph, and obtain the model training data set after preprocessing;

[0009] S2: Build an abnormal false positive detection model based on microservice call graph, which consists of a microservice call graph feature extraction module based on gated graph neural network (GGNN) and a false positive detection module based on Deep SVDD;

[0010] S3: Use the dataset of S1 to train the S2 model, obtain the normal / abnormal learning parameters of the gated graph neural network (GGNN), and obtain the normal / abnormal Deep SVDD hypersphere respectively;

[0011] S4: Use the trained false positive detection model to input the suspected abnormal microservice link data to be detected into the S3 model, calculate the abnormality metric of the microservice call graph, and determine the normal / abnormal state of the overall graph structure of the current microservice call graph.

[0012] Further, the step S1 specifically includes the following steps:

[0013] S11: Obtain a call graph according to the traceId, request initiating node, and requested node in the microservice call link data set, and preprocess the call graph to remove graphs that cannot be traversed through any node with an in-degree of 0 to find all other nodes, and remove completely identical graphs;

[0014] S12: Perform BFS traversal on each call graph starting from the microservice node with in-degree 0 to obtain the node sequence and the initial feature vector of the node, including degree betweenness, closeness betweenness, and PageRank value;

[0015] S13: Calculate the degree betweenness, proximity betweenness and PageRank value of each node in the node sequence corresponding to each call graph respectively to form a node feature sequence.

[0016] Further, the step S2 specifically includes the following steps:

[0017] S21: The Gated Graph Neural Network (GGNN) microservice call graph feature extraction module performs multiple rounds of message passing on the initial features of each node in the call graph, inputs them into the GRU (Gated Neural Unit), updates the node features after each round of message passing, and outputs the feature values ​​of the entire call graph;

[0018] S22: Deep SVDD false positive detection module, defines a hypersphere, controls the hypersphere boundary to tightly wrap the "normal" data, controls the distance between the data point and the hypersphere boundary, back-propagates the gradient of the parameters of each layer of Deep SVDD, and uses the Adam optimization algorithm to update the parameters of the gated graph neural network (GGNN) and the center point of the hypersphere, so that all data are as close to the center point of the hypersphere as possible after passing through the model. After the training is completed, if the sample data is outside the hypersphere boundary, it is determined that the data set does not belong to the hypersphere;

[0019] Further, the step S3 specifically includes the following steps:

[0020] S31: Input the node feature sequences of the normal / abnormal microservice call graphs obtained in S13 into the microservice call graph feature extraction module of step S21 to obtain the call graph feature sequence;

[0021] S32: Input the call graph feature sequence obtained in step S31 into the Deep SVDD in step S22 to calculate the center and radius of the hypersphere, and obtain a normal / abnormal hypersphere;

[0022] Further, the step S4 specifically includes the following steps:

[0023] S41: Input the microservice call graph data to be detected (suspicious abnormal data) into the model of step S3, and obtain the feature representation of the call graph through the forward propagation process of the model;

[0024] S42: Calculate the distance between the feature representation of the call graph and the center point of the hypersphere formed by the abnormal call graph data. If it is within the sphere, it is determined to be abnormal data and the process ends; if it is outside the sphere, enter S43;

[0025] S43: Calculate the distance between the characteristic representation of the call graph and the center point of the hypersphere formed by the normal call graph data. If it is located inside the sphere, it is determined to be false positive data and the process ends; if it is located outside the sphere, it is determined to be abnormal data that has never been found and the data is recorded for the next detection.

[0026] The beneficial effects of the present invention are:

[0027] 1) Automated microservice false positive detection: Combining gated graph neural network (GGNN) and Deep SVDD, it can effectively identify whether the service call graph is normal, abnormal, or false positive, solving the problem that the current microservice detection algorithm has high false positive and false alarm rates in actual production environments.

[0028] 2) Adaptability to complex structures: The gated graph neural network (GGNN) model is used to learn the node relationships and dependency information in the graph structure, so that the model can adapt to complex call relationships.

[0029] 3) Unsupervised learning: The model can directly detect the status of the service call graph through unsupervised learning, without the need for a large amount of labeled data, thus reducing the actual application cost.

[0030] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0032] Figure 1 Generate process graphs for service call graphs and feature acquisition;

[0033] Figure 2 is a flow chart of the method of the present invention; DETAILED DESCRIPTION

[0034] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0035] Figure 1 Generate service call graphs and feature acquisition process graphs, Figure 2 The present invention is a flow chart of the method.

[0036] The present invention provides a tunnel deformation prediction method integrating spatiotemporal data, which mainly comprises the following steps:

[0037] Step 1: Obtain the microservice call graph through the data set composed of link data of the microservice log, obtain the characteristics of each service node through the microservice call graph, and obtain the model training data set after preprocessing;

[0038] Step 2: Build an abnormal false positive detection model based on microservice call graph, which consists of a microservice call graph feature extraction module based on gated graph neural network (GGNN) and a false positive detection module based on Deep SVDD;

[0039] Step 3: Use the dataset of S1 to train the S2 model, obtain the normal / abnormal learning parameters of the gated graph neural network (GGNN), and obtain the normal / abnormal Deep SVDD hyperspheres respectively;

[0040] Step 4: Use the trained false positive detection model to input the suspected abnormal microservice link data to be detected into the S3 model, calculate the abnormality metric of the microservice call graph, and determine the normal / abnormal state of the overall graph structure of the current microservice call graph.

[0041] like Figure 1 As shown, the steps of obtaining a microservice call graph through a data set composed of link data of a microservice log, obtaining the characteristics of each service node through the microservice call graph, and obtaining a model training data set after preprocessing include:

[0042] (1) Obtain the call graph based on the traceid, request initiating node, and requested node in the microservice call link data set. Preprocess the call graph data (including normal / abnormal data) by adding a directed edge from the request initiating node to the requested node to the graph based on traceid. Eliminate graphs that cannot be traversed through any node with an in-degree of 0 to find all other nodes, and eliminate completely identical graphs.

[0043] (2) Perform BFS traversal on each call graph starting from the microservice node with in-degree 0 to obtain the node sequence; construct the initial feature vector of the node, including degree betweenness, closeness betweenness, and PageRank value;

[0044] (3) Calculate the degree betweenness, proximity betweenness and pagerank value of each node in the node sequence corresponding to each call graph respectively to form a node feature sequence.

[0045] like Figure 2 As shown in FIG. 1 , the steps of constructing a false positive detection method model based on the fusion of gated graph neural network (GGNN) and Deep SVDD include:

[0046] (1) The Gated Graph Neural Network (GGNN) microservice call graph feature extraction module performs multiple rounds of message transmission on the initial features of each node in the call graph, inputs them into the GRU (Gated Neural Unit), updates the node features after each round of message transmission, and outputs the feature values ​​of the entire call graph;

[0047] (2) Deep SVDD false positive detection module, defines a hypersphere, controls the hypersphere boundary to tightly wrap the "normal" data, controls the distance between the data point and the hypersphere boundary, back-propagates the gradient of the parameters of each layer of Deep SVDD, and uses the Adam optimization algorithm to update the parameters of the gated graph neural network (GGNN) and the center point of the hypersphere, so that all data are as close to the center point of the hypersphere as possible after passing through the model. After the training is completed, if the sample data is outside the hypersphere boundary, it is determined that the data set does not belong to the hypersphere;

[0048] The node feature matrix is ​​constructed and input into the gated graph neural network (GGNN). The dependency between nodes in the graph is captured through multiple rounds of message passing. The calculation method is as follows:

[0049] (1) Node feature initialization

[0050] Each node v i Initialized as a set of feature vectors These features include: point betweenness, closeness betweenness, PageRank;

[0051] The feature matrix is in is node v i The initial features of , N is the number of nodes, and d is the feature dimension.

[0052] (2) Building a message delivery mechanism

[0053] Message passing occurs between nodes through edges {ij} To spread information. A node receives messages from its neighbor nodes and updates its own features. The formula for message passing is:

[0054]

[0055] in, is node v i The message received in round t+1, Represents node v i The set of neighbor nodes, W (t) is the weight matrix of the tth round, is the neighbor node v j Features at round t.

[0056] (3) Node feature update

[0057] After each round of message passing, the node features are updated through GRU (Gated Recurrent Unit):

[0058]

[0059] in is node v i In the update feature of the t+1th round; GRU will update the node based on the previous round feature and the current round of messages To determine new features

[0060] The update formula of GRU is as follows:

[0061]

[0062] in It is the reset gate that determines how much past information is forgotten; is the update gate, which determines how much information to update; W r , W z , W h is a weight matrix, and ⊙ represents element-level multiplication. Through multiple rounds of message passing and GRU updates, the features of the node will gradually integrate the information of the neighboring nodes.

[0063] (4) The calculation of generating the feature representation of the entire call graph is as follows:

[0064]

[0065] in It is a soft-attention mechanism, which is used to select which nodes are more relevant to the entire graph; f i and f j is a neural network; T is the number of layers of the gated graph neural network (GGNN);

[0066] The data set of step 1 is used to train the model of step 2 to obtain the normal / abnormal learning parameters of the gated graph neural network (GGNN), and the normal / abnormal Deep SVDD hyperspheres are obtained respectively, including:

[0067] (1) Inputting the node feature sequences of the normal / abnormal microservice call graph obtained in step S13 into the microservice call graph feature extraction module in step S21 to obtain the call graph feature sequence;

[0068] (2) Input the call graph feature sequence obtained in (1) into the Deep SVDD of step S22 to calculate the center and radius of the hypersphere and obtain the normal / abnormal hypersphere;

[0069] The loss function for calculating the hypersphere in Deep SVDD is as follows:

[0070]

[0071] (1) The first term with a larger R 2 will form a larger "normal" area and a smaller R 2 This will tighten the region. Minimize R 2 Helps to pack the hypersphere boundaries tightly around the "normal" data.

[0072] (2) The second boundary loss term is used to control the distance between the data point and the boundary of the hypersphere, where: N g : the total number of training samples; h g : potential representation of the g-th service call graph; c: the center point of the hypersphere; || h g-c|| 2 : indicates h g The square of the Euclidean distance to the center c; max{0,||h g -c|| 2 -R 2}: When || h g -c|| 2 >R 2 When a non-zero value is generated, it means that the sample point is beyond the boundary of the hypersphere. By minimizing this term, the number of sample points beyond the boundary can be reduced. μ is a hyperparameter used to control the volume of the hypersphere (by adjusting the radius R 2 ) and boundary violation. The setting of this parameter can allow some training data to be outside the boundary (for example, there may be some abnormal data in the training set).

[0073] (3) The third term is a weight decay (regularization) term used to prevent overfitting: N l : The number of layers of the GGNN model; θ (l) : Model parameters of the lth layer; The square of the Frobenius norm of the parameter is used to measure the size of the parameter; λ is a regularization hyperparameter used to control the impact of this weight decay term to help the model be more generalized.

[0074] Further, if Figure 2 , use the trained false positive detection model, input the suspected abnormal microservice link data to be detected into the model in step 3, calculate the abnormality metric of the microservice call graph, and judge the normal / abnormal state of the overall graph structure of the current microservice call graph, which specifically includes the following steps:

[0075] (1) Input the microservice call graph data to be detected (suspicious abnormal data) into the model of step S3, and obtain the feature representation of the call graph through the forward propagation process of the model;

[0076] (2) Calculate the distance between the feature representation of the call graph after Deep SVDD and the center point of the false positive detection, determine the normal / abnormal state of the call graph according to the preset threshold, and output the predicted normal / abnormal measurement, so as to determine whether the service call graph belongs to normal call, abnormal call, or false positive.

[0077] (3) Calculate the distance between the feature representation of the call graph and the center point of the hypersphere formed by the normal call graph data. If it is within the sphere, it is determined to be false positive data and the process ends; if it is outside the sphere, it is determined to be abnormal data that has never been found and is recorded for the next detection.

[0078] The call graph features represent the distance from the center point of the false positive detection after Deep SVDD, such as Figure 2 As shown, the calculation method is as follows:

[0079]

[0080] Among them, h g is the feature representation learned in the model; c is the center of the Deep SVDD hypersphere (the center point obtained through model training); is the final radius of the trained hypersphere. It can be judged that sample h g Relative to the boundary position of the hypersphere.

[0081] Furthermore, it is determined whether the sample (determined as abnormal during anomaly detection) is a false positive or a false alarm, and the sample is input into the hypersphere of the abnormal data set:

[0082] (1) In the hypersphere of the abnormal data set, if ans(h g )>0, it means that the sample point h g If the sample is outside the boundary of the abnormal data set hypersphere, it further indicates that the sample does not belong to the abnormal data set or belongs to an anomaly that has never appeared before. It may be regarded as a false positive or false alarm and enter the next judgment process.

[0083] (2) In the hypersphere of the abnormal data set, if ans(h g )≤0, it means that the sample point h g If the sample point is inside or on the boundary of the abnormal data set hypersphere, it is further indicated that the sample point is an anomaly and is regarded as an anomaly that is correctly reported, and the process ends.

[0084] Furthermore, we determine whether a sample that may be considered a false positive or false alarm belongs to a normal call and enter the sample into the hypersphere of the normal data set:

[0085] (1) In the hypersphere of the normal data set, if ans(h g )>0, it means that the sample point h g If the sample point is outside the boundary of the normal data set hypersphere, it further indicates that the sample point is neither abnormal data nor normal data. The sample is regarded as a new anomaly that has never appeared in history, and the anomaly is included in the abnormal data set for the next new sample point detection. The process ends.

[0086] (2) In the hypersphere of the normal data set, if ans(h g )≤0, it means that the sample point h g If the sample is inside or on the boundary of the normal dataset hypersphere, it further indicates that the sample belongs to the normal call graph. In this case, the sample is considered a false positive and the process ends.

Claims

1. A false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD based on microservice call graph, characterized in that: The method comprises the following steps: S1: Obtain the microservice call graph through the data set composed of link data of the microservice log, obtain the characteristics of each service node through the microservice call graph, and obtain the model training data set after preprocessing; S2: Build an abnormal false positive detection model based on microservice call graph, which consists of a microservice call graph feature extraction module based on gated graph neural network (GGNN) and a false positive detection module based on Deep SVDD; S3: Use the dataset of S1 to train the S2 model, obtain the normal / abnormal learning parameters of the gated graph neural network (GGNN), and obtain the normal / abnormal Deep SVDD hypersphere respectively; S4: Use the trained false positive detection model to input the suspected abnormal microservice link data to be detected into the S3 model, calculate the abnormality metric of the microservice call graph, and determine the normal / abnormal state of the overall graph structure of the current microservice call graph.

2. According to claim 1, a false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD, characterized in that: In step S2, it specifically includes: S11: Obtain a call graph according to the traceId, request initiating node, and requested node in the microservice call link data set, and preprocess the call graph to remove graphs that cannot be traversed through any node with an in-degree of 0 to find all other nodes, and remove completely identical graphs; S12: Perform BFS traversal on each call graph starting from the microservice node with in-degree 0 to obtain the node sequence and the initial feature vector of the node, including degree betweenness, closeness betweenness, and PageRank value; S13: Calculate the degree betweenness, proximity betweenness and PageRank value of each node in the node sequence corresponding to each call graph respectively to form a node feature sequence.

3. According to claim 2, a false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD, characterized in that: In step S1, it specifically includes: S21: The Gated Graph Neural Network (GGNN) microservice call graph feature extraction module performs multiple rounds of message passing on the initial features of each node in the call graph, inputs them into the GRU (Gated Neural Unit), updates the node features after each round of message passing, and outputs the feature values ​​of the entire call graph; S22: Deep SVDD false positive detection module, defines a hypersphere, controls the hypersphere boundary to tightly wrap the "normal" data, controls the distance between the data point and the hypersphere boundary, back-propagates the gradient of the parameters of each layer of Deep SVDD, and uses the Adam optimization algorithm to update the parameters of the gated graph neural network (GGNN) and the center point of the hypersphere, so that all data are as close to the center point of the hypersphere as possible after passing through the model. After the training is completed, if the sample data is outside the hypersphere boundary, it is determined that it does not belong to the data set of the hypersphere.

4. According to claim 3, a false positive detection method for microservice call graph anomalies based on the fusion of graph neural network (GGNN) and Deep SVDD is characterized in that: In step S3, it specifically includes: S31: Input the node feature sequences of the normal / abnormal microservice call graphs obtained in S13 into the microservice call graph feature extraction module of step S21 to obtain the call graph feature sequence; S32: Input the call graph feature sequence obtained in step S31 into the Deep SVDD in step S22 to calculate the center and radius of the hypersphere to obtain a normal / abnormal hypersphere.

5. According to claim 4, a false positive detection method for microservice call graph anomalies based on the fusion of gated graph neural network (GGNN) and Deep SVDD is characterized in that: In step S4, it specifically includes: S41: Input the microservice call graph data to be detected (suspicious abnormal data) into the model of step S3, and obtain the feature representation of the call graph through the forward propagation process of the model; S42: Calculate the distance between the feature representation of the call graph and the center point of the hypersphere formed by the abnormal call graph data. If it is within the sphere, it is determined to be abnormal data and the process ends; if it is outside the sphere, enter S43; S43: Calculate the distance between the characteristic representation of the call graph and the center point of the hypersphere formed by the normal call graph data. If it is located inside the sphere, it is determined to be false positive data and the process ends; if it is located outside the sphere, it is determined to be abnormal data that has never been found and the data is recorded for the next detection.

Images