Microservice fault diagnosis method, device and equipment based on graph convolutional neural network

Abstract

The present invention discloses a microservice fault diagnosis method, device and equipment based on a graph convolutional neural network, which is mainly capable of quickly and accurately locating the root cause microservice node and determining the type of microservice fault. The method includes: collecting alarm events of the target microservice to be diagnosed within a preset time period before and after the fault, wherein the alarm event is generated based on multimodal data; determining the alarm event sequence of each microservice node in the target microservice based on the alarm event; generating an event vector for each alarm event in the alarm event sequence, and calculating the node vector of the microservice node corresponding to each alarm event based on the event vector; inputting the node vector of each microservice node into a trained fault diagnosis model to obtain a fault diagnosis result, which at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a microservice fault diagnosis method, apparatus, and device based on a graph convolutional neural network. Background Art

[0002] In recent years, with the growing popularity of microservices software architecture, internet companies have begun to split large, monolithic applications into multiple microservices with independent functions to address complex and changing business needs and the ever-expanding scale of their operations. However, the increased scale, complexity, and flexibility of microservices have posed greater challenges to the operations and maintenance (O&M) departments of internet companies. These departments continuously record various status information within microservice systems. When an anomaly is detected, they must quickly and accurately identify the root cause so they can take swift action and avoid excessive losses. Therefore, fault diagnosis is a crucial component of the entire O&M process. How to quickly and accurately locate the microservice node responsible for the fault and analyze the fault type have become pressing challenges.

[0003] In contemporary internet companies, existing fault diagnosis methods often rely on only one or two types of data. Typically, dedicated data platforms are used to collect runtime data related to microservice systems (such as logs, metrics, and call chain data). During system operation, monitoring systems monitor the operational status of microservice systems in real time. When a system anomaly is detected, multimodal data such as logs, metrics, and call chains generates a large number of alarms. Faced with the massive amount of alarm information generated by large microservice systems, fault diagnosis still relies heavily on manpower and expert experience. This approach is no longer sufficient to meet the rapidly expanding and increasingly complex needs of large-scale microservice systems. Summary of the Invention

[0004] The present invention provides a microservice fault diagnosis method, device and equipment based on graph convolutional neural network, which can quickly and accurately locate the root cause microservice node and determine the type of microservice fault.

[0005] According to a first aspect of the present invention, a microservice fault diagnosis method based on a graph convolutional neural network is provided, comprising:

[0006] Collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data;

[0007] Determine the alarm event sequence of each microservice node in the target microservice according to the alarm event;

[0008] Generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector of a microservice node corresponding to each alarm event based on the event vector;

[0009] The node vector of each microservice node is input into the trained fault diagnosis model to obtain a fault diagnosis result, which at least includes a location result of the root cause microservice node and a prediction result of the microservice fault type.

[0010] According to a second aspect of the present invention, a microservice fault diagnosis device based on a graph convolutional neural network is provided, the device comprising:

[0011] A collection module is used to collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data;

[0012] A determination module, configured to determine an alarm event sequence of each microservice node in the target microservice according to the alarm event;

[0013] A calculation module, configured to generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector for the microservice node corresponding to each alarm event based on the event vector;

[0014] An input module is used to input the node vector of each microservice node into the trained fault diagnosis model to obtain a fault diagnosis result, which at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type.

[0015] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the following steps:

[0016] Collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data;

[0017] Determine the alarm event sequence of each microservice node in the target microservice according to the alarm event;

[0018] Generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector of a microservice node corresponding to each alarm event based on the event vector;

[0019] The node vector of each microservice node is input into the trained fault diagnosis model to obtain a fault diagnosis result, which at least includes a location result of the root cause microservice node and a prediction result of the microservice fault type.

[0020] According to a fourth aspect of the present invention, there is provided a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the following steps are implemented:

[0021] Collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data;

[0022] Determine the alarm event sequence of each microservice node in the target microservice according to the alarm event;

[0023] Generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector of a microservice node corresponding to each alarm event based on the event vector;

[0024] The node vector of each microservice node is input into the trained fault diagnosis model to obtain a fault diagnosis result, which at least includes a location result of the root cause microservice node and a prediction result of the microservice fault type.

[0025] The present invention provides a microservice fault diagnosis method, device and equipment based on graph convolutional neural network. Compared with the current single-dimensional data fault detection method, it can collect alarm events of the target microservice to be diagnosed within a preset time period before and after the fault based on multimodal data, and then determine the alarm event sequence of each microservice node in the target microservice according to the alarm event; then generate the event vector of each alarm event in the alarm event sequence, calculate the node vector of the microservice node corresponding to each alarm event based on the event vector; finally, input the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, which at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type. Through the technical solution of the present invention, the three modal microservice alarm events can be combined and used in a topological structure, and the graph convolutional neural network can be used to accurately locate the root cause microservice node and accurately predict the microservice fault type, thereby reducing the time spent by operation and maintenance personnel to restore the fault, accelerating the process from fault occurrence to fault recovery of the microservice system, and thus minimizing the loss.

[0026] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0028] Figure 1 A schematic diagram of a microservice fault diagnosis method based on a graph convolutional neural network provided by an embodiment of the present invention is shown;

[0029] Figure 2 A schematic diagram of another microservice fault diagnosis method based on a graph convolutional neural network provided by an embodiment of the present invention is shown;

[0030] Figure 3 A schematic diagram illustrating an example of a call chain provided by an embodiment of the present invention is shown;

[0031] Figure 4 A schematic diagram showing a span in a call chain on a timeline provided by an embodiment of the present invention is shown;

[0032] Figure 5 A schematic diagram of a model structure used in a training event vector generator provided by an embodiment of the present invention is shown;

[0033] Figure 6 A schematic structural diagram of a fault diagnosis model provided by an embodiment of the present invention is shown;

[0034] Figure 7 A microservice system business process diagram provided by an embodiment of the present invention is shown;

[0035] Figure 8 A microservice deployment diagram provided by an embodiment of the present invention is shown;

[0036] Figure 9 A schematic diagram of a homogeneous graph topology structure provided by an embodiment of the present invention is shown;

[0037] Figure 10 A schematic diagram of a heterogeneous graph topology structure provided by an embodiment of the present invention is shown;

[0038] Figure 11 A schematic diagram of the structure of a first convolutional neural network model provided by an embodiment of the present invention is shown;

[0039] Figure 12 A schematic diagram of the structure of a second convolutional neural network model provided by an embodiment of the present invention is shown;

[0040] Figure 13 A schematic diagram of the structure of a microservice fault diagnosis device based on a graph convolutional neural network provided by an embodiment of the present invention is shown;

[0041] Figure 14 A schematic diagram of the structure of another microservice fault diagnosis device based on a graph convolutional neural network provided by an embodiment of the present invention is shown;

[0042] Figure 15 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0043] Faced with complex, large-scale microservice systems, an increasing number of enterprises are building intelligent operations and maintenance platforms. These platforms, leveraging vast amounts of operational data and artificial intelligence (AI) methods (machine learning and deep learning), aim to solve problems that traditional operations and maintenance (O&M) cannot. Faced with massive amounts of multimodal data and complex, ever-changing topologies, rapid and accurate fault diagnosis can help enterprises promptly resolve issues and avoid excessive losses. Therefore, research on fault diagnosis in microservice systems is of paramount importance and practical value.

[0044] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0045] In order to quickly and accurately locate the root cause microservice node and accurately determine the type of microservice failure, the present invention provides a microservice fault diagnosis method based on graph convolutional neural network, such as Figure 1 As shown, the method includes:

[0046] 101. Collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data.

[0047] In recent years, research on fault diagnosis can be categorized into four types based on the types of data used: log data, metrics data, call chain data, and the combination of multiple modal data. However, most existing fault diagnosis methods only utilize one or two types of data, and relatively few studies have combined multiple modal data and utilized topological structures. In the fault diagnosis process of internet enterprises, the data acquired by operation and maintenance data collection platforms is multi-source and heterogeneous, including but not limited to log data, platform metrics (such as CPU utilization and network throughput), business metrics (such as the number of successful and failed user orders per unit time for an e-commerce app), call chain data, and configuration management data. When applying various algorithms for fault diagnosis, it is important to utilize as much multimodal data as possible. This is because using data from a single channel can lead to the "blind men and the elephant" problem. For example, when CPU utilization or memory usage spikes, these anomalies are often reflected in metrics data but not in log data. Similarly, when a user login fails, such information may be recorded in log data or call chain data but not in metrics data. Compared to using only a single channel of data, using multimodal data can usually obtain more data features, and the effect of fault diagnosis is often more obvious. In other applications of artificial intelligence, using multimodal data to build a system can often achieve better performance (for example, in video classification tasks, a multimodal model that uses video information, audio information, and letter title information is better than a single modal model), which has also been theoretically proven by researchers. In view of this, the present invention can simultaneously combine multimodal operation and maintenance data (logs, indicators, call chain data) for fault diagnosis.

[0048] In this embodiment, when a microservice anomaly is detected, multimodal data within a preset time period before and after the anomaly is first collected, and an alarm event is further generated based on the multimodal data. The multimodal data used includes, but is not limited to, log data, indicator data, and call chain data. Accordingly, the alarm events include, but are not limited to, log alarm events, indicator alarm events, and call chain alarm events. The length of the preset time period can be set according to the actual application scenario, such as 5 minutes, and is not specifically limited here.

[0049] Log data is generated during microservice runtime and communication between microservices. Logs are typically semi-structured data, consisting of a fixed log template and variable parameters. During the fault period, the log alert event generated by the log data contains the following fields:

[0050] a. Microservice node name: This field indicates which microservice node generated the log entry;

[0051] b. Timestamp: This field indicates the time when the log entry was generated;

[0052] c. Log template ID: This field is obtained by the Drain algorithm

[19] and is a hash value substring of the template corresponding to the log entry.

[0053] Here is an example of an alert event generated from log data:

[0054] ('dbservice1',1625122737,'d071a2a6')

[0055] Metric data can often be divided into business metrics and system metrics. Business metrics refer to metrics related to specific businesses. For example, for an e-commerce website, the number of visits within one minute, the number of successful transactions, the number of failed transactions, and the number of new user registrations can all be considered business metrics. System metrics refer to metrics related to servers, containers, and other components. For example, metrics such as CPU utilization, network traffic, and disk utilization are all platform metrics. The operations and maintenance department continuously records metrics and visualizes them for easy review. Furthermore, the operations and maintenance department detects anomalies in the metrics and automatically generates alerts when anomalies occur.

[0056] During the fault occurrence period, the indicator alarm event generated by the indicator data contains the following fields:

[0057] a. Microservice Node Name: This field indicates the microservice to which the indicator belongs;

[0058] b. Timestamp: This field indicates the time when the indicator abnormal value was recorded;

[0059] c. Indicator ID: This field can uniquely identify an indicator curve;

[0060] d.Indicator abnormality type: This field indicates the abnormality type of the indicator.

[0061] Here is an example of an alert event generated from metric data:

[0062] ('webservice2',1625122737,'webservice2_0.0.0.3_docker_cpu_kernel_ticks','+')

[0063] For call chain data, a call chain represents the execution process of a transaction or process in a (distributed) system. In the OpenTracing standard, a call chain is a directed acyclic graph composed of multiple spans, and each span represents a named and timed continuous execution segment in the call chain. The service calls in the call chain are initially recorded separately in different microservice nodes, and then they are integrated into a complete call chain through the fields in the record. Figure 3 As shown, the call chain is presented in a tree structure. This call chain contains all the spans related to a service call. Each span has a unique span ID. Similarly, each call chain can be uniquely identified by the call chain ID. A span records the microservice instances of the caller and the callee, and usually also records the corresponding operation name and latency. Except for the root span, under normal circumstances, each span has a corresponding parent span. Figure 3 In the example, Span A is a synchronous call, and during its execution, two new synchronous calls, Span B and Span C, are generated. Therefore, Span A is the parent span of Span B and Span C. Figure 4 Shown on the timeline Figure 3 This timing diagram can more specifically illustrate the execution process of spans in a call chain. Span B and Span C are parallel calls generated by Span A, so their execution times overlap. Span F, Span G, and Span H are synchronous calls generated by Span C, so they do not overlap on the timeline.

[0064] During the fault time period, the call chain alarm event generated by the call chain data contains the following fields:

[0065] a. Timestamp: This field indicates the time when the call was initiated;

[0066] b. Caller microservice node name: This field indicates the caller microservice node;

[0067] c. Callee microservice node name: This field indicates the callee microservice node;

[0068] Here is an example of an alert event generated by call chain data:

[0069] (1625131245,'webservice1','loginservice2')

[0070] 102. Determine the alarm event sequence of each microservice node in the target microservice based on the alarm event.

[0071] In this embodiment, after collecting alarm events from the target microservice to be diagnosed within a preset time period before and after a failure, the alarm events can be divided according to microservice nodes. Log alarms and indicator alarm events only need to be attributed to the microservice node where the event occurred, while call chain alarm events need to be attributed to both the caller microservice node and the callee microservice node. Furthermore, on each microservice node, the alarm events can be uniformly arranged according to the order in which they occurred, thereby obtaining a unified sequence of alarm events of the three modalities, namely, the alarm event sequence.

[0072] 103. Generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector of a microservice node corresponding to each alarm event based on the event vector.

[0073] In a specific application scenario, in order to be able to map the microservice node to a low-dimensional vector space, it is first necessary to express the events on the microservice node in a vectorized manner. Through step 102 of the embodiment, each microservice node corresponds to an alarm event sequence. Let m represent the number of fault cases used for training in the historical fault case library, and n represent the number of microservice nodes, then a total of m×n event sequences can be obtained in the training set. As time series data, event sequences can be trained for event embedding in a manner similar to fastText training word embedding. FastText can obtain vector representations of words through both supervised learning and unsupervised learning. The model structure used by the event vector generator for training the present invention is as follows: Figure 5 As shown, the input of the model is the event sequence corresponding to a single microservice node in a failure, and the label is the failure type of the microservice node (if the microservice node is normal, it is marked as normal). The training data in the training data set is used to perform the training process of the event vector generator. The training results of the event vector generator are verified using the verification data and test data in the training data set. The vector generation accuracy of the event vector generator is calculated based on the verification results and the training labels. If the vector generation accuracy is greater than the first preset threshold, the event vector generator training is determined to be complete. Among them, the first preset threshold can be any value from 0 to 1. The closer the set value is to 1, the higher the training accuracy of the event vector generator. The specific value can be set according to the actual application scenario and is not specifically limited here.

[0074] In this embodiment, the alarm event sequence of each microservice node can be input into a trained event vector generator, which then outputs the event vector for each alarm event in the alarm event sequence. Furthermore, the node vector for the microservice node corresponding to each alarm event can be calculated based on the event vector. However, during a microservice failure, each microservice node generates a large number of alarm events. Analysis of these alarm events reveals that some alarm events are generated in large quantities but are not associated with the actual failure, adding a significant amount of noise to the description of the microservice node. Furthermore, while some alarm events occur in small numbers, actual observation of the alarm content reveals that a small number of these alarms have a strong correlation with the ultimate root cause node and failure type. Therefore, when calculating the node vector for the microservice node corresponding to each alarm event, different weights can be assigned to each alarm event according to a certain rule. Next, the weighted mean of the event vectors for each microservice node is calculated based on the event weights, which serves as the vector representation of the microservice node, i.e., the node vector.

[0075] 104. Input the node vector of each microservice node into the trained fault diagnosis model to obtain a fault diagnosis result. The fault diagnosis result at least includes a location result of the root cause microservice node and a prediction result of the microservice fault type.

[0076] Among them, the fault diagnosis model can be a graph convolutional neural network model, which can specifically include a first convolutional neural network model based on a homogeneous graph and a second convolutional neural network model based on a heterogeneous graph.

[0077] In this embodiment, to ensure the reliability and service quality of microservices, the operations and maintenance team needs to consider how to quickly recover from a failure. To recover the microservice system from a failure, the first step is to locate the root microservice node that caused the failure. During a failure, the failure of the root microservice node may propagate along the call chain, causing other microservices to also exhibit anomalies. Due to the large number of microservices in a microservice system and the complex call relationships between them, the monitoring system typically generates a large number of alarm messages during a failure. Using this alarm information to quickly and accurately locate the root microservice node is the first problem that needs to be solved in fault diagnosis. After locating the root microservice node, operations and maintenance personnel urgently need to recover from the failure. If the fault diagnosis system can automatically determine the microservice failure type, the time spent by operations and maintenance personnel on recovery will be greatly reduced. Quickly and accurately determining the microservice failure type can accelerate the process from failure occurrence to recovery in the microservice system, thereby minimizing losses. Therefore, the second problem that needs to be solved in fault diagnosis is to quickly and accurately determine the type of microservice failure by using the large amount of alarm information generated by the monitoring system during the fault.

[0078] In a specific application scenario, before executing the steps of this embodiment, it is necessary to pre-train the fault diagnosis model offline. Figure 6 As shown, during offline training, historical alarm events from historical fault cases need to be collected and the training dataset is divided into training data, validation data, and test data. For each fault in the training dataset of the historical fault case library, logs, metrics, and call chain alarm events are collected based on the time period of the fault. Then, the alarm event sequence of each microservice node is obtained. By analyzing the alarm event sequence on each microservice node in each fault case, an event vector generator is generated to generate event vectors for each microservice node. At the same time, different weights are assigned to each event according to certain rules. Next, the weighted mean of the event vectors of each node is calculated based on the event weights, which serves as the vector representation of the node, namely the training dataset. Finally, the fault diagnosis model is trained using the training dataset of each node, along with the corresponding topology and labels. Specifically, the cross entropy loss function can be used for training, and the training data in the training data set is used to execute the training process of the fault diagnosis model. The training results of the fault diagnosis model are verified using the verification data and test data in the training data set. The fault diagnosis accuracy of the fault diagnosis model is calculated based on the verification results and the training labels. If the fault diagnosis accuracy is greater than the second preset threshold, the fault diagnosis model training is determined to be complete. Among them, the second preset threshold and the first preset threshold can be the same or different, and can be any value from 0 to 1. The closer the set value is to 1, the higher the training accuracy of the fault diagnosis model. The specific value can be set according to the actual application scenario and is not specifically limited here. It should be noted that in order to avoid the risk of data leakage to the fault diagnosis model, the same training data set is used when training the event vector generator and the fault diagnosis model.

[0079] As a preferred approach, during offline training, each fault case corresponds to a training sample for the graph convolutional neural network. Because training samples in historical fault cases are unevenly distributed across fault type labels and root cause node labels, data augmentation is used to improve the model's generalization capabilities. For multiple training samples with the same root cause node label (fault type label), the vector representations of each sample at a specific microservice node are randomly swapped to obtain more training samples with the same root cause label (fault type label). These additional training samples are then combined with the original samples as training data for training.

[0080] After determining that the fault diagnosis model training is complete, for this embodiment, the node vectors of each microservice node can be input into the trained fault diagnosis model to obtain the fault diagnosis results. The fault diagnosis results at least include the location results of the root cause microservice node and the prediction results of the microservice fault type. Among them, the location results of the root cause microservice node can be a ranking of the root cause microservice nodes, and the root cause microservice node ranked first can be further determined as the root cause microservice node causing the target microservice fault; the prediction results of the microservice fault type can be a ranking of the fault types, and the fault type ranked first can be further determined as the fault type of the target microservice. Furthermore, the location results of the root cause microservice node and the prediction results of the microservice fault type can be reported to the relevant operation and maintenance personnel.

[0081] The microservice fault diagnosis method based on graph convolutional neural network provided by the embodiment of the present invention can collect alarm events of the target microservice to be diagnosed within a preset time period before and after the fault based on multimodal data, and then determine the alarm event sequence of each microservice node in the target microservice according to the alarm event; then generate the event vector of each alarm event in the alarm event sequence, and calculate the node vector of the microservice node corresponding to each alarm event based on the event vector; finally, input the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, which at least includes the positioning result of the root cause microservice node and the prediction result of the microservice fault type. Through the technical solution of the present invention, the microservice alarm events of three modes can be combined and used in a topological structure, and the graph convolutional neural network can be used to accurately locate the root cause microservice node and accurately predict the microservice fault type, thereby reducing the time spent by operation and maintenance personnel to restore the fault, accelerating the process from fault occurrence to fault recovery of the microservice system, and thus minimizing the loss.

[0082] Furthermore, in order to better illustrate the above microservice fault detection process, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another microservice fault diagnosis method based on graph convolutional neural network, such as Figure 2 As shown, the method includes:

[0083] 201. Collect alarm events of a target microservice to be diagnosed within a preset time period before and after a fault, wherein the alarm events are generated based on multimodal data.

[0084] For this embodiment, the specific implementation process can refer to the relevant description in step 101 of the embodiment, which will not be repeated here.

[0085] 202. Determine an alarm event sequence for each microservice node in the target microservice based on the alarm event.

[0086] For this embodiment, in a specific application scenario, step 202 of the embodiment may specifically include: attributing the indicator alarm event and the log alarm event to the microservice node where the event occurs, and attributing the call chain alarm event to both the caller microservice node and the callee microservice node; arranging the alarm events uniformly on each microservice node according to the order in which the events occurred to obtain the alarm event sequence of each microservice node.

[0087] 203. Input the alarm event sequence into the trained event vector generator to generate an event vector for each alarm event in the alarm event sequence.

[0088] Among them, the event vector generator is trained using the training data set after data enhancement processing.

[0089] In specific application scenarios, the number of fault cases is typically small, and the distribution of fault types is uneven, with significant differences in the number of fault types. In this context, before executing the steps of this embodiment, after acquiring the training dataset, the present invention can use a data augmentation strategy to expand the number of samples corresponding to the smaller number of fault types in the training dataset to address the issues of the small number of fault cases themselves and the uneven distribution of fault categories.

[0090] On this basis, a method for pre-training the event vector generator is provided, which may specifically include:

[0091] (1) Train the event vector generator in the original training dataset of the fault case library, thereby mapping all alarm events appearing in the fault case library into a low-dimensional vector space and obtaining the vector representation of each alarm event;

[0092] (2) Considering the small number of fault cases and the uneven distribution of fault categories, the event vector generator trained in (1) cannot accurately depict the differences and similarities between alarms and is easily affected by a large number of fault categories. Therefore, it is necessary to expand the number of samples corresponding to a small number of fault categories. The specific approach is: given a sequence of events in a microservice node in a certain fault, randomly select an event in the sequence, and in the vector space obtained in (1), find an event corresponding to the vector closest to the event vector (measured using cosine similarity), and exchange the event in the original sequence with the found event to obtain a new sample, the label of the new sample is still the original fault category;

[0093] (3) Use the new training samples obtained in (2) and the original training samples to input the model for training again. It is worth noting that in this step of training, both unsupervised training and supervised training (the label is the fault type) can be performed.

[0094] Accordingly, the steps of the embodiment may specifically include: generating a sample node vector of each microservice node in the sample microservice based on the historical alarm events of the sample microservice in the initial training data set; exchanging at least one sample node vector of the sample microservice in the initial training data set based on the root cause microservice node label or the microservice fault type label of the sample microservice to obtain a training data set after data enhancement processing; and training an event vector generator using the training data set after data enhancement processing to obtain a trained event vector generator.

[0095] Through the above steps, an event vector generator can be trained. For events that appear in the training data, the corresponding vector of the event can be found in the obtained vector space. However, for a small number of events in the verification data, training data, and online fault diagnosis, the trained event vector generator cannot generate corresponding vectors. These small number of events can be ignored in fault diagnosis. When the number of unknown events is large, you can choose to use a new batch of data to retrain the model (usually it is necessary to obtain data regularly and update the model regularly). Furthermore, the alarm event sequence of each microservice node can be input into the trained event vector generator, and the event vector generator can be used to output the event vector of each alarm event in the alarm event sequence.

[0096] 204. Calculate the node vector of the microservice node corresponding to each alarm event based on the event vector.

[0097] In this embodiment, when calculating the node vectors for the microservice nodes corresponding to each alarm event, different weights can be assigned to each alarm event according to certain rules. Next, the weighted mean of the event vectors for each microservice node is calculated based on the event weights, which serves as the vector representation of the microservice node, i.e., the node vector. In this embodiment, two exemplary rules are proposed for calculating the weights of event alarms on microservice nodes during a fault:

[0098] (1) Event Frequency: The frequency of alarm events occurring on a microservice node during a fault period. When an event frequency is high, its corresponding weight does not necessarily need to be set high. This is because the alarm events generated during the fault period may contain many unrelated events, so it is necessary to analyze the importance of the events.

[0099] (2) Event Importance: Through the analysis of all historical failure cases, if an event only occurs in a few cases, then the event is generally considered to be of high importance. That is, if an event occurs on both the normal microservice node and the root cause microservice node, then the event is generally considered to be of low importance. If an event occurs on the root cause microservice node but not on the normal microservice node, then the event is considered to be of high importance.

[0100] For an event e during a certain fault i The microservice node c corresponding to the event j , the event frequency and event importance can be calculated using the following formula:

[0101]

[0102]

[0103] Among them, n i,j It is event e i In microservice node c j The number of occurrences in the event sequence of Is the microservice node c j The number of alarm events that occurred in the system; |C| is the product of the number of historical fault cases and the number of system microservice nodes, |{k|e i ∈c k}|The historical fault case contains the alarm event e i The number of microservice nodes.

[0104] Finally, considering the event frequency and event importance, the event e i Relative to microservice node c j The event weight calculation formula is:

[0105] EW i,j =EF i,j ×EI i

[0106] With the event weight, we can get the microservice node c j Vector representation of

[0107]

[0108] Among them, microservice node c j There are K events on the , and the K-th event vector is V k express.

[0109] Correspondingly, for this embodiment, the steps of the embodiment may specifically include: calculating the weight index of each alarm event, the weight index including at least the event frequency and the event importance; according to the weight index of each alarm event, calculating the event weight of each alarm event relative to the microservice node corresponding to each alarm event; based on the event vector and the event weight, calculating the weighted mean of each alarm event as the node vector of the microservice node corresponding to each alarm event.

[0110] 205. Model and generate a target homogeneous graph topology structure and a target heterogeneous graph topology structure of the target microservice, and obtain the relationship between microservice nodes in the target microservice.

[0111] For this embodiment, we can first obtain the target microservice corresponding to Figure 7 The microservice system business process diagram shown in the figure and Figure 8 The microservice deployment diagram shown in the figure is further modeled by the microservice system business process diagram to obtain the following Figure 9 The target homogeneous graph topology shown in the figure is obtained by modeling the microservice deployment diagram. Figure 10 The target heterogeneous graph topology is shown.

[0112] 206. Input the node vector of each microservice node and the target homogeneous graph topology structure into the trained first convolutional neural network model to obtain the positioning result of the root cause microservice node.

[0113] In a specific application scenario, before executing the steps of this embodiment, it is necessary to pre-train the first convolutional neural network model. Accordingly, the steps of the embodiment may specifically include: generating sample node vectors of each microservice node in the sample microservice based on the historical alarm events of the sample microservice; using the sample node vectors and the homogeneous graph topology of the sample microservice as input features, and using the root cause microservice node labels of the sample microservice as training labels, to train the first convolutional neural network model to obtain a trained first convolutional neural network model. It should be noted that in order to ensure accurate training of the first convolutional neural network model, the initial training data set may first be subjected to data enhancement processing, and the first convolutional neural network model may be further trained using the sample microservices in the training data set after data enhancement processing. Among them, the implementation process of performing data enhancement processing on the initial training data set can be referred to the relevant description in step 203 of the embodiment, which will not be repeated here.

[0114] Among them, the structure of the first convolutional neural network model (which can also be understood as a fault diagnosis model based on homogeneous graph) is as follows: Figure 11As shown in the figure, when using the trained first convolutional neural network model to determine the positioning result of the root cause microservice node, the target homogeneous graph topology structure and each node vector obtained in the previous step can be used as input, and after two graph convolution layers (each graph convolution layer is followed by a ReLu activation layer), each node embedding is obtained. The mean of each node vector is used as the vector representation of the graph. The graph vector representation is used as the input of the linear layer, and the output dimension is the positioning result of the root cause microservice node. In the first convolutional neural network model, the convolution layer can use the GCN graph convolution operator. For node i, in the lth layer of the graph convolutional neural network, the convolution operation is performed using the following formula:

[0115]

[0116] in, represents the state of node i in layer l, N(i) represents the set of neighbor nodes of node i, W (l) is the weight used for training, initialized using Xavier, σ is the activation function, b (l) is the bias term.

[0117] 207. The node vectors of each microservice node, the target heterogeneous graph topology, and the relationship between microservice nodes in the target microservice are input into the trained second convolutional neural network model to obtain a prediction result of the microservice fault type.

[0118] Microservice failure types include but are not limited to:

[0119] (1) CPU abnormality: related to CPU system indicators;

[0120] (2) Memory abnormality: related to memory system indicators;

[0121] (3) Memory release anomaly: anomaly injection manifests as a change in the way the system memory is used over a period of time;

[0122] (4) Call chain exception: Exception injection manifests itself in the lack of call chain integrity;

[0123] (5) File movement anomaly: Abnormal injection is manifested in changes in log patterns or sudden changes in the number of log errors;

[0124] (6) Access Denial Exception: Exception injection is reflected in the average success rate of business flows.

[0125] In a specific application scenario, before executing the steps of this embodiment, the second convolutional neural network model needs to be pre-trained. Accordingly, the steps of the embodiment may specifically include: generating sample node vectors of each microservice node in the sample microservice based on the historical alarm events of the sample microservice; using the sample node vectors, the heterogeneous graph topology of the sample microservice, and the relationship between the microservice nodes in the sample microservice as input features, and using the microservice fault type labels of the sample microservices as training labels, to train the second convolutional neural network model to obtain a trained second convolutional neural network model. It should be noted that in order to ensure accurate training of the second convolutional neural network model, the initial training data set may first be subjected to data enhancement processing, and the second convolutional neural network model may be further trained using the sample microservices in the training data set after data enhancement processing. Among them, the implementation process of data enhancement processing of the initial training data set can be referred to the relevant description in step 203 of the embodiment, which will not be repeated here.

[0126] Among them, the structure of the second convolutional neural network model (which can also be understood as a fault diagnosis model based on heterogeneous graphs) is as follows: Figure 12 As shown in Figure 2, the model input consists of three parts: the target heterogeneous graph topology, node vectors, and edge types. The target heterogeneous graph topology and node vector inputs are the same as those in the first convolutional neural network model. Edge types are derived from the different relationships between microservice nodes, and the output dimension is the predicted microservice failure type. The following uses the GAIA dataset as an example to illustrate how edge types are constructed.

[0127] In the GAIA dataset, there are three types of relationships between microservice nodes:

[0128] (1) Call. When a fault occurs in the upstream call during microservice communication, incorrect information (such as incorrect parameters) may be transmitted to the downstream call, causing the fault to propagate. Therefore, in the topology, the edge (A, 'call', B) is used to characterize the call from node A to node B.

[0129] (2) Being called. When a fault occurs in the called party during microservice communication, it may return an incorrect result to the caller, or may not return a result, causing the fault to propagate. Therefore, in the topology, the edge (B, 'called', A) is used to characterize the call of node B by node A.

[0130] (3) Shared containers. In order to describe the relationship between different microservice nodes sharing containers, virtual machines, hosts, etc., in the topology structure, two edges (A, 'share', 'B') and (B, 'share', A) are used to describe the shared container relationship between node A and node B.

[0131] The above three types of edges are more common in microservice systems. For other microservice systems, other types of edges can be defined according to specific circumstances as input to the second convolutional neural network model.

[0132] In the second convolutional neural network model, the convolution layer uses the R-GCN graph convolution operator. For node i, in the lth layer of the graph convolutional neural network, the convolution operation is performed using the following formula:

[0133]

[0134] in, represents the set of neighbor nodes of node i under relationship r, c i,r is a normalization constant (which can be ), σ is the activation function, is the self-loop weight, Used to regularize the weights in the R-GCN layer.

[0135] The microservice fault diagnosis method based on graph convolutional neural network provided by the embodiment of the present invention can collect alarm events of the target microservice to be diagnosed within a preset time period before and after the fault based on multimodal data, and then determine the alarm event sequence of each microservice node in the target microservice according to the alarm event; then generate the event vector of each alarm event in the alarm event sequence, and calculate the node vector of the microservice node corresponding to each alarm event based on the event vector; finally, input the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, which at least includes the positioning result of the root cause microservice node and the prediction result of the microservice fault type. Through the technical solution of the present invention, the microservice alarm events of three modes can be combined and used in a topological structure, and the graph convolutional neural network can be used to accurately locate the root cause microservice node and accurately predict the microservice fault type, thereby reducing the time spent by operation and maintenance personnel to restore the fault, accelerating the process from fault occurrence to fault recovery of the microservice system, and thus minimizing the loss.

[0136] Further, as Figure 1 The specific implementation of the present invention provides a microservice fault diagnosis device based on graph convolutional neural network, such as Figure 13 As shown, the device includes: a collection module 31, a determination module 32, a calculation module 33, and an input module 34.

[0137] The collection module 31 can be used to collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data;

[0138] A determination module 32 may be used to determine an alarm event sequence of each microservice node in a target microservice according to an alarm event;

[0139] The calculation module 33 can be used to generate an event vector for each alarm event in the alarm event sequence, and calculate the node vector of the microservice node corresponding to each alarm event based on the event vector;

[0140] The input module 34 can be used to input the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result. The fault diagnosis result at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type.

[0141] In a specific application scenario, alarm events include at least indicator alarm events, log alarm events, and call chain alarm events. When determining the alarm event sequence of each microservice node in the target microservice based on the alarm events, the determination module 32 can be specifically used to attribute the indicator alarm events and log alarm events to the microservice node where the event occurs, and to attribute the call chain alarm events to both the caller microservice node and the callee microservice node; on each microservice node, the alarm events are uniformly arranged in the order in which the events occur to obtain the alarm event sequence of each microservice node.

[0142] In a specific application scenario, when generating event vectors for each alarm event in an alarm event sequence, the calculation module 33 can be specifically used to input the alarm event sequence into a trained event vector generator to generate event vectors for each alarm event in the alarm event sequence, wherein the event vector generator is trained using a training data set after data enhancement processing.

[0143] In a specific application scenario, in order to realize data enhancement processing of the initial training data set and train the event vector generator using the training data set after data enhancement processing, such as Figure 14 As shown, the device further includes: a first training module 35:

[0144] The first training module 35 can be specifically used to: generate a sample node vector for each microservice node in the sample microservice based on the historical alarm events of the sample microservice in the initial training data set; exchange at least one sample node vector of the sample microservice in the initial training data set based on the root cause microservice node label or the microservice fault type label of the sample microservice to obtain a training data set after data enhancement processing; and use the training data set after data enhancement processing to train an event vector generator to obtain a trained event vector generator.

[0145] In a specific application scenario, when calculating the node vector of the microservice node corresponding to each alarm event based on the event vector, the calculation module 33 can be specifically used to calculate the weight index of each alarm event, and the weight index includes at least the event frequency and the event importance; according to the weight index of each alarm event, the event weight of each alarm event relative to the microservice node corresponding to each alarm event is calculated; based on the event vector and the event weight, the weighted mean of each alarm event is calculated as the node vector of the microservice node corresponding to each alarm event.

[0146] In a specific application scenario, the fault diagnosis model includes a first convolutional neural network model based on a homogeneous graph and a second convolutional neural network model based on a heterogeneous graph. Before the node vectors of each microservice node are input into the trained fault diagnosis model and the fault diagnosis results are obtained, in order to realize the pre-training of the fault diagnosis model, Figure 14 As shown, the device further includes: a second training module 36;

[0147] The second training module 36 can be specifically used to: generate a sample node vector of each microservice node in the sample microservice based on the historical alarm events of the sample microservice; use the sample node vector and the homogeneous graph topology structure of the sample microservice as input features, and use the root cause microservice node label of the sample microservice as a training label to train the first convolutional neural network model to obtain a trained first convolutional neural network model; and use the sample node vector, the heterogeneous graph topology structure of the sample microservice, and the relationship between the microservice nodes in the sample microservice as input features, and use the microservice fault type label of the sample microservice as a training label to train the second convolutional neural network model to obtain a trained second convolutional neural network model.

[0148] In specific application scenarios, such as Figure 14 As shown, the device further includes: a generating module 37;

[0149] A generation module 37 may be used to model and generate a target homogeneous graph topology structure and a target heterogeneous graph topology structure of a target microservice, and obtain the relationship between microservice nodes in the target microservice;

[0150] In a specific application scenario, the input module 34 can be used to input the node vectors of each microservice node and the target homogeneous graph topology structure into the trained first convolutional neural network model to obtain the positioning result of the root cause microservice node; and input the node vectors of each microservice node, the target heterogeneous graph topology structure and the relationship between the microservice nodes in the target microservice into the trained second convolutional neural network model to obtain the prediction result of the microservice fault type.

[0151] It should be noted that for other corresponding descriptions of the functional modules involved in the microservice fault diagnosis device based on graph convolutional neural network provided by the embodiment of the present invention, please refer to Figure 1 The corresponding description of the method shown will not be repeated here.

[0152] Based on the above Figure 1 The method shown, accordingly, an embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored, which implements the following steps when executed by a processor: collecting alarm events of the target microservice to be diagnosed within a preset time period before and after the fault, wherein the alarm events are generated based on multimodal data; determining the alarm event sequence of each microservice node in the target microservice according to the alarm events; generating an event vector for each alarm event in the alarm event sequence, and calculating the node vector of the microservice node corresponding to each alarm event based on the event vector; inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, which at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type.

[0153] Based on the above Figure 1 、 Figure 2 The method shown and Figure 13 、 Figure 14 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 15 As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43. When the processor 41 executes the program, the following steps are implemented: collecting alarm events of the target microservice to be diagnosed within a preset time period before and after the fault, wherein the alarm events are generated based on multimodal data; determining the alarm event sequence of each microservice node in the target microservice according to the alarm events; generating an event vector for each alarm event in the alarm event sequence, and calculating the node vector of the microservice node corresponding to each alarm event based on the event vector; inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, which at least includes the location result of the root cause microservice node and the prediction result of the microservice fault type.

[0154] The embodiment of the present invention can combine the three modal microservice alarm events and use the topological structure, and utilize the graph convolutional neural network to accurately locate the root cause microservice node and accurately predict the type of microservice failure, thereby reducing the time spent by operation and maintenance personnel to restore the failure, speeding up the process of microservice system from failure occurrence to failure recovery, and thus minimizing losses.

[0155] Obviously, those skilled in the art will appreciate that the various modules or steps of the present invention described above can be implemented using a general-purpose computing device, centralized on a single computing device, or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. In some cases, the steps shown or described can be performed in a different order than that shown, or can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0156] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A microservice fault diagnosis method based on graph convolutional neural network, characterized in that: include: Collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data and include at least indicator alarm events, log alarm events, and call chain alarm events; Determining an alarm event sequence for each microservice node in the target microservice based on the alarm event, including: attributing the indicator alarm event and the log alarm event to the microservice node where the event occurs, and attributing the call chain alarm event to both the caller microservice node and the callee microservice node; and uniformly arranging the alarm events on each microservice node according to the order in which the events occurred to obtain an alarm event sequence for each microservice node; Generating an event vector for each alarm event in the alarm event sequence, and calculating a node vector of a microservice node corresponding to each alarm event based on the event vector, including: calculating a weight index for each alarm event, the weight index including at least event frequency and event importance; calculating an event weight of each alarm event relative to the microservice node corresponding to each alarm event based on the weight index of each alarm event; and calculating a weighted mean of each alarm event based on the event vector and the event weight as the node vector of the microservice node corresponding to each alarm event; Inputting the node vector of each microservice node into the trained fault diagnosis model to obtain a fault diagnosis result, wherein the fault diagnosis result at least includes a location result of the root cause microservice node and a prediction result of the microservice fault type; Among them, before inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, it also includes: modeling and generating the target homogeneous graph topology structure and target heterogeneous graph topology structure of the target microservice, and obtaining the relationship between the microservice nodes in the target microservice; inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, including: inputting the node vector of each microservice node and the target homogeneous graph topology structure into the trained first convolutional neural network model to obtain the positioning result of the root cause microservice node; inputting the node vector of each microservice node, the target heterogeneous graph topology structure and the relationship between the microservice nodes in the target microservice into the trained second convolutional neural network model to obtain the prediction result of the microservice fault type.

2. The method according to claim 1, characterized in that Generating an event vector for each alarm event in the alarm event sequence includes: The alarm event sequence is input into a trained event vector generator to generate event vectors for each alarm event in the alarm event sequence, wherein the event vector generator is trained using a training data set after data enhancement processing.

3. The method according to claim 2, characterized in that Before inputting the alarm event sequence into a trained event vector generator to generate an event vector for each alarm event in the alarm event sequence, the method further includes: Generate a sample node vector for each microservice node in the sample microservice based on historical alarm events of the sample microservice in the initial training data set; swapping at least one sample node vector of the sample microservice in the initial training data set according to the root cause microservice node label or the microservice fault type label of the sample microservice to obtain a training data set after data enhancement processing; The event vector generator is trained using the training data set after the data enhancement process to obtain a trained event vector generator.

4. The method according to claim 1, wherein The fault diagnosis model includes a first convolutional neural network model based on a homogeneous graph and a second convolutional neural network model based on a heterogeneous graph. Before inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, the model further includes: Generate a sample node vector for each microservice node in the sample microservice based on historical alarm events of the sample microservice; Using the sample node vector and the homogeneous graph topology of the sample microservice as input features, and using the root cause microservice node label of the sample microservice as a training label, training the first convolutional neural network model to obtain a trained first convolutional neural network model; and The sample node vector, the heterogeneous graph topology of the sample microservice, and the relationship between microservice nodes in the sample microservice are used as input features, and the microservice fault type label of the sample microservice is used as a training label to train the second convolutional neural network model to obtain a trained second convolutional neural network model.

5. A microservice fault diagnosis device based on graph convolutional neural network, characterized in that: include: A collection module is used to collect alarm events of the target microservice to be diagnosed within a preset time period before and after the failure, wherein the alarm events are generated based on multimodal data and include at least indicator alarm events, log alarm events, and call chain alarm events; A determination module is configured to determine an alarm event sequence for each microservice node in the target microservice based on the alarm event, including: attributing the indicator alarm event and the log alarm event to the microservice node where the event occurs, and attributing the call chain alarm event to both the caller microservice node and the callee microservice node; and uniformly arranging the alarm events on each microservice node in the order in which the events occur to obtain an alarm event sequence for each microservice node; A calculation module is configured to generate an event vector for each alarm event in the alarm event sequence, and calculate a node vector of a microservice node corresponding to each alarm event based on the event vector, including: calculating a weight index for each alarm event, wherein the weight index includes at least event frequency and event importance; calculating, based on the weight index of each alarm event, an event weight of each alarm event relative to the microservice node corresponding to each alarm event; and calculating a weighted mean of each alarm event based on the event vector and the event weight as the node vector of the microservice node corresponding to each alarm event. An input module, configured to input the node vector of each microservice node into the trained fault diagnosis model to obtain a fault diagnosis result, wherein the fault diagnosis result includes at least a location result of the root cause microservice node and a prediction result of the microservice fault type; Among them, before inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, it also includes: modeling and generating the target homogeneous graph topology structure and target heterogeneous graph topology structure of the target microservice, and obtaining the relationship between the microservice nodes in the target microservice; inputting the node vector of each microservice node into the trained fault diagnosis model to obtain the fault diagnosis result, including: inputting the node vector of each microservice node and the target homogeneous graph topology structure into the trained first convolutional neural network model to obtain the positioning result of the root cause microservice node; inputting the node vector of each microservice node, the target heterogeneous graph topology structure and the relationship between the microservice nodes in the target microservice into the trained second convolutional neural network model to obtain the prediction result of the microservice fault type.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images