Method and device for determining fault root cause

Abstract

The embodiment of the invention discloses a fault root cause determination method and device, and is applied to the technical field of computers. According to the determination method, firstly, a calling relation graph and an index causal graph are obtained, and then the fault root cause is determined by using the calling relation graph and the index causal graph. Wherein the calling relation graph is used for representing calling relations among the plurality of micro-services; the index causal graph is used for representing the causal relationship among the multiple pieces of monitoring index data. That is to say, the root cause causing the fault of the micro-service system is accurately positioned by obtaining the calling relationship between the micro-services in the micro-service system and the causal relationship between the monitoring index data of each micro-service. Namely, according to the embodiment of the invention, on the premise that a plurality of micro-services give an alarm at the same time, the calling link with the fault can be positioned through the calling relationship among the micro-services, and the fault root cause can be accurately positioned from the positioned calling link in combination with the causal relationship among the monitoring index data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a fault root cause determination method and device. BACKGROUND

[0002] A microservice system includes a plurality of microservices having a dependency relationship, for example, an e-commerce system includes user services, commodity services, order services, shopping cart services and other microservices having a dependency relationship. When the microservice system fails, multiple microservices often simultaneously alarm, which makes it difficult for an operation and maintenance personnel to accurately locate the fault root cause of the failure. The fault root cause refers to the root cause of the failure of the microservice system, that is, the microservice or other component that causes the failure of the microservice system. SUMMARY

[0003] Embodiments of the present application provide a fault root cause determination method and device for accurately locating the fault root cause of the failure of a microservice.

[0004] In a first aspect, embodiments of the present application provide a fault root cause determination method applied to a microservice system, the microservice system including a plurality of microservices, and the method including:

[0005] obtaining a call relationship graph and an index causal graph;

[0006] The call relationship graph is used to represent the call relationship between the plurality of microservices. The index causal graph is used to represent the causal relationship between a plurality of monitoring index data, the plurality of monitoring index data including monitoring index data of each microservice in the plurality of microservices, the monitoring index data of a first microservice being used to monitor the performance of the first microservice, the first microservice being any one of the plurality of microservices.

[0007] According to the call relationship graph and the index causal graph, the fault root cause of the microservice system is determined.

[0008] Optionally, the call relationship graph includes service nodes and call relationship edges, one service node being used to represent one microservice. The call relationship edge is used to represent a call link from a calling microservice to a called microservice, the start point of the call relationship edge being the service node corresponding to the calling microservice, and the end point of the call relationship edge being the service node corresponding to the called microservice.

[0009] Optionally, the indicator causal diagram comprises monitoring nodes and causal relationship edges, one monitoring node corresponding to one monitoring indicator data; the causal relationship edge is used to represent a causal relationship chain in which a first monitoring indicator data causes a second monitoring indicator data to be generated, a starting point of the causal relationship edge being a monitoring node corresponding to the first monitoring indicator data, and a terminal point of the causal relationship edge being a monitoring node corresponding to the second monitoring indicator data.

[0010] Optionally, the determining of the fault root cause of the microservice system according to the call relationship diagram and the indicator causal diagram comprises:

[0011] fusing the call relationship diagram and the indicator causal diagram to obtain a node fusion diagram;

[0012] The node fusion diagram is used to indicate the call relationship among the plurality of microservices, the causal relationship among the plurality of monitoring indicator data, and the mapping relationship between the plurality of microservices and the plurality of monitoring indicator data.

[0013] The fault root cause of the microservice system is determined according to the node fusion diagram.

[0014] Optionally, the determining of the fault root cause of the microservice system according to the node fusion diagram comprises:

[0015] The fault root cause of the microservice system is determined according to the node fusion diagram and the time sequence feature of each microservice in the plurality of microservices.

[0016] The time sequence feature of the first microservice indicates a data feature of monitoring indicator data of the first microservice changing with time, and the time sequence feature of the first microservice comprises at least two of a short-term feature, a medium-term feature and a long-term feature of the first microservice, the time scale corresponding to the short-term feature being smaller than the time scale corresponding to the medium-term feature, and the time scale corresponding to the medium-term feature being smaller than the time scale corresponding to the long-term feature.

[0017] Optionally, the determining of the fault root cause of the microservice system according to the node fusion diagram and the time sequence feature of each microservice in the plurality of microservices comprises:

[0018] The fault propagation path is determined according to the node fusion diagram and the time sequence feature of each microservice, and a fault propagation probability matrix of the fault propagation between the plurality of microservices is obtained.

[0019] The fault propagation path comprises the propagation of the fault between microservices, the propagation of the fault between monitoring indicator data, and the propagation of the fault between microservices and monitoring indicator data.

[0020] determine a fault root cause of the microservice system based on the fault propagation path and the fault propagation probability matrix.

[0021] Optionally, the determining the fault root cause of the microservice system based on the fault propagation path and the fault propagation probability matrix comprises:

[0022] determine a path probability of each node in the fault propagation path to a target node based on the fault propagation path and the fault propagation probability matrix; the target node is a node corresponding to a microservice that has been determined to have a fault in the microservice system;

[0023] for each node in the fault propagation path, determine a posterior probability of the node based on path probabilities of the node to all target nodes;

[0024] determine the fault root cause based on the posterior probability of each microservice node.

[0025] Optionally, the determining the fault root cause of the microservice system based on the node fusion graph and the time sequence feature of each microservice in the plurality of microservices comprises:

[0026] processing the node fusion graph and the time sequence feature of each microservice using a preset network model to determine the fault root cause of the microservice system;

[0027] The preset network model is a network model processed based on a causal enhancement attention mechanism, and the causal enhancement attention mechanism is a mechanism that adjusts attention weights by taking a causal relationship parameter as an adjustment factor of the attention weights.

[0028] Optionally, the obtaining the call relationship graph and the indicator causal graph comprises:

[0029] dynamically updating the call relationship graph and the indicator causal graph based on a sliding time window; wherein the following operations are performed within one sliding time window:

[0030] obtaining call relationship data between the plurality of microservices and monitoring indicator data of each microservice;

[0031] The call relationship data is used to describe data of a call relationship between the plurality of microservices.

[0032] obtaining the call relationship graph according to the call relationship data, and obtaining an indicator causal graph based on the monitoring indicator data of each microservice.

[0033] In a second aspect, an embodiment of the present application provides a fault root cause determination apparatus applied to a microservice system, the microservice system comprising a plurality of microservices, and the apparatus comprising:

[0034] an acquisition unit configured to acquire a call relationship graph and an index causal graph;

[0035] The call relationship graph is used to represent a call relationship between the plurality of microservices. The index causal graph is used to represent a causal relationship between a plurality of monitoring index data, the plurality of monitoring index data including monitoring index data of each microservice in the plurality of microservices, monitoring index data of a first microservice being used to monitor performance of the first microservice, the first microservice being any one of the plurality of microservices.

[0036] a determination unit configured to determine a fault root cause of the microservice system according to the call relationship graph and the index causal graph.

[0037] In a third aspect, an embodiment of the present application provides a computer program product, which includes a computer program (also referred to as code or instructions), when the computer program is executed, causes a computer to execute the method in any possible implementation manner of any one of the aspects above.

[0038] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program (also referred to as code or instructions), when the computer program is executed on a computer, causes the computer to execute the method in any possible implementation manner of any one of the aspects above.

[0039] In a fifth aspect, an embodiment of the present application provides a chip system, which includes one or more processors configured to call and execute instructions stored in a memory, so that the method in each aspect or any possible implementation manner of each aspect is executed. The chip system can be composed of a chip, or can include a chip and other discrete devices.

[0040] Beneficial effects: The embodiment of the present application provides a fault root cause determination method and device. When the method is executed, first, a call relationship graph and an index causal graph are acquired, and then the call relationship graph and the index causal graph are used to determine a fault root cause. The call relationship graph is used to represent a call relationship between the plurality of microservices; the index causal graph is used to represent a causal relationship between a plurality of monitoring index data, the plurality of monitoring index data including monitoring index data of each microservice in the plurality of microservices, and the monitoring index data of a first microservice is used to monitor the performance of the first microservice, the first microservice being any microservice in the plurality of microservices. That is, the present application accurately locates a root cause of causing a fault of a microservice system by acquiring a call relationship between microservices in the microservice system and a causal relationship between monitoring index data of each microservice. That is, the embodiment of the present application can locate a call link that appears to be faulty by using the call relationship between the microservices, and accurately locate a fault root cause from the located call link by combining the causal relationship between the monitoring index data, on the premise that a plurality of microservices simultaneously alarm. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0042] Figure 1 A system architecture schematic diagram of a positioning system 10 provided by the embodiment of the present application;

[0043] Figure 2 A flowchart of a fault root cause determination method provided by the embodiment of the present application;

[0044] Figure 3 A schematic diagram of a call relationship graph provided by the embodiment of the present application;

[0045] Figure 4 A schematic diagram of an index causal graph provided by the embodiment of the present application;

[0046] Figure 5 A schematic diagram of a node fusion graph 500 provided by the embodiment of the present application;

[0047] Figure 6 A flowchart of another fault root cause determination method provided by the embodiment of the present application;

[0048] Figure 7 A structural schematic diagram of a fault root cause determination device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0049] In the following, the technical solutions in the embodiments will be described clearly and completely in combination with the drawings in the embodiments so that those skilled in the art can better understand the solutions in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0050] The method for determining a fault root cause provided by the embodiments of the present application is used to determine a fault root cause of a micro-service system.

[0051] Exemplarily, the method for determining a fault root cause provided by the embodiments of the present application can be applied in a cloud-native micro-service system. The cloud-native micro-service system refers to a micro-service system architecture constructed based on a cloud-native environment. The cloud-native micro-service system includes multiple small and independent micro-services, and each micro-service focuses on a specific business function and is independently developed, deployed and expanded. For example, the cloud-native micro-service system is an e-commerce system, which can include multiple micro-services such as user service, commodity service, order service, payment service and shopping cart service.

[0052] It should be noted that the user service is used for a micro-service responsible for handling operations related to user accounts, including but not limited to user registration, login verification, personal information management, etc. The order service is used for a micro-service handling a series of operations such as order creation, confirmation, cancellation and refund. The shopping cart service refers to a micro-service that allows users to select and save desired goods before purchase. The payment service is used for a micro-service handling all payment-related transactions. The commodity service refers to a micro-service managing and handling all information and processes related to commodities.

[0053] The micro-service system may fail, for example, the e-commerce system has an order service exception. When the current micro-service system fails, multiple micro-services may simultaneously alarm, which makes it difficult for the operation and maintenance personnel to determine the fault root cause of the micro-service system. For example, when the e-commerce system has an order service exception, the order service, payment service, user service and commodity service simultaneously alarm, but the root cause of the order service exception may be caused by the user service exception, and the operation and maintenance personnel cannot accurately locate the fault root cause through the existing alarm condition.

[0054] Further analysis of the prior art shows that at present, simple rule matching methods are often used to locate the root cause of the fault of the microservice system. For example, a threshold alarm method is used to locate the root cause of the fault. When the monitoring index of a microservice reaches a preset threshold, the microservice alarms. Based on the alarm information, an operation and maintenance personnel can locate the root cause of the fault. However, there is a dependency relationship between multiple microservices. For example, in an e-commerce system, to execute an order service, a payment service needs to be called to handle payment matters. To execute the payment service, a user service needs to be called to obtain relevant information of a user, such as an account balance. This makes it so that when one of the microservices fails, the failure will affect other microservices through the call link. For example, if the user service fails, it will affect the payment service, and further affect the order service, thereby causing multiple microservices to alarm at the same time.

[0055] Therefore, the embodiments of the present application provide a method for determining a root cause of a fault. By obtaining the call relationship between microservices in a microservice system and the causal relationship between monitoring index data of each microservice, the root cause of the fault of the microservice system can be accurately located. That is, under the premise that multiple microservices alarm at the same time, the call link that fails can be located through the call relationship between the microservices, and the root cause of the fault can be accurately located from the located call link in combination with the causal relationship between the monitoring index data.

[0056] It should be noted that the method for determining the root cause of the fault provided by the embodiments of the present application is not limited to the cloud-native microservice system described above, but also applies to other types of microservice systems, such as hybrid cloud microservice systems, systems in which monolithic applications are gradually microserviced, and the like. The embodiments of the present application are not specifically limited. In these different types of microservice systems, the method for determining the root cause of the fault faces different architecture environments and application scenarios, but the core working principle remains the same or basically consistent. Hereinafter, the cloud-native microservice system is taken as an example for description. Unless otherwise specified, the microservice system described hereinafter is a cloud-native microservice system.

[0057] The system architecture of the embodiments of the present application is described below.

[0058] The Figure 1 A system architecture diagram of a positioning system 10 provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the positioning system 10 is used to locate the root cause of the fault of a microservice system 20. The system architecture of the positioning system 10 includes a graph construction layer 101 and a root cause reasoning layer 102. Figure 1

[0059] The graph construction layer 101 can directly obtain a call relationship graph and an index causal graph.

[0060] ​It should be noted that the graph construction layer 101 can obtain the constructed call relationship graph and the indicator causal graph from other systems or components, or can first obtain call relationship data for constructing the call relationship graph and monitoring indicator data for constructing the indicator causal graph, and then construct the call relationship graph based on the call relationship data and construct the indicator causal graph based on the monitoring indicator data. The embodiments of the present application are not specifically limited.

[0061] The graph construction layer 101 sends the obtained call relationship graph and the indicator causal graph to the root cause reasoning layer 102. Correspondingly, the root cause reasoning layer 102 receives the call relationship graph and the indicator causal graph. Then the root cause reasoning layer 102 determines the fault root cause of the microservice system based on the received call relationship graph and the indicator causal graph.

[0062] In an example, the graph construction layer 101 is further configured to fuse the call relationship graph and the indicator causal graph, so that a fused graph includes both the call relationship connection information between the plurality of microservices and the causal relationship connection information between the plurality of monitoring indicators. The graph construction layer 101 sends the fused graph to the next level for processing.

[0063] The system architecture of the positioning system 10 provided in the embodiments of the present application can also include other layers.

[0064] For example, continuing to refer to Figure 1 As shown, the system architecture of the positioning system 10 further includes a data acquisition layer B01, which is in communication connection with the graph construction layer 101, and is configured to acquire multi-dimensional data of the microservice system 20, and send the acquired multi-dimensional data to the graph construction layer 101.

[0065] In the embodiments of the present application, the data acquisition layer B01 can acquire call relationship data and monitoring indicator data between a plurality of microservices in the microservice system 20. For example, the data acquisition layer B01 acquires the call relationship data and the monitoring indicator data of the microservice system 20 through a distributed tracking system. The distributed tracking system includes but is not limited to Jaeger and Zipkin systems.

[0066] In addition, the data acquisition layer B01 can also acquire log event data of the microservice system 20. For example, the data acquisition layer B01 can acquire log event data of the microservice system 20 based on a semantic structured log acquisition method. Specifically, the data acquisition layer B01 first classifies the log event data, for example, divides the log event data into error level data, warning level data and general data. Then, the data acquisition layer B01 extracts key abnormal information from unstructured logs using a keyword extraction method, for example, the data acquisition layer B01 uses natural language processing (NLP) technology to extract key abnormal information from unstructured logs.

[0067] It should be noted that, in the embodiments of the present application, in order to ensure the consistency of the collected data and accurately locate the root cause of the fault, the log time of each microservice in the microservice system 20 can be synchronized based on a distributed clock synchronization mechanism.

[0068] In yet another example, continuing to refer to Figure 1 As shown, the positioning system 10 further includes a feature extraction layer B02, one end of the feature extraction layer B02 being connected to the graph construction layer 101 and the other end being connected to the root cause inference layer 102.

[0069] The feature extraction layer B02 can obtain the index causal graph and the call relationship graph, and perform feature extraction on the index causal graph and the call relationship graph, and send the extracted features to the root cause inference layer 102 for processing.

[0070] Alternatively, the feature extraction layer B02 is further configured to obtain a node fusion graph from the graph construction layer 101, and perform extraction processing on the node fusion graph to obtain fusion features. The feature extraction layer B02 sends the fusion features to the root cause inference layer 102 for processing. The node fusion graph is a fusion graph obtained by fusing the index causal graph and the call relationship graph.

[0071] Alternatively, the feature extraction layer B02 is configured to obtain a node fusion graph from the graph construction layer 101, and extract time sequence features of nodes corresponding to each microservice in the node fusion graph to determine fusion features, which are also referred to as graph time sequence fusion features. The time sequence features are used to reflect the data features of the monitoring index data of the microservice changing over time.

[0072] It should be noted that, in the embodiments of the present application, the time sequence features can be determined based on time sequence data, and the time sequence data is determined based on log event data.

[0073] In the embodiments of the present application, the root cause inference layer 102 can receive the fusion features sent by the feature extraction layer B02. The root cause inference layer 102 includes a Causal-Enhanced Graph Attention Network (CEGAT) algorithm, which processes the fusion features to determine the root cause of the fault.

[0074] In yet another example, continuing to refer to Figure 1 As shown, the positioning system 10 can further include a model training layer B03. One end of the feature extraction layer B02 is connected to the graph construction layer 101, and the other end can be connected to the model training layer B03, and the model training layer B03 is connected to the root cause inference layer 102.

[0075] The feature extraction layer B02 is also configured to send the graph fusion feature to the model training layer B02, so that the model training layer B02 trains the CEGAT algorithm in the root cause inference layer 102 by using the graph fusion feature, so that the CEGAT algorithm can learn the fault propagation mode and the fault propagation rule.

[0076] The feature extraction layer B02 can also be configured to send the graph fusion feature to the root cause inference layer 102 through the model training layer B02.

[0077] In yet another example, as shown in FIG. 10, the positioning system 10 further includes a visualization layer B04 connected with the root cause inference layer 102, configured to receive the fault root cause sent by the root cause inference layer 102 and visualize the fault root cause. Figure 1

[0078] In the embodiments of the present application, the layers can interact through standardized interfaces to achieve high cohesion and low coupling, or can be achieved through other means, which are not specifically limited in the embodiments of the present application. The layers can be designed in a plug-in manner to support flexible replacement of different algorithms and components to adapt to different micro-service architectures.

[0079] It should be noted that the fault root cause determination method provided by the embodiments of the present application can also be applied to other system architectures, which are not specifically limited in the embodiments of the present application. In addition, the system architecture provided by the embodiments of the present application can process one micro-service system to determine the fault root cause of the micro-service system, and can also simultaneously process the architectures of multiple micro-service systems to determine the fault root cause in each micro-service system. Since the fault root cause determination method is the same or basically the same, for the convenience of those skilled in the art, the present application will be described in detail with reference to the positioning system 10 shown in FIG. 10 as an example. Figure 1

[0080] The above describes the system architecture of the positioning system for determining the fault root cause of the micro-service system, and the following will be described in detail with reference to the positioning system 10 shown in FIG. 10 as an example. Figure 1 Figure 2 Figure 5 The fault root cause determination method provided by the embodiments of the present application will be described in detail and completely.

[0081] FIG. 11 is a flowchart of a fault root cause determination method provided by an embodiment of the present application, which is applied to the micro-service system described above, the micro-service system including a plurality of micro-services, and the method including the following contents. Figure 2

[0082] S210, acquiring a call relationship graph and an index causal graph.

[0083] First, the call relationship graph is introduced.

[0084] ​​​​​In the embodiments of the present application, the call relationship graph is used to represent the call relationship between the plurality of microservices in the microservice system, or also referred to as the service dependency relationship. Exemplarily, a schematic diagram of a call relationship graph is provided in the embodiments of the present application. As shown in the figure, the call relationship graph 300 is a call relationship graph involved in the execution of an order service in an e-commerce system. Specifically, when the e-commerce system executes the order service, the payment service is called to perform payment, and the user service is called when the payment is performed. Figure 3 Figure 3 As shown in the figure, the call relationship graph 300 is a call relationship graph involved in the execution of an order service in an e-commerce system. Specifically, when the e-commerce system executes the order service, the payment service is called to perform payment, and the user service is called when the payment is performed.

[0085] In the embodiments of the present application, the call relationship graph can be a directed graph, including service nodes and call relationship edges. One service node corresponds to one microservice, and the call relationship edge is used to represent the call link of the calling microservice pointing to the called microservice. The starting point of the call relationship edge is the service node corresponding to the calling microservice, and the ending point of the call relationship edge is the service node corresponding to the called microservice.

[0086] Exemplarily, continuing to refer to the figure, the call relationship graph involves the user service node 301 corresponding to the user service, the payment service node 302 corresponding to the payment service, and the order service node 303 corresponding to the order service. It also involves a first call relationship edge and a second call relationship edge, wherein the starting point of the first call relationship edge is the user service node 301, and the ending point is the payment service node 302. The starting point of the second call relationship edge is the payment service node 302, and the ending point is the order service node 303. Figure 3

[0087] The call relationship graph provided in the embodiments of the present application includes the call relationship between the microservices. When a microservice fails, the positioning system 10 can determine the microservice that may cause the microservice failure based on the call relationship graph.

[0088] In addition, the call relationship graph provided in the embodiments of the present application can also include service node attribute parameters and call attribute parameters.

[0089] The service node attribute parameter is used to uniquely identify the microservice corresponding to the service node. In the embodiments of the present application, the node attribute parameter includes but is not limited to the microservice name, the microservice identifier, and the deployment location of the microservice.

[0090] The call attribute parameter is used to describe the call frequency and call quality between the microservices. It can be understood that the higher the call frequency of a microservice, the more likely the microservice has a large load pressure, and the microservice is prone to failure. The worse the call quality of a microservice, the more likely the microservice has a reliability problem, i.e., the microservice is prone to failure.

[0091] ​​In the embodiments of the present application, the calling attribute parameters include but are not limited to: calling frequency, calling response time and calling error rate. The calling frequency refers to the frequency of calling the called microservice by the calling microservice. The higher the frequency, the more frequent the calling. The calling response time is the time for the called microservice to respond to the calling request sent by the calling microservice. The calling error rate indicates the probability of error (i.e. unsuccessful response) of the called microservice in processing the calling request.

[0092] In a possible implementation, due to the dynamic scaling of the number of microservices of the cloud-native microservice system, and the frequent version update of the services, it is necessary to dynamically update the calling relationship graph, so that the calling relationship graph can better reflect the calling relationship of the current microservice system.

[0093] In the embodiments of the present application, as shown in Figure 1 The data collection layer B01 can collect the calling relationship data between the plurality of microservices in the microservice system based on the distributed tracing system. The calling relationship data includes but is not limited to calling microservice, called microservice, calling time, calling delay and calling status code, etc.

[0094] For example, the data collection layer B01 can collect the calling relationship data of the microservice system in real time or at intervals of a preset time period through automatic and / or manual burying. Further, the data collection layer B01 can also store the collected data to a time series database (such as InfluxDB). The time series database can support efficient data query and acquisition operations within a preset time range.

[0095] Then, the graph construction layer 101 can obtain the calling relationship data from the data collection layer B01 based on the sliding time window strategy. The sliding time window is a continuously moving time interval, for example, the window slides forward once every certain time interval, so as to ensure that the positioning system is always processing the calling relationship data in the latest time period.

[0096] It should be noted that in the current time window, the graph construction layer 101 obtains the calling relationship data collected in the corresponding time range in the sliding time window from the data collection layer B01. Then, the graph construction layer 101 dynamically constructs the calling relationship graph based on the calling relationship data.

[0097] The index causal graph is described below.

[0098] In the embodiments of the present application, the index causal graph is used to represent the causal relationship between the plurality of monitoring index data. The plurality of monitoring index data includes the monitoring index data of each microservice in the plurality of microservices in the microservice system.

[0099] The monitoring indicator data of each microservice, for example, the monitoring indicator data of the first microservice, is used to monitor the performance of the first microservice. In the embodiments of the present application, the monitoring indicator data of each microservice includes, but is not limited to, the corresponding response time, error rate, garbage collection time, memory occupation, network throughput, and the like of the microservice.

[0100] In an example, the monitoring indicator data of each microservice is specifically multi-level resource usage indicator data. The multi-level resource usage indicator data includes resource usage indicator data of an infrastructure layer, resource usage indicator data of a middleware layer, and resource usage indicator data of an application layer. The resource usage indicator data of the infrastructure layer includes, but is not limited to, a central processing unit (CPU) usage rate, memory occupation, network throughput, and disk I / O usage rate. The resource usage indicator data of the middleware layer includes, but is not limited to, message queue backlog, cache hit rate, and connection pool usage rate. The resource usage indicator data of the application layer includes, but is not limited to, the calling frequency of the called microservice (also referred to as queries per second (QPS)), calling response time, calling error rate, and garbage collection time.

[0101] In the embodiments of the present application, the indicator causal diagram can include monitoring nodes and causal relationship edges. One monitoring node corresponds to one monitoring indicator data, and the causal relationship edge represents a causal relationship chain in which the change of a first monitoring indicator data causes the change of a second monitoring indicator data. The starting point of the causal relationship edge is the monitoring node corresponding to the first monitoring indicator data, and the terminal point of the causal relationship edge is the monitoring node corresponding to the second monitoring indicator data.

[0102] Exemplarily, the monitoring indicator data corresponding to the monitoring nodes in the indicator causal diagram 400 is shown in Table 1. Figure 4 An example of an indicator causal diagram is provided in the embodiments of the present application. As shown in Table 1, the indicator causal diagram 400 includes monitoring node A, monitoring node B, monitoring node C, monitoring node D, and monitoring node E. The monitoring indicator data corresponding to the monitoring nodes in the indicator causal diagram 400 is shown in Table 1. Figure 4 As shown in Table 1, the monitoring indicator data corresponding to the monitoring nodes in the indicator causal diagram 400 is shown in Table 1.

[0103] In addition, the index causal diagram 400 further includes causal relationship edges AB, BC, CD and AE. The causal relationship edge AB indicates that the monitoring index data corresponding to the monitoring node A has a causal influence on the monitoring index data corresponding to the monitoring node B. The causal relationship edge BC indicates a causal influence of the monitoring index data corresponding to the monitoring node B on the monitoring index data corresponding to the monitoring node C. The causal relationship edge CD indicates that the monitoring index data corresponding to the monitoring node C has a causal influence on the monitoring index data corresponding to the monitoring node D. The causal relationship edge AE indicates that the monitoring index data corresponding to the monitoring node A has a causal influence on the monitoring index data corresponding to the monitoring node E.

[0104] In the embodiment of the present application, the index causal diagram 400 further includes a monitoring node attribute parameter and a causal strength parameter. The monitoring node attribute parameter is used to identify the monitoring index data corresponding to the node, including but not limited to: a data type of the monitoring index data, an identifier of a microservice where the monitoring index data is from, a time when the monitoring index data is collected, and the like.

[0105] The causal strength parameter is used to measure the strength of the causal relationship between the monitoring index data. In the embodiment of the present application, the higher the causal strength parameter, the stronger the causal relationship between the two monitoring index data. For example, the QPS of the user service is likely to be the main cause of the long response time of the order service, and therefore, the positioning system 10 can quickly focus on the root cause of the fault, i.e., the QPS of the user service, based on the causal strength parameter.

[0106] In a specific implementation, the positioning system 10 can determine the causal relationship between the monitoring index data based on historical data, and obtain the index causal diagram.

[0107] In another implementation, the positioning system 10 can further construct the index causal diagram through an improved Peter-Clark algorithm (PC) algorithm and a Granger causality test method. The PC algorithm is a statistical learning method for constructing a causal diagram, which can infer the potential causal relationship structure from the observation data. The Granger causality test method is based on the comparison of prediction errors to determine the causal strength parameter of one data causing another data.

[0108] Specifically, the positioning system 10 firstly assumes that there can be a causal relationship between all the monitoring indicator data, and obtains an initial causal graph (the initial indicator causal graph is an undirected graph, i.e., without direction). Then, the positioning system 10 determines the causal relationship between the monitoring indicator data by using the improved PC algorithm, and obtains an intermediate causal graph. Then, the positioning system 10 determines a causal strength parameter of one monitoring indicator data to another monitoring indicator data by using the Granger causality test method through time series characteristics. Then, the positioning system 10 deletes the causal relationship edge whose causal strength parameter is less than a first preset strength threshold. Finally, the positioning system 10 can obtain the target causal graph (i.e., the indicator causal graph).

[0109] It should be noted that the first preset strength threshold is a strength threshold set by a person skilled in the art according to needs, which can be adjusted according to needs in actual use. For example, the first preset strength threshold is 0.2.

[0110] In a specific implementation, the indicator causal graph can be dynamically updated based on a sliding time window strategy, and the specific updating manner is the same as the dynamic updating of the calling relationship graph described above, which will not be described here.

[0111] S220, determining a fault root cause of the microservice system based on the calling relationship graph and the indicator causal graph.

[0112] In the embodiment of the application, the positioning system 10 can use the calling relationship graph and the indicator causal graph to accurately determine the fault root cause of the microservice system in combination with the calling relationship information among the microservices and the causal relationship information of the plurality of monitoring indicator data.

[0113] For example, at a certain moment, the user service, the order service and the payment service all have alarms. The graph construction layer 101 obtains the calling relationship graph among the user service, the order service and the payment service. The calling relationship graph specifically indicates that the order service calls the payment service, and the payment service calls the user service. The graph construction layer 101 is also used to obtain the indicator causal graph, for example, the indicator causal graph indicates that the CPU usage of the payment service increases, which causes the response time of the payment service to increase, the response time of the payment service increases, which causes the response time of the order service to increase, and further causes the order service to fail.

[0114] In an example, the positioning system 10 can firstly perform fusion processing on the calling relationship graph and the indicator causal graph, and obtain a node fusion graph. The positioning system 10 determines the fault root cause of the microservice system based on the node fusion graph.

[0115] The node fusion graph is used to indicate the calling relationship among the plurality of microservices, the causal relationship among the plurality of monitoring indicator data, and the mapping relationship between the plurality of microservices and the plurality of monitoring indicator data.

[0116] For example, the positioning system 10 can obtain the node fusion graph as shown in FIG. 6.Figure 5 A schematic diagram of a node fusion graph 500 is provided in an embodiment of the present application. As shown, the node fusion graph 500 includes a user service node 301, a payment service node 302, and an order service node 303, and further includes a calling relationship edge 12 and a calling relationship edge 23. The node fusion graph 500 also involves a monitoring node A, a monitoring node B, a monitoring node C, a monitoring node D, and a monitoring node E, and a causal relationship edge AB, an edge BC, an edge CD, and an edge AE. In addition, the node fusion graph 500 also includes a mapping relationship edge 3A, a mapping relationship edge 3C, a mapping relationship edge 3E, and a mapping relationship edge 2B. Figure 5

[0117] In an embodiment of the present application, the mapping relationship edge is used to indicate the mapping relationship between the monitoring index data and the microservice, and through the mapping relationship edge, the microservice where the monitoring index data comes from can be determined.

[0118] It should be noted that the monitoring node A is a node corresponding to the monitoring index data A, the monitoring node B is a node corresponding to the monitoring index data B, the monitoring node C is a node corresponding to the monitoring index data C, and the monitoring node D is a node corresponding to the monitoring index data D.

[0119] The positioning system 10 can obtain the calling relationship information and the causal relationship information from one graph at the same time during fault positioning by fusing the index causal graph and the calling relationship graph, which can significantly improve the data processing efficiency compared with cross-querying between different graphs.

[0120] In an embodiment of the present application, the positioning system 10 can fuse the index causal graph and the calling relationship graph in multiple ways.

[0121] Exemplarily, since the monitoring index data carries a source identification tag of the microservice where it comes from, such as the service name of the microservice, the microservice identifier, etc., when the monitoring index data is collected, the positioning system 10 can obtain the mapping relationship between the service nodes in the calling relationship graph and the index nodes in the index causal graph based on the source identification tag. Then, the positioning system 10 fuses the index causal graph and the calling relationship graph based on the mapping relationship to obtain a node fusion graph.

[0122] The node fusion graph specifically includes a service calling structure, a causal relationship structure, and a node mapping structure. The service calling structure is a calling structure corresponding to the calling relationship graph, the causal relationship structure is a causal structure corresponding to the index causal graph, and the node mapping structure is a mapping structure determined based on the mapping relationship between the service nodes in the calling relationship graph and the index nodes in the index causal graph. Exemplarily, continuing to refer to FIG. 1, the service calling structure 101 is a service calling structure in the calling relationship graph as shown in FIG. 2, and the causal relationship structure 102 is a causal relationship structure in the index causal graph as shown in FIG. 3. Figure 5 Figure 3 The node fusion graph specifically includes a service calling structure, a causal relationship structure, and a node mapping structure. The service calling structure is a calling structure corresponding to the calling relationship graph, the causal relationship structure is a causal structure corresponding to the index causal graph, and the node mapping structure is a mapping structure determined based on the mapping relationship between the service nodes in the calling relationship graph and the index nodes in the index causal graph. Exemplarily, continuing to refer to FIG. 1, the service calling structure 101 is a service calling structure in the calling relationship graph as shown in FIG. 2, and the causal relationship structure 102 is a causal relationship structure in the index causal graph as shown in FIG. 3. Figure 4 ​​The causal relationship structure in the illustrated causal relationship diagram. The node mapping structure 103 specifically points to the monitoring node A, monitoring node B and monitoring node C structure of the order service node 303, and the structure of the payment service node 302 pointing to the monitoring node B.

[0123] In another specific implementation, since the call relationship diagram can involve a large number of microservices, the causal relationship diagram includes a large amount of monitoring index data, and the mapping relationship between the two also involves a large amount of data, which leads to a large amount of data in the node fusion graph. Directly processing the node fusion graph will affect the determination efficiency. Therefore, the positioning system 10 first performs sparse processing on the node fusion graph, reduces the number of edges, and reduces the number of node fusion graphs to save computing cost and improve determination efficiency.

[0124] In an example, the positioning system 10 can be implemented by removing edges with an attribute parameter below a preset parameter threshold. For example, the call relationship edge with a call frequency below a preset frequency threshold in the node fusion graph is removed, and the causal relationship edge with a causal strength parameter below a second preset strength threshold is deleted.

[0125] It should be noted that the preset frequency threshold and the second preset strength threshold are set by those skilled in the art as needed, and the second preset strength threshold is greater than the first preset strength threshold, for example, the second preset strength threshold is 0.5.

[0126] In another example, the positioning system 10 can also use a graph sparsity optimization algorithm based on spectral theory to reduce the number of edges while ensuring the connectivity and spectral properties of the node fusion graph.

[0127] It should be noted that the embodiments of the present application can also use other ways to perform sparse processing on the fusion graph, and the embodiments of the present application are not specifically limited.

[0128] Further, the positioning system 10 can determine the fault root cause based on the call relationship between the microservices and the causal relationship between the monitoring index data. In addition, the timing characteristics of the microservice system also affect the determination of the fault root cause. Among them, the timing characteristics of the microservice system are used to reflect the change characteristics of the microservice system at different time scales, such as short-term change characteristics or long-term gradual change characteristics of the microservice system.

[0129] In the embodiments of the present application, the timing characteristics of each microservice, also known as multi-scale timing characteristics, at least include at least two of short-term characteristics, medium-term characteristics and long-term characteristics.

[0130] The time scale of the short-term feature is smaller than the time scale of the medium-term feature, and the time scale of the medium-term feature is smaller than the time scale of the long-term feature. For example, the time scale of the short-term feature is 1 minute, the time scale of the medium-term feature is 10 minutes, and the time scale of the long-term feature is 60 minutes.

[0131] It can be understood that the short-term feature can reflect the instantaneous change of the microservice in the short term, and the short-term feature is very important for capturing transient faults such as sudden errors and service crashes, and can help to quickly respond to emergencies. The medium-term feature can focus on the trend change of the microservice in a medium time period, and the medium-term feature can reflect the gradual change of the microservice. The long-term feature focuses on the long-term and periodic change of the microservice.

[0132] In the embodiments of the present application, the time series feature of each microservice is specifically the change feature of the monitoring indicator data of the microservice at different time scales. The microservice A includes monitoring indicator data C1, monitoring indicator data C2, and monitoring indicator data C3, and the time series feature of the microservice A includes the medium-term feature, the long-term feature, and the short-term feature of the monitoring indicator data C1, the medium-term feature, the long-term feature, and the short-term feature of the monitoring indicator data C2, and the medium-term feature, the long-term feature, and the short-term feature of the monitoring indicator data C3.

[0133] Further, the positioning system 10 can adaptively extract the time series features of different monitoring indicator data of each microservice to improve the identification ability of the microservice to faults. Specifically, the positioning system 10 can dynamically adjust the feature extraction strategy based on the features of different monitoring indicators.

[0134] Since different monitoring indicator data have different characteristics and abnormal patterns, for example, when the microservice system fails, the response time of the call may appear a sharp abnormality, while the resource usage rate may exhibit a slowly growing characteristic that needs to be controlled for a long time. In order to improve the identification ability to faults, the positioning system 10 can assign different weight values to different time series features based on the characteristics and abnormal patterns of different monitoring indicator data.

[0135] For example, for the indicator of the response time that appears a sharp abnormality, the positioning system can collect the short-term feature of the monitoring indicator with a high collection weight. For the resource usage rate that needs to be controlled for a long time, the medium-term feature and the long-term feature can be collected with a high collection weight to slowly identify the process of exceeding the threshold. In the above manner, the positioning system 10 can make the monitoring indicator data more focused on the period of abnormality occurrence, thereby enhancing the sensitivity to faults.

[0136] It should be noted that the high collection weight refers to a weight that is greater than a preset weight threshold.

[0137] In an example, the positioning system 10 can determine the root cause of the fault based on the node fusion graph and the time series features of each microservice. That is, by comprehensively considering the time series features of each microservice and the calling relationship and causal relationship in the node fusion graph, the probability of the fault in each microservice and the fault propagation path are comprehensively controlled, so as to accurately locate the root cause of the fault.

[0138] In a specific implementation, the positioning system 10 first determines the fault propagation probability matrix based on the node fusion graph and the time series features of each microservice. The fault propagation probability matrix includes the probability of the fault propagation between the plurality of microservices.

[0139] For example, the positioning system 10 can determine the rules of the propagation fault probability between any two microservices in the plurality of microservices based on historical data. For example, rule 1: if the short-term resource usage (such as CPU, memory) of a microservice is abnormally high, the probability of the fault propagation to the downstream microservices (other microservices calling the microservice) increases. Rule 2: if the medium-term response time of a microservice shows an upward trend, the probability of the fault propagation of other services calling the service increases. Rule 3: if the long-term error rate of a microservice is higher than the historical baseline, the probability of the microservice becoming a fault source increases. The positioning system 10 can calculate the propagation probability of the fault between the plurality of microservices based on the rules of the fault probability.

[0140] For example, the positioning system 10 first specifies any fault source as an initial fault source, and assigns a fault probability value of m% to the initial fault source, where m is a number greater than 0 and less than 100. The positioning system 10 checks the node fusion graph to find all microservices (such as order service, refund service) directly connected to the payment service. Then, using rules 1 and 2, the probability of the fault propagation from the payment service to these directly calling services is calculated in combination with the multi-scale time series features of these microservices. For example, if the short-term CPU usage of the order service is also abnormally high, the probability of the fault propagation from the payment service to the order service will increase accordingly.

[0141] Then, the positioning system 10 iteratively applies rules 1 and 2 to calculate the propagation probability of the indirect calling relationship (such as the payment service to the user service, and then from the user service to the commodity service). With each layer of indirect calling, the fault propagation probability is attenuated or enhanced according to the time series features and the probability of the previous level, until the entire node fusion graph is evaluated.

[0142] Finally, the positioning system 10 applies rule 3 and the long-term time series features to adjust the calculated propagation probability, especially for those services that have shown abnormalities in the long term. For example, if the commodity service has a continuously high error rate higher than the normal level in the past 60 minutes, the propagation path probability of all paths pointing to or passing through the commodity service will increase accordingly. Thus, the fault propagation probability of each microservice is obtained.

[0143] Then, the positioning system 10 can determine the fault propagation path based on the fault propagation probability. The fault propagation path is used to indicate a specific link through which the fault propagates among the plurality of microservices.

[0144] Exemplarily, the positioning system 10 can determine the fault propagation path in an iterative manner. Specifically, the implementation includes marking the service node corresponding to the initial fault source (referred to as a marked node), and selecting a service node corresponding to a microservice having a calling relationship with the initial fault source (referred to as a current service node) from all unanalyzed microservices for analysis. The positioning system 10 determines whether the current service node is a service node on the fault propagation path according to the fault propagation probability of the current service node and the marked node. For example, when the fault propagation probability is greater than a preset probability threshold, the current service node is determined as a key node on the fault propagation path. In this way, the positioning system 10 can determine the fault propagation path. For example, if the propagation probability from the payment service to the order service is 0.8, the propagation probability from the order service to the user service is 0.7, and the preset probability threshold is 0.5, the positioning system 10 can obtain a fault propagation path of “payment service-order service-user service”.

[0145] Finally, the positioning system 10 determines the fault root cause of the microservice system based on the fault propagation path. For example, the positioning system 10 can calculate the probability of each service node as a fault root cause based on the fault propagation path. The root cause probability = the fault probability of the node itself x the fault propagation probability from the target node to the node. The fault probability of the node itself is determined based on the short-term characteristics of the service node. The positioning system 10 can determine the microservice corresponding to the service node with the highest root cause probability as the fault root cause. For another example, the positioning system 10 can display the fault propagation path, and an operation and maintenance personnel can determine the fault root cause of the microservice system based on experience or step-by-step troubleshooting. It should be noted that the embodiments of the present application can also determine the fault root cause in other ways, and the embodiments of the present application are not specifically limited.

[0146] In summary, the embodiments of the present application provide a method for determining a fault root cause, which can accurately determine the root cause of a fault in a microservice system by obtaining the calling relationship among microservices in the microservice system and the causal relationship between the monitoring index data of each microservice. That is, the embodiments of the present application can locate the calling link that has a fault under the premise that multiple microservices simultaneously alarm, and accurately locate the fault root cause from the located calling link by combining the causal relationship between the monitoring index data.

[0147] Further, the embodiment of the present application can also utilize a Causal-Enhanced Graph Attention Network (CEGAT) algorithm to process the node fusion graph and the multi-scale time sequence features of each microservice to determine the fault root cause. Specifically, first, the node fusion graph and the multi-scale time sequence features of each microservice are fused to obtain graph time sequence fusion features. Then, the CEGAT algorithm is used to process the graph time sequence fusion features to determine the fault root cause.

[0148] In the formula, the connection information of each service node in the node fusion graph and the multi-scale time sequence features are spliced together by the positioning system 10 to obtain the graph time sequence fusion features. The graph time sequence fusion features include the connection information of the nodes (service nodes or monitoring nodes) and the multi-scale time sequence features.

[0149] It should be noted that the CEGAT algorithm provided by the embodiment of the present application is a trained CEGAT algorithm. Specifically, the CEGAT algorithm is trained using training sample data, wherein the training sample data includes historical graph time sequence fusion features and labeled or semi-labeled historical fault data.

[0150] Further, in the embodiment of the present application, the CEGAT algorithm can be trained once every preset time interval. Specifically, the model training layer B03 can obtain the stored graph time sequence fusion features from the preset database every preset time interval, wherein the preset database is used to store the graph time sequence fusion features within the preset time interval from the current time point. It can be understood that the training sample data obtained in this way can improve the model's ability to distinguish different fault modes and adapt to the few-sample scenario.

[0151] The CEGAT algorithm will be described below.

[0152] In the embodiment of the present application, the CEGAT algorithm determines the fault root cause of the microservice system by fusing the causal reasoning mechanism and the graph neural network. Specifically, the CEGAT model is used to obtain graph time sequence fusion features (also referred to as initial features), and based on the attention mechanism, the graph time sequence fusion features are processed to obtain target features containing more information. Then, the target features are modeled for anomaly propagation to obtain a fault propagation path and a fault propagation probability matrix. Next, the fault propagation path and the fault propagation probability matrix are processed to determine the posterior probability that each service or each component becomes a root cause. The core of the CEGAT algorithm is the CEGAT model.

[0153] The CEGAT model includes three layers, namely an input layer, an attention layer, and an output layer.

[0154] The input layer is configured to receive initial features of nodes, and send the received initial features to the attention layer. For example, in the embodiments of the present application, the input layer can receive initial features of each service node and neighbor nodes in the graph temporal fusion features, and process the initial features to obtain intermediate features corresponding to the initial features.

[0155] For example, the service node is an "order service node", and the neighbor nodes corresponding to the order service node include a "payment service node", a "user service node", a "monitoring node of CPU usage of the order service", and a "monitoring node of memory indicators of the order service".

[0156] The initial features of the nodes specifically refer to connection information and temporal features of the nodes in the graph temporal fusion features. For example, the node is an order service node, and the initial features of the node include, but are not limited to, connection information such as call response time, call error rate, and CPU usage rate of the order service, and multi-scale temporal features.

[0157] The attention layer processes the initial features through an attention mechanism to obtain intermediate features corresponding to the initial features. The intermediate features are used to determine information focused by the initial features during feature fusion.

[0158] The attention mechanism is described below.

[0159] Firstly, the initial features of each node are transformed into three features of query, key, and value: specifically, the attention mechanism converts the initial features of each node into a single different vector through learned parameters (also referred to as a weight matrix), specifically query (Q), key (K), and value (V).

[0160] Then, the basic attention score is obtained through feature splicing: the basic attention score of a node refers to the basic correlation score between the node (as a query) and its neighbor nodes (as keys). In the embodiments of the present application, the basic attention score can be determined by calculating vector operations, such as dot product operations, between the node and the neighbor nodes.

[0161] Further, the causal strength parameter between the nodes is introduced as an adjustment factor into the attention calculation. For example, the attention score (or attention weight) can be determined by the product of the basic attention score and the causal strength parameter. It can be understood that the attention score considers the causal relationship between the causes and effects, and considers the calling relationship between multiple microservices.

[0162] Next, for ease of processing, the attention weights can be normalized to obtain normalized attention weights. In one example, the CEGAT model can use a softmax function to convert the attention weights of all neighbor nodes into probability scores to ensure that the sum of all attention weights is equal to 1.

[0163] Finally, the CEGAT model uses the normalized attention weights to perform a weighted summation of the features of all neighboring nodes of a node to obtain the intermediate features of the node. The intermediate features of the node incorporate the information of the neighboring nodes.

[0164] In an embodiment of the present application, because multiple data are involved, for example, the response time of the order service being called by other microservices, the error rate of calling other microservices, the causal relationship between its own CPU usage and other indicator nodes, etc., in order to enable the CEGAT model to accurately process multiple types of data, the CEGAT model can adopt a multi-head attention mechanism. Exemplarily, an 8-head attention mechanism is adopted, that is, the attention layer includes head 1 to head 8. Among them, each attention head independently learns different feature representations. In other words, each head attention outputs an intermediate feature, which focuses on different neighboring nodes, different indicator types and different causal relationships, and independently calculates the attention weight based on the intermediate feature to determine the neighboring nodes or indicators that the node pays more attention to when performing information aggregation in the output layer.

[0165] For example, the first step is to learn to determine the response time relationship between "order analysis" and the downstream microservice (payment service) it directly calls; the first step is to learn to determine the error rate relationship between "order service" and the upstream microservice (shopping cart service) it directly calls; the first step is to learn to determine the causal relationship between the CPU usage of "order service" and its adjacent indicator node (memory usage); the first step is to learn to determine the indicator change trend of medium-term characteristics and / or long-term characteristics; the first step is to learn to determine the sudden anomaly of short-term characteristics, etc.

[0166] The attention layer is used to input intermediate features into the output layer. Correspondingly, the output layer integrates the intermediate features of the microservices, for example, by concatenating or averaging, to generate the target features corresponding to the microservices. It can be understood that the target features include information such as the call relationship of the node in the graph, the causal relationship parameters with the neighboring nodes, and the changes in multi-scale time series features relative to the initial features. Among them, the causal relationship parameters between the node and the neighboring nodes are used to describe the causal relationship parameters between the node and the neighboring nodes. Therefore, using this target feature, the fault propagation probability and fault propagation path that reflect the actual fault propagation can be accurately determined, thereby accurately locating the root cause of the fault.

[0167] The output layer is also used for abnormal propagation modeling calculation. Specifically, based on the target features, a fault propagation probability, i.e., a probability of fault propagation from one node to another node, is determined. The following is analyzed in detail:

[0168] Step 1: Obtain an adjacency matrix and a causal relationship matrix.

[0169] The adjacency matrix is used to represent a matrix of connection relationships between nodes, i.e., whether two nodes are connected. For example, an adjacency matrix A, where an element Aij represents that a node i is connected to a node j. It should be noted that i and j are integers.

[0170] The causal relationship matrix is a matrix composed of causal strength parameters between nodes. For example, a causal relationship matrix C, where an element Cxy represents a causal strength parameter of a node x to a node y.

[0171] Step 2: Calculate the similarity between nodes.

[0172] For example, the cosine similarity can be used to calculate the similarity between two nodes in terms of features. It can be understood that the higher the similarity between two nodes, the more likely they are to influence each other.

[0173] Step 3: Calculate the fault propagation probability.

[0174] The similarity between nodes and the causal strength parameters between them are combined to determine the fault propagation probability of the node. Wherein, the higher the similarity between two nodes and the higher the causal strength parameter, the higher the probability of fault propagation between the two nodes, i.e., the higher the fault propagation probability of the two nodes.

[0175] Step 4: Normalize the fault propagation probability of each node, and use the normalized propagation probability matrix as the fault propagation probability matrix.

[0176] Wherein, after normalization, the sum of the fault propagation probabilities of each node is 1.

[0177] Then, the output layer determines the fault propagation path based on the fault propagation probability matrix. The following is described in detail:

[0178] Step 5: Use the fault propagation probability matrix to construct a directed graph, which is also called a node propagation probability graph.

[0179] Wherein, the nodes in the node propagation probability graph can be service nodes or intermediate components, etc., which are not specifically limited by the embodiments of the present application. The edges in the node propagation probability graph are propagation probability edges, which represent the fault propagation probability of fault propagation from a first node to a second node. The starting point of the propagation probability edge is the first node, and the end point is the second node.

[0180] Step ⑥: Based on the node propagation probability graph, determine the path probability of each non-target node to the target node.

[0181] It should be noted that the target node can be an abnormal node in the microservice system that has been monitored, such as a service node corresponding to a microservice with a response time process. The positioning system 10 reversely traces and calculates the path probability of each non-target node in the microservice system to reach the abnormal node along the node propagation probability graph. Among them, a path probability is usually the product of the probabilities of all edges on the path or the sum of the probabilities of all edges.

[0182] It should be noted that the non-target node includes other nodes in the node probability graph except the target node, which can be a monitoring node or a service node.

[0183] Exemplarily, if the node propagation probability graph includes the following service nodes, respectively, the user service node, the commodity service node, the order service node, and the payment service node. The positioning system 10 detects that the order service response time abnormally increases. The positioning system 10 marks the order service node as the target node, and if the propagation probability matrix determined by the positioning system 10 includes: the fault propagation probability from the user service node to the order service node is 0.2, the fault propagation probability from the commodity service node to the order service node is 0.15, the fault propagation probability from the payment service node to the order service node is 0.3, and the fault propagation probability from the commodity service node to the payment service node is 0.4. The positioning system 10 determines the path probability of other nodes to reach the order service node. Specifically, the path probability of the user service node directly to the order service node is 0.2, the path probability of the commodity service node directly to the order service node is 0.15, and the path probability of the payment service node directly to the order service node is 0.3. In addition, the embodiment of the present application also includes the path probability of the commodity service node affecting the order service node through the payment service node: the probability of the fault from the commodity service node to the payment service node * the probability of the payment service node to the order service node. That is, the path probability of the commodity service node affecting the order service node through the payment service node is 0.12.

[0184] Step ⑦: Synthesize all path probabilities of each non-target node to obtain the posterior probability of the non-target node as the root cause of the fault.

[0185] Considering that a node can be the source of multiple target nodes, or multiple paths can propagate to the target node, the positioning system 10 considers the probability of all possible paths of the node to all target nodes for each non-target node, and calculates the total root cause score (i.e., the posterior probability of each node as the root cause of the fault). The higher the posterior probability, the higher the probability of the node becoming the root cause of the fault.

[0186] Continuing to refer to the embodiment shown in step ⑥, the posterior probability of the user service node is 0.2, the posterior probability of the commodity service node is 0.15+0.12=0.27, and the posterior probability of the payment service is 0.3.

[0187] It should be noted that the posterior probability of each node can be obtained in the above manner, and can also be obtained by using other manners, for example, using the Bayesian inference method to calculate the posterior probability of each micro service or intermediate component becoming a root cause.

[0188] Step ⑧: determining the fault root cause based on the posterior probability of each non-fault point of the fault root cause.

[0189] For example, the positioning system can take the non-fault point with the highest posterior probability as the fault root cause.

[0190] In an example, the positioning system 10 can display the top K nodes with the highest posterior probability scores in the visualization layer, or display the top K nodes with posterior probability greater than a preset probability threshold. Wherein, K is a positive integer. Further, it can be displayed in order. In a specific implementation, the fault root cause can be highlighted in the call relationship graph or the node fusion graph.

[0191] In the embodiment of the application, the visualization layer can also display the fault propagation path, and the operator can understand the process of fault occurrence and diffusion based on the fault propagation path. In another specific implementation, an interactive node can be provided to allow the user to graph the structure, understand the detailed indicators and log information of the micro service node, etc. The visualization layer can also display other content, which is not specifically limited in the embodiment of the application.

[0192] It should be noted that, to avoid overfitting, the CEGAT algorithm can use a dropout mechanism (default 0.1) to prevent overfitting.

[0193] The above introduces a method for determining a fault root cause based on a CEGAT algorithm, and the following introduces a specific implementation of a method for determining a fault root cause. The method is applied to an e-commerce micro service system, and the micro service system includes user services, commodity services, order services, and shopping cart services, and multiple micro services have a mutual dependency relationship. The method is applied to Figure 1 The positioning system 10 shown in the figure includes a data acquisition layer B01, a graph construction layer 101, a feature extraction layer B02, a model training layer B03, a root cause inference layer 102, and a visualization layer B04.

[0194] The Figure 6 Another method for determining a fault root cause is provided in the embodiment of the application, and the method includes the following contents:

[0195] S610, the data collection layer B01 collects the calling relationship data and the monitoring index data of each microservice.

[0196] The monitoring index data of each microservice refers to the monitoring index data of each microservice in the microservice system, including the calling response time of the microservice, the calling error rate, the resource usage rate of the microservice itself, and the middleware index involved.

[0197] In the embodiment of the application, the calling response time can be represented by using percentiles, for example, P95 and P99 are used to measure the response time of most requests. Among them, P95 delay represents that 95% of the request response time does not exceed this value. If the P95 delay is 500ms, it means that 95% of the requests can be completed within 500ms. P99 delay represents that 99% of the request response time does not exceed this value, which provides more stringent performance guarantee.

[0198] The calling error rate, for example, the proportion of 5xx errors, reflects the failure of the microservice when processing requests. 5xx errors usually refer to server errors, such as 500 Internal Server Error, 502 Bad Gateway, 504 Gateway Timeout, etc. High 5xx error rate may indicate that there is a problem inside the microservice, such as code error, resource exhaustion or dependent service failure.

[0199] The resource usage rate includes but is not limited to CPU usage rate, memory occupancy rate and disk I / O usage rate.

[0200] The middleware index includes but is not limited to connection pool usage rate and cache hit rate. Among them, the connection pool usage rate refers to the usage of the connection pool directly connected to the database, message queue and other external nodes.

[0201] Exemplarily, the data collection layer B01 can adopt a non-invasive design, automatically collect the calling relationship data between microservices by using OpenTelemetry, and collect the monitoring index data of each microservice by using Prometheus.

[0202] S620, the data collection layer B01 sends the collected calling relationship data and the monitoring index data of each microservice to the graph construction layer 101.

[0203] Correspondingly, the graph construction layer B101 receives the calling relationship data and the monitoring index data of each microservice, constructs a calling relationship graph by using the calling relationship data, and constructs a causal relationship graph by using the monitoring index data of each microservice.

[0204] S630, the graph construction layer 101 constructs a calling relationship graph and a index causal graph.

[0205] S640, the graph construction layer 101 fuses the index causal graph and the calling relationship graph to obtain a node fusion graph.

[0206] The specific implementation of steps S630-S640 is described above and will not be repeated here.

[0207] S650, the graph construction layer 101 sends the node fusion graph to the feature extraction layer B02.

[0208] Correspondingly, the feature extraction layer B02 receives the node fusion graph and extracts features of the received node fusion graph.

[0209] S660, the feature extraction layer B02 extracts graph timing fusion features and sends the graph timing fusion features to the model training layer B03 and the root cause reasoning layer 102.

[0210] Specifically, the feature extraction layer B02 processes the node fusion graph and the timing features of each node to obtain graph timing fusion features. Then, the feature extraction layer B02 sends the graph timing fusion features to the model training layer B03 and the root cause reasoning layer 102.

[0211] In the embodiments of the present application, the timing features of each node include 1-minute features, 10-minute features, and 60-minute features.

[0212] Correspondingly, the model training layer B03 can be used to train the CEGAT algorithm using the graph timing fusion features. The root cause reasoning layer 102 can process the current graph timing fusion features based on the trained CEGAT algorithm to obtain a fault propagation path.

[0213] S670, the root cause reasoning layer 102 processes the graph timing fusion features using the trained CEGAT model to obtain a fault propagation path.

[0214] S680, the root cause reasoning layer 102 determines a fault root cause based on the fault propagation path.

[0215] In the embodiments of the present application, the root cause reasoning layer 102 finally determines that the database connection pool configuration of the order service (specifically, the maximum connection number is set too small) is the fault root cause.

[0216] This fault root cause causes the order service to respond slowly, the commodity service inventory lock to timeout, and the user service to fail to query the order status, etc.

[0217] Further, the positioning system 10 can display the fault propagation path on the visualization layer to help the operation and maintenance personnel understand the fault propagation process.

[0218] The implementation environment composed of 50 microservices is tested by the above example, and 10 different types of configurations are injected, including but not limited to configuration errors, resource exhaustion, and dependent service exceptions. Figure 6 Compared with the fault root cause determined by the prior art, the accuracy of the fault root cause determined in the manner shown in the above example is greatly improved, the average diagnosis time is significantly reduced, and the false positive rate is greatly reduced.

[0219] In summary, the fault root cause determination method provided by the embodiments of the present application constructs a unified graph model that integrates service dependency relationships and index causal relationships; applies adaptive time sequence feature extraction technology to process node data; and uses a causal augmented graph attention network to model abnormal propagation and root cause reasoning on the unified graph model, thereby achieving accurate positioning of fault root causes in complex microservice architectures.

[0220] In addition, the embodiments of the present application also provide a fault root cause determination device.

[0221] The structure of the fault root cause determination device provided by the embodiments of the present application is shown in the accompanying drawings. Figure 7 The fault root cause determination device provided by the embodiments of the present application is applied to a microservice system, and the microservice system includes a plurality of microservices. The device 700 includes:

[0222] An acquisition unit 701 is configured to acquire a call relationship graph and an index causal graph.

[0223] The call relationship graph is used to represent the call relationship between the plurality of microservices. The index causal graph is used to represent the causal relationship between a plurality of monitoring index data, and the plurality of monitoring index data includes monitoring index data of each microservice in the plurality of microservices. The monitoring index data of a first microservice is used to monitor the performance of the first microservice, and the first microservice is any microservice in the plurality of microservices.

[0224] A determination unit 702 is configured to determine the fault root cause of the microservice system according to the call relationship graph and the index causal graph.

[0225] Optionally, the call relationship graph includes service nodes and call relationship edges. One service node is used to represent one microservice. The call relationship edge is used to represent a call link from a calling microservice to a called microservice. The starting point of the call relationship edge is the service node corresponding to the calling microservice, and the terminal point of the call relationship edge is the service node corresponding to the called microservice.

[0226] Optionally, the indicator causal diagram comprises monitoring nodes and causal relationship edges, one monitoring node corresponding to one monitoring indicator data; the causal relationship edge is used to represent a causal relationship chain in which first monitoring indicator data causes second monitoring indicator data to be generated, a starting point of the causal relationship edge being a monitoring node corresponding to the first monitoring indicator data, and a terminal point of the causal relationship edge being a monitoring node corresponding to the second monitoring indicator data.

[0227] Optionally, the determining unit 702 is specifically configured to:

[0228] fuse the calling relationship diagram and the indicator causal diagram to obtain a node fusion diagram;

[0229] The node fusion diagram is used to indicate the calling relationship among the plurality of microservices, the causal relationship among the plurality of monitoring indicator data, and the mapping relationship between the plurality of microservices and the plurality of monitoring indicator data.

[0230] According to the node fusion diagram, determine the fault root cause of the microservice system.

[0231] Optionally, the determining unit 702 is further configured to:

[0232] According to the node fusion diagram and the time sequence characteristics of each microservice in the plurality of microservices, determine the fault root cause of the microservice system.

[0233] The time sequence characteristics of the first microservice indicate data characteristics of monitoring indicator data of the first microservice changing over time, and the time sequence characteristics of the first microservice include at least two of short-term characteristics, medium-term characteristics and long-term characteristics of the first microservice, the time scale corresponding to the short-term characteristics being smaller than the time scale corresponding to the medium-term characteristics, and the time scale corresponding to the medium-term characteristics being smaller than the time scale corresponding to the long-term characteristics.

[0234] Optionally, the determining unit 702 is further configured to:

[0235] According to the node fusion diagram and the time sequence characteristics of each microservice, determine a fault propagation path and obtain a fault propagation probability matrix of the fault propagating among the plurality of microservices;

[0236] The fault propagation path includes propagation of the fault among microservices, propagation of the fault among monitoring indicator data, and propagation of the fault between microservices and monitoring indicator data.

[0237] Based on the fault propagation path and the fault propagation probability matrix, determine the fault root cause of the microservice system.

[0238] Optionally, the determining unit 702 is further configured to:

[0239] Based on the fault propagation path and the fault propagation probability matrix, determining a path probability from each node in the fault propagation path to a target node; the target node is a node corresponding to a microservice in the microservice system that has been determined to have a fault;

[0240] For each node in the fault propagation path: determining a posterior probability of the node based on the path probabilities from the node to all target nodes;

[0241] The root cause of the fault is determined based on the posterior probability of each microservice node.

[0242] Optionally, the determining unit 702 is further configured to:

[0243] Using a preset network model, the node fusion graph and the timing characteristics of each microservice are processed to determine the root cause of the failure of the microservice system;

[0244] The preset network model is a network model that is processed based on a causal enhanced attention mechanism, and the causal enhanced attention mechanism is a mechanism that adjusts the attention weight by using causal relationship parameters as adjustment factors of the attention weight.

[0245] Optionally, obtaining the call relationship graph and the indicator cause-effect graph includes:

[0246] Based on the sliding time window, the call relationship graph and the metric causal graph are dynamically updated. The following operations are performed within a sliding time window:

[0247] Obtaining call relationship data between the multiple microservices and monitoring indicator data of each microservice;

[0248] The call relationship data is used to describe the call relationship between the multiple microservices;

[0249] The call relationship graph is obtained according to the call relationship data; and an indicator cause-effect graph is obtained based on the monitoring indicator data of each microservice.

[0250] The present application provides a device for determining the root cause of a fault. By obtaining the call relationships between microservices in a microservice system and the causal relationship between the monitoring indicator data of each microservice, the device can accurately locate the root cause of the fault in the microservice system. In other words, the present application can locate the faulty call link based on the call relationships between microservices when multiple microservices are simultaneously alerted. In combination with the causal relationship between the monitoring indicator data, the device can accurately locate the root cause of the fault from the located call link.

[0251] According to the method provided in the embodiments of the present application, the present application further provides a chip system, which comprises one or more processors, and is configured to call and run instructions stored in a memory, so that the method provided in the embodiments of the present application is executed. The chip system can be composed of a chip, or can comprise a chip and other discrete devices.

[0252] The chip system can comprise an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0253] According to the method provided in the embodiments of the present application, the present application further provides a computer program product, which comprises computer program codes, and when the computer program codes are run on a computer, the computer is caused to execute each step or flow of the network device and the terminal device in any of the method embodiments.

[0254] According to the method provided in the embodiments of the present application, the present application further provides a computer readable storage medium, which stores program codes, and when the program codes are run on a computer, the computer is caused to execute each step or flow of the network device and the terminal device in any of the method embodiments.

[0255] The computer readable storage medium can be the volatile memory or the non-volatile memory, or can comprise the volatile memory and the non-volatile memory.

[0256] In the embodiments of the present application, each term and English abbreviation is an exemplary example given for convenience of description, and should not constitute any limitation on the present application. The present application does not exclude the possibility of defining other terms capable of achieving the same or similar functions in the existing or future protocols.

[0257] In the above embodiments, all or part of the embodiments can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the embodiments can be realized in the form of a computer program product. The computer program product comprises one or more computer instructions. When the computer instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated.

[0258] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other manners. For example, the described device embodiments are merely schematic. For example, the division of the units is only a logical function division. For example, there can be another division manner for the actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms.

Claims

1. A method for determining the root cause of a fault, characterized in that: Applied to a microservice system, the microservice system includes multiple microservices, and the method includes: Obtain call relationship graph and indicator cause-effect graph; The call relationship graph is used to represent the call relationship between the multiple microservices; the indicator causal graph is used to represent the causal relationship between multiple monitoring indicator data, the multiple monitoring indicator data include monitoring indicator data of each microservice in the multiple microservices, and the monitoring indicator data of the first microservice is used to monitor the performance of the first microservice, and the first microservice is any microservice in the multiple microservices; Determine the root cause of the failure of the microservice system based on the call relationship graph and the indicator cause-effect graph.

2. The determination method according to claim 1, characterized in that The call relationship graph includes service nodes and call relationship edges, where one service node is used to represent one microservice; the call relationship edge is used to represent the call link from the calling microservice to the called microservice, the starting point of the call relationship edge is the service node corresponding to the calling microservice, and the end point of the call relationship edge is the service node corresponding to the called microservice.

3. The determination method according to claim 1, characterized in that The indicator causal graph includes monitoring nodes and causal edges, where one monitoring node corresponds to one monitoring indicator data; the causal edge is used to represent the causal chain in which the first monitoring indicator data causes the second monitoring indicator data to be generated, the starting point of the causal edge is the monitoring node corresponding to the first monitoring indicator data, and the end point of the causal edge is the monitoring node corresponding to the second monitoring indicator data.

4. The determination method according to any one of claims 1 to 3, characterized in that: Determining the root cause of the failure of the microservice system according to the call relationship graph and the indicator cause-effect graph includes: Fusing the call relationship graph and the indicator causal graph to obtain a node fusion graph; The node fusion graph is used to indicate the calling relationship between the multiple microservices, the causal relationship between the multiple monitoring indicator data, and the mapping relationship between the multiple microservices and the multiple monitoring indicator data; Determine the root cause of the failure of the microservice system according to the node fusion graph.

5. The determination method according to claim 4, characterized in that: Determining the root cause of the failure of the microservice system according to the node fusion graph includes: Determining a root cause of a failure of the microservice system according to the node fusion graph and a timing characteristic of each microservice in the plurality of microservices; The timing feature of the first microservice indicates a data feature of monitoring indicator data of the first microservice that changes over time. The timing feature of the first microservice includes at least two of a short-term feature, a medium-term feature, and a long-term feature of the first microservice. The time scale corresponding to the short-term feature is smaller than the time scale corresponding to the medium-term feature, and the time scale corresponding to the medium-term feature is smaller than the time scale corresponding to the long-term feature.

6. The determination method according to claim 5, characterized in that: Determining the root cause of the failure of the microservice system according to the node fusion graph and the timing characteristics of each microservice in the multiple microservices includes: Determine a fault propagation path according to the node fusion graph and the timing characteristics of each microservice, and obtain a fault propagation probability matrix of the fault propagating between the multiple microservices; The fault propagation path includes the propagation of the fault between microservices, the propagation between monitoring indicator data, and the propagation between microservices and monitoring indicator data; Based on the fault propagation path and the fault propagation probability matrix, a root cause of the fault of the microservice system is determined.

7. The determination method according to claim 6, characterized in that: The determining the root cause of the fault of the microservice system based on the fault propagation path and the fault propagation probability matrix includes: Based on the fault propagation path and the fault propagation probability matrix, determining a path probability from each node in the fault propagation path to a target node; the target node is a node corresponding to a microservice in the microservice system that has been determined to have a fault; For each non-target node in the fault propagation path: determining a posterior probability of the non-target node based on the path probabilities from the non-target node to all target nodes; The root cause of the fault is determined based on the posterior probability of each non-target node.

8. The determination method according to claim 6, characterized in that: Determining the root cause of the failure of the microservice system according to the node fusion graph and the timing characteristics of each microservice in the multiple microservices includes: Using a preset network model, the node fusion graph and the timing characteristics of each microservice are processed to determine the root cause of the failure of the microservice system; The preset network model is a network model that is processed based on a causal enhanced attention mechanism, and the causal enhanced attention mechanism is a mechanism that adjusts the attention weight by using causal relationship parameters as adjustment factors of the attention weight.

9. The determination method according to any one of claims 1 to 8, characterized in that: The obtaining of the call relationship graph and the indicator causal graph includes: Based on a sliding time window, the call relationship graph and the indicator causal graph are dynamically obtained; wherein the following operations are performed within a sliding time window: Obtaining call relationship data between the multiple microservices, and obtaining monitoring indicator data of each microservice; wherein the call relationship data is used to describe the call relationship between the multiple microservices; The call relationship graph is obtained according to the call relationship data, and the indicator causal graph is obtained according to the monitoring indicator data of each microservice.

10. A device for determining the root cause of a fault, characterized in that: Applied to a microservice system, the microservice system includes multiple microservices, and the device includes: Acquisition unit, used to obtain call relationship graph and indicator causal graph; The call relationship graph is used to represent the call relationship between the multiple microservices; the indicator causal graph is used to represent the causal relationship between multiple monitoring indicator data, the multiple monitoring indicator data include monitoring indicator data of each microservice in the multiple microservices, and the monitoring indicator data of the first microservice is used to monitor the performance of the first microservice, and the first microservice is any microservice in the multiple microservices; A determination unit is used to determine the root cause of the failure of the microservice system based on the call relationship graph and the indicator cause-effect graph.

Images