An actionable and explainable root cause location method for recurring fault types

By adopting a root cause positioning method based on decision trees and fault dependency graphs in the online service system, the problem of insufficient operability and interpretability of repeated type fault positioning in the prior art is solved, and the operation and maintenance personnel can directly from The repair operation is derived from the positioning results and the automated fault location of the algorithm logic is understood.

CN114661504BActive Publication Date: 2024-11-19TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210168856.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-23
Publication Date
2024-11-19
Estimated Expiration
2042-02-23

AI Technical Summary

Technical Problem

The prior art has insufficient operability and interpretability of the repetitive type failure in the online service system, which makes it difficult for operation and maintenance personnel to obtain repair operations directly from the positioning results, and lacks understanding and trust in algorithm logic. .

Method used

Using fault monitoring and trained root cause positioning model based on monitoring system, global interpretation is performed through decision trees, and local interpretation is performed using fault dependency graphs. Combining feature extractors, feature aggregators and classifiers, we design operable and capable The explained root cause positioning method realizes detailed positioning and repair operation guidance for fault instances.

Benefits of technology

The operational and explainable root cause positioning is achieved. Operations and maintenance personnel can directly obtain repair operations from the positioning results and understand the positioning results of the automation method, which improves the accuracy and trustworthiness of fault positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114661504B_ABST
    Figure CN114661504B_ABST
Patent Text Reader

Abstract

The present application discloses an operational and explainable root cause location method for recurring faults, wherein the method comprises: using a monitoring system to monitor faults, triggering a trained root cause location model based on the faults monitored by the monitoring system, wherein the trained root cause location model is globally explained through a trained decision tree; obtaining a root cause fault instance based on the indicator data and fault dependency graph corresponding to the fault instance when the fault occurs, and locally explaining the location result of the root cause fault instance; wherein the fault dependency graph is constructed based on the fault instance. The present application can realize operational and explainable root cause location, can directly obtain how to perform repair operations based on the location results, and can accurately locate the results.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the technical field of fault diagnosis and root cause location, and in particular to an operable and explainable root cause location method and device for repeatedly occurring type faults. Background Art

[0002] Online service systems, such as online banking and search engines, have become an indispensable part of our lives. Online services are composed of many components, such as load balancers, web servers, containers, databases, etc. These components work together and call each other to provide services, and there are complex dependencies between components. Due to the huge scale and high complexity of online service systems, failures in online service systems are inevitable, which can lead to huge economic losses and reduced user satisfaction. Therefore, in order to reduce the negative impact caused by failures, root cause location is critical.

[0003] In actual online service systems, most faults are of repeated types. For example, more than 94% of faults are caused by a small number of repeated root causes. Examples of such repeated faults include high service response time due to lack of indexes for corresponding fields in the database, reduced service success rate due to unavailability of third-party services, etc. Therefore, it is very important to locate repeated faults.

[0004] In order to diagnose faults and locate root causes of online service systems, various types of monitoring data are continuously collected. Among them, indicator data is the continuous monitoring of key performance indicators of each component, stored in the form of time series. Compared with log data, indicator data has a single type, simple analysis, small data volume, and is sufficient to reflect the abnormal status of components. Compared with call chain data, indicator data can reflect the underlying performance indicators (such as CPU, memory, etc.).

[0005] In existing practice, the root cause of repeated types of failures is located based on the perception accumulated by experienced operation and maintenance personnel from years of troubleshooting experience. Experienced operation and maintenance personnel can guess what kind of failure may be based on the symptoms of the online service system, such as the form of indicators, and make further confirmation based on this. Due to the complex dependencies between components, the root cause may cause problems in a series of components. Based on their experience, operation and maintenance personnel can analyze the relationship between such related anomalies to find the real root cause. On the other hand, after finding the root cause of the failure, operation and maintenance personnel can find references from the repair experience of other similar failures in history, find more accurate repair solutions faster, and perform repair operations.

[0006] At present, the problem of locating the root cause service of a fault is modeled as a multi-classification problem (each class is a possible root cause service) through machine learning methods. The data used is based on the call chain, the global configuration of the system, the service indicators, etc. A multi-classification model is trained based on historical fault data, in which the specific machine learning algorithm uses random forest, multi-layer perceptron or k-nearest neighbor, etc. Then when a fault occurs, the trained model is used to determine which service is more likely to be the root cause service. The granularity of the root cause located by MEPFL is the root cause service, and no explanation is given for the classification results.

[0007] The current method of indirectly locating the root cause by searching for similar historical faults only recommends similar historical faults, and the root cause of the similar historical faults obtained may be the root cause of the current fault. It models each fault as a fault graph, where the nodes of the fault graph are components in the system, the node features are various key indicators of the component, and the edges are the deployment relationships between components.

[0008] Existing methods are not operable. The granularity of existing root cause location methods is either too coarse (for example, service) or too fine (for example, a single indicator), which is not enough for operation and maintenance personnel to directly determine how to take repair actions from such location results. There is no explainability. Existing root cause location methods lack explainability and cannot allow operation and maintenance personnel to understand how the algorithm gives the root cause. The explainability here includes two aspects: local explanation, that is, explaining how the location result of a single fault is given; global explanation, that is, explaining the logic of the entire location method or model.

[0009] Application Contents

[0010] The present application aims to solve one of the technical problems in the related art at least to some extent.

[0011] To this end, the purpose of this application is to propose an operational and explainable root cause location method for recurring types of failures, which can achieve operational and explainable root cause location. The operability enables operation and maintenance personnel to directly obtain how to perform repair operations from the location results of this application, and the explainability enables operation and maintenance personnel to understand and trust the location results given by the automated method of this application.

[0012] Another object of the present application is to provide an operational and explainable root cause location device for recurring faults.

[0013] To achieve the above objectives, the present application proposes, on one hand, an operational and explainable root cause location method for recurring faults, comprising the following steps:

[0014] A monitoring system is used to monitor faults, and a trained root cause location model is triggered based on the faults monitored by the monitoring system; the trained root cause location model is globally explained through a trained decision tree; a root cause fault instance is obtained based on indicator data and a fault dependency graph corresponding to the fault instance when the fault occurs, and a local explanation is given to the location result of the root cause fault instance; the fault dependency graph is constructed based on the fault instance.

[0015] The method of the embodiment of the present application can achieve operational and explainable root cause positioning. The operability enables operation and maintenance personnel to directly obtain how to perform repair operations from the positioning results of the present application, and the explainability enables operation and maintenance personnel to understand and trust the positioning results given by the automated method of the present application.

[0016] In addition, the operable and explainable root cause location method for recurring type faults according to the above embodiment of the present application may also have the following additional technical features:

[0017] Furthermore, in one embodiment of the present application, a loss function preset by a stochastic gradient descent method is used, and the root cause location model is trained based on historical fault data to obtain the trained root cause location model.

[0018] Furthermore, in one embodiment of the present application, the root cause localization model includes three components: a feature extractor, a feature aggregator, and a classifier. Then, the feature extractor is used to represent each fault instance as a vector of a fixed length; wherein the vector is an instance-level feature; the feature aggregator is used to aggregate the instance-level features of the related fault instances of each fault instance based on the structure of the fault dependency graph to obtain an aggregated feature; the classifier is used to score based on the aggregated feature, so as to design the root cause localization model according to the three components.

[0019] Furthermore, in one embodiment of the present application, the loss function preset by the stochastic gradient descent method includes:

[0020] The output score s of the fault instance v of fault T is measured using binary cross entropy BCE T (v) and the true label r T (v) The gap between the gaps:

[0021] BCE(r T (v),s T (v)) = r T (v)·log(s T (v))+(1-r T (v))·log(1-s T (v))

[0022] The preset loss function is obtained by using additional weights to perform weighted average on the BCEs of different fault instances. The preset loss function is L s ,but:

[0023]

[0024] in, is the training set, T i represents a fault in the training set, N H is the size of the training set, V represents all fault instances,

[0025] Further, in an embodiment of the present application, the local interpretation method is used to explain the location result of the root cause fault instance, including: presetting two faults, a first fault and a second fault; wherein the first fault is a current fault to be explained, and the second fault is a historical fault; for each fault instance in the first fault, comparing its aggregated features with the distance of the aggregated features of each fault instance in the second fault that belongs to the same fault category, and taking the smallest one as the distance between the fault instance and the second fault; taking a weighted average of the distances from all fault instances in the first fault to the second fault, and explaining the location result of the root cause fault instance by comparing the distance formula between the two faults.

[0026] Further, in one embodiment of the present application, the first fault T is preset 1 and the second fault T 2 , the distance formula between the two faults is:

[0027]

[0028] Among them, N c (v) represents the set of all fault instances that belong to the same fault category as fault instance v. Indicates fault T 1 Aggregate features of fault instance v in, Indicates fault T 2 Aggregate features of fault instance v′ in ||·|| 1 represents the L1 norm.

[0029] Furthermore, in one embodiment of the present application, the trained root cause location model is globally interpreted through a trained decision tree, including: extracting a first time series feature from the indicator data using a plurality of time series curve feature extractions; selecting the first time series feature to obtain a second time series feature using feature extraction based on indicator reconstruction; training a decision tree based on the second time series feature to obtain the trained decision tree, and performing root cause judgment on the fault instance using the trained root cause location model according to the trained decision tree; wherein the rules in the trained decision tree serve as a global interpretation of the trained root cause location model.

[0030] Furthermore, in one embodiment of the present application, the feature extraction of the indicator reconstruction includes: using the feature extractor and feature aggregator of the trained root cause location model as the encoder part of the autoencoder, and freezing the parameters of the encoder part; and constructing a decoder part composed of a deconvolution layer and a fully connected layer; training the autoencoder to minimize its reconstruction error; comparing the values ​​of the first time series feature in the original indicator data and the indicator data after reconstruction by the autoencoder, and if the difference is greater than a preset value, filtering out the feature.

[0031] Furthermore, in one embodiment of the present application, the multiple time series curve feature extractions include: multiple ones of mean, standard deviation, range count, maximum value, minimum value, autocorrelation coefficient and peak count.

[0032] To achieve the above objectives, the present application proposes, on the other hand, an operable and explainable root cause location device for recurring type faults, comprising:

[0033] A fault triggering module is used to perform fault monitoring using a monitoring system, and trigger a trained root cause location model based on the faults monitored by the monitoring system; wherein the trained root cause location model is globally interpreted through a trained decision tree; a root cause location module is used to obtain a root cause fault instance based on indicator data and a fault dependency graph corresponding to the fault instance when the fault occurs, and locally interpret the location result of the root cause fault instance; wherein the fault dependency graph is constructed based on the fault instance.

[0034] The operable and explainable root cause locating device for recurring types of faults of the embodiments of the present application can realize operable and explainable root cause locating. The operability enables operation and maintenance personnel to directly obtain how to perform repair operations from the locating results of the present application, and the explainability enables operation and maintenance personnel to understand and trust the locating results given by the automated method of the present application.

[0035] The beneficial effects of this application are:

[0036] 1) Based on the root cause location framework of the present application, operational and explainable root cause location can be achieved. The operability enables operation and maintenance personnel to directly obtain how to perform repair operations from the location results, and the explainability enables operation and maintenance personnel to understand and trust the location results given by the automated method.

[0037] 2) The model of this application can provide a unified encoding for fault instances of any fault category, so that these fault instances can be further analyzed in a unified manner; it can consider the dependencies between fault instances, so that the root cause can still be accurately located in the scenario of fault propagation; it can be generalized to fault instances that have not been seen before

[0038] 3) The explainability method provides local and global explanations for the fault location model. On the one hand, it explains how the location result of each fault is given, and on the other hand, it explains what the entire model has learned from historical data, so that operation and maintenance personnel can understand and trust the location results of this application.

[0039] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0041] Figure 1 The present invention is a flowchart of an operational and explainable root cause location method for recurring faults according to an embodiment of the present application;

[0042] Figure 2 The overall architecture diagram of root cause location according to an embodiment of the present application;

[0043] Figure 3 is a schematic diagram of the structure of a root cause location model according to an embodiment of the present application;

[0044] Figure 4 It is a schematic diagram of the structure of an operable and explainable root cause locating device for repeatedly occurring type faults according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0047] The following describes an operable and explainable root cause location method and device for recurring type faults proposed in accordance with an embodiment of the present application with reference to the accompanying drawings. First, the operable and explainable root cause location method for recurring type faults proposed in accordance with an embodiment of the present application will be described with reference to the accompanying drawings.

[0048] Figure 1 The flowchart is a method for locating the root cause of recurring faults that is operable and explainable according to an embodiment of the present application.

[0049] like Figure 1 As shown, the method includes but is not limited to the following steps:

[0050] Step S1, using a monitoring system to perform fault monitoring, triggering a trained root cause location model based on the faults monitored by the monitoring system; wherein the trained root cause location model is globally explained by a trained decision tree;

[0051] Step S2, obtaining the root cause fault instance according to the indicator data corresponding to the fault instance when the fault occurs and the fault dependency graph, and locally interpreting the location result of the root cause fault instance; wherein the fault dependency graph is constructed according to the fault instance.

[0052] The embodiments of the present application are further described below in conjunction with the accompanying drawings.

[0053] Figure 2 This is the overall architecture diagram of the root cause location of this application. According to the architecture of this application, operational and explainable root cause location can be achieved. The operability allows operation and maintenance personnel to directly obtain how to perform repair operations from the location results of this application, and the explainability allows operation and maintenance personnel to understand and trust the location results given by the automated method of this application.

[0054] First, the granularity of the positioning of the present application is explained. It can be understood that in order to achieve operability, the granularity of the positioning of the present application is a fault instance. A fault instance is a set of indicators on a component. For example, the golden indicators (transaction volume, success rate, response rate, response time) on the service Service1 are a fault instance; the CPU-related indicators (for example, CPU usage, wio, etc.) on the container Container1 are a fault instance. A group of fault instances composed of similar indicators on similar components is called a fault category. For example, the CPU-related indicators on the container constitute a fault category, and these indicators on any specific container are a fault instance in this fault category.

[0055] When the operation and maintenance personnel determine that a set of indicators on a component is the root cause, the operation and maintenance personnel are sufficient to determine what problem has occurred with the component and what further repair actions should be taken. Therefore, such a combination of components and indicators is defined as a fault instance and a fault category. When the operation and maintenance personnel locate a fault instance defined in this way as the root cause of a fault, they can determine what repair actions should be taken. Therefore, by setting the positioning granularity to the fault instance, the positioning framework of the present application becomes operational.

[0056] Unlike existing methods, fault instances are finer-grained than components and individual indicators, because this application does not only locate the components and indicators corresponding to the fault instance, but determines their combined root causes. The number of indicator types and indicator patterns corresponding to different fault categories are different.

[0057] In order to model the complex dependencies in online service systems, all fault instances in a system are organized into a fault dependency graph. The nodes on the fault dependency graph are fault instances, and the edges represent the correlation between fault instances. The fault dependency graph is an undirected graph.

[0058] When there is a call relationship between the components corresponding to two fault instances, or a deployment relationship, or a causal relationship between the indicators, there will be an edge between the two fault instances on the fault dependency graph. For example, if service A calls service B, then there will be an edge between the instance composed of the golden indicators of service A and the instance composed of the golden indicators of service B. If service A is deployed on container C, then there will be an edge between the instance composed of the golden indicators of service A and the fault instances composed of various indicators of container C.

[0059] The structural design of the positioning model of this application is further explained. The positioning model of this application is a deep learning model. Its input is the indicators corresponding to all fault instances when a fault occurs, and the fault dependency graph. Its output is a score for each fault instance. The score range is [0,1]. The larger the score, the more likely it is that a fault instance is the root cause. Figure 3 As shown, Figure 3 The structure of the root cause location model for this application.

[0060] It can be understood that for any fault instance v, its input A () is the M on the fault instance v The value of an indicator at the latest w time points when the fault occurs. (v) is a shape of w×M v In practice, w = 20. The input of the entire model is the matrix A corresponding to all fault instances and the fault dependency graph.

[0061] The positioning model of the present application includes three components, a feature extractor, a feature aggregator, and a classifier. The feature extractor is responsible for representing each fault instance as a vector of fixed length, which is called an instance-level feature. The feature aggregator is responsible for aggregating the instance-level features of related fault instances of each fault instance based on the structure of the fault dependency graph to form an aggregated feature. Finally, the classifier scores based on the aggregated features of each fault instance. Because the aggregated features already include the features of the fault dependency graph and related instances, the classifier can also consider the propagation relationship of the fault between instances based only on the aggregated features of a single instance.

[0062] As an example, the feature extractor is a neural network composed of three layers: GRU (gated recurrent unit) layer, convolutional layer and fully connected layer. GRU is a recurrent neural network that extracts the features from A (v) The temporal information of the indicators is extracted from the network. Then, through the convolutional layer and the fully connected layer, the association between different time points and different indicators can be modeled.

[0063] It is understandable that because the number of indicators contained in the fault instances in each fault category is different, a fault extractor module is used for each fault category. The input dimensions (corresponding to the number of indicators) of these fault extractors are different, but all other configurations, including the output dimensions, are the same. Therefore, it is guaranteed that for any fault instance, the instance-level features of the same dimension are obtained through the fault extractor. The instance-level feature corresponding to the fault instance v is denoted as f (v) .

[0064] As an example, the feature aggregator uses a multi-head and multi-layer graph attention network (GAT). For each GAT, it first maps the instance-level features of each fault instance to a new space through a neural network, then calculates the weights between each pair of connected fault instances on the fault dependency graph, and finally performs a weighted average of the mapped features of the neighbors of each fault instance on the fault dependency graph to obtain the aggregated features of the fault instance. The aggregated features corresponding to the fault instance v are denoted as In order to improve the model capability of the feature aggregator, multiple GATs (multi-head GATs) are stacked in parallel, and then their outputs are concatenated as the aggregated features of each fault instance. On the other hand, multiple layers of multi-head GATs are stacked in series, that is, the output of the previous layer of multi-head GAT is used as the input of the next layer to model dependencies of more than one hop.

[0065] As an example, the classifier uses a two-layer fully connected network. The final output layer uses the sigmoid function to ensure that the output score belongs to [0,1]. For a fault instance v, its classifier output is s(v)∈[0,1].

[0066] The training and positioning of the model of this application are explained in detail. This application can be divided into two parts: offline training and online positioning.

[0067] In the offline training part, the proposed root cause localization model is trained periodically (e.g., weekly, monthly) based on historical fault data. The trained localization model will be explained through a global explanation method to show the operation and maintenance personnel what the trained localization model has learned.

[0068] In the online location part, when a fault occurs, the trained location model is triggered. After being triggered, the trained root cause location model gives the root cause fault instance based on the current indicator data and fault dependency graph, and at the same time, the local explanation method gives the explanation of the location result.

[0069] As an example, the model of the present application is trained by minimizing the loss function using the stochastic gradient descent method commonly used in training neural networks. However, in order to apply the stochastic gradient descent method, a key point is how to design the corresponding loss function.

[0070] The core idea of ​​the loss function of this application is to measure the output score s of each fault instance v of each fault (denoted as T) T (v) and the true label r T (v). If v is a root cause fault instance of T, then r T (v) = 1, otherwise r T (v)=0.

[0071] First, binary cross-entropy (BCE) is used to measure the gap between each pair:

[0072] BCE(r T (v),s T (v)) = r T (v)·log(s T (v))+(1-r T (v))·log(1-s T (v))

[0073] Then, because for each fault, the number of root cause fault instances and non-root cause fault instances is very different, in order to balance, additional weights are used to perform weighted average of the BCEs of different fault instances to obtain the final loss function.

[0074] Let the loss function be L s ,but

[0075]

[0076] in, is the training set, T i represents a fault in the training set, N H is the size of the training set, V represents all fault instances,

[0077] As an example, this application explains how the current fault model is located by finding similar historical faults, and proposes a method for comparing the distance between two faults.

[0078] Given two faults T 1 and T 2 , where T 1 is the current fault to be explained, T 2 is a historical malfunction. 1 For each fault instance in T, compare its aggregated features with T 2 The distance between the aggregate features of each fault instance belonging to the same fault category in the fault class, and the smallest one is taken as the distance between the fault instance and T 2 Then for T 1 All fault instances in T 2 The weight is the model score corresponding to the fault instance. The above distance calculation method can be formalized as:

[0079]

[0080] Among them, N c(v) represents the set of all fault instances that belong to the same fault category as v. Indicates fault T 1 Aggregate features of fault instance v in, Indicates fault T 2 Aggregate features of fault instance v′ in ||·|| 1 represents the L1 norm.

[0081] Therefore, for a current fault T to be explained 1 , calculate T through function d 1 The distance between each historical fault and the historical fault with the smallest distance is found as the similar historical fault.

[0082] Furthermore, we can globally explain the trained model by training some decision trees that can mimic the output of neural networks. Decision trees are rule-based and interpretable machine learning models. Through decision trees, we can see what root cause location rules the trained model may have learned.

[0083] In order to train the decision tree, we first need to extract time series features from the indicators. We selected several classic time series curve feature extraction methods, including mean, standard deviation, range count, maximum value, minimum value, self-correlation, peak count, etc.

[0084] Since the number of time series features extracted from the indicators is very large, the efficiency and accuracy of decision tree training are seriously affected. Therefore, it is hoped that the features that the trained positioning model may pay attention to can be selected from them. To this end, a feature extraction method based on indicator reconstruction is designed. First, the feature extractor and feature aggregator of the trained model are taken out as the encoder part of an autoencoder, and its parameters are frozen. Then, a decoder part consisting of a deconvolution layer and a fully connected layer is constructed. This autoencoder is trained to minimize its reconstruction error, and then the value of each feature in the original indicator and the indicator after reconstruction by the autoencoder is compared. If the difference is large, the feature is screened out.

[0085] Then, the decision tree is trained based on the selected features. The input of the decision tree is the time series features of the selected indicators of a fault instance. The training target output is the judgment of the trained positioning model on whether the fault instance is the root cause. This application uses the rules in the trained decision tree as a global interpretation of the trained positioning model.

[0086] Through the operable and explainable root cause location method for recurring types of faults of the embodiments of the present application, it is possible to achieve operable and explainable root cause location, and it is possible to directly determine how to perform repair operations based on the location results, and the location results can be accurately determined.

[0087] In order to implement the above embodiment, Figure 4 As shown, this embodiment also provides an operable and explainable root cause location device 10 for repeatedly occurring type faults. The device 10 includes: a rule learning module 100 and a root cause explanation module 200 .

[0088] A fault triggering module 100 is used to perform fault monitoring using a monitoring system, and trigger a trained root cause location model based on the fault monitored by the monitoring system; wherein the trained root cause location model is globally explained by a trained decision tree;

[0089] The root cause location module 200 is used to obtain the root cause fault instance according to the indicator data corresponding to the fault instance when the fault occurs and the fault dependency graph, and to locally interpret the location result of the root cause fault instance; wherein the fault dependency graph is constructed according to the fault instance.

[0090] Through the operable and explainable root cause locating device for recurring types of faults of the embodiment of the present application, it is possible to achieve operable and explainable root cause determination, and it is possible to directly determine how to perform repair operations based on the locating results, and the locating results can be accurately obtained.

[0091] It should be noted that the explanation of the aforementioned embodiment of the operational and explainable root cause locating method for recurring type faults is also applicable to the operational and explainable root cause locating device for recurring type faults of this embodiment, which will not be repeated here.

[0092] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0093] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0094] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. An operational and explainable root cause location method for recurring faults, characterized in that: The following steps are involved: Using a monitoring system to monitor faults, triggering a trained root cause location model based on the faults monitored by the monitoring system; wherein the trained root cause location model is globally explained by a trained decision tree; Obtaining a root cause fault instance according to the indicator data corresponding to the fault instance when the fault occurs and the fault dependency graph, and locally interpreting the location result of the root cause fault instance; wherein the fault dependency graph is constructed according to the fault instance; Using a loss function preset by a stochastic gradient descent method and training a root cause location model based on historical fault data, to obtain the trained root cause location model; The loss function preset by the stochastic gradient descent method includes: The output score s of the fault instance v of fault T is measured using binary cross entropy BCE T (v) and the true label r T (v) The gap between the gaps: BCE(r T (v),s T (v))=r T (v)·log(s T (v))+(1-r T (v)) log(1-s T (v)) The preset loss function is obtained by using additional weights to perform weighted average on the BCEs of different fault instances. The preset loss function is L s ,but: in, is the training set, T i represents a fault in the training set, N H is the size of the training set, V represents all fault instances, 2. The method according to claim 1, characterized in that The root cause localization model includes three components: feature extractor, feature aggregator and classifier. Then, Using the feature extractor to represent each fault instance as a vector of fixed length; wherein the vector is an instance-level feature; aggregating instance-level features of related fault instances of each fault instance by using the feature aggregator based on the structure of the fault dependency graph to obtain aggregated features; The classifier is used to score based on the aggregated features, so as to design the root cause localization model according to the three components.

3. The method according to claim 1, characterized in that The local explanation method is used to explain the location result of the root cause fault instance, including: Two faults, a first fault and a second fault, are preset; wherein the first fault is a current fault to be explained, and the second fault is a historical fault; For each fault instance in the first fault, compare the distance between its aggregate feature and the aggregate feature of each fault instance in the second fault that belongs to the same fault category, and take the smallest one as the distance between the fault instance and the second fault; A weighted average is taken of the distances from all fault instances in the first fault to the second fault, and a location result of the root fault instance is explained by comparing the distance formula between the two faults.

4. The method according to claim 3, characterized in that A first fault T1 and a second fault T2 are preset, and the distance formula between the two faults is: Among them, N c (v) represents the set of all fault instances that belong to the same fault category as fault instance v. represents the aggregate features of the fault instance v in fault T1, represents the aggregated features of the fault instance v′ in fault T2, and ||·||1 represents the L1 norm.

5. The method according to claim 1, characterized in that The trained root cause localization model is globally explained through the trained decision tree, including: Extracting a first time series feature from the indicator data using a plurality of time series curve feature extractions; Selecting the first time series feature by using feature extraction based on indicator reconstruction to obtain a second time series feature; A decision tree is trained based on the second time series feature to obtain the trained decision tree, and a root cause judgment is performed on the fault instance using the trained root cause localization model according to the trained decision tree; wherein the rules in the trained decision tree serve as a global interpretation of the trained root cause localization model.

6. The method according to claim 5, characterized in that The feature extraction of the indicator reconstruction includes: Using the trained feature extractor and feature aggregator of the root cause localization model as an encoder part of the autoencoder, and freezing the parameters of the encoder part; and constructing a decoder part consisting of a deconvolution layer and a fully connected layer; Training the autoencoder to minimize its reconstruction error; Compare the values ​​of the first time series feature in the original indicator data and the indicator data reconstructed by the automatic encoder, and if the difference is greater than a preset value, filter out the feature.

7. The method according to claim 5, characterized in that The multiple time series curve feature extractions include: multiple ones of mean value, standard deviation, range count, maximum value, minimum value, autocorrelation coefficient and peak count.

8. An operational and explainable root cause location device for recurring faults, characterized in that: include: A fault triggering module is used to perform fault monitoring using a monitoring system, and trigger a trained root cause location model based on the fault monitored by the monitoring system; wherein the trained root cause location model is globally explained by a trained decision tree; A root cause location module, used to obtain a root cause fault instance according to the indicator data corresponding to the fault instance when the fault occurs and the fault dependency graph, and to perform a local interpretation on the location result of the root cause fault instance; wherein the fault dependency graph is constructed according to the fault instance; The fault trigger module is further used to train the root cause location model based on historical fault data using a loss function preset by a stochastic gradient descent method to obtain the trained root cause location model; The loss function preset by the stochastic gradient descent method includes: The output score s of the fault instance v of fault T is measured using binary cross entropy BCE T (v) and the true label r T (v) The gap between the gaps: BCE(r T (v),s T (v))=r T (v)·log(s T (v))+(1-r T (v)) log(1-s T (v)) The preset loss function is obtained by using additional weights to perform weighted average on the BCEs of different fault instances. The preset loss function is L s ,but: in, is the training set, T i represents a fault in the training set, N H is the size of the training set, V represents all fault instances,