Fault root cause analysis method and device, electronic equipment and medium

Abstract

The present disclosure provides a fault root cause analysis method, which relates to the technical field of artificial intelligence. The method comprises: acquiring a link structure; constructing a fault rule tree according to the link structure, wherein the fault rule tree represents the correlation relationship of each node in the link structure to generate alarm information; acquiring a plurality of to-be-detected indicators of each node in the link structure; acquiring first alarm information; acquiring a set of alarm information issued by a plurality of nodes in the link structure within a specified time window; determining at least one node in the link structure as a root cause node corresponding to the first alarm information according to a plurality of historical alarm information and the fault rule tree; using a multivariate time series detection method to calculate the deviation degree of each to-be-detected indicator of the root cause node, and determining the to-be-detected indicator corresponding to the highest deviation degree as the root cause indicator of the root cause node. The present disclosure also provides a fault root cause analysis device, equipment, storage medium and program product.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a fault root cause analysis method, device, electronic device, and medium. Background Art

[0002] In the era of distributed cloud computing, application links are becoming increasingly complex, with more and more nodes and increasingly complex calling relationships between nodes. When a link or a node fails, many nodes often issue alarms simultaneously. Finding which node and which indicator has failed is particularly important for fault location and resolution, which directly affects the availability and stability of the application.

[0003] Currently, root cause identification, a common method used in fault analysis, relies primarily on manual analysis by development and operations personnel to identify the root cause of a fault or alarm from massive amounts of data. This method requires manual effort to sift through vast amounts of files and data to identify traces of the fault, and relies on expert experience, making it relatively time-consuming and labor-intensive. Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a fault root cause analysis method, device, electronic device and medium that improve the efficiency of root cause location and the ability to troubleshoot fault indicators.

[0005] According to a first aspect of the present disclosure, a fault root cause analysis method is provided, comprising: obtaining a link structure, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates alarm information corresponding to a fault event; constructing a fault rule tree based on the link structure, wherein the fault rule tree represents the association relationship between the alarm information generated by each node in the link structure; obtaining multiple indicators to be detected for each node in the link structure; obtaining first alarm information, wherein the first alarm information is issued by at least one first node in the link structure at a first moment; obtaining an alarm information set issued by multiple nodes in the link structure within a specified time window, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window; determining at least one node in the link structure as a root cause node corresponding to the first alarm information based on the multiple historical alarm information and the fault rule tree; and calculating the deviation of each indicator to be detected of the root cause node using a multivariate time series detection method, and determining the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node.

[0006] According to an embodiment of the present disclosure, the multiple nodes include upstream nodes and downstream nodes, and the link structure includes a call link in which the upstream node calls the downstream node; constructing a fault rule tree based on the link structure specifically includes: obtaining upstream alarm information of the upstream node; and according to the call link in which the upstream node calls the downstream node, causing the downstream node to generate downstream alarm information, wherein the downstream alarm information and the upstream alarm information are of the same alarm type.

[0007] According to an embodiment of the present disclosure, the method of determining at least one node in the link structure as a root cause node corresponding to the first alarm information based on the multiple historical alarm information and the fault rule tree specifically includes: screening valid alarm information in the multiple historical alarm information according to the fault rule tree, wherein the valid alarm information is multiple and has the same alarm type as the first alarm information; obtaining multiple second nodes corresponding to the valid alarm information; and determining the upstreammost node among the multiple second nodes as the root cause node corresponding to the first alarm information.

[0008] According to an embodiment of the present disclosure, the method of using a multivariate time series detection method to calculate the deviation of each indicator to be detected of the root cause node specifically includes: normalizing each indicator to be detected of the root cause node; and determining the deviation of each indicator to be detected based on the standard deviation of the normalized indicator to be detected.

[0009] According to an embodiment of the present disclosure, at least one of the multiple nodes runs in a container, and the multiple indicators to be detected include at least one of container CPU usage, container memory usage, container I / O usage, number of network connections, access time and access success rate.

[0010] According to an embodiment of the present disclosure, when the multiple indicators to be detected include the number of network connections and access time, the indicator to be detected corresponding to the highest deviation is determined as the root cause indicator of the root cause node, specifically including: comparing the deviation of the number of network connections and the access time of the root cause node with their respective preset lower limit thresholds; if the deviation of the number of network connections and the access time are both greater than the lower limit threshold, it is determined that there is an abnormality in the network; and determining the one with the higher deviation between the number of network connections and the access time as the root cause indicator of the root cause node.

[0011] According to an embodiment of the present disclosure, the method further includes: sorting other indicators to be detected of the root cause node except the root cause indicator according to the size of the deviation; and sending the other indicators to be detected sorted according to the size of the deviation.

[0012] The second aspect of the present disclosure provides a fault root cause analysis device, comprising: a link acquisition module for acquiring a link structure, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates alarm information corresponding to a fault event; a rule tree construction module for constructing a fault rule tree based on the link structure, wherein the fault rule tree represents the association relationship between the alarm information generated by each node in the link structure; a to-be-detected indicator acquisition module for acquiring multiple to-be-detected indicators of each node in the link structure; an alarm information acquisition module for acquiring first alarm information, wherein the first alarm information is an alarm information issued by at least one first node in the link structure at a first moment. ; A historical alarm information acquisition module, used to obtain a set of alarm information issued by multiple nodes in the link structure within a specified time window, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window; a root cause node determination module, used to determine at least one node in the link structure as a root cause node corresponding to the first alarm information based on the multiple historical alarm information and the fault rule tree; and a root cause indicator determination module, used to use a multivariate time series detection method to calculate the deviation of each indicator to be detected of the root cause node, and determine the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node.

[0013] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned fault root cause analysis method.

[0014] A fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned fault root cause analysis method.

[0015] A fifth aspect of the present disclosure further provides a computer program product, including a computer program, which implements the above-mentioned fault root cause analysis method when executed by a processor.

[0016] Compared with the prior art, the fault root cause analysis method, device, electronic device, and medium provided by the present disclosure have at least the following beneficial effects:

[0017] The present disclosure utilizes a fault rule tree and a multivariate time series detection algorithm to locate nodes. The fault rule tree is established offline, so the node location is fast and accurate. At the same time, for the positioning of indicators, the multivariate time series detection algorithm is used to comprehensively consider the deviation of indicators, which also has high accuracy and high timeliness of detection. Therefore, the fault root cause analysis method disclosed in the present disclosure has the characteristics of high accuracy and fast response. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] Figure 1 The system architecture of the fault root cause analysis method and device according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 2 The following schematically shows a flow chart of a fault root cause analysis method according to an embodiment of the present disclosure;

[0021] Figure 3 Schematically shows a flow chart of constructing a fault rule tree according to an embodiment of the present disclosure;

[0022] Figure 4 Schematically shows a flowchart of determining a root cause node according to an embodiment of the present disclosure;

[0023] Figure 5 Schematically shows a flow chart of deviation calculation according to an embodiment of the present disclosure;

[0024] Figure 6 The flowchart of determining the root cause index of the root cause node according to the embodiment of the present disclosure is schematically shown;

[0025] Figure 7 Schematically shows a flow chart of a fault root cause analysis method according to another embodiment of the present disclosure;

[0026] Figure 8 The following schematically shows a structural block diagram of a fault root cause analysis device according to an embodiment of the present disclosure;

[0027] Figure 9 A block diagram of an electronic device suitable for implementing a fault root cause analysis method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] Embodiments of the present disclosure provide a fault root cause analysis method, device, electronic device, and medium, relating to the field of artificial intelligence technology. The method includes: obtaining a link structure, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates alarm information corresponding to a fault event; constructing a fault rule tree based on the link structure, wherein the fault rule tree represents the association relationship between the alarm information generated by each node in the link structure; obtaining multiple indicators to be detected for each node in the link structure; obtaining first alarm information, wherein the first alarm information is issued by at least one first node in the link structure at a first moment; obtaining an alarm information set issued by multiple nodes in the link structure within a specified time window, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window; determining at least one node in the link structure as a root cause node corresponding to the first alarm information based on the multiple historical alarm information and the fault rule tree; and calculating the deviation of each indicator to be detected of the root cause node using a multivariate time series detection method, and determining the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node. At the moment an alarm occurs, this method locates the root cause node from the alarm information set within a reference time window on the link structure. Once the root cause node is found, it performs multivariate time series testing on each of the indicators defined for detection on that node to determine the specific indicator causing the fault. This method thus integrates alarm rules and node indicator detection, making root cause analysis simple and time-efficient.

[0033] Before describing specific embodiments of the present disclosure in detail, technical terms are first explained to facilitate a better understanding of the present disclosure.

[0034] Time series: A sequence of numerical values ​​of a statistical indicator arranged in chronological order. In a time series, it is necessary to predict its future trend based on the current data of the time series.

[0035] Prophet algorithm: It is an open source time series forecasting algorithm developed by Facebook. It is based on a decomposable (trend, seasonal, or holiday) model and supports customized seasonal and holiday impacts. Compared with the Holt-Winters (Triple / Three Order Exponential Smoothing) algorithm and the ARIMA algorithm, it has more flexible parameter configuration.

[0036] Rule tree: It is a classification method based on human experience. It uses a tree structure to construct a rule tree with flexible rule configuration.

[0037] Figure 1 The system architecture 100 of the method and apparatus for fault root cause analysis according to an embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0038] like Figure 1 As shown, the system architecture 100 according to this embodiment may include access terminals 101 and 102, a client server 103, an intermediate server 104, and a main server 105. These node devices are connected together through a network to form a distributed system architecture. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0039] The access terminals 101 and 102 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0040] The user-side server 103, the intermediate server 104, and the main server 105 can each be a server that provides various services, such as a backend management server (for example only) that supports websites browsed by users using the access terminals 101 and 102. The backend management server can analyze and process received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices in sequence according to nodes.

[0041] Based on the distributed system architecture, the topological network structure of the link can be pre-set. In addition, the operation and maintenance personnel can pre-build a fault rule tree based on the causal relationship between the alarm types that may occur in each node device of the distributed system architecture.

[0042] It should be noted that the fault root cause analysis method provided in the embodiment of the present disclosure can generally be executed by the main server 105. Accordingly, the fault root cause analysis device provided in the embodiment of the present disclosure can generally be set in the main server 105.

[0043] Specifically, the network status of all node devices can be uploaded to the main server 105 for use in root cause analysis. The main server 105 regularly collects information about each node device in the distributed system architecture and determines whether each node device has a fault based on the collected information. For example, if the main server 105 detects that the access terminals 101, 102 and the user-side server 103 have successively experienced downtime, it will generate corresponding alarm data a0, b0 and c0 and upload them to the main server 105. The main server 105 receives the above alarm data and puts it into the alarm list. The alarm list may also contain other alarm data that was previously put in, such as alarm data b1 and b2. The main server 105 needs to analyze the alarms to determine which alarm is a root cause alarm. Therefore, it is necessary to extract each alarm information from all alarm lists for root cause analysis.

[0044] It should be understood that Figure 1 The number of access terminals, user-side servers, intermediate servers, and main servers in the embodiment is merely illustrative. Any number of access terminals, user-side servers, intermediate servers, and main servers may be provided as required.

[0045] The following will be based on Figure 1 The system architecture described by Figures 2 to 7 The fault root cause analysis method of the disclosed embodiment is described in detail.

[0046] Figure 2 The flowchart of the fault root cause analysis method according to the embodiment of the present disclosure is schematically shown.

[0047] like Figure 2 As shown, the fault root cause analysis method of this embodiment includes operations S210 to S270.

[0048] In operation S210 , a link structure is acquired, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates alarm information corresponding to a fault event.

[0049] In operation S220 , a fault rule tree is constructed according to the link structure, wherein the fault rule tree represents the association relationship between alarm information generated by each node in the link structure.

[0050] The fault rule tree can be formulated by operation and maintenance support personnel or experts based on the link structure. The fault rule tree can reflect the correlation relationship between alarms occurring at other nodes when various types of alarms occur in the link structure.

[0051] Fault rule trees can also be built from historical fault event-related data sets using association analysis algorithms. For example, these association analysis algorithms can include the currently popular Apriori algorithm, the Maximum Frequent Itemsets Approach, and the Galois Closure-Based Approach.

[0052] In operation S230 , a plurality of indicators to be detected of each node in the link structure are obtained.

[0053] In an embodiment of the present disclosure, at least one of the multiple nodes runs in a container, the data generated by the link structure may be transaction data, and the indicators to be detected of each node are selected from the fault indicators pre-stored in the node. For example, the multiple indicators to be detected may be selected from at least one of the container CPU usage, container memory usage, container I / O usage, number of network connections, access time, and access success rate.

[0054] In operation S240, first alarm information is acquired, where the first alarm information is sent by at least one first node in the link structure at a first moment.

[0055] In operation S250, an alarm information set issued by multiple nodes in the link structure within a specified time window is obtained, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by multiple nodes within the specified time window.

[0056] The first alarm information indicates that a link failure occurred at the first moment, i.e., the moment the alarm occurred. The alarm information set consists of multiple historical alarm information collected during a specified time period before the first moment. It is understood that this specified time window can be determined based on expert experience and serves as a reference time window for generating alarms. For example, the specified time window can be 5 minutes before the first moment. Thus, at the first moment, all alarm information within the 5 minutes before the first moment is extracted from the link structure to form an alarm information set, which is used to analyze the root cause of the failure event at the first moment.

[0057] In the embodiments of the present disclosure, the alarm information set includes different types of alarm information. For ease of understanding, the alarm types are pre-set, for example, the root cause alarm type and the object alarm type mentioned above. In other embodiments, different alarm types can be set according to actual needs, and the present disclosure is not limited to these two types.

[0058] In operation S260, at least one node in the link structure is determined as a root cause node corresponding to the first alarm information based on the plurality of historical alarm information and the fault rule tree.

[0059] In operation S270 , a multivariate time series detection method is used to calculate the deviation of each to-be-detected indicator of the root cause node, and the to-be-detected indicator corresponding to the highest deviation is used as the root cause indicator of the root cause node.

[0060] In an embodiment of the present disclosure, the multivariate time series detection method includes a Prophet algorithm, and the deviation of each to-be-detected indicator of the root cause node includes the degree to which each to-be-detected indicator deviates from an average value.

[0061] The Prophet algorithm is a time series algorithm suitable for business behavior data with clear internal patterns. For example, business problems with the following characteristics: historical data with hourly, daily, or weekly observations spanning at least several months (preferably a year) and known, irregularly occurring important holidays (such as National Day). Therefore, using the Prophet algorithm to calculate the deviation of various indicators offers high accuracy and timely detection.

[0062] Through the disclosed embodiments, at the moment an alarm occurs, the root cause node is located based on the alarm information set within a reference time window on the link structure. Once the root cause node is found, multivariate time series testing is performed on each of the indicators defined for detection on that node to determine the specific indicator causing the fault. Thus, this embodiment comprehensively considers alarm rules and node indicator testing to propose a simple and efficient method for locating the root cause of a fault.

[0063] In the embodiment of the present disclosure, the multiple nodes include upstream nodes and downstream nodes, and the link structure includes a call link in which the upstream node calls the downstream node.

[0064] Figure 3 The flowchart of constructing a fault rule tree according to an embodiment of the present disclosure is schematically shown.

[0065] like Figure 3 As shown, the above operation S220 of constructing a fault rule tree according to the link structure may specifically include operations S2201 and S2202.

[0066] In operation S2201, upstream alarm information of an upstream node is acquired.

[0067] In operation S2202 , according to a call link of the upstream node calling the downstream node, the downstream node generates downstream alarm information, wherein the downstream alarm information and the upstream alarm information are of the same alarm type.

[0068] For example, if the link is composed of A->B->C->D, node D is at the upstream of the link and node A is at the downstream of the link. If node D sends the first alarm message at the first moment, according to the construction method of the fault rule tree, the first alarm message will cause node C, node B, and node A to all issue alarm messages of the same type as the first alarm message.

[0069] Therefore, the fault rule tree is constructed in such a way that the alarm information of the upstream node automatically triggers the downstream node to generate alarm information of the same alarm type, thereby making the alarm information generated between different nodes in the link structure have corresponding association relationships.

[0070] Figure 4 The flowchart of determining the root cause node according to an embodiment of the present disclosure is schematically shown.

[0071] like Figure 4 As shown, the above operation S260 of determining at least one node in the link structure as the root cause node corresponding to the first alarm information based on multiple historical alarm information and the fault rule tree may specifically include operations S2601 to S2603.

[0072] In operation S2601, valid alarm information in a plurality of historical alarm information is screened according to a fault rule tree, wherein the valid alarm information is multiple and has the same alarm type as the first alarm information.

[0073] In operation S2602, multiple second nodes corresponding to valid alarm information are obtained.

[0074] In operation S2603 , the most upstream node among the plurality of second nodes is determined as a root cause node corresponding to the first alarm information.

[0075] It should be noted that the multiple second nodes corresponding to the multiple valid alarm information are continuous and uninterrupted in the link structure, and generally there are no isolated nodes with discontinuous intervals.

[0076] Thus, after obtaining multiple valid alarm information, multiple second nodes are correspondingly obtained, and the node most upstream among the multiple second nodes is determined as the root cause node corresponding to the first alarm information issued at the first moment. Since there may be multiple first nodes at the first moment, there may also be multiple root cause nodes, each root cause node corresponding to the first alarm information issued by a first node at the first moment.

[0077] The root cause node corresponding to the first alarm information is iteratively updated by the alarm information set according to the fault rule tree, and generally tends to be the most upstream node generating the alarm in the link structure.

[0078] Figure 5 The flowchart of deviation calculation according to an embodiment of the present disclosure is schematically shown.

[0079] like Figure 5 As shown, the above operation S270 of calculating the deviation of each to-be-detected indicator of the root cause node using the multivariate time series detection method may specifically include operations S2701 and S2702.

[0080] In operation S2701 , normalization processing is performed on each to-be-detected indicator of the root cause node.

[0081] In operation S2702 , the deviation of each indicator to be detected is determined according to the standard deviation of the indicator to be detected after normalization.

[0082] Therefore, each indicator to be detected of the root cause node can be a stationary indicator data, and the proportion of the absolute value of the difference between each indicator to be detected and the standard deviation to the standard deviation is used as the deviation of each indicator to be detected.

[0083] Figure 6 The flowchart of determining the root cause indicator of the root cause node according to an embodiment of the present disclosure is schematically shown.

[0084] like Figure 6 As shown, in order to further improve the accuracy of the root cause indicator, when multiple indicators to be detected include the number of network connections and access time, the above operation S270 determines the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node, which can specifically include operations S2703 to S2706.

[0085] In operation S2703 , the number of network connections and the deviation of the access time of the root cause node are compared with respective preset lower thresholds.

[0086] In operation S2704 , if the deviations of the number of network connections and the access time are both greater than the lower threshold, it is determined that there is an abnormality in the network.

[0087] In operation S2705 , the one with a higher deviation between the number of network connections and the access duration is determined as the root cause indicator of the root cause node.

[0088] Therefore, when it is detected that both the number of network connections and the access time are in an abnormal state, the one with the higher deviation between the two indicators is used as the root cause indicator of the root cause node of the fault event.

[0089] Figure 7 The flowchart of a fault root cause analysis method according to another embodiment of the present disclosure is schematically shown.

[0090] like Figure 7 As shown, in another embodiment, after the root cause indicator of the root cause node is obtained in the above operation S270, operations S280 to S290 may be further included.

[0091] In operation S280 , the other to-be-detected indicators of the root cause node except the root cause indicator are sorted according to the degree of deviation.

[0092] In operation S290, other indicators to be detected are sent after being sorted according to the degree of deviation.

[0093] Therefore, other indicators to be detected on the root cause node except the root cause indicator can be sorted according to the degree of deviation and then sent to development or related personnel.

[0094] The disclosed embodiments utilize a fault rule tree and a multivariate time series detection algorithm for node location. The fault rule tree is established offline, resulting in fast node location and high accuracy. Furthermore, the multivariate time series detection algorithm is used to comprehensively analyze indicator deviations for indicator location, achieving high accuracy and timeliness. Therefore, the root cause analysis method of this embodiment boasts high accuracy and rapid response.

[0095] Based on the above-mentioned fault root cause analysis method, the present disclosure also provides a fault root cause analysis device. Figure 8 The device is described in detail.

[0096] Figure 8 A block diagram of a fault root cause analysis device according to an embodiment of the present disclosure is schematically shown.

[0097] like Figure 8 As shown, the fault root cause analysis device 800 of this embodiment includes a link acquisition module 810, a rule tree construction module 820, a to-be-detected indicator acquisition module 830, an alarm information acquisition module 840, a historical alarm information acquisition module 850, a root cause node determination module 860 and a root cause indicator determination module 870.

[0098] Link acquisition module 810 is configured to acquire a link structure, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates an alarm corresponding to a fault event. In one embodiment, link acquisition module 810 can be configured to perform operation S210 described above, and will not be further described here.

[0099] The rule tree construction module 820 is used to construct a fault rule tree based on the link structure, wherein the fault rule tree represents the association relationship between the alarm information generated by each node in the link structure. In one embodiment, the rule tree construction module 820 can be used to perform the operation S220 described above, which will not be repeated here.

[0100] The indicator acquisition module 830 is configured to acquire multiple indicators to be detected for each node in the link structure. In one embodiment, the indicator acquisition module 830 can be configured to execute the operation S230 described above, which will not be described in detail here.

[0101] The alarm information acquisition module 840 is configured to acquire first alarm information, wherein the first alarm information is sent by at least one first node in the link structure at a first moment. In one embodiment, the alarm information acquisition module 840 may be configured to execute the operation S240 described above, which will not be described in detail herein.

[0102] Historical alarm information acquisition module 850 is configured to acquire a set of alarm information issued by multiple nodes in the link structure within a specified time window, where the specified time window is a specified period of time before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window. In one embodiment, historical alarm information acquisition module 850 can be used to perform operation S250 described above, and will not be further described here.

[0103] Root cause node determination module 860 is configured to determine, based on the plurality of historical alarm information and the fault rule tree, at least one node in the link structure as the root cause node corresponding to the first alarm information. In one embodiment, root cause node determination module 860 may be configured to perform operation S260 described above, which will not be further described here.

[0104] Root cause indicator determination module 870 is configured to use a multivariate time series detection method to calculate the deviation of each to-be-detected indicator of the root cause node and determine the to-be-detected indicator corresponding to the highest deviation as the root cause indicator of the root cause node. In one embodiment, root cause indicator determination module 870 can be configured to perform operation S270 described above and will not be further described here.

[0105] Through the embodiments of the present disclosure, at the moment an alarm occurs, the root cause node is located from the alarm information set of a certain reference time window on the link. After the root cause node is found, multivariate time series detection is performed on each indicator to be detected defined on the node to determine which specific indicator causes the fault. Therefore, this embodiment comprehensively considers alarm rules and node indicator detection to propose a simple and highly efficient fault root cause location method.

[0106] According to an embodiment of the present disclosure, any multiple modules among the link acquisition module 810, the rule tree construction module 820, the to-be-detected indicator acquisition module 830, the alarm information acquisition module 840, the historical alarm information acquisition module 850, the root cause node determination module 860, and the root cause indicator determination module 870 can be combined into a single module for implementation, or any one of these modules can be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules can be combined with at least part of the functionality of other modules and implemented in a single module. According to an embodiment of the present disclosure, at least one of the link acquisition module 810, the rule tree construction module 820, the to-be-detected indicator acquisition module 830, the alarm information acquisition module 840, the historical alarm information acquisition module 850, the root cause node determination module 860, and the root cause indicator determination module 870 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable method of integrating or packaging the circuit, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the link acquisition module 810, the rule tree construction module 820, the to-be-detected indicator acquisition module 830, the alarm information acquisition module 840, the historical alarm information acquisition module 850, the root cause node determination module 860, and the root cause indicator determination module 870 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0107] Figure 9 A block diagram of an electronic device suitable for implementing a fault root cause analysis method according to an embodiment of the present disclosure is schematically shown.

[0108] like Figure 9 As shown, the electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. The processor 901 may, for example, include a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include an onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0109] Various programs and data required for the operation of the electronic device 900 are stored in the RAM 903. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 902 and / or the RAM 903. It should be noted that the programs may also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0110] According to an embodiment of the present disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 may further include one or more of the following components connected to the I / O interface 905: an input portion 906 including a keyboard, a mouse, etc.; an output portion 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage portion 908 including a hard disk, etc.; and a communication portion 909 including a network interface card such as a LAN card or a modem. The communication portion 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 910 as needed, so that a computer program read therefrom can be installed into the storage portion 908 as needed.

[0111] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the root cause analysis method according to the embodiments of the present disclosure.

[0112] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 902 and / or RAM 903 described above and / or one or more memories other than ROM 902 and RAM 903.

[0113] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is executed in a computer system, the program code is used to enable the computer system to implement the fault root cause analysis method provided by the embodiments of the present disclosure.

[0114] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the processor 901 executes the computer program. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0115] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 909, and / or installed from a removable medium 911. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0116] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from a removable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0117] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0120] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A method for analyzing root causes of failures, characterized in that: include: Acquire a link structure, wherein the link structure includes call links between multiple nodes, and at least one node in the link structure generates alarm information corresponding to a fault event; Constructing a fault rule tree according to the link structure, wherein the fault rule tree represents the association relationship between alarm information generated by each node in the link structure; Acquire multiple indicators to be detected for each node in the link structure; Obtaining first alarm information, wherein the first alarm information is sent by at least one first node in the link structure at a first moment; Acquire an alarm information set issued by multiple nodes in the link structure within a specified time window, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window; Determining, based on the plurality of historical alarm information and the fault rule tree, at least one node in the link structure as a root cause node corresponding to the first alarm information; and Normalizing each indicator to be detected of the root cause node; and According to the standard deviation of the normalized indicators to be detected, the deviation of each indicator to be detected is determined, and the indicator to be detected corresponding to the highest deviation is determined as the root cause indicator of the root cause node.

2. The method according to claim 1, wherein The multiple nodes include upstream nodes and downstream nodes, and the link structure includes a call link in which the upstream node calls the downstream node; The constructing of a fault rule tree according to the link structure specifically includes: Obtaining upstream alarm information of the upstream node; and According to the calling link of the upstream node calling the downstream node, the downstream node generates downstream alarm information, wherein the downstream alarm information and the upstream alarm information are of the same alarm type.

3. The method according to claim 2, wherein: The determining, based on the multiple historical alarm information and the fault rule tree, at least one node in the link structure as a root cause node corresponding to the first alarm information specifically includes: Filtering valid alarm information from the plurality of historical alarm information according to the fault rule tree, wherein the valid alarm information is multiple and has the same alarm type as the first alarm information; Acquire multiple second nodes corresponding to the valid alarm information; and The most upstream node among the plurality of second nodes is determined as a root cause node corresponding to the first alarm information.

4. The method according to claim 1, wherein At least one of the multiple nodes runs in a container, and the multiple indicators to be detected include at least one of container CPU usage, container memory usage, container I / O usage, number of network connections, access time and access success rate.

5. The method according to claim 4, wherein When the multiple indicators to be detected include the number of network connections and the access time, determining the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node specifically includes: Comparing the number of network connections and the deviation of access time of the root cause node with respective preset lower limit thresholds; If the deviations of the number of network connections and the access time are both greater than the lower threshold, it is determined that there is a network anomaly; and The one with a higher deviation between the number of network connections and the access time is determined as the root cause indicator of the root cause node.

6. The method according to claim 1, wherein The method further comprises: Sort the other to-be-detected indicators of the root cause node except the root cause indicator according to the degree of deviation; and Send the other indicators to be detected sorted by deviation.

7. A fault root cause analysis device, characterized in that: include: a link acquisition module, configured to acquire a link structure, wherein the link structure includes call links between a plurality of nodes, and at least one node in the link structure generates alarm information corresponding to a fault event; A rule tree construction module is used to construct a fault rule tree according to the link structure, wherein the fault rule tree represents the association relationship between the alarm information generated by each node in the link structure; A module for obtaining indicators to be detected, used to obtain multiple indicators to be detected of each node in the link structure; an alarm information acquisition module, configured to acquire first alarm information, wherein the first alarm information is sent by at least one first node in the link structure at a first moment; a historical alarm information acquisition module, configured to acquire an alarm information set issued by multiple nodes in the link structure within a specified time window, wherein the specified time window is a specified time period before the first moment, and the alarm information set includes multiple historical alarm information issued by the multiple nodes within the specified time window; a root cause node determination module, configured to determine, based on the plurality of historical alarm information and the fault rule tree, at least one node in the link structure as a root cause node corresponding to the first alarm information; and The root cause indicator determination module is used to normalize the various indicators to be detected of the root cause node; and determine the deviation of each indicator to be detected based on the standard deviation of the normalized indicators to be detected, and determine the indicator to be detected corresponding to the highest deviation as the root cause indicator of the root cause node.

8. An electronic device comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images