Fault root cause determination method and apparatus, electronic device, and readable storage medium

Abstract

The disclosure provides a fault root cause determination method and device, electronic equipment and readable storage medium, which can be applied to the fields of computer technology, Internet technology and electronics. The method comprises the following steps: in the case of detecting that a fault occurs, executing a sub-automation script of a sub-node in each domain node of an automation script set to obtain a first health check result of each sub-node, wherein the domain nodes of the automation script set have a one-way hierarchical dependency relationship; determining an abnormal sub-node corresponding to the domain node according to the first health check result of each sub-node; determining at least one abnormal domain node according to the correspondence between the abnormal sub-node and the domain node; and determining a fault root cause causing the fault to occur from the at least one abnormal domain node based on the one-way hierarchical dependency relationship between the domain nodes of the automation script set.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology, internet technology, and electronic technology, and specifically to a method, apparatus, electronic device, readable storage medium, and program product for determining the root cause of a fault. Background Technology

[0002] With the rapid development of artificial intelligence, automatic control, communication and computer technologies, the scale of data center networks is getting larger and larger. When the system fails, multiple alarms will break out in multiple areas of the data center, which will affect the normal operation of applications and service equipment.

[0003] Currently, when a system failure occurs, the maintenance personnel responsible for each area need to conduct fault checks in their respective areas based on their historical experience and accumulated knowledge. The results of these checks are then summarized to determine the root cause of the system failure. However, when faced with a large number of alarms, maintenance personnel often find it difficult to pinpoint the root cause in a timely manner relying solely on their historical experience and accumulated knowledge. Manual processing is inefficient and cannot accurately report the root cause, thus affecting the continuity of business operations. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a method, apparatus, electronic device, readable storage medium, and program product for determining the root cause of a fault.

[0005] According to one aspect of this disclosure, a method for determining the root cause of a fault is provided, comprising: upon detecting a fault, executing sub-automated scripts of child nodes in each domain node of an automated script set to obtain a first health check result for each child node, wherein the domain nodes of the automated script set have a unidirectional hierarchical dependency relationship; determining abnormal child nodes corresponding to the domain nodes based on the first health check result of each child node; determining at least one abnormal domain node based on the correspondence between the abnormal child nodes and the domain nodes; and determining the root cause of the fault from the at least one abnormal domain node based on the unidirectional hierarchical dependency relationship between the domain nodes of the automated script set.

[0006] According to embodiments of this disclosure, executing the sub-automation scripts of child nodes in each domain node of the automation script set includes: executing the sub-automation scripts of child nodes in each domain node sequentially based on the unidirectional hierarchical dependency relationship between domain nodes.

[0007] According to embodiments of this disclosure, the sequential execution of sub-automation scripts of child nodes in each domain node further includes: for executing sub-automation scripts of child nodes in the current domain node, when executing the sub-automation script of the current child node, retrieving the third health check result of the sub-automation scripts of the already executed child nodes in the current domain node; determining the second health check result of the sub-automation scripts of the already executed child nodes associated with the current child node based on the association relationship between child nodes in the current domain node; if the third health check result of the sub-automation scripts of the already executed child nodes associated with the current child node is determined to be abnormal, executing the sub-automation script of the next child node of the current child node; if the third check result of the sub-automation scripts of the already executed child nodes associated with the current child node is determined to be normal, executing the sub-automation script of the current child node.

[0008] According to embodiments of this disclosure, the sequential execution of sub-automation scripts of child nodes in each domain node includes: for executing sub-automation scripts of child nodes between each domain node, when executing the sub-automation script of a child node in the current domain node, retrieving the third health check result of the sub-automation script of the child node of the executed domain node corresponding to the current domain node; and executing the sub-automation script of the child node in the current domain node based on the association relationship between the child node in the current domain node and the child node in the executed domain node, and based on the third health check result of the sub-automation script of the child node of the executed domain node.

[0009] According to embodiments of this disclosure, the execution of a sub-automation script of a child node in the current domain node, based on the association relationship between the child nodes in the current domain node and the child nodes in the executed domain node, and based on the third health check result of the sub-automation script of the child node of the executed domain node, includes: if it is determined that the third health check result of the sub-automation script of the child node in the executed domain node associated with the child node in the current domain node is abnormal, executing the sub-automation script of the next child node in the current domain node; if it is determined that the third health check result of the sub-automation script of the child node in the executed domain node associated with the child node in the current domain node is normal, executing the sub-automation script of the child node in the current domain node.

[0010] According to embodiments of this disclosure, the method further includes: determining child nodes in each domain node based on each domain node; writing sub-automation scripts for the child nodes in each domain node based on the standard operating status indicators of the child nodes; arranging the sub-automation scripts for the child nodes in each domain node according to preset arrangement rules to obtain the arranged sub-automation scripts for the child nodes; and performing set processing on the arranged sub-automation scripts for the child nodes in each domain node based on the unidirectional hierarchical dependency relationship between domain nodes to obtain an automation script set.

[0011] Another aspect of this disclosure provides a fault root cause determination apparatus, comprising: an execution module, configured to, upon detecting a fault, execute sub-automation scripts of child nodes in each domain node of an automation script set to obtain a first health check result for each child node, wherein the domain nodes of the automation script set have a unidirectional hierarchical dependency relationship; a first determination module, configured to determine an abnormal child node corresponding to a domain node based on the first health check result of each child node; a second determination module, configured to determine at least one abnormal domain node based on the correspondence between the abnormal child node and the domain node; and a third determination module, configured to determine the fault root cause causing the fault from at least one abnormal domain node based on the unidirectional hierarchical dependency relationship between the domain nodes of the automation script set.

[0012] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the above-described fault root cause determination method.

[0013] Another aspect of this disclosure provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the aforementioned fault root cause determination method.

[0014] Another aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for determining the root cause of a fault.

[0015] According to embodiments of this disclosure, based on the unidirectional hierarchical dependency relationship between nodes in each domain of the automated script set, the sub-automated scripts of the child nodes corresponding to each domain node in the automated script set are executed sequentially to obtain the abnormal child nodes of each domain node. Based on the correspondence between the abnormal child nodes and the domain nodes, at least one abnormal domain node is determined. Since there is a unidirectional hierarchical dependency relationship between the domain nodes, the root cause of the fault can be determined from at least one abnormal domain node. This solves at least some of the technical problems that currently require maintenance personnel in each domain to investigate faults in each domain before summarizing the problems and finally judging the root cause of the problem, such as poor timeliness of manual processing, inability to accurately report the root cause of the problem, delay in determining the root cause of the problem, and impact on business continuity. This achieves the technical effects of improving the timeliness of determining the root cause of the fault, accurately reporting the root cause of the fault, narrowing the scope of fault investigation, and targeting the analysis of the root cause of the fault. Attached Figure Description

[0016] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0017] Figure 1 This diagram illustrates an application scenario of the fault root cause determination method and apparatus according to embodiments of the present disclosure.

[0018] Figure 2 A schematic diagram illustrating various domain nodes and sub-nodes according to embodiments of the present disclosure is provided.

[0019] Figure 3 A flowchart illustrating a fault determination method according to an embodiment of the present disclosure is shown schematically.

[0020] Figure 4 The diagram illustrates the unidirectional hierarchical dependency relationships between domain nodes according to embodiments of the present disclosure and the child nodes included in each domain node.

[0021] Figure 5 A schematic block diagram of a fault root cause determination apparatus according to an embodiment of the present disclosure is shown; and

[0022] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a fault root cause determination method according to an embodiment of the present disclosure. Detailed Implementation

[0023] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0024] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0025] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0026] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0027] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0028] In the technical solution disclosed herein, the acquisition, collection, storage, use, processing, transmission, provision, disclosure, and application of data all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0029] Embodiments of this disclosure provide a method for determining the root cause of a fault, comprising: upon detecting a fault, executing sub-automation scripts of child nodes in each domain node of an automation script set to obtain a first health check result for each child node, wherein the domain nodes of the automation script set have a unidirectional hierarchical dependency relationship; determining abnormal child nodes corresponding to the domain nodes based on the first health check result of each child node; determining at least one abnormal domain node based on the correspondence between the abnormal child nodes and the domain nodes; and determining the fault root that caused the fault from the at least one abnormal domain node based on the unidirectional hierarchical dependency relationship between the domain nodes of the automation script set.

[0030] Figure 1 The illustration shows an application scenario of the fault root cause determination method and apparatus according to embodiments of the present disclosure.

[0031] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as a medium for providing a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0032] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0033] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0034] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0035] It should be noted that the fault root cause determination method provided in this disclosure embodiment can generally be executed by server 105. Correspondingly, the fault root cause determination device provided in this disclosure embodiment can generally be located in server 105. The fault root cause determination method provided in this disclosure embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the fault root cause determination device provided in this disclosure embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] Figure 2 The diagram illustrates various domain nodes and sub-nodes according to embodiments of the present disclosure.

[0038] like Figure 2 As shown, according to embodiments of this disclosure, the system may include multiple data centers, and each data center 200 may include multiple domain nodes, namely, domain node 1, domain node 2, domain node 3, ..., domain node n. Each domain node may include multiple child nodes. For example, domain node 1 may include child node 11, child node 12, child node 13, ...; domain node 2 may include child node 21, child node 22, child node 23, ...; ...; domain node n may include child node n1, child node n2, child node n3, ... , where n is a positive integer greater than or equal to 1.

[0039] Based on the network functions of each domain node, each domain node implements different functions and provides a specific service to its next adjacent domain node, thus establishing a unidirectional hierarchical dependency relationship between domain nodes. For example, there is a unidirectional hierarchical dependency relationship between domain node 1 and domain node n. Domain node 1 can be a data center, providing a specific service to domain node 2; domain node 2 can be a network, providing a specific service to domain node 3; domain node 3 can be an operating system, and so on. There are associations between the child nodes of each domain node and among the child nodes of each domain node.

[0040] It should be noted that, Figure 1 In the fault root cause determination method executed by server 105 or a server or server cluster different from server 105 but capable of communicating with terminal devices 101, 102, 103 and / or server 105, the domain node and child nodes can be... Figure 2 The domain nodes and child nodes shown are illustrated.

[0041] Figure 3 A flowchart illustrating a fault determination method according to an embodiment of the present disclosure is shown schematically.

[0042] like Figure 3 As shown, the method 300 may include operations S310 to S340.

[0043] When operating S310, if a fault is detected, the sub-automation scripts of the child nodes in each domain node of the automation script set are executed to obtain the first health check result of each child node. There is a unidirectional hierarchical dependency relationship between the domain nodes of the automation script set.

[0044] According to embodiments of this disclosure, the automated script set can be a collection of programs for detecting the root causes of failures. Domain nodes can be nodes in various domains of a data center, and child nodes within a domain node can be specific elements included in that domain node.

[0045] According to embodiments of this disclosure, the sub-automation scripts of child nodes in each domain node of the automation script set are executed sequentially based on the unidirectional hierarchical dependencies between the domain nodes of the automation script set. The unidirectional hierarchical dependencies between the domain nodes of the automation script set and the child nodes included in each domain node can be exemplified as follows: Figure 4 As shown.

[0046] For example, Figure 4 This illustration schematically depicts the unidirectional hierarchical dependencies between domain nodes according to embodiments of the present disclosure, and a diagram illustrating the child nodes included in each domain node. For example... Figure 4As shown in the diagram 400, each domain node can include a data center 401, a network 402, an operating system 403, a database 404, middleware 405, and applications 406. Each domain node can include multiple sub-nodes. For example, data center 401 (domain node 1) can include temperature and humidity, cold aisle, rack power supply, voltage / current, etc.; network 402 (domain node 2) can include firewall status, switches, routers, network protocols, etc.; operating system 403 (domain node 3) can include CPU / ME (CPU management engine), M / DISK (disk), DNS (Domain Name System), NTP (Network Time Protocol), cloud management, etc.; database 404 (domain node 4) can include Master / slave (master-slave mode), tablespaces, locks, SQL (Structured Query Language), etc. Middleware 405 (domain node 5) may include Tomact (application server), Wenlogic (logic), heap memory, JVM (Java Virtual Machine), ... Application 406 (domain node 6) may include application processes, SaaS, logs, business green lights, ...

[0047] According to embodiments of this disclosure, based on the network functions of each domain node, each domain node provides a specific service to its next adjacent domain node. This can also be understood as the current domain node using the services provided by the domain nodes preceding it. For example, data center 401 can provide services to network 402, and both data center 401 and network 402 simultaneously provide services to operating system 403. Similarly, domain nodes 401-405 preceding application 406 provide services to application 406, thus reflecting a unidirectional hierarchical dependency relationship between domain nodes. Each domain node's child nodes are associated with each other, and the child nodes of each domain node are also associated with each other.

[0048] According to embodiments of this disclosure, in the event of a detected fault, an automated script set 400' can be executed, such that each domain node 401-406 in the automated script set executes the sub-automated scripts of its child nodes according to the unidirectional hierarchical dependency relationship between the domain nodes.

[0049] It should be noted that, Figure 4 The illustrated domain nodes and child nodes are exemplary and do not represent all domain nodes and child nodes.

[0050] According to embodiments of this disclosure, the first health check result can be used to output the health check result of each child node in each domain node by executing the sub-automation script of each child node in each domain node.

[0051] In operation S320, based on the first health check result of each child node, the abnormal child node corresponding to the domain node is determined.

[0052] In operation S330, at least one abnormal domain node is determined based on the correspondence between abnormal child nodes and domain nodes.

[0053] According to embodiments of this disclosure, an abnormal child node can be a child node whose first health check result is abnormal in each domain node.

[0054] According to embodiments of this disclosure, each domain node and its child nodes have a node identifier, and the node identifier of each domain node has a corresponding relationship with the child node identifier of its child node. Based on the correspondence between the domain node and its child nodes, at least one domain node corresponding to an abnormal child node can be determined.

[0055] According to embodiments of this disclosure, the domain node corresponding to an abnormal child node can be called an abnormal domain node. At least one abnormal domain node is determined based on the correspondence between abnormal child nodes and domain nodes. Each abnormal domain node may include multiple child nodes in abnormal states.

[0056] In operation S340, based on the unidirectional hierarchical dependencies between domain nodes of the automated script set, the root cause of the failure is determined from at least one abnormal domain node.

[0057] According to embodiments of this disclosure, the unidirectional hierarchical dependency between domain nodes can also be such that when a child node of a domain node providing services experiences an abnormal state, it will affect the health status of the child node of the domain node being provided with services that is associated with the child node of the domain node providing services.

[0058] According to embodiments of this disclosure, for example, if a child node in a network is associated with a child node in a data center, and the first health check result of the child node in the data center is abnormal, it will affect the health status of a child node in the network nodes associated with the child node in the data center. In this case, the abnormal domain nodes can be the data center and the network. Due to the unidirectional hierarchical dependency between domains, it can be determined that the abnormal state of a child node in the network is caused by the abnormal state of a child node in the data center; therefore, the root cause of the fault is considered to be the child node in the data center.

[0059] According to embodiments of this disclosure, based on the unidirectional hierarchical dependency relationship between nodes in each domain of the automated script set, the sub-automated scripts of the child nodes corresponding to each domain node in the automated script set are executed sequentially to obtain the abnormal child nodes of each domain node. Based on the correspondence between the abnormal child nodes and the domain nodes, at least one abnormal domain node is determined. Since there is a unidirectional hierarchical dependency relationship between the domain nodes, the root cause of the fault can be determined from at least one abnormal domain node. This solves at least some of the technical problems that currently require maintenance personnel in each domain to investigate faults in each domain before summarizing the problems and finally judging the root cause of the problem, such as poor timeliness of manual processing, inability to accurately report the root cause of the problem, delay in determining the root cause of the problem, and impact on business continuity. This achieves the technical effects of improving the timeliness of determining the root cause of the fault, accurately reporting the root cause of the fault, narrowing the scope of fault investigation, and targeting the analysis of the root cause of the fault.

[0060] According to embodiments of this disclosure, executing sub-automation scripts of child nodes in each domain node of an automation script set includes: sequentially executing sub-automation scripts of child nodes in each domain node based on unidirectional hierarchical dependencies between domain nodes.

[0061] According to embodiments of this disclosure, when a fault is detected, in response to a request to execute an automated script set, the sub-automated scripts of the child nodes of each domain node included in the automated script set can be executed. When executing the sub-automated scripts of each domain node's child nodes, the unidirectional hierarchical dependencies between domain nodes can be utilized for sequential execution. After executing the sub-automated scripts of the first domain node's child nodes, the sub-automated scripts of the second domain node's child nodes are executed, and so on, until the sub-automated scripts of the last domain node's child nodes are executed, outputting the execution result corresponding to each domain node.

[0062] According to embodiments of this disclosure, sub-automation scripts of child nodes in each domain node are executed sequentially, including:

[0063] For executing sub-automation scripts of child nodes in the current domain node, if the sub-automation script of the current child node is being executed, the second health check result of the sub-automation scripts of the already executed child nodes in the current domain node is retrieved; based on the association relationship between child nodes in the current domain node, the second health check result of the sub-automation scripts of the already executed child nodes associated with the current child node is determined; if the second health check result of the sub-automation script of the already executed child node associated with the current child node is determined to be abnormal, the sub-automation script of the next child node of the current child node is executed; if the second health check result of the sub-automation script of the already executed child node associated with the current child node is determined to be normal, the sub-automation script of the current child node is executed.

[0064] According to embodiments of this disclosure, the second health check result can be determined by executing the sub-automation script of the child node of the domain node to determine whether the running status parameters of the child node meet the preset running status parameters set by the sub-automation script of the child node. If the preset running status parameters are met, the second health check result is normal; otherwise, the second health check result is abnormal.

[0065] According to an embodiment of this disclosure, if the current child node is the starting child node of the domain node, the sub-automation result of the current child node is directly executed to obtain the second health check result of the starting child node.

[0066] According to an embodiment of this disclosure, if the current child node is a non-starting child node of a domain node, when executing the sub-automation script of the current child node, the second health check result of the sub-automation script of the child node that has been executed in the domain node is first retrieved.

[0067] According to embodiments of this disclosure, there are associations between each child node in a domain node. It can be determined which child nodes of executed sub-automation scripts in the domain node are associated with the current child node, so as to determine the second health check result of the child nodes of executed sub-automation scripts associated with the current child node.

[0068] According to embodiments of this disclosure, if the second health check result of the child node associated with the current child node that has executed an automated script is abnormal, it indicates that the second health check result of the current child node has been affected by the second health check result of the associated child node. If the current child node is an abnormal child node, then the child automated script of the current child node does not need to be executed, but the child automated script of the next child node of the current child node is executed, and the second health check result of the next child node is output.

[0069] According to embodiments of this disclosure, the process of executing the sub-automation script of the next child node of the current child node is similar to the process of executing the sub-automation script of the current child node described above, and will not be repeated here.

[0070] According to embodiments of this disclosure, if the second health check result of the child node associated with the current child node that has executed an automated script is normal, it means that the second health check result of the current child node will not be affected by the second health check result of the associated child node. If it is necessary to determine the third health check result of the child node, then the sub-automated script of the current child node is executed, and the second health check result of the current child node is output.

[0071] According to embodiments of this disclosure, combined with Figure 4As shown, for example, a server room and its child nodes are used for illustration. When a fault is detected, the automated script set component is triggered, executing the child automated scripts of the server room's child nodes. This can involve executing the initial child node's temperature and humidity child automated script to obtain the second health check results for temperature and humidity. When executing the child automated scripts of the cold aisle child nodes, the second health check results of the child nodes associated with the cold aisle and whose executed child automated scripts are determined first. If the second health check results of the associated child nodes are abnormal, the cold aisle child node's child automated script is not executed, and the cold aisle's second health check result is directly output as abnormal. If the second health check results of the associated child nodes are normal, the cold aisle child node's child automated script is executed, and the cold aisle's second health check result is output. When executing the child automated scripts of the remaining child nodes in the server room, the second health check results of the child nodes associated with the remaining child nodes and whose executed child automated scripts are executed are similarly determined, and the corresponding child node's child automated script is executed, outputting the corresponding second health check results. The specific execution process is similar to the above and will not be repeated here.

[0072] It should be noted that since the data center domain node executed above is domain node 1, which is equivalent to the first layer of a one-way hierarchical dependency relationship and provides services to subsequent domain nodes, the process of executing the sub-automation scripts of the data center domain node's child nodes only considers the relationships between child nodes within the data center domain node. However, when executing the sub-automation scripts of each subsequent domain node's child nodes, it is necessary to consider not only the relationships between child nodes within the domain node, but also the relationships between the child nodes of different domain nodes. This will be explained in detail later.

[0073] According to embodiments of this disclosure, the sub-automation scripts of child nodes in each domain node are executed sequentially, further comprising: for executing the sub-automation scripts of child nodes between each domain node, when executing the sub-automation script of a child node in the current domain node, retrieving the third health check result of the sub-automation script of the child node of the executed domain node corresponding to the current domain node; and executing the sub-automation script of the child node in the current domain node based on the association relationship between the child node in the current domain node and the child node in the executed domain node, and based on the third health check result of the sub-automation script of the child node of the executed domain node.

[0074] According to embodiments of this disclosure, the third health check result can be determined by executing the sub-automation script of each sub-node in the domain node based on the association relationship between the sub-nodes of the domain node and the association relationship between the sub-nodes in the domain node, and by determining whether the running status parameters of each sub-node in the domain node meet the preset running status parameters set by the sub-automation script of the sub-node. If the preset running status parameters are met, the third health check result is normal; otherwise, the third health check result is abnormal.

[0075] According to the embodiments of this disclosure, if the current domain node is the starting domain node in the unidirectional hierarchical dependency relationship of the domain node, the process of executing the sub-automation script of the child node of the starting domain node only considers the mutual relationship between the child nodes in the domain node. The execution process is similar to the above-described sub-automation script execution process for the child nodes in each domain node, and will not be repeated here.

[0076] According to embodiments of this disclosure, if the current domain node is a non-starting domain node in a unidirectional hierarchical dependency relationship of domain nodes, the third health check results of the child nodes of the previously executed domain nodes are first retrieved. Then, it is determined which child nodes of the previously executed domain nodes are associated with the child nodes of the current domain node, in order to determine the third health check results of the sub-automation scripts of the child nodes of the previously executed domain nodes associated with the child nodes of the current domain node.

[0077] According to embodiments of this disclosure, a third health check result is used to determine whether to execute the sub-automation script of the child node in the current domain node, based on the determined sub-automation script of the child node of the domain node associated with the child node of the current domain node.

[0078] According to embodiments of this disclosure, combined with Figure 4 For example, when a fault is detected, after executing the sub-automation scripts of the child nodes of the data center domain node, the third health check results of each child node in the data center domain node are output. When executing the sub-automation scripts of the child nodes of the network domain node, it is necessary to first retrieve the third health check results of each child node in the domain node in front of the network (i.e., the data center) and put them into the network domain node. When executing the sub-automation scripts of each child node in the network domain node, it is necessary to determine the child nodes of the data center domain node that are associated with each child node that have already been executed. Based on the third health check results of the child nodes of the associated data center domain nodes that have already been executed, it is determined whether to execute the sub-automation scripts of the associated child nodes in the network domain node.

[0079] According to embodiments of this disclosure, when executing a sub-automation script of a child node of a domain node behind a network, it is necessary to simultaneously retrieve the third health check results of the child nodes of the network and the child nodes of the data center in front of the domain node to determine whether to execute the sub-automation script of the associated child node in the domain node.

[0080] According to embodiments of this disclosure, based on the association between child nodes in the current domain node and child nodes in the executed domain node, and based on the third health check result of the child automation script of the child node of the executed domain node, the execution of the child automation script of the current domain node includes: if it is determined that the third health check result of the child automation script of the child node of the executed domain node associated with the child node in the current domain node is abnormal, executing the child automation script of the next child node of the current domain node; if it is determined that the third health check result of the child automation script of the child node of the executed domain node associated with the child node in the current domain node is normal, executing the child automation script of the child node in the current domain node.

[0081] According to embodiments of this disclosure, sub-automation scripts of child nodes in the current domain node can be executed sequentially. When it is determined that the third health check result of the child node in the executed domain node associated with the child node in the current domain node is abnormal, it indicates that the third health check status of the child node in the current domain node is affected by the third health check result of the child node in the executed domain node associated with it. If the child node in the current domain node is an abnormal child node, then the sub-automation script of that child node does not need to be executed, and the sub-automation script of the next child node in the current domain node is executed.

[0082] According to an embodiment of this disclosure, if the third health check result of the child node in the executed domain node associated with the child node in the current domain node is normal, it is necessary to determine the third health check result of the child node in the current domain node. Then, the sub-automation script of the child node in the current domain node is executed, and the third health check result of the child node is output.

[0083] It should be noted that when executing a child automation script of a child node that is not part of the starting domain node, the health check results of not only the executed child automation scripts of the domain node related to that child node in the non-starting domain node, but also the health check results of the executed child automation scripts of the child node associated with that child node in the non-starting domain node must be considered. If at least one of these two sets of health check results for its associated child node is abnormal, then that child node of the non-starting domain node is considered an abnormal child node.

[0084] According to embodiments of this disclosure, before detecting a fault, the process may further include: determining child nodes in each domain node based on each domain node; writing sub-automation scripts for the child nodes in each domain node based on standard operational status indicators of the child nodes; arranging the sub-automation scripts for the child nodes in each domain node according to preset orchestration rules to obtain orchestrated sub-automation scripts for the child nodes; and performing set processing on the orchestrated sub-automation scripts for the child nodes in each domain node based on the unidirectional hierarchical dependency relationship between domain nodes to obtain an automation script set.

[0085] According to embodiments of this disclosure, multiple elements included in each domain node can be determined based on the attributes of the domain nodes, and these multiple elements can be designated as child nodes of the domain node. For example, Figure 4 The middle domain node is a node of the data center. Depending on the attributes of the data center, it can include elements such as temperature and humidity, cold aisle, and rack power supply as child nodes of the data center.

[0086] According to embodiments of this disclosure, the standard operational health metric can be the parameter range for each child node in a domain node during normal operation. The sub-automation scripts for each child node in a domain node can be programs that detect whether the operational health of each child node is normal.

[0087] According to embodiments of this disclosure, the preset orchestration rules can be used to orchestrate the sub-automation scripts of child nodes based on their importance in each domain node; or they can be used to orchestrate the sub-automation scripts of each child node based on historical experience. Specific preset orchestration rules are not limited here.

[0088] According to embodiments of this disclosure, the orchestrated child scripts of each domain node can be collected and encapsulated into an automation script set. During the collection process, a one-way hierarchical dependency relationship between each domain node is determined, and the automation scripts of the child nodes in each domain node are unidirectionally collected according to this one-way hierarchical dependency relationship to form an automation script set.

[0089] Based on the above-described method for determining the root cause of a fault, this disclosure also provides a device for determining the root cause of a fault. The following will be combined with... Figure 5 The device is described in detail.

[0090] Figure 5 A schematic block diagram of a fault root cause determination apparatus according to an embodiment of the present disclosure is shown.

[0091] like Figure 5 As shown, the fault root cause determination device 500 of this embodiment may include: an execution module 510, a first determination module 520, a second determination module 530 and a third determination module 540.

[0092] The execution module 510 is used to execute the sub-automation scripts of the child nodes in each domain node of the automation script set when a fault is detected, and to obtain the first health check result of each child node, wherein the domain nodes of the automation script set have a unidirectional hierarchical dependency relationship.

[0093] The first determination module 520 is used to determine the abnormal child node corresponding to the domain node based on the first health check result of each child node.

[0094] The second determining module 530 is used to determine at least one abnormal domain node based on the correspondence between abnormal child nodes and domain nodes.

[0095] The third determination module 540 is used to determine the root cause of the failure from at least one abnormal domain node based on the unidirectional hierarchical dependency relationship between domain nodes based on the automated script set.

[0096] According to embodiments of this disclosure, based on the unidirectional hierarchical dependency relationship between nodes in each domain of the automated script set, the sub-automated scripts of the child nodes corresponding to each domain node in the automated script set are executed sequentially to obtain the abnormal child nodes of each domain node. Based on the correspondence between the abnormal child nodes and the domain nodes, at least one abnormal domain node is determined. Since there is a unidirectional hierarchical dependency relationship between the domain nodes, the root cause of the fault can be determined from at least one abnormal domain node. This solves at least some of the technical problems that currently require maintenance personnel in each domain to investigate faults in each domain before summarizing the problems and finally judging the root cause of the problem, such as poor timeliness of manual processing, inability to accurately report the root cause of the problem, delay in determining the root cause of the problem, and impact on business continuity. This achieves the technical effects of improving the timeliness of determining the root cause of the fault, accurately reporting the root cause of the fault, narrowing the scope of fault investigation, and targeting the analysis of the root cause of the fault.

[0097] According to embodiments of this disclosure, the execution module may include an execution submodule.

[0098] The execution submodule is used to execute the sub-automation scripts of the child nodes in each domain node in turn, based on the unidirectional hierarchical dependency relationship between domain nodes.

[0099] According to embodiments of this disclosure, the execution submodule may include: a first invocation unit, a determination unit, a first execution unit, and a second execution unit. This is for executing sub-automation scripts of child nodes within the current domain node.

[0100] The first retrieval unit is used to retrieve the second health check result of the child automation script that has been executed in the current domain node when executing the child automation script of the current child node.

[0101] The determining unit is used to determine the second health check result of the sub-automation script of the executed sub-node associated with the current sub-node based on the association relationship between the sub-nodes in the current domain node.

[0102] The first execution unit is used to execute the sub-automation script of the next child node of the current child node if the second health check result of the sub-automation script of the executed child node associated with the current child node is abnormal.

[0103] The second execution unit is used to execute the sub-automation script of the current child node if the second health check result of the sub-automation script of the executed child node associated with the current child node is normal.

[0104] According to embodiments of this disclosure, the execution submodule may further include a second fetching unit and a third execution unit.

[0105] For each domain node, execute the sub-automation scripts for its child nodes.

[0106] The second retrieval unit is used to retrieve the third health check result of the sub-automation script of the child node of the currently executed domain node when executing the sub-automation script of the child node in the current domain node.

[0107] The third execution unit is used to execute the sub-automation scripts of the child nodes in the current domain node based on the association relationship between the child nodes in the current domain node and the child nodes in the executed domain node, and based on the third health check result of the sub-automation scripts of the child nodes of the executed domain node.

[0108] According to embodiments of this disclosure, the third execution unit may include: a first execution subunit and a second execution subunit.

[0109] The first execution subunit is configured to execute the sub-automation script of the next child node of the current domain node if the third health check result of the sub-automation script of the child node of the executed domain node associated with the child node in the current domain node is abnormal.

[0110] The second execution subunit is used to execute the sub-automation script of the child node in the current domain node if the third health check result of the sub-automation script of the child node in the executed domain node associated with the child node in the current domain node is normal.

[0111] According to embodiments of this disclosure, the device 500 may further include: a fourth determining module, a writing module, an arrangement module, and a processing module.

[0112] The fourth determination module is used to determine the child nodes in each domain node based on each domain node.

[0113] Write modules to generate sub-automation scripts for child nodes in each domain node based on the standard operational status indicators of the child nodes.

[0114] The orchestration module is used to orchestrate the sub-automation scripts of the child nodes in each domain node according to preset orchestration rules, so as to obtain the orchestrated sub-automation scripts of the child nodes.

[0115] The processing module is used to process the sub-automation scripts of the orchestrated child nodes in each domain node based on the unidirectional hierarchical dependency relationship between domain nodes, and obtain an automation script set.

[0116] According to embodiments of this disclosure, any plurality of modules among execution module 510, first determining module 520, second determining module 530, and third determining module 540 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of execution module 510, first determining module 520, second determining module 530, and third determining module 540 may be at least partially implemented as hardware circuitry, such as field-programmable gate array (FPGA), programmable logic array (PLA), system-on-a-chip, system-on-a-substrate, system-on-package, application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these methods. Alternatively, at least one of the execution module 510, the first determining module 520, the second determining module 530, and the third determining module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0117] Figure 6 A block diagram schematically illustrates an electronic device suitable for implementing a fault root cause determination method according to an embodiment of the present disclosure.

[0118] like Figure 6As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0119] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0120] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0121] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0122] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 602 and / or RAM 603 and / or one or more memories other than ROM 602 and RAM 603 described above.

[0123] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the root cause determination method provided in the embodiments of this disclosure.

[0124] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0125] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0126] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0127] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0128] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0129] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0130] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A method for determining a fault root cause, comprising: in a case where a fault is detected, executing a sub-automation script of a child node in each domain node of a set of automation scripts to obtain a first health check result of each of the child nodes, wherein the domain nodes of the set of automation scripts have a unidirectional hierarchical dependency relationship therebetween; determining an abnormal child node corresponding to the domain node according to the first health check result of each of the child nodes; determining at least one abnormal domain node according to a corresponding relationship between the abnormal child node and the domain node; determining a fault root cause causing the fault to occur from the at least one abnormal domain node based on the unidirectional hierarchical dependency relationship between the domain nodes of the set of automation scripts, wherein the execution of the sub-automation script of the child node in each domain node of the set of automation scripts comprises: in execution of a sub-automation script of a current child node of a non-starting domain node, determining both a second health check result of a sub-automation script of an executed child node associated with the current child node in the non-starting domain node and a third health check result of a sub-automation script of a child node of an executed domain node associated with the current child node in the non-starting domain node, determining that the current child node of the non-starting domain node is an abnormal child node when at least one of the first health check result, the second health check result and the third health check result is abnormal, and in a case where it is determined that the current child node of the non-starting domain node is an abnormal child node, not executing the sub-automation script of the current child node, but executing a sub-automation script of a next child node of the current child node, and outputting a second health check result of the next child node.

2. The method of claim 1, wherein, the execution of the sub-automation script of the child node in each domain node of the set of automation scripts comprises: based on the unidirectional hierarchical dependency relationship between the domain nodes, executing the sub-automation script of the child node in each of the domain nodes in sequence.

3. The method of claim 2, wherein, the execution of the sub-automation script of the child node in each of the domain nodes in sequence comprises: for execution of a sub-automation script of a child node in a current domain node, in execution of the sub-automation script of the current child node, calling a second health check result of a sub-automation script of the child node executed in the current domain node; determining the second health check result of the sub-automation script of the child node executed in association with the current child node according to an association relationship between the child nodes in the current domain node; in a case where it is determined that the second health check result of the sub-automation script of the child node executed in association with the current child node is normal, executing the sub-automation script of the current child node.

4. The method of claim 3, wherein, the execution of the sub-automation script of the child node in each of the domain nodes in sequence further comprises: for execution of a sub-automation script of a child node between each of the domain nodes, In the case of executing the child automation script of the child node in the current domain node, the third health check result of the child automation script of the child node of the executed domain node corresponding to the current domain node is invoked; According to the association relationship between the child node in the current domain node and the child node in the executed domain node, the third health check result of the child automation script of the child node of the executed domain node is executed based on the child automation script of the child node in the current domain node.

5. The method of claim 4, wherein, According to the association relationship between the child node in the current domain node and the child node in the executed domain node, the third health check result of the child automation script of the child node of the executed domain node is executed based on the child automation script of the child node in the current domain node. In the case of determining that the third health check result of the child automation script of the child node of the executed domain node associated with the child node in the current domain node is normal, the child automation script of the child node in the current domain node is executed.

6. The method of claim 1, further comprising: According to each of the domain nodes, determining the child nodes in each of the domain nodes; According to the standard health indicators of the child nodes, compiling the child automation scripts of the child nodes in each of the domain nodes; According to a preset arrangement rule, arranging the child automation scripts of the child nodes in each of the domain nodes to obtain the arranged child automation scripts of the child nodes; Based on the one-way hierarchical dependency relationship between the domain nodes, performing set processing on the arranged child automation scripts of the child nodes in each of the domain nodes to obtain the automation script set.

7. A fault root cause determination apparatus, comprising: An execution module configured to execute the child automation script of the child node in each domain node of an automation script set to obtain a first health check result of each of the child nodes in the case of detecting that a fault occurs, wherein the domain nodes of the automation script set have a one-way hierarchical dependency relationship therebetween; A first determination module configured to determine an abnormal child node corresponding to the domain node according to the first health check result of each of the child nodes; A second determination module configured to determine at least one abnormal domain node according to the corresponding relationship between the abnormal child node and the domain node; A third determination module configured to determine a fault root cause causing the fault to occur from at least one of the abnormal domain nodes based on the one-way hierarchical dependency relationship between the domain nodes of the automation script set, The execution module is further configured to perform the following steps: In the case of executing the child automation script of the current child node of the non-starting domain node, determining both the second health check result of the child automation script of the executed child node associated with the current child node in the non-starting domain node and the third health check result of the child automation script of the child node of the executed domain node associated with the current child node in the non-starting domain node, determine that the current child node of the non-starting domain node is an abnormal child node when at least one of the first health check result, the second health check result and the third health check result is abnormal, and in a case where it is determined that the current child node of the non-starting domain node is an abnormal child node, not execute a child automation script of the current child node, but execute a child automation script of a next child node of the current child node, and output a second health check result of the next child node. 8.An electronic device, comprising: one or more processors; memory storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-6. 9.A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-6. 10.A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.

Images