Hypothesis-Driven Diagnosis of Network Systems

Abstract

Hypothesis-driven diagnosis of a network system. An example method includes: obtaining, by one or more processors, data indicating resource dependencies between multiple resources in a network and event dependencies between multiple network events and one or more of the multiple resources; generating a Bayesian model based on the resource types of the multiple resources and the event types of the multiple network events; receiving an indication of a fault in the network; collecting fault data, and generating, based on the Bayesian model and the fault data, multiple root cause hypotheses for the fault; ranking the multiple root cause hypotheses based on corresponding root cause probabilities associated with the multiple root cause hypotheses; and outputting the ranked multiple root cause hypotheses.

Description

Technical Field

[0001] This disclosure relates to computer networks and, more particularly, to using hypotheses to guide diagnostics for diagnosing network problems. Background Art

[0002] A computer network is a collection of interconnected computing devices that can exchange data and share resources. Various devices operate to facilitate communication between computing devices. For example, a computer network can include routers, switches, gateways, firewalls, and various other devices to provide and facilitate network communication. These devices can provide information such as current status, error events, and resource usage information to applications as an aid in analyzing network performance and for analyzing errors in the system. As an example, such information can be useful for determining the root cause of an error in the system.

[0003] The proliferation of the number of Internet of Things (IoT) devices and the need for large data centers to host cloud and web applications have led to a need for mechanisms to effectively analyze, evaluate, and / or control complex, heterogeneous, and distributed networks. Network systems often require distributed and fast diagnostic solution techniques to analyze relevant events. In complex networks, a failure in the underlying layer of the network can cause a large number of higher layer service failures, which may or may not be directly connected to the failed component. Summary of the Invention

[0004] This disclosure describes techniques for generating and evaluating root cause hypotheses that attempt to explain the cause of a failure or adverse event in a network. In some examples, a system (such as a network health monitor or controller) can be configured with or construct a Bayesian model based on resources and events defined for the network. The Bayesian model can be defined using resource types and event types rather than using specific instances of resources and events. When a network failure occurs, the system will apply the Bayesian model to provide a set of potential root cause hypotheses that explain the failure data received for the failure and the probability for each hypothesis in the set. The system can use detectors defined for the resources to refute one or more hypotheses in the set. Then, in some cases, the system can provide the user (such as a network administrator) with a ranked set of the remaining hypotheses that have not been refuted by the detectors, which the user can use to perform a manual investigation and perform improvement measures to resolve the failure or adverse event in the network.

[0005] In one example, the present disclosure describes a method that includes: obtaining, by one or more processors, data indicative of: resource dependencies among a plurality of resources in a network, and event dependencies between a plurality of network events and one or more of the plurality of resources; generating a Bayesian model based on the resource types of the plurality of resources and the event types of the plurality of network events; receiving an indication of a fault in the network; collecting fault data, and generating, based on the Bayesian model and the fault data, a plurality of root cause hypotheses for the fault; ranking the plurality of root cause hypotheses based on corresponding root cause probabilities associated with the plurality of root cause hypotheses; and outputting the ranked plurality of root cause hypotheses.

[0006] In another example, the present disclosure describes an apparatus that includes a memory and processing circuitry configured to: obtain data indicative of: resource dependencies among a plurality of resources in a network, and event dependencies between a plurality of network events and one or more of the plurality of resources; generate a Bayesian model based on the resource types of the plurality of resources and the event types of the plurality of network events; receive an indication of a fault in the network; collect fault data, and generate, based on the Bayesian model and the fault data, a plurality of root cause hypotheses for the fault; rank the plurality of root cause hypotheses based on corresponding root cause probabilities associated with the plurality of root cause hypotheses; and output the ranked plurality of root cause hypotheses.

[0007] In a further example, the present disclosure describes a computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: obtain data indicative of: resource dependencies among a plurality of resources in a network, and event dependencies between a plurality of network events and one or more of the plurality of resources; generate a Bayesian model based on the resource types of the plurality of resources and the event types of the plurality of network events; receive an indication of a fault in the network; collect fault data, and generate, based on the Bayesian model and the fault data, a plurality of root cause hypotheses for the fault; rank the plurality of root cause hypotheses based on corresponding root cause probabilities associated with the plurality of root cause hypotheses; and output the ranked plurality of root cause hypotheses.

[0008] Details of one or more techniques of the present disclosure are set forth in the accompanying drawings and the following description. Other features, objects, and advantages of the invention will become apparent from the specification, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram illustrating an example network configured to perform hypothesis-driven diagnosis in accordance with one or more aspects of the present disclosure.

[0010] Figure 2is a block diagram illustrating an example network health monitor according to one or more aspects of the present disclosure, which can be used to perform root cause analysis.

[0011] Figure 3 is a conceptual diagram illustrating an example portion of a resource dependency model according to one or more aspects of the present disclosure.

[0012] Figure 4 is a conceptual diagram illustrating an example dependency graph representing a resource dependency model according to one or more aspects of the present disclosure.

[0013] Figure 5 illustrates an example Bayesian model according to one or more aspects of the present disclosure, which can be generated by a Bayesian network system based on Figure 4 the example resource dependencies illustrated in the resource dependency graph of.

[0014] Figure 6 is a flowchart according to one or more aspects of the present disclosure, which illustrates operations performed by an example network health monitor to generate potential root cause hypotheses.

[0015] Figure 7A and Figure 7B illustrates the software definition of resource types and detector attributes.

[0016] Like reference numerals throughout the text and the drawings indicate like elements. Detailed Description

[0017] Root cause analysis (RCA), also known as fault diagnosis, is a process of identifying the initiating condition or event that triggers a network component failure from a set of possible candidate events / conditions that are generated or exist within a discrete time window. For an operator, RCA can be an important task to maintain the normal operation of the network. Various techniques that can be used to perform RCA include model traversal techniques and dependency graph techniques.

[0018] Model traversal techniques can use an object model to determine fault propagation. A network can be represented using various components and the relationships between the components. Based on this model representing the network, fault dependencies can be inferred and used to identify the root cause of the problem. Model traversal techniques do not directly specify fault dependencies but derive fault dependencies from the model during runtime. These techniques are suitable for networks that change frequently. However, by themselves, model traversal techniques cannot handle more complex fault propagation scenarios (e.g., model traversal techniques typically assume that fault propagation occurs one problem at a time, etc.).

[0019] Dependency graph techniques use directed graphs to model dependencies between object events. Nodes represent network elements (e.g., devices such as network devices and / or host devices). A directed edge from node A: event to node B: event indicates that a failure in node A can cause a failure in node B. Dependency graphs are typically used in networks that do not change frequently. In networks that change frequently, the dependencies need to be updated frequently. As described above, especially in view of the rapid increase in the number of connected devices, network complexity is increasing the relatively complex topologies of distributed networks and is increasing the adoption of the Internet of Things (IoT). Due to differences in device capabilities and configurations, these factors also contribute to network heterogeneity.

[0020] The size, complexity, and heterogeneity of typical networks can result in graphs and models representing the network being equally large and complex. Root cause analysis can be likened to determining the best move in a chess game. Each chess piece can have multiple options to move, and more optional moves can follow each move, resulting in millions of possible moves during the game. Similarly, considering the number of possible options at each node in a graph or each object in a model, performing root cause analysis using such a graph or model can be even more complex. As a result, performing root cause analysis on a model and / or graph representing a network configuration using model traversal techniques or dependency graph techniques may be impractical due to the computational resources required to perform such an analysis.

[0021] The techniques described herein can use a Bayesian model of a network. Nodes in the Bayesian model can represent resource types and event types rather than representing specific instances of resources and events. The Bayesian model can be initially created using scenarios and modified using other network models such as resource dependency models and diagnostic models. The Bayesian model can include probability values that indicate the probability that a specific resource type is the root cause of a network failure. In other words, the probability is associated with the resource type rather than with a specific instance of the resource.

[0022] Modern data centers and / or networks are often evaluated, compared, and differentiated by reliability and service levels. In some cases, for data centers that provide network and computing services for lease, data center customers tend to expect service level agreements ("SLAs") that specify or guarantee the end-to-end service performance of the data center. Therefore, it may be important to handle faults or other events that can affect network performance and service SLAs. Sometimes metrics such as mean time between failures (MTBF) and mean time to repair (MTTR) are used as the basis for SLAs. For data center providers subject to such SLAs, it is particularly important to respond quickly and accurately when a failure of a network resource is detected.

[0023] Practical applications of the technology described herein include systems that can automatically identify potential root causes of network failures. Additionally, the system can receive input from a user to confirm the root cause or add a new root cause. Thus, another practical application of the technology described herein is that the system can be trained to better predict root causes. The technology described herein can provide technical advantages over current systems. For example, since there are fewer instances of resources of a resource type than all resource types, a Bayesian model constructed using resource types as described herein is less complex than a Bayesian model based on instances of resources. As a result, the evaluation of potential root causes can be completed faster and with fewer computational resources compared to systems that evaluate the probability for each instance of a resource. This can reduce the MTTR (a potentially important factor in the SLA). Additionally, root cause observations can be applied across multiple instances of a resource type rather than only to individual resources. This provides the advantage of being able to utilize observations across different instances of the same resource type.

[0024] Figure 1 is a block diagram illustrating an example network in accordance with one or more aspects of the present disclosure, the example network being configured to perform hypothesis-driven diagnosis. Network 102 includes devices, components, or elements that can be managed using a control system or controller device such as controller 110. In some examples, network 102 can be an enterprise network operated or managed by an enterprise or other organization. The managed elements 114A - 114G (collectively referred to as "elements 114") of network 102 include devices interconnected via communication links to form a communication topology that enables the exchange of resources and information. Elements 114 can be organized into one or more resource groups and can include, for example, routers, switches, gateways, bridges, hubs, servers, firewalls or other intrusion detection systems (IDS) or intrusion prevention systems, computing devices, computing terminals, printers, storage devices, other network devices, or combinations of such devices. Devices directly or indirectly coupled to network 102 can transmit data in packets, frames, or any other discrete data unit defined by any other protocol (such as cells defined by the asynchronous transfer mode (ATM) protocol or datagrams defined by the user datagram protocol (UDP)). The communication links interconnecting elements 114 can be physical links (e.g., optical, copper, etc.), wireless links, or any combination thereof.

[0025] Network 102 is shown as being coupled to a public network 118 (e.g., the Internet) via a communication link. The public network 118 can include, for example, one or more client computing devices. The public network 118 can provide access to web servers, application servers, public databases, media servers, end-user devices, and other types of network resource devices and content.

[0026] The controller 110 can be communicatively coupled to the elements 114. In some examples, the controller 110 forms part of a device management system, and only one device of the device management system is shown for ease of illustration in Figure 1 Once the elements 114 are deployed and activated, the administrator 112 can use the controller 110 (or such management devices) to manage the devices using a device management protocol. One example device protocol is the Simple Network Management Protocol (SNMP), which allows the controller 110 to traverse and modify the Management Information Base (MIB), which stores configuration data within each of the managed elements in the managed elements 114. Additional details of the SNMP protocol can be found in: Harrington et al., RFC 3411, “An Architecture for Describing Simple Network Management Protocol (SNMP) Management Frameworks”, Network Working Group, the Internet Engineering Task Force draft, December 2002, which is available at tools.ietf.org / html / rfc3411, the entire content of which is incorporated herein by reference.

[0027] The controller 110 (also referred to as a Network Management System (NMS) or NMS device) and the elements 114 can be maintained by an enterprise's Information Technology (IT) group. The administrator 112 can interact with the controller 110 to remotely monitor and configure the elements 114. For example, the administrator 112 can: receive an alert from the controller 110 regarding any of the elements 114; view the configuration data of the elements 114; modify the configuration data of the elements 114; add a new device to the network 102; remove an existing device from the network 102; or otherwise manipulate the network 102 and the devices therein. Although described herein with respect to an enterprise network, the techniques of the present disclosure also apply to other network types, including public and private, including LANs, VLANs, VPNs, etc.

[0028] In some examples, the administrator 112 uses the controller 110 or a local workstation to directly interact with the component 114, for example, through telnet, Secure Shell (SSH), or other such communication sessions. That is, the component 114 typically provides interfaces for direct interaction, such as a command-line interface (CLI), a web-based interface, a graphical user interface (GUI), etc., through which users can interact with the device to directly issue text-based commands. For example, these interfaces generally allow users to directly interact with the device, such as through telnet, Secure Shell (SSH), Hypertext Transfer Protocol (HTTP), or other network sessions, to enter text according to a defined syntax to submit commands to the managed component. In some examples, the user uses the controller 110 to initiate an SSH session 115 with one of the components 114 (e.g., component 14F) to directly configure the component 14F. In this way, the user can provide commands to the component 114 in a format for direct execution.

[0029] In addition, the administrator 112 can also create scripts that can be submitted by the controller 110 to any one or all of the components 114. For example, in addition to the CLI interface, the component 114 can also provide an interface for receiving scripts, and the scripts specify commands according to a scripting language. In a sense, the scripts can be output by the controller 110 to automatically invoke the corresponding remote procedure calls (RPCs) on the managed component 114. The scripts can conform to, for example, Extensible Markup Language (XML) or another data description language.

[0030] The administrator 112 can use the controller 110 to configure the component 114 to specify certain operational characteristics that further achieve the goals of the administrator 112. For example, the administrator 112 can specify specific operational policies for the component 114 regarding security, device accessibility, service engineering, quality of service (QoS), network address translation (NAT), packet filtering, packet forwarding, rate limiting, or other policies. The controller 110 uses one or more network management protocols designed to manage configuration data within the managed network component 114 to perform the configuration, such as the SNMP protocol or the Network Configuration Protocol (NETCONF) protocol, or derivatives thereof (such as the Juniper device management interface). The controller 110 can establish a NETCONF session with one or more of the components 114.

[0031] The network health monitor 106 can monitor the network 102 to determine whether the network 102 is experiencing problems such as failures, throughput issues, or other network problems. The network health monitor 106 can use the information received from the controller 110, the administrator 112, and the components 114 to create or receive one or more data models of the network 102, and the one or more data models can be stored in the model database 116. As described above, the data models can be used to describe the components in the network and the dependencies between the components in the network. The data models can be used by the network health monitor 106 in hypothesis-driven diagnostics to determine the potential root causes of events. For example, the network health monitor 106 can receive event information indicating network performance problems, such as throughput problems or error conditions in the network 102. The network health monitor 106 can use the data models created by the controller 110 together with the data models created by the network health monitor 106 to generate multiple hypotheses about the root cause of the problem. The network health monitor 106 can eliminate the impossible hypotheses and select the most likely root cause from the remaining hypotheses based on the probability ranking of the remaining hypotheses.

[0032] The network health monitor 106 can use YANG modeling for data models stored in the model database 116 and for low-level device configuration models that can be stored in the configuration database 108. The data can include relationships across YANG entities, such as list items and containers. As discussed in more detail herein, the controller 110 can convert the YANG data model into a graph data model and convert YANG validation into data validation. YANG is described in: (i) Bjorklund, “YANG—A Data Modeling Language for the Network Configuration Protocol (NETCONF)”, Internet Engineering Task Force, RFC 6020, October 2010, available at tools.ietf.org / html / rfc6020, (ii) Clemm et al., “A YANG Data Model for Network Topologies”, Internet Engineering Task Force, RFC 8345, March 2018, available at the Uniform Resource Locator (URL) tools.ietf.org / html / rfc8345 (sometimes referred to as “RFC 8345”). Techniques for managing network devices using a graph model for high-level configuration data are described in U.S. Patent Application No. 15 / 462465, filed on Mar. 17, 2017, entitled: “Configuring And Managing Network Devices Using Program Overlay On Yang-Based Graph Database”, the entire content of which is incorporated herein by reference.

[0033] Figure 2 is a block diagram illustrating an example network health monitor in accordance with one or more aspects of the present disclosure, which example network health monitor can be used to perform root cause analysis. In Figure 2 the illustrated example, the network health monitor 106 includes a control unit 202, a network interface 214, and a user interface 212. The network interface 214 represents an example interface that can communicatively couple the network health monitor 106 to the controller 110 or other external devices (e.g., Figure 1 any one of the elements 114). The network interface 214 can represent a wireless and / or wired interface, such as Ethernet An interface or radio wave, which is configured to communicate according to a wireless standard, such as one or more of the IEEE 802.11 wireless networking protocols (such as 802.11a / b / g / n or other such wireless protocols). Although only one network interface is illustrated in the non-limiting example of Figure 2 , in various examples, the network health monitor 106 may include multiple network interfaces.

[0034] The control unit 202 represents any combination of hardware, hardware implementing software, and / or firmware for implementing the functionality attributed to the control unit 202 and its constituent modules and elements. When the control unit 202 includes software or firmware, the control unit 202 also includes any necessary hardware for storing and executing the software or firmware, such as one or more processors or processing units. Generally, the processing unit may include one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), fixed function circuitry, programmable processing circuitry, or any other equivalent integrated or discrete logic circuitry, and any combination of such components. Fixed and / or programmable processing circuitry is typically used to implement the processing unit.

[0035] The user interface 212 represents one or more interfaces through which a user, such as the administrator 112, interacts with the network health monitor 106, for example, to provide input and receive output. For example, the user interface 212 may represent one or more of the following: a monitor, a keyboard, a mouse, a touch screen, a touchpad, a trackpad, a speaker, a camera, a microphone, etc. Additionally, although in this example, the network health monitor 106 includes the user interface 212, the administrator 112 does not need to directly interact with the health monitor 106, but may, for example, remotely access the health monitor 106 via the network interface 214.

[0036] In Figure 2In the illustrated example, the control unit 202 executes the network root cause analyzer 206, the Bayesian network system 204, the user interface module 208, the network interface module 210, and the detectors 222A - 222N (collectively referred to as "detectors 222") to perform a root cause analysis for the network 102. The control unit 202 executes the user interface module 208 to receive input from and / or provide output via the user interface 212. The control unit 202 also executes the network interface module 210 to send and receive data (e.g., in packetized form) via the network interface 214. The network root cause analyzer 206, the Bayesian network system 204, the user interface module 208, the network interface module 210, and the detectors 222A - 222N may be implemented as corresponding hardware units, or implemented in software or firmware implemented by an appropriate hardware infrastructure, or implemented as a combination of the above.

[0037] The Bayesian network system 204 may create and maintain a Bayesian model 218 that describes the relationship between network resource types and event types and describes the root cause based on such dependencies. The Bayesian model may also be referred to as a Bayesian network, a belief network, a decision network, or a probabilistic directed acyclic graph model. In some aspects, the Bayesian model 218 is a probabilistic graphical model that represents a set of variables and the conditional dependencies of the variables via a directed acyclic graph (DAG). A DAG is a graph with directed links and does not contain directed cycles.

[0038] In some aspects, the Bayesian network system 204 creates a Bayesian model 218 based on information in a resource dependency model 216 and a diagnostic model 220. The resource dependency model 216 can be a graph model, where the nodes in the graph model represent network resources. The resources can include, for example, network services, hardware and / or software units, and a table of components 114. The resources can be physical resources or logical resources, which can generate alerts, events, or other types of fault data. Thus, the resources can include, for example, network tunnels (such as label switched paths (LSPs)), virtual private networks (VPNs), virtual routing and forwarding (VRF) tables, hardware interfaces, logical interfaces, and the like. The nodes in the resource dependency model 216 can represent resources, and the edges between the nodes can represent dependencies. The dependency edges can span device resources, from device resources to services, and across services. In some aspects, the resource dependency model 216 can be programmable such that the model can be created and changed as the resources and services change over time. In some aspects, the resource dependency model 216 and / or the Bayesian model 218 can be programmed by a playbook 224. More details regarding the creation and maintenance of the resource dependency model 216 can be found in U.S. Patent Application No. 16 / 947,507, filed on August 4, 2020, entitled "USING A PROGRAMMABLE RESOURCE DEPENDENCY MATHEMATICAL MODEL TO PERFORM ROOTCAUSE ANALYSIS", the entire content of which is incorporated herein by reference.

[0039] Figure 3 is a conceptual diagram showing an example portion of a resource dependency model. In this example, the resource dependency model includes a resource A node 302, a resource C node 304, and a resource B node 306. In some aspects, the network health monitor 106 determines dependencies in the reverse order:

[0040] Dependency (resource) → [list of resources]

[0041] Thus, using Figure 3 the resource dependency model as an example, if resource A uses resource B (as shown by the edge 310 from the resource A node 302 to the resource B node 306), the network health monitor can generate the following resource dependency model:

[0042] Dependency (resource B) → [resource A, resource C]

[0043] Resource nodes A 302, C 304, and B 306 each have a detector 222 associated with that node. In some aspects, the detector 222 can be associated with a resource node in the resource dependency model 216. The detector can be a software, firmware, or hardware module that can be programmed to perform operations to test the resources represented by the resource node. In some examples, the detector 222 can: execute a ping command; perform bidirectional forwarding detection (BFD); execute a "Traceroute" command; correlate counters associated with the resource; or perform other operations designed to evaluate the operability of the resources associated with the resource node. The detector can be involved in message passing, querying resources, obtaining values, calculating values, and otherwise determining the operating conditions of the resources. A threshold can be associated with the detector. In some aspects, the threshold can be a pre-determined or configurable value. In some aspects, a machine learning algorithm can be used, for example, to automatically learn the threshold. The Bayesian network system 204 can use the threshold to determine whether a resource "passes" the test provided by the detector 222. For example, a threshold of 999 for a ping-based detector for a resource can specify that 999 out of 1000 ping packets must be successfully returned from the resource in order for the Bayesian network system 204 to consider the resource to have passed the detector test. Similarly, a threshold of 100 ms can specify that 99% of the ping attempts must have a round-trip time of less than 100 ms. For a BFD-based detector, a threshold of four (4) can specify that the routing information on the remote router resource needs to be updated no later than four times the BFD time period.

[0044] In Figure 3 the illustrated example, resource node A 302 is associated with detectors 222A and 222B, resource node C 304 is associated with detector 222C, and resource node B 306 is associated with detectors 222A and 222D. As Figure 3 shown, the same detector (e.g., detector 222A) can be assigned to more than one resource node.

[0045] In Figure 7A and Figure 7B examples of XML code are provided that include data fields defining resource nodes and detectors. In Figure 7A it, part 702 of the XML code defines a detector labeled "ping" and defines the location (e.g., path), parameters, and conditions that trigger a detector error. In Figure 7B it, part 704 includes fields defining an LSP resource object. The definition of the LSP resource object in part 704 includes an association 706 that associates the ping detector with the LSP resource.

[0046] Figure 4 is a conceptual diagram illustrating an example dependency graph 400 representing a resource dependency model according to one or more aspects of the present disclosure. Dependency graph 400 shows a resource group having multiple resource layers, including slots 401A, 401B, 401C, and 401D (“slot 401”), and each of the slots can be one of a number of port concentrators or modular port concentrators associated with a network or network device. In Figure 4 are also illustrated PFEs 402A to 402C (“PFE 402”), Ethernet interfaces 403A to 403D (“interface 403”), label switched paths (LSPs) 404A to 404C (“LSP 404”), external border gateway protocol (eBGP) services 405A to 405C (“eBGP service 405”), virtual routing and forwarding instances (VRFs) 406A to 406C (“VRF 406”), and customer edge devices or delays 407A to 407C (“customer edge delay 407”). Although in Figure 4 a limited number of slots 401, PFEs 402, interfaces 403, LSPs 404, VRFs 406, and customer edge delays 407 are illustrated, the techniques described herein can be applied to other networks having any number of such resources or other types of resources.

[0047] Dependency graph 400 can correspond to a resource dependency model of a network (such as, for example, network 102 described in conjunction with Figure 1 ). Dependency graph 400 illustrates dependencies across each set of instances or across each layer (e.g., across slot 401 to PFE 402). Any instance shown in dependency graph 400 (e.g., any one of PFE 402, interface 403, LSP 404, eBGP service 405, VRF 406, and / or customer edge delay 407) can experience a failure, degradation, error, or other event that can affect the operation of the network. For example, each PFE in slot 401A and PFE 402 has a dependency relationship, where an event affecting slot 401A can affect each PFE in PFE 402. Similarly, an event affecting PFE 402B can affect some or all of the interfaces in interface 403.

[0048] Return Figure 2, the diagnostic model 220 captures causal (sometimes referred to herein as "correlation") relationships between various resources. For example, the diagnostic model 220 can reflect causal relationships across events that occur on the network 102. Events can be physical events or logical events, which can be derived from physical events and / or network states. Causal relationships are defined between resources and resource alerts / events. When causal relationships are defined between resources, alerts or events on a resource will have an impact on "supporting resources". When causal relationships are defined between resources and resource alerts / events, events on a resource will have an impact on "supported resource" events. In some aspects, the diagnostic model 220 can be a programmable diagnostic model, which can be implemented by a controller such as the controller 110 and can be provided by the controller 110 to the network health monitor 106. Details regarding the creation and use of an example programmable diagnostic model can be found in U.S. Patent Application 16 / 821745, filed on March 17, 2020, entitled "PROGRAMMABLE DIAGNOSIS MODEL FOR CORRELATION OF NETWORK EVENTS" and U.S. Patent Application 16 / 946994, filed on July 14, 2020, entitled "FAILURE IMPACT ANALYSIS OF NETWORK EVENTS", the entire contents of which are incorporated herein by reference in their entirety.

[0049] As described above, the Bayesian network system 204 generates the Bayesian model 218 based on the data in the resource dependency model 216 and the diagnostic model 220. The controller 110 can generate the resource dependency model 216 and the diagnostic model 220 for use by the network health monitor 106, and the network health monitor 106 can use the resource dependency model 216 and the diagnostic model 220 to create the Bayesian model 218. In some aspects, the Bayesian network system 204 can generate the Bayesian model 218 based on a subset of the data in the models 216 and 220, rather than including all resources and events in the resource dependency model 216 and the diagnostic model 220. The subset of data can represent a higher-level abstraction of the resources and events in the resource dependency model 216 and the diagnostic model 220. As an example, the Bayesian network system 204 can generate the Bayesian model 218 based on the types of resources and the types of events in the resource dependency model 216 and the diagnostic model 220, rather than based on the instances of resources and events. For example, the Bayesian model 218 can include nodes that are abstract representations of resources. Thus, a single edge router node can represent any different type of edge router in the system, regardless of the manufacturer of the edge router or the version of the edge router, rather than including a node for each edge router in the system.

[0050] Figure 5 illustrates an example Bayesian model 500 in accordance with one or more aspects of the present disclosure, which may be generated by a Bayesian network system 204 based on Figure 4 the example resource dependencies illustrated in the resource dependency graph 400 of. The example Bayesian model 500 includes nodes for each type of resource in the resource dependency model, rather than including nodes for each resource instance. As Figure 5 shown, the example Bayesian model 500 includes an MPC slot node 502, a PFE node 504, an ET interface node 506, an LSP node 508, an E-BGP node 510, a VRF node 512, and a CE latency node 514. Each of the nodes in the figure corresponds to a resource type, which represents all resource instances of that type in the dependency model. As a result, the Bayesian network system 204 generates a Bayesian model 218 that is: less complex and easier to maintain than a model that includes nodes for each instance of a resource. For example, the MPC slot node 502 is a type node that represents the resource type for MPC slots 401A - 401D. The PFE node 504 is a type node that represents the resource type for PFEs 402A - 402C. The ET interface node 506 is a type node that represents the resource type for ET interfaces 403A - 403D. The LSP node 508 is a type node that represents the resource type for LSPs 404A - 404C. The E-BGP node 510 is a type node that represents the resource type for eBGP services 405A - 405C. The VRF node 512 is a type node that represents the resource type for VRFs 406A - 406C. Finally, the CE latency node 514 is a type node that represents the type for event CE latency 407A - 407C. Table 1 illustrates example probabilities associated with the VRFs 512 for the example Bayesian model 500.

[0051] Table 1

[0052]

[0053]

[0054] As shown in the example Bayesian model 500, the VRF 512 is associated with two nodes, the LSP 508 and the E - BGP 510. The LSP and E - BGP columns in Table 1 indicate whether there is fault data for the corresponding resource, where "T" indicates the presence of fault data and "F" indicates the absence of fault data. Given the four possible combinations of fault data for the LSP and E - BGP, the "T" and "F" columns provide the probability that the LSP 508 or the E - BGP 510 is the root cause of the VRF problem. Thus, if neither the LSP 508 nor the E - BGP 510 indicates a fault, it is unlikely that either the LSP 508 or the E - BGP 510 is the root cause of the fault in the VRF 512. If the LSP 508 does not indicate a fault while the E - BGP 510 indicates a fault, there is an 80% probability that the E - BGP 510 is the root cause of the fault. If the LSP 508 indicates a fault and the E - BGP 510 does not indicate a fault, there is a 90% probability that the LSP 508 is the root cause of the fault in the VRF 512. Finally, if both the LSP 508 and the E - BGP 510 indicate a fault, there is a 99% probability that the LSP 508 is the root cause of the fault.

[0055] As can be seen by comparing Figure 3 and Figure 4 the diagrams illustrated in, generating a Bayesian model based on the types of resources and events rather than the instances of resources and events results in lower complexity and, therefore, a more manageable model. For example, the number of nodes in the example Bayesian model 500 (7 nodes) is less than the number of nodes in the resource dependency graph 400 (23 nodes). Thus, the Bayesian model 500 can be evaluated and maintained more efficiently than the resource dependency model 400.

[0056] Return Figure 2 , as described above, the Bayesian model 218 can be a probabilistic graphical model. The probabilities in the Bayesian model can indicate the probabilities associated with the nodes, i.e., the probability that the type of resource indicated by the node is the root cause of the fault data that matches the hypothesis. In other words, the probabilities are associated with the resource types rather than the specific instances of the resources. In some aspects, the Bayesian network system 204 generates the Bayesian model 218, where each node is initialized with equal probability between the nodes.

[0057] As described above with respect to Figure 2The discussion generally describes operational aspects associated with configuring and / or initializing the Bayesian network system 204 for the network health monitor 106 or the controller 110. These operational aspects can occur during the initialization of the network health monitor 106 and / or the controller 110. Aspects of the use of the Bayesian model 218 by the Bayesian network system 204 and / or the network health monitor 106 during the operation of the network 102 will now be discussed.

[0058] The network health monitor 106 can receive fault data from the devices 114 and the controller 110 of the network 102. The fault data can describe events and alerts that can be associated with the devices 114 and / or the links in the network 102 ( Figure 1 ). The control unit 202 can execute the network root cause analyzer 206 and the Bayesian network system 204 to analyze the fault data and provide potential root causes for the fault data to the administrator 112 via, for example, the user interface module 208 and the user interface 212.

[0059] In some aspects, the network health monitor 106 can determine whether fault data collection is complete. If the fault data collection is complete, the network health monitor 106 can use the Bayesian network system 204 to determine one or more root cause hypotheses based on the fault data. If the fault data collection is not complete, the network health monitor 106 can extend the observation period and wait for additional fault data to arrive. Such data can be referred to as "late" fault data. In some aspects, if a programmable percentage or a predetermined percentage of the resources corresponding to the fault node and the sub-resources corresponding to the child nodes of the fault node report faults, the fault data collection can be considered complete. As an example, consider a network system with ten LSPs. Suppose fault data indicating an alert is received from five of the LSPs. This could be the case where only five LSPs are faulty. However, this could also be the case where all ten LSPs are faulty, but fault data for the five LSPs that have not reported any fault data has not been received. Alternatively, the network interface associated with the ten LSPs could be faulty, and fault data associated with that interface has not been reported. In this case, it may be premature to initiate a root cause analysis because more fault data that can support a better or more specific root cause analysis may be received.

[0060] The network root cause analyzer 206 can determine whether the fault data is complete enough and, if so, generate a set of root cause hypotheses corresponding to the fault data. For example, the set of root cause hypotheses are potential root causes where the fault data matches the conditions for the root cause. If the fault data is incomplete, the network root cause analyzer 206 can extend the observation period and wait for additional fault data to arrive. As an example, a faulty node may not send an alert for a relatively long time. However, root cause analysis can start at some point after an adjacent node detects an interruption that may be caused by the faulty node. In that case, it may be necessary to wait some time for the alert to arrive from a dedicated node, but if the alert does not arrive in time, the network root cause analyzer 206 can start the root cause analysis. Once the first fault is received (many other faults are not yet clear), the root cause analysis can start. However, since the "fault mode" is incomplete, the root cause analysis can involve considering many possible root causes that could easily be dismissed if enough fault information is available.

[0061] The Bayesian network system 204 can evaluate the set of root cause hypotheses based on the fault data received by the network health monitor 106. Each hypothesis in the set can identify one or more nodes in the Bayesian model 218 that caused the hypothesis to be generated. The Bayesian network system 204 can use the Bayesian model 218 to assign probabilities to each root cause hypothesis in the generated set of root cause hypotheses. For example, the Bayesian network system 204 can compare the fault data with resource and event dependencies to determine the set of potential root cause hypotheses and determine the probabilities associated with each root cause hypothesis in the set.

[0062] The Bayesian network system 204 can cause the probe 222 to be executed for the resources indicated in the set of root cause hypotheses. In some aspects, the probe 222 can be used to attempt to refute the hypotheses. For example, one of the root cause hypotheses in the set of root cause hypotheses might be "if no packets are received from the source node, the routing engine might be down". There are two possibilities, there might be no packets to be forwarded by the routing engine (which is not a fault), or there might be an internal fault in the routing engine that prevents packets from being forwarded. The probe 222 configured to execute a ping command can be associated with the routing engine resource node in the resource dependency model 216. The Bayesian network system 204 can execute the probe for the routing engine, for example, the Bayesian network system can issue a ping command to the routing engine. If a response to the ping command is received, the root cause hypothesis is refuted because if a ping response is received, the routing engine is operating. The Bayesian network system 204 can thus remove the refuted root cause hypothesis from the set of root cause hypotheses. In this case, the ping probe can be used to distinguish between the case where the routing engine has no packets to forward and an internal fault that prevents packets from being forwarded.

[0063] After the Bayesian network system 204 has executed the probes associated with the resource types and removed any refuted root cause hypotheses from the set of root cause hypotheses, the Bayesian network system 204 can rank the remaining root cause hypotheses in the set. In some aspects, the root cause hypotheses can be ranked according to the probabilities assigned in the Bayesian model 218.

[0064] In some aspects, the network health monitor 106 can utilize the user interface module 208 to present the ranked root cause hypotheses to the administrator 112 via the user interface 212. In some other aspects, the network health monitor 106 can utilize the network interface module 210 to transmit the ranked root cause hypotheses to another system (e.g., a logging or management system) via the network interface 214. The administrator 112 can view the root cause hypotheses and determine which of the root cause hypotheses (if any) in the root cause hypotheses is correct.

[0065] In some aspects, the network health monitor 106 can receive feedback from a user (e.g., the administrator 112) regarding a set of root cause hypotheses presented to the user in response to a fault in the network. For example, the user can provide feedback confirming that a hypothesis is correct or indicating that a hypothesis is incorrect. In response to this feedback, the Bayesian network system 204 can adjust the probabilities associated with the corresponding root cause hypotheses. For example, in response to receiving feedback that a root cause hypothesis correctly describes the fault, the Bayesian network system 204 can increase the probability associated with the corresponding root cause hypothesis. Similarly, in response to receiving feedback that a root cause hypothesis is incorrect, the Bayesian network system 204 can decrease the probability associated with the corresponding root cause hypothesis. As another example, the user can add a new probe to a resource node in the resource model 216, and the new probe can assist in validating a root cause hypothesis associated with the resource type of the node. For example, it may be the case that all potential root cause hypotheses presented to the user are incorrect. The user can create a new root cause hypothesis and can select a resource probe that can be used to refute the hypothesis. The new root cause hypothesis can be added to the Bayesian model 218, for example, via the user interface 212.

[0066] In some aspects, the network health monitor 106 can select the most likely root cause hypothesis in a sorted set of root cause hypotheses and automatically perform a remediation action based on the selected root cause hypothesis. For example, if the selected root cause hypothesis indicates that a network node is down, the network health monitor 106 can take an action to reset or reboot the network node to bring the network node back to an operational state. After the Bayesian model 218 has been adequately trained based on the above user feedback, the automatic selection of the remediation action can be performed. In some other aspects, a Bayesian 218 that has been adequately trained on a different network can be imported or installed for use on the current network, thus avoiding the need to train the Bayesian model 218 for the current network.

[0067] An example of the operation of the above system will now be given. For purposes of illustration, the example network includes three nodes N1, N2, and N3. The set of LSPs from N1 to N3 and from N3 to N1 are labeled LSP13 and LSP31, respectively. In this example, the fault causes the PFE1 to "forget" the label for LSP13. As a result, the network packets for LSP13 are dropped and do not appear on the expected interface counters.

[0068] Before the fault, a Bayesian model is derived from the resource model and the diagnostic model. Probes are associated with the available probes in the resource model in the Bayesian network model. In this example, the probes can include ping, BFD, traceroute probes, etc.

[0069] The symptoms of the fault can be that the Ethernet Operations, Administration, and Maintenance (OAM) for the Ethernet Virtual Private Line (EVPL) times out on node N3. When the fault is reported to the network health monitor, the network root cause analyzer 206 can determine whether the available fault information is complete. In this example, the network root cause analyzer 206 can determine whether the underlying resources (e.g., resources associated with sub-resource nodes) have also generated fault data (e.g., alarms). In this example, node N1 can also report an OAM timeout.

[0070] The network root cause analyzer 206 can generate multiple root cause hypotheses based on the server resources in the resource dependency model. In addition to the above alarms, the following situations also exist in the network.

[0071] 1. LDP and BGP do not show errors or anomalies

[0072] 2. LSP13 and LSP31 are configured and do not provide alarms

[0073] 3. The PFE has no alarms

[0074] 4. The Ethernet interface ETH-x does not issue an alarm

[0075] The Bayesian network system 204 can score the hypotheses based on the Bayesian model 218. As described above, in the initial state, each resource in the Bayesian model has an equal probability. The Bayesian network system 204 can execute detectors associated with the resources to attempt to refute each hypothesis in the hypotheses. In this example, assume that the detectors produce the following results:

[0076] 1. The detector associated with the PFE resource obtains packet forwarding statistics indicating that the PFE is forwarding packets. Therefore, the PFE is operational, and the hypothesis that the PFE is the root cause is refuted.

[0077] 2. The detector associated with the node resource pings node N3 from node N1 and vice versa. Both nodes respond to the ping, indicating that the Ethernet interfaces of the nodes are operational. Therefore, nodes N3 and N1 are operational, and the root cause hypotheses indicating that node N3 or node N1 is the root cause are refuted.

[0078] 3. The detector associated with the LSP resource pings LSP31. A response to the ping is received, indicating that LSP31 is operational. Therefore, the hypothesis indicating that LSP31 is the root cause is refuted.

[0079] 4. The detector associated with the LSP resource pings LSP13. No response to the ping is received. Therefore, the hypothesis indicating LSP13 is not refuted.

[0080] The Bayesian network system 204 ranks the unrefuted hypotheses (e.g., LSP13 is the root cause), and the network health monitor 106 presents the hypotheses in probabilistic order. The network health monitor 106 may seek user input to confirm the hypotheses, possibly based on manual root cause analysis techniques. In this example, the actual root cause is the faulty PFE, and thus different from the root cause predicted by the Bayesian network system 204 (e.g., LSP13 is down). The user may provide input to select different nodes in the Bayesian model, where different nodes indicate that the PFE is a potential root cause, and may select probes aimed at improving the prediction of the Bayesian network system 204. The network health monitor 106 may receive this input and modify the Bayesian model 218 and / or the resource dependency model 216 accordingly.

[0081] Figure 2 The modules illustrated herein and / or illustrated or described elsewhere in this disclosure (e.g., the network health monitor 106, the network root cause analyzer 206, the Bayesian network system 204, the UI module 208, the network interface module 210, the probes 222A - 222N (collectively referred to as "the probes 222")) may perform the described operations using software, hardware, firmware, or a combination of hardware, software, and firmware residing in and / or executed at one or more computing devices. For example, a computing device may execute one or more of such modules with multiple processors or multiple devices. A computing device may execute one or more of such modules as virtual machines executing on the underlying hardware. One or more of such modules may execute as one or more services of an operating system or a computing platform. One or more of such modules may execute as one or more executable programs at the application layer of a computing platform. In other examples, the functionality provided by the modules may be implemented by dedicated hardware devices.

[0082] Although certain modules, data stores, components, programs, executable files, data items, functional units, and / or other items included within one or more storage devices may be separately illustrated, one or more of such items may be combined and operate as a single module, component, program, executable file, data item, or functional unit. For example, one or more modules or data stores may be combined or partially combined such that they operate as a single module or provide functionality. Additionally, one or more modules may interact and / or operate in conjunction with each other such that, for example, one module acts as a service or extension of another module. Further, each module, data store, component, program, executable file, data item, functional unit, or other item illustrated within the storage device may include multiple components, sub - components, modules, sub - modules, data stores, and / or other components or modules or data stores not illustrated.

[0083] In addition, each module, data store, component, program, executable file, data item, functional unit, or other item depicted within a storage device may be implemented as a downloadable or pre-installed application or "app". In some other examples, each module, data store, component, program, executable file, data item, functional unit, or other item depicted within a storage device may be implemented as part of an operating system executing on a computing device.

[0084] Figure 6 is a flowchart in accordance with one or more aspects of the present disclosure that illustrates operations performed by an example network health monitor to generate potential root cause hypotheses. The following is described in the context of the controller 110 and the network health monitor 106 of Figure 1 and Figure 2 . In some other examples, the operations described in Figure 6 . In some other examples, the operations described in Figure 6 may be performed by one or more other components, modules, systems, or devices. Additionally, in some other examples, the operations described in conjunction with Figure 6 may be combined, performed in a different order, performed in parallel, omitted, or may include additional operations not specifically shown or described.

[0085] In the process illustrated in Figure 6 and in accordance with one or more aspects of the techniques described herein, the network health monitor 106 may obtain data that indicates resource dependencies between resources in the network and event dependencies between a network event and one or more of the resources (605). For example, the network health monitor 106 may obtain data indicating resource and event dependencies from the controller 110.

[0086] The network health monitor 106 generates a Bayesian model that is at a higher level than a resource dependency model or a diagnostic model (610). For example, the nodes in the Bayesian model may be resource types of resources and event types of network events, rather than specific resource instances or event instances.

[0087] The network health monitor 106 receives an indication of a fault in the network (615). The indication may be an alert or an event that occurs in the network. In response to the indication of the fault, the network health monitor 106 collects fault data and, based on the Bayesian model, generates a set of root cause hypotheses (620). The fault data may include data provided as part of an alert or an event that initially indicates a fault in the network.

[0088] For each root cause hypothesis in the set, the network health monitor 106 can execute a probe that is associated with the resources identified in the fault data and is associated with the root cause hypothesis. In some aspects, the output of the probe can be used to refute the hypothesis. If the output of the probe refutes the hypothesis (625), then the root cause hypothesis is removed from the set of root cause hypotheses (630, the "yes" branch of 625). If the output of the probe does not refute the hypothesis, then the next root cause hypothesis in the set, if any, is obtained (the "no" branch of 625).

[0089] Optionally, the remaining hypotheses in the set of root cause hypotheses can be ranked based on the probability of the root cause hypothesis as determined by a Bayesian model (635). The set of remaining root cause hypotheses can be output (640). For example, the set of remaining hypotheses can be displayed to the user via a user interface.

[0090] For the processes, apparatuses, and other examples or illustrations described herein that are included in any flowchart or flow diagram, certain operations, actions, steps, or events included in any of the techniques described herein can be executed in a different sequence, can be added, combined, or entirely omitted (e.g., not all described actions or events are necessary for the practice of the technique). Additionally, in certain examples, operations, actions, steps, or events can be executed concurrently rather than sequentially, such as by multithreading, interrupt processing, or multiple processors. Additionally, certain other operations, actions, steps, or events can be automatically executed even if not explicitly identified as being automatically executed. Additionally, certain operations, actions, steps, or events described as being automatically executed can alternatively not be automatically executed, but rather, in some examples, such operations, actions, steps, or events can be executed in response to an input or another event.

[0091] For ease of illustration, only a limited number of devices are shown in the figures and / or other illustrations referenced herein. However, the techniques according to one or more aspects of the present disclosure can be executed with more such systems, components, devices, modules, and / or other items, and the collective reference to such systems, components, devices, modules, and / or other items can represent any number of such systems, components, devices, modules, and / or other items.

[0092] The figures included herein each illustrate at least one example implementation of an aspect of the present disclosure. However, the scope of the present disclosure is not limited to such implementations. Thus, other examples or alternative implementations of the systems, methods, or techniques described herein may be suitable in other instances. Such implementations can include a subset of the devices and / or components included in the figures and / or can include additional devices and / or components not shown in the figures.

[0093] The foregoing detailed description is intended as a description of various configurations and is not intended to represent the only configurations in which the concepts described herein may be practiced. The detailed description includes specific details to provide a thorough understanding of the various concepts. However, the concepts may be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form in the accompanying figures to avoid obscuring such concepts.

[0094] Accordingly, although one or more implementations of various systems, devices, and / or components may be described with reference to specific figures, such systems, devices, and / or components may be implemented in a variety of different ways. For example, one or more devices illustrated herein as separate devices may alternatively be implemented as a single device; one or more components illustrated as separate components may alternatively be implemented as a single component. Additionally, in some examples, one or more devices illustrated as a single device in the figures herein may alternatively be implemented as multiple devices; one or more components illustrated as a single component may alternatively be implemented as multiple components. Each of such multiple devices and / or components may be directly coupled via wired or wireless communication and / or remotely coupled via one or more networks. Additionally, one or more devices or components illustrated in the various figures herein may alternatively be implemented as part of another device or component not shown in such figures. In this and other ways, some of the functionality described herein may be performed via distributed processing by more than two devices or components.

[0095] Additionally, certain operations, techniques, features, and / or functions may be described herein as being performed by specific components, devices, and / or modules. In some other examples, such operations, techniques, features, and / or functions may be performed by different components, devices, or modules. Thus, some operations, techniques, features, and / or functions described herein as belonging to one or more components, devices, or modules may in some other examples belong to other components, devices, and / or modules, even if not specifically described herein in that manner.

[0096] Although specific advantages have been identified in connection with the description of some examples, various other examples may include some, none, or all of the recited advantages. Other advantages of the technology or other aspects will become apparent to those skilled in the art in accordance with the present disclosure. Additionally, although specific examples have been disclosed herein, any number of techniques may be used to implement aspects of the present disclosure, whether currently known or not, and thus the present disclosure is not limited to the examples specifically described and / or illustrated herein.

[0097] In one or more examples, the described functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functionality may be stored on and / or transmitted via a computer-readable medium as one or more instructions or code and executed by a hardware-based processing unit. The computer-readable medium may include: a computer-readable storage medium corresponding to a tangible medium such as a data storage medium, or a communication medium including any medium that facilitates transfer of a computer program from one place to another (e.g., according to a communication protocol). In this manner, the computer-readable medium generally may correspond to (1) a non-transitory tangible computer-readable storage medium or (2) a communication medium such as a signal or carrier wave. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to obtain instructions, code, and / or data structures for implementing the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0098] By way of example, and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and that can be accessed by a computer. Additionally, any connection is properly termed a computer-readable medium. For example, if the instructions are transmitted using coaxial cable, fiber optic cable, twisted pair, or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but instead are directed to non-transitory, tangible storage media. Combinations of the above should also be included within the scope of computer-readable media.

[0099] The instructions may be executed by one or more processors such as one or more digital signal processors (DSPs), general microprocessors, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Thus, the term “processor” or “processing circuitry” as used herein may both refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described. Additionally, in some examples, the described functionality may be provided within dedicated hardware and / or software modules. Additionally, these techniques may be implemented entirely within one or more circuits or logic elements.

[0100] The techniques of the present disclosure can be implemented in a variety of devices or apparatuses, including wireless telephones, mobile or non-mobile computing devices, wearable or non-wearable computing devices, integrated circuits (ICs) or collections of ICs (e.g., chip sets). Various components, modules or units are described in the present disclosure to emphasize the functional aspects of the devices configured to perform the disclosed techniques, but do not necessarily need to be implemented by different hardware units. Instead, as described above, the various units can be combined in a hardware unit or provided by a collection of interoperating hardware units including one or more processors as described above in conjunction with suitable software and / or firmware.

Claims

1. A method for providing root cause hypotheses, comprising: obtaining, by one or more processors, data indicative of: resource dependencies among a plurality of resources in a network, and event dependencies between a plurality of network events and one or more of the plurality of resources; generating a Bayesian model based on resource types associated with the plurality of resources and event types associated with the plurality of network events; receiving an indication of a fault in the network; collecting fault data and generating, based on the Bayesian model and the fault data, a plurality of root cause hypotheses for the fault, wherein each root cause hypothesis of the plurality of root cause hypotheses has an associated probability; receiving confirmation of a root cause hypothesis among the plurality of root cause hypotheses; increasing the probability associated with each node corresponding to the confirmed root cause hypothesis; responding to refuting a root cause hypothesis among the plurality of root cause hypotheses by removing the root cause hypothesis from the plurality of root cause hypotheses to form an updated plurality of root cause hypotheses; ranking the updated plurality of root cause hypotheses based on corresponding root cause probabilities associated with the updated plurality of root cause hypotheses to form a ranked plurality of root cause hypotheses; and outputting the ranked plurality of root cause hypotheses.

2. The method according to claim 1, wherein removing the root cause hypothesis from the plurality of root cause hypotheses to form an updated plurality of root cause hypotheses comprises: executing a detector associated with the root cause hypothesis among the plurality of root cause hypotheses; and responding to determining that the detector refutes the root cause hypothesis by removing the root cause hypothesis from the plurality of root cause hypotheses to form the updated plurality of root cause hypotheses.

3. The method according to any one of claims 1 to 2, further comprising determining whether the fault data is complete; and responding to determining that the fault data is complete by waiting for a first time period and, after the first time period has elapsed, generating the plurality of root cause hypotheses.

4. The method according to claim 3, further comprising: responding to determining that the fault data is incomplete by waiting for a second time period that is longer than the first time period and, after the second time period has elapsed, generating the plurality of root cause hypotheses.

5. The method according to claim 3, wherein determining that the fault data is complete includes: Determining that a threshold percentage of sub-resources have provided fault information, wherein the sub-resources correspond to child nodes of a resource node in a resource dependency model, and the resource node corresponds to the resource that provided the fault data.

6. The method according to any one of claims 1 to 2, further comprising: receiving a user-generated root cause hypothesis among the plurality of root cause hypotheses; receiving an indication of a detector associated with the user-generated root cause hypothesis; and adding the user-generated root cause hypothesis to the Bayesian model.

7. The method according to claim 6, wherein the detector comprises a new detector, and wherein the method further comprises: Receiving a mapping of resource characteristics of a resource node to inputs of the new detector.

8. The method according to any one of claims 1 to 2 further comprises: Initializing the probability associated with each node of the Bayesian model to an equal probability.

9. An apparatus for providing root cause hypotheses, comprising: A memory and a processing circuit device, the processing circuit device being configured to: Obtain data indicating resource dependencies among a plurality of resources in a network and event dependencies between a plurality of network events and one or more of the plurality of resources; Generate a Bayesian model based on the resource types associated with the plurality of resources and the event types associated with the plurality of network events; Receive an indication of a fault in the network; Collect fault data and generate, based on the Bayesian model and the fault data, a plurality of root cause hypotheses for the fault, wherein each root cause hypothesis among the plurality of root cause hypotheses has an associated probability; Receive confirmation of a root cause hypothesis among the plurality of root cause hypotheses; Increase the probability associated with each node corresponding to the confirmed root cause hypothesis; In response to refuting a root cause hypothesis among the plurality of root cause hypotheses, remove the root cause hypothesis from the plurality of root cause hypotheses to form an updated plurality of root cause hypotheses; Rank the updated plurality of root cause hypotheses based on the respective root cause probabilities associated with the updated plurality of root cause hypotheses to form a ranked plurality of root cause hypotheses; and Output the ranked plurality of root cause hypotheses.

10. The apparatus according to claim 9, wherein in order to remove the root cause hypothesis from the plurality of root cause hypotheses to form an updated plurality of root cause hypotheses, the processing circuit device is configured to: Execute a detector associated with the root cause hypothesis among the plurality of root cause hypotheses; and In response to determining that the detector refutes the root cause hypothesis, remove the root cause hypothesis from the plurality of root cause hypotheses to form the updated plurality of root cause hypotheses.

11. The apparatus according to any one of claims 9 to 10, wherein the processing circuit device is further configured to: Determine whether the fault data is complete; and In response to determining that the fault data is complete, wait for a first time period and, after the first time period has elapsed, generate the plurality of root cause hypotheses.

12. The apparatus according to claim 11, wherein the processing circuit device is further configured to: In response to determining that the fault data is incomplete, wait for a second time period that is longer than the first time period and, after the second time period has elapsed, generate the plurality of root cause hypotheses.

13. The apparatus according to claim 11, wherein determining that the fault data is complete includes: Determine that a threshold percentage of child nodes have provided fault information.

14. The apparatus according to any one of claims 9 to 10, wherein the processing circuit device is further configured to: Receive a user-generated root cause hypothesis among the plurality of root cause hypotheses; Receive an indication of a detector associated with the user-generated root cause hypothesis; and Add the user-generated root cause hypothesis to the Bayesian model.

15. The apparatus according to claim 14, wherein the detector includes a new detector, and wherein the processing circuit device is further configured to receive a mapping of resource characteristics of a resource node to an input of the new detector.

16. The apparatus according to any one of claims 9 to 10, wherein the processing circuitry is further configured to: initialize the probabilities associated with each node of the Bayesian model to equal probabilities.

17. A computer-readable storage medium encoded with instructions for causing one or more programmable processors to perform the method according to any one of claims 1 to 8.

Images