Framework for automated application-to-network root cause analysis

Abstract

A computing system including memory and processing circuitry may perform the techniques. The memory may store time series data including measurements of one or more performance metrics. The processing circuitry may determine anomalies in performance of the network system based on the time series data, and create a knowledge graph based on the time series data. In response to detecting an anomaly, and based on the knowledge graph and a machine learning (ML) model trained with previous time series data, the processing circuitry may determine a causal graph. The processing circuitry may determine a weight for each edge in the causal graph, determine candidate root causes associated with the anomaly based on the edges in the causal graph, and determine a ranking of the candidate root causes based on the weights. The analysis framework system may output at least a portion of the ranking.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of the following provisional applications: U.S. Provisional Application No. 63 / 367,457, filed on June 30, 2022, entitled “FRAMEWORK FOR AUTOMATED APPLICATION-TO-NETWORK ROOT CAUSE ANALYSIS”; U.S. Provisional Application No. 63 / 367,456, filed on June 30, 2022, entitled “GRAPH ANALYTICS ENGINEFOR APPLICATION-TO-NETWORK TROUBLESHOOTING”; and U.S. Provisional Application No. 63 / 367,452, filed on June 30, 2022, entitled “APPLICATION-AWARE ACTIVE MEASUREMENT FOR MONITORING NETWORK HEALTH”, the entire contents of each of the above applications are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to computer networks, and more particularly, to root cause analysis of anomalies in computer networks. Background Art

[0004] A computer network is a collection of interconnected computing devices that can exchange data and share resources. Various devices operate to facilitate communications between computing devices. For example, a computer network may include routers, switches, gateways, firewalls, and various other devices to provide and facilitate network communications. Summary of the invention

[0005] In general, the present disclosure describes techniques, systems, and frameworks for performing root cause analysis of network problems. Root cause analysis may include analysis of data collected at several different layers of a network system, including the application layer to the network layer. Application performance may be based on many factors, including the dynamics of application behavior, computing / memory / storage infrastructure, and network performance behavior. Any performance bottleneck in the underlying physical layer is likely to reduce the performance of the application perceived by the end user. The emergence of cloud native application architectures based on microservices and their highly distributed nature is likely to make troubleshooting any performance bottlenecks a highly challenging task. Network performance may be even more critical to the performance of microservices-based applications, which may operate in reduced time frames (e.g., on the order of milliseconds, seconds, and / or minutes), initialized and deinitialized based on demand or other factors. Given a reduced time frame, quickly determining the root cause of application performance degradation to reduce MTTR (mean time to response) and MTTI (mean time to identify) issues in the network may allow for more efficient operation of microservices-based applications. Disclosed herein is a framework and system configured to locate the root cause of performance degradation of a microservices-based application by correlating application performance with the performance of underlying components, such as computing resources and network devices used by the application.

[0006] In this regard, aspects of the technology can improve the operation of microservices-based applications themselves and computing systems that execute such microservices-based applications. That is, troubleshooting microservices-based applications that are quickly launched in response to demand and are therefore likely to be short-lived (e.g., on the order of milliseconds, seconds, and / or minutes), and automatic identification of root causes that degrade the performance of microservices-based applications can allow for more efficient (e.g., in terms of computing resources, such as processing cycles, memory storage consumption, memory bus bandwidth utilization, associated power consumption, etc.) operation of the underlying computing devices that execute the microservices-based applications. In other words, by automatically performing root cause analysis, various issues that affect the execution of microservices-based applications can more quickly identify the causes of degraded operation of the microservices-based applications, thereby resulting in more efficient operation of the microservices-based applications and the underlying computing devices that execute the microservices-based applications.

[0007] In one example, aspects of the present technology relate to a method comprising: receiving, by an analysis framework system, time series data including measurements of one or more performance indicators from a device of a network system; determining, by the analysis framework system and based on the time series data, one or more anomalies in the performance of the network system; creating a knowledge graph based on the time series data, the knowledge graph including a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; determining, by the analysis framework system in response to detecting one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with previous time series data, a causal graph, wherein the causal graph includes a second node associated with the performance indicator, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more first nodes in the first node and one or more second nodes in the second node, the second node being associated with an element residing at a different network service layer of the plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0008] In another example, aspects of the present technology relate to a computing system comprising: a memory configured to store time series data, the time series data including measurements of one or more performance indicators; and a processing circuit system configured to execute an analysis framework system, the analysis framework system configured to: determine one or more anomalies in the performance of a network system based on the time series data; create a knowledge graph based on the time series data, the knowledge graph including a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; in response to detecting one or more anomalies, and based on the knowledge graph and a knowledge graph trained with previous time series data, A method for determining a causal graph using a machine learning (ML) model, wherein the causal graph includes a second node associated with a performance indicator, wherein an edge between a first node and a second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more of the first nodes and one or more of the second nodes, the second node being associated with an element residing at a different network service layer of a plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0009] In another example, aspects of the present technology relate to a non-transitory computer-readable storage medium having instructions stored thereon, which, when executed, cause one or more processors to execute a framework analysis system, wherein the framework analysis system is configured to: receive time series data including measurements of one or more performance indicators; determine one or more anomalies in performance of a network system based on the time series data; create a knowledge graph based on the time series data, the knowledge graph including a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; in response to detecting one or more anomalies, and based on the knowledge graph and a knowledge graph trained with previous time series data, determine one or more anomalies in performance of a network system; A method for determining a causal graph using a machine learning (ML) model, wherein the causal graph includes a second node associated with a performance indicator, wherein an edge between a first node and a second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more of the first nodes and one or more of the second nodes, the second node being associated with an element residing at a different network service layer of a plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0010] In another example, aspects of the present technology relate to a method, comprising: receiving, by an analysis framework system, time series data including measurements of one or more performance indicators from a device of a network system; creating, by the analysis framework system, a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in the network system referenced in the time series, the first nodes representing elements residing at one or more of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates communication or computational dependency between the two nodes; receiving, by a graph analysis service of the analysis framework system, a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine the impact of a simulated fault, or a request to determine an application to a network path; determining, by the graph analysis service, a response to the graph analysis request; and outputting, by the analysis framework working system, a response to the graph analysis request.

[0011] In another example, aspects of the present technology relate to a computing device comprising: a processing circuit system configured to execute an analysis framework system; and a memory configured to store time series data including measurements of one or more performance indicators, wherein the analysis framework system is configured to: create a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in a network system referenced in the time series, the first nodes representing elements residing at one or more layers of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates communication or computational dependency between the two nodes; enable a graph analysis service of the analysis framework system to receive a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine the impact of a simulated fault, or a request to determine an application to a network path; enable the graph analysis service to determine a response to the graph analysis request; and output a response to the graph analysis request.

[0012] In another example, aspects of the present technology relate to a non-transitory computer-readable storage medium having instructions stored thereon, which when executed cause one or more processors to execute an analysis framework system, wherein the analysis framework system is configured to: receive time series data including measurements of one or more performance indicators; create a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in a network system referenced in the time series, the first nodes representing elements residing at one or more layers of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates a communication or computational dependency between the two nodes; cause a graph analysis service of the analysis framework system to receive a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine the impact of a simulated fault, or a request to determine an application to a network path; cause the graph analysis service to determine a response to the graph analysis request; and output a response to the graph analysis request.

[0013] In another example, aspects of the present technology relate to a method, comprising: determining, by a network system, a topology map for a network, the topology map representing multiple logical layers of the network; receiving, by the network system, an identifier associated with an application that communicates using the network; determining, by the network system and based on the topology map, a subgraph of the topology map based on the positions of nodes representing computing nodes that are hosts of the application in the topology map; determining, by the network system and based on the subgraph, a set of one or more probe modules to measure performance metrics associated with network communications associated with the application; generating, by the network system, for each of the one or more probe modules, configuration data corresponding to the type of the probe module; and outputting, by the network system and to the probe module, the configuration data corresponding to the probe module.

[0014] In another example, aspects of the present technology relate to a network system, including: a processing circuit system configured to determine a topology map for a network, the topology map representing multiple logical layers of the network; and a memory configured to store the topology map, wherein the processing circuit system is further configured to: receive an identifier associated with an application that communicates using the network; based on the topology map, determine a subgraph of the topology map based on the position of a node representing a computing node that is a host of the application in the topology map; based on the subgraph, determine a set of one or more detection modules to measure performance metrics associated with network communications associated with the application; for each of the one or more detection modules, generate configuration data corresponding to the type of the detection module; and output the configuration data corresponding to the detection module to the detection module.

[0015] In another example, aspects of the present technology relate to a non-transitory computer-readable storage medium having instructions stored thereon that, when executed, cause one or more processors to: determine a topology map for a network, the topology map representing multiple logical layers of the network; receive an identifier associated with an application that communicates using the network; based on the topology map, determine a subgraph of the topology map based on the positions of nodes representing computing nodes that serve as hosts for the application in the topology map; based on the subgraph, determine a set of one or more probe modules to measure performance metrics associated with network communications associated with the application; for each of the one or more probe modules, generate configuration data corresponding to the type of the probe module; and output the configuration data corresponding to the probe module to the probe module.

[0016] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a block diagram illustrating an example network system including a framework system for application to network root cause analysis in accordance with one or more aspects of the techniques described in this disclosure.

[0018] Figure 2 is a block diagram illustrating a root cause analysis framework system in accordance with aspects of the present disclosure.

[0019] Figure 3 is a block diagram illustrating a root cause analysis pipeline in accordance with aspects of the present disclosure.

[0020] Figure 4 is a block diagram illustrating an example network node in accordance with techniques described in this disclosure.

[0021] Figure 5is a flow diagram illustrating example operation of a root cause analysis framework system in accordance with one or more techniques of this disclosure.

[0022] Fig. 6A is a conceptual diagram illustrating an example knowledge graph that may be generated by a knowledge graph generator according to techniques described in this disclosure.

[0023] Figure 6B is a conceptual diagram illustrating an example pruned knowledge graph that may be generated by a knowledge graph generator and pruned by a graph pruner.

[0024] Fig. 7A is a diagram with added KPI nodes Fig. 6A A conceptual diagram of an example graph of a knowledge graph.

[0025] Figure 7B The diagram shows Fig. 7A Conceptual diagram of an example graph showing cause-and-effect relationships between KPI metrics.

[0026] Figure 7C The diagram shows Figure 7B Conceptual diagram of an example graph of weighted edges created by .

[0027] FIG. 8A to FIG. 8C is a conceptual diagram illustrating an example graph illustrating weights of edges in a causal graph in accordance with techniques of this disclosure.

[0028] 9A to 9C is a conceptual diagram illustrating an example graph illustrating weights of edges in a causal graph in accordance with techniques of this disclosure.

[0029] Fig.10 is a conceptual diagram illustrating a visual representation of an example cause-effect graph associated with the examples described herein.

[0030] Fig.11 is a conceptual diagram illustrating an example collection of APIs for accessing and updating a knowledge graph.

[0031] Fig.12 is a conceptual diagram illustrating an example set of APIs for accessing and updating a causal graph.

[0032] Fig.13 The diagram is from Figure 2 and Figure 3 Conceptual diagram of example page ranking output from the root cause analysis pipeline described in .

[0033] Fig.14 is a conceptual diagram of an example knowledge graph illustrating anomaly alerts detected at multiple layers in the graph.

[0034] Fig.15 It is shown Fig.14 Concept diagram of top-down analysis of the knowledge graph.

[0035] Fig.16 It is shown Fig.14 Concept diagram of bottom-up analysis of knowledge graphs.

[0036] Fig.17A and Fig. 17B is a conceptual diagram illustrating two knowledge graphs, which shows how the graphs evolve between two time windows.

[0037] Fig.18 is a conceptual diagram illustrating an example knowledge graph including a microservices-based application.

[0038] Fig.19A is a conceptual diagram illustrating the API exposed by the graph analytics service.

[0039] Fig.19B is a conceptual diagram illustrating a set of APIs exposed by a graph analysis service, which can be used to initiate graph change analysis and query the results of graph change analysis.

[0040] Fig.19C is a conceptual diagram illustrating an API for performing application-to-network mapping analysis.

[0041] Fig. 20 It is a graphic Figure 2 A flowchart of example operations of the graph analysis service shown in the example of FIG in performing various aspects of the graph analysis techniques described in the present disclosure.

[0042] Fig.21 is a block diagram illustrating additional details of a network system utilizing a root cause analysis framework system in accordance with aspects of the present disclosure.

[0043] Fig. 22 is a conceptual diagram illustrating an example knowledge graph with nodes across multiple layers of a network topology in accordance with techniques of this disclosure.

[0044] Fig.23 is a block diagram illustrating a data pipeline for application-aware telemetry processing in accordance with techniques of this disclosure.

[0045] Fig.24 is a conceptual diagram illustrating an example of mapping exceptions for an application according to the technology of the present disclosure.

[0046] Fig.25 is a conceptual diagram illustrating example endpoints for a probe generator service in accordance with techniques of this disclosure.

[0047] Fig.26 is a flow diagram illustrating example operation of a network system in performing aspects of application-aware active measurement techniques.

[0048] In the drawings and text, like reference numerals refer to like elements. DETAILED DESCRIPTION

[0049] Figure 1 1 is a block diagram illustrating an example network system 2 including a framework system for application to network root cause analysis according to one or more aspects of the techniques described in the present disclosure. The network system 2 can provide packet-based network services to a subscriber device 16. That is, the network system 2 can provide authentication and establishment of network access to the subscriber device 16 so that the subscriber device can begin to exchange data packets with a public network 12, which can be a packet-based internal or external network such as the Internet.

[0050] exist Figure 1 In the example of , the network system 2 includes an access network 6 that provides connectivity to a public network 12 via a wide area network 7 (hereinafter referred to as "WAN 7"). The WAN 7 and the public network 12 can provide packet-based services that can be requested and used by subscriber devices 16. As examples, the WAN 7 and / or the public network 12 can provide bulk data delivery, Internet Protocol Voice (VoIP), Internet Protocol Television (IPTV), Short Message Service (SMS), Wireless Application Protocol (WAP) services, or customer-specific application services. The public network 12 can include, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a layer 3 virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the access network 6, an enterprise IP network, or some combination thereof. In various examples, the public network 12 is connected to a public WAN, the Internet, or other networks. The public network 12 implements one or more packet data protocols (PDPs), such as IP (IPv4 and / or IPv6), X.25, or Point-to-Point Protocol (PPP), to enable public network 12 services to be based on packet transmissions.

[0051] In general, subscriber devices 16 are connected to network devices 18A to 18B (collectively referred to as "network devices 18") via access network 6 to receive connectivity for subscriber services for applications hosted by public network 12. A subscriber may represent, for example, an enterprise, a residential subscriber, or a mobile subscriber. Subscriber devices 16 may be, for example, a personal computer, a laptop, or other type of computing device located behind user equipment (CE) 11, which may provide local routing and switching functions. Each subscriber device 16 may run various software applications, such as word processing and other office support software, web browsing software, software supporting voice calls, video games, video conferencing, and email, etc. For example, subscriber devices 16 may be various network-enabled devices, commonly referred to as "Internet of Things" (IoT) devices, such as cameras, sensors (S), televisions, appliances, etc. In addition, subscriber devices 16 may include mobile devices that access data services of network system 2 via radio access network (RAN) 4. Example mobile subscriber devices include mobile phones, laptop or desktop computers with, for example, a wireless card, wireless-enabled netbooks, tablets, video gaming devices, pagers, smart phones, personal digital assistants (PDAs), and the like.

[0052] The network service provider operates or, in some cases, leases elements of the access network 6 (e.g., Figure 1 ) to provide packet transmission between subscriber device 16 and network device 18A. Access network 6 represents a network that aggregates data traffic from one or more of subscriber devices 16 for transmission to / from WAN 7 of a service provider. Access network 6 includes network nodes that execute communication protocols to transmit control and user data to facilitate communication between subscriber device 16 and network device 18A. Access network 6 may include a broadband access network, a wireless LAN, a public switched telephone network (PSTN), a customer premises equipment (CPE) network, or other types of access networks, and may include or otherwise provide connectivity for a cellular access network such as a radio access network (RAN), e.g., RAN 4. Examples of RANs include networks compliant with 5th generation (5G) mobile networks, 4th generation (4G) mobile networks Universal Mobile Telecommunications System (UMTS) architecture, the evolution of UMTS known as Long Term Evolution (LTE), 5G including enhanced mobile broadband, Mobile IP standardized by the Internet Engineering Task Force (IETF), and other standards proposed by the 3rd Generation Partnership Project (3GPP), 3rd Generation Partnership Project 2 (3GGP / 2), and the WiMAX Forum.

[0053] The network devices 18 may each be a customer edge (CE) router, a provider edge (PE) router, an SD-WAN edge device, a service device, a network appliance, a server that performs a virtualized network function, or other computing devices that provide connectivity between networks (e.g., access network 6 and public network 12) or network services. WAN 7 provides packet-based connectivity to subscriber devices 16 attached to access network 6 for access to public network 12 (e.g., the Internet). WAN 7 may represent a public network owned and operated by a service provider to interconnect multiple networks, which may include access network 6. In some examples, WAN 7 may implement multi-protocol label switching (MPLS) forwarding, and in this case may be referred to as an MPLS network or MPLS backbone. In some cases, WAN 7 represents multiple interconnected autonomous systems, such as the Internet, that provide services from one or more service providers. Public network 12 may represent the Internet. Public network 12 may represent an edge network that is coupled to WAN 7 via a transit network 22 and one or more network devices (e.g., a customer edge device such as a customer edge switch or router). The public network 12 may include a data center. Figure 1 In the example of , network device 18B may exchange packets with computing nodes 10A to 10D (“computing nodes 10 ”) via virtual network 20 , and network device 18B may forward the packets to public network 12 via transit network 22 .

[0054] In the example of the network system 2 including the wired / broadband access network, the network device 18A or 18B can represent a broadband network gateway (BNG), a broadband remote access server (BRAS), an MPLS PE router, a core router or gateway, or a cable modem termination system (CMTS). In the example of the network system 2 including the cellular access network as the access network 6, the network device 18A or 18B can represent a mobile gateway, such as a gateway general packet radio service (GPRS) service node (GGSN), an access gateway (aGW), or a packet data network (PDN) gateway (PGW). In other examples, the functions described about the network device 18B can be implemented in a switch, a service card, or another network element or component. In some examples, the network device 18B itself can be a service node.

[0055] A network service provider that manages at least a portion of the network system 2 typically provides services to subscribers associated with devices (e.g., subscriber devices 16) that access the network system 2. The services provided may include, for example, traditional Internet access, VoIP, video and multimedia services, and security services. As described above with respect to WAN 7, WAN 7 may support multiple types of access network infrastructures that connect to a service provider network access gateway to provide access to the provided services (e.g., services provided by service node 10). In some examples, the network system may include subscriber devices 16 attached to multiple different access networks 6 having different architectures.

[0056] Typically, any one or more of the subscriber devices 16 may request authorization and data services by sending a session request to a gateway device such as network device 18A or 18B. Accordingly, the network device may access a central server (not shown) such as an authentication, authorization, and accounting (AAA) server to authenticate one of the subscriber devices 16 requesting network access. Once authenticated, any of the subscriber devices 16 may send subscriber data traffic to WAN 7 to access and receive services provided by public network 12, and such packets may pass through network device 18A or 18B as part of at least one packet flow. In some examples, network device 18A may forward all authenticated subscriber traffic to public network 12, and if the subscriber traffic requires services on computing node 10, network device 18B may apply services and / or direct specific subscriber traffic to data center 9. Service applications to be applied to subscriber traffic may be hosted on computing node 10.

[0057] For example, the network system 2 includes a data center 9 having a cluster of computing nodes 10 that provide an execution environment for a virtualized network service. In some examples, each computing node of the computing node 10 represents a service instance. Each node of the computing node 10 can apply one or more services to a traffic flow. In this way, the network device 18B can guide the subscriber packet flow through the defined service set provided by the computing node 10. That is, in some examples, each subscriber packet flow can be forwarded by a specific ordered combination of services provided by the computing node 10, and each ordered set is referred to as a "service chain" in this article. As an example, the computing node 10 can apply a stateful firewall (SFW) and security service, deep packet inspection (DPI), carrier-grade network address translation (CGNAT), traffic destination function (TDF) service, media (voice / video) optimization, Internet Protocol Security (IPSec) / virtual private network (VPN) service, hypertext transfer protocol (HTTP) filtering, counting, accounting, billing and / or load balancing of packet flows, or other types of services applied to network traffic.

[0058] In some examples, the network system 2 includes a software defined network (SDN) and a network function virtualization (NFV) architecture. In these examples, an SDN controller (not shown) can provide a controller for configuring and managing the routing and switching infrastructure of the network system 2.

[0059] Although illustrated as part of the data center 9, the computing node 10 can be a network device coupled by one or more switches or virtual switches of the WAN 7. In one example, each computing node in the computing node 10 can be run as a virtual machine (VM) in a virtual computing environment. In addition, the computing environment can include an extensible cluster of general-purpose computing devices, such as services based on x86 processors. As another example, the computing node 10 can include a combination of general-purpose computing devices and special-purpose devices. As a virtualized network service, the various network services provided by the computing node 10 can be horizontally scaled by the allocation of virtualized memory, processor utilization, storage and network policies, and by adding additional load balancing VMs, just like in a modern data center. In other examples, the computing node 10 can be a gateway device or other router. In another example, the functions described for each computing node in the computing node 10 can be implemented in a switch, a service card, or another network element or component.

[0060] Subscriber device 16 can be configured to utilize services provided by one or more applications in application 30 hosted on a server, which is part of cloud-based service 26. In some aspects, cloud-based service 26 can be provided from one or more data centers including data center 9 by computing node 10. Application 30 can be configured to provide a single service, or it can be configured as multiple microservices. For the purpose of illustration, it is assumed that application 30 is configured as multiple microservices. Subscriber device 16 can utilize the service of application 30 by communicating a request to application 30 via public network 12 and receiving a response from application 30. In some aspects, application 30 can be a containerized application (or microservice). Containerization is a virtualization scheme based on operating system-level virtualization. Containers are lightweight and portable execution elements for applications isolated from each other and from the host. Such isolation systems represent containers, such as those provided by open source DOCKER container applications or CoreOs Rkt ("rocket"). Like a virtual machine, each container is virtualized and can remain isolated from the host and other containers. However, unlike virtual machines, each container can omit a separate operating system and instead provide an application suite and application-specific libraries. In general, containers are executed by the host as isolated user space instances, and can share an operating system and common libraries with other containers executed on the host. Therefore, containers can require less processing power, storage, and network resources than virtual machines. A group of one or more containers can be configured to share one or more virtual network interfaces for communicating on a corresponding virtual network.

[0061] Because containers are not tightly coupled to the host hardware computing environment, applications can be bound to a container image and executed as a single lightweight package on any host or virtual host that supports the underlying container architecture. In this way, containers solve the problem of how to make software work in different computing environments. Containers offer the promise of running consistently from one computing environment to another (virtual or physical).

[0062] As described herein, computing devices within the network system 2 can provide network monitoring services. For example, the network device 18 and / or the computing node 10 are configured as measurement points (e.g., with a detection module) to provide network monitoring services to determine, for example, network performance and functionality and the interconnection of service chains. The detection module can provide telemetry data (e.g., in the form of time series data, which can also be referred to as "time series data"), which can be used to determine the health of some or all networks. The detection module can be implemented as a plug-in module provided as part of the cloud (as a software as a service (SAAS) solution), software deployed locally in the NFV environment, software modules installed to the host computing device, or other implementations. A computing device configured with a detection module can send and / or receive test packets to calculate one or more key performance indicators (KPIs) of the network, such as latency, delay (inter-frame gap), jitter, packet loss, throughput, etc. The probe module can send test packets according to various protocols, such as Hypertext Transfer Protocol (HTTP), Internet Control Message Protocol (ICMP), Speedtest, User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Operation, Administration, and Maintenance (OAM) functions (e.g., Y.1731), Two-Way Activity Measurement Protocol (TWAMP), Internet Protocol Television (IPTV) and Over-the-Top (OTT) protocols, VoIP phones and Session Initiation Protocol (SIP), mobile radio, remote packet inspection, and other protocols that measure network performance. The probe module on the computing node can calculate KPIs related to resource utilization such as CPU utilization, memory utilization, etc.

[0063] exist Figure 1 In the example of , network device 18A is configured with a detection module (“PM”) 19A (“PM 19A”), network device 18B is configured with a detection module 19B (“PM 19B”), computing node 10A is configured with a detection module 19C (“19C”), computing node 10B is configured with a detection module 19D (“19D”), computing node 10C is configured with a detection module 19E (“19E”), computing node 10D is configured with a detection module 19F (“19F”), and at least one application 30 of cloud-based service 26 is configured with a detection module 19G (“PM 19G”). Although each of network device 18 and computing node 10 is described as being configured with a detection module, any number of network devices and / or service nodes may be configured with detection modules to provide monitoring services. Detection modules 19A to 19G are collectively referred to as “detection modules 19”. Detection modules 19 may generate and / or collect performance statistics (e.g., KPIs) about application and network performance.

[0064] The detection modules 19 and other devices and services send their KPIs 24 as time series data to the cloud-based analysis framework system 28. The cloud-based analysis framework system 28 can receive the KPIs and other data from the detection modules 19 and use the received data to detect anomalies in the network and / or computing node operations. In some aspects, the cloud-based analysis framework system 28 can be implemented at least in part as a containerized framework system. Further details of the processing of time series data are described below with respect to Figure 2 and Figure 3 is provided.

[0065] Figure 2 2 is a block diagram illustrating a root cause analysis framework system according to aspects of the present disclosure. The root cause analysis framework system 200 may be implemented as a cloud-based analysis framework system 28 ( Figure 1 ). In some aspects, the root cause analysis framework system 200 includes a detection service 201, a telemetry service 207, a telemetry collector 216, an event stream engine 218, and a fault locator 240.

[0066] Probe service 201 includes creating, configuring, and / or controlling probe modules (e.g., Figure 1 In some aspects, the probe service 201 may include a probe generator service 202, a probe configuration service 204, and a probe control service 206. The probe generator service 202 analyzes the network configuration and may generate a probe module suitable for a specific network device or a specific type of network device (both physical and virtual) that may be present in the network system 2.

[0067] The probe configuration service 204 can configure the probe modules generated by the probe generation service 202 for execution on the network system 2. The probe control service 206 can deploy the probe modules and monitor the execution of the probe modules (e.g., the probe module 19). More details about the probe generator service 202 and the probe configuration service 204 are described below in relation to Figure 21 to Figure 26 is provided.

[0068] Fig.2121 is a block diagram illustrating additional details of a network system 2100 configured to use a root cause analysis framework system according to various aspects of the present disclosure. In some aspects, the network system 2100 may include a topology service 2110 and an application location service 2118 in addition to a probe generator service 202, a probe configuration service 204, a probe control service 206, and a knowledge graph generator 220. The topology service 2110 may determine the network topology for which the network probe module needs to be configured to measure end-to-end network performance. Examples of topologies that the topology service 2110 may support include an underlying network topology 2112, an overlay network topology 2114, and other topologies 2116 such as a Kubernetes cluster topology.

[0069] The application location service 2118 can determine the subgraph in the network topology that needs to be monitored from the application perspective. That is, the application location service can determine the subset of the network topology used to send and receive network data for the application. This can be determined based on the service nodes on which different instances of the application are hosted and the network responsible for the communication between the set of endpoint hosts for the network traffic associated with the application. The application location service 2118 can use the system's knowledge graph as a graph data structure to represent all applications and corresponding infrastructure components. In some aspects, the knowledge graph generator 220 generates a knowledge graph. Application-to-network queries on this knowledge graph help determine the subset of the network responsible for application communication.

[0070] The probe generator service 202 is responsible for determining a set of network probe modules that need to be configured for a given network, the set of network probe modules being determined based on the network topology and the corresponding application layout in the network. The probe generator service 202 may receive the network topology and application location as input, and may determine a set of network probe modules that may obtain performance measurements associated with the application. The set of network probe modules generated by the probe generator service is localized to the portion of the network topology that is used to communicate the network traffic of the application.

[0071] Depending on the environment, different types of network probe modules may need to be generated. For example, if the application has specific QoS requirements, a probe module needs to be generated to detect specific QoS bits (for example, DSCP bits in the IP header) to actively measure the performance of the network from the application's perspective. In addition, there are different types of topologies, such as underlying networks, overlay networks, kubernetes clusters, etc., and probe modules need to be generated for different types of topologies. Different probe generator plug-ins can be supported depending on the topology, which are responsible for determining a set of probe modules while taking into account application requirements, coverage, and the cost for active probing for a given topology. Examples of different plug-ins include CLOS data center network topology plug-in, overlay network plug-in, SD-WAN topology plug-in, Kubernetes cluster plug-in, etc. In Fig.21 In the example of , the underlying probe generator 2104 determines a set of probe modules to be used to measure the performance of network components in a subset of nodes of the underlying network, which is used to pass data to the application. The overlay probe generator 2106 determines a set of probe modules to be used to measure the performance of network components in a subset of nodes of the overlay network, which is used to pass data to the application. The custom probe generator 2108 can generate probe modules that are used to measure the performance of components utilized by applications such as Kubernetes components. Each of generators 2104, 2106, and 2108 can generate a list of probe modules, wherein the list indicates the probes to be used, the servers performing the probes, and a set of one or more servers to be probed. In some aspects, the probe modules are generated only for the portion of the network topology used by the application.

[0072] Independent of the input topology or the application mapped to the network area, different probe generation service 202 adapters can have a consistent format to represent the probe module that needs to be configured. It is represented as a source server with a pointer to a list of destination servers for the probe module. Note here that the server can be a bare metal server, a VM, or a container depending on the topology.

[0073] server1_uuid:[server2_uuid,server3_uuid]

[0074] server2_uuid:[server4_uuid]

[0075] server3_uuid:[server1_uuid,server4_uuid]

[0076] In some aspects, the probe generator service 202 may expose Fig.25 The endpoints are shown.

[0077] After determining the probe modules that need to be configured, the probe generation service 202 can publish the definitions for the probe modules on the message bus 2116. The probe definitions can be maintained in a probe definition database 2138. If it is desired to perform an analysis of the probe definition to determine whether it was generated correctly, and whether the probe modules were correctly configured and pushed to their final destinations, the probe definition database 2138 can store the probe definitions. In addition, the probe definition database 2138 can store the status of probe generation, configuration, and distribution. If it becomes desirable to disable a probe module later, the information in the probe definition database can be used to appropriately disable the probe module and update the probe distribution status for the application.

[0078] In network system, can have many types of detection module available.In some cases, equipment or system manufacturer can provide the detection module that is dedicated to its equipment.In other cases, the detection module that is used for equipment or component can be obtained via open source or other mechanism.Detection configuration service 204 can receive detection definition, and is responsible for converting desired detection intention (as expressed in detection definition) into specific detection configuration based on actual detection service.In some respects, detection configuration service 204 can have plug-in architecture, and wherein plug-in can be for the detection module adaptation detection configuration service of specific type.

[0079] exist Fig.21 In the example shown, the probe configuration service 204 includes probe compiler plug-ins 2122A to 2122N and probe configuration plug-ins 2124A to 2124N. The probe compiler plug-in 2122 receives the probe definition and compiles the probe definition into a probe configuration for a specific type of probe module. The probe configuration plug-in 2124 can receive the configuration (as configuration data) from its corresponding probe compiler plug-in 2122 and push (e.g., send or transmit) the configuration to the appropriate probe module. As an example, the Netrounds testing and service assurance platform available from Juniper Networks, Inc. provides various types of probe modules.

[0080] In this example, a probe compiler plugin 2122A can receive a probe definition (as configuration data) and compile the probe definition into a configuration suitable for a Netrounds probe module. A corresponding probe configuration plugin 2124A can receive the probe configuration (as configuration data) from the probe compiler plugin 2122A and send the probe configuration data to the appropriate Netrounds probe module. Similarly, there are other probe modules for generating probe modules (e.g., custom probe modules or probe modules from other vendors), and then other probe compilers 2122 and corresponding probe configuration plugins can be provided to the probe configuration service 204, which can generate and provide the probe configuration data to its corresponding probe module.

[0081] In some aspects, the detection module is associated with an application. The system can use this association to determine applications that may be affected by an anomaly detected by the system. For example, assume that the detection module returns telemetry data indicating an anomaly. The system can identify the application associated with the detection module as being potentially affected by the anomaly.

[0082] In some aspects, the probe configuration service 204 can send the probe configuration data directly to the appropriate probe module. In some aspects, the probe configuration service can send the probe configuration to the probe control service 206, which can in turn send the probe configuration to its appropriate probe module.

[0083] In this regard, aspects of the above technology enable Fig.21 The network system shown is capable of determining a topology map for a network, wherein the topology map represents multiple logical layers of the network. The topology service 2110 can receive an identifier associated with an application that communicates using the network. The topology service 2110 can determine, based on the topology map, a subgraph of the topology map based on the position of a node in the topology map representing a computing node that is a host of the application. The probe generation service 202 can determine, based on the subgraph, a set of one or more probe modules to measure performance metrics associated with network communications associated with the application, and generate configuration data corresponding to the type of the probe module for each of the one or more probe modules. The probe generator service 202 can output configuration data corresponding to the probe module to the probe module.

[0084] Fig. 22 is a conceptual diagram illustrating an example knowledge graph 2200 having nodes across multiple layers of a network topology, including nodes representing detection modules associated with an application. The layers include a microservice layer, a container group (pod) layer, a computer layer, a network detection layer, and an underlying network layer. Fig. 22 In the example shown, two different microservice applications 1 and 2 are mapped to the set of nodes responsible for a given application in the network region. Fig. 22 As shown in the example, the application location service 2118 has determined that application 1 is mapped to partition 1 of the network topology, and application 2 is mapped to partition 2 of the network topology. The probe generator service 202 can use the mapping of applications to network sub-areas to determine a set of probe modules to be configured to probe network areas in an application-aware manner. As seen, probe module PR1 is used to measure the network performance of network partition 1 between host 1 (H1) and host 2 (H2). Probe module PR2 is used to measure the performance of network partition 2 between host 3 (H3) and host 4 (H4). Note that depending on the application deployment scenario, multiple applications can share the same network. Therefore, from the perspective of all multiple applications, the same set of network probe modules can be used to measure the performance of the network.

[0085] Back to Figure 2 , the telemetry service 207 includes cross-domain Figure 1Multiple layers of a networked system of system 2 collect telemetry services from detection modules. As used herein, a layer may refer to a set of components (real or virtual) that use resources and services provided by another set of components at a different layer. Where a first component uses resources or services of a second component, the first component may be said to be dependent on the second component. For example, an application at the application layer may use resources and services of computing nodes at the computing node layer. The computing node layer, in turn, may use resources and services provided by network devices at the network layer, and so on. Therefore, applications at the application layer depend on computing nodes at the computing node layer, which in turn may depend on physical network devices at the physical layer.

[0086] In some aspects, the telemetry service 207 includes a telemetry collector 216, a network telemetry collector 212, and a device / structure telemetry collector 214. The telemetry collector 216 can collect application telemetry 208 such as application performance data. Application telemetry can be collected using a service mesh, Istio, etc. The telemetry collector 216 can also collect compute / container group telemetry 210. The compute / container group telemetry 210 can include Kubernetes container group and node performance telemetry data, such as central processor unit (CPU) usage, memory usage, network statistics, etc. In some aspects, the telemetry collector 212 can be implemented using a Prometheus monitoring system combined with a Thanos high availability and data storage system. Both Prometheus and Thanos are open source components.

[0087] The network telemetry collector 212 can collect telemetry data that measures end-to-end network performance. For example, the network telemetry collector 212 can collect network performance measurements obtained or created by the detection module 19. The device / structure telemetry collector 214 can collect telemetry data from network devices (e.g., network device 18) in the network structure of the network system 2.

[0088] The event stream engine 218 can receive telemetry data collected by the telemetry service 207 (such as the telemetry collector 216, the network telemetry collector 212, the device telemetry collector 214, and other telemetry collectors), and can store the data as time series data in the time series database (TSDB) 238. The data received from the telemetry service 207 can be in different formats, depending on the source of the telemetry data and the collector used to collect the telemetry data. In some aspects, the data transformer 232 applies data transformations to the input data to normalize the data and / or put the data in a standardized format. In some aspects, the data is transformed into an open TSDB format. In some aspects, the event stream engine 218 can be implemented at least in part using a Kafka stream processing system.

[0089] The event stream engine can apply alert rules 234 to the incoming telemetry data. Alert rules 234 can be a set of one or more rules that indicate that the telemetry value is abnormal, such as outside of an expected range, above or below a limit, etc.

[0090] Anomaly detector 236 can apply machine learning (ML) model 240 to the time series of telemetry data from TSDB 238 to determine anomalies associated with the time series of telemetry data. In some aspects, ML model 240 can be trained using supervised or unsupervised learning techniques to discover KPIs in telemetry data that are related to, for example, Figure 1 The relationships between abnormal conditions within one or more layers of the multi-layer network system of system 2 can span multiple layers of the network model. For example, a KPI associated with a network device can be related to a KPI of an application that utilizes the network device to communicate with another service. As an example, a KPI indicating a bottleneck at a network device or a CPU overload on a host device can affect a response time KPI of an application running on the host and / or utilizing the network device for communicating with another endpoint in the network.

[0091] Fig.23 23 is a block diagram illustrating a data pipeline for application-aware telemetry processing. The data pipeline may include one or more probe telemetry collectors 2301, one or more probe telemetry transformers 2307, data aggregation 2314, anomaly detection 2316, and anomaly mapping 2318 applications. The probe telemetry collector supports the collection of telemetry for different types of probe services such as Netrounds probe modules, custom probe modules, etc. In general, the probe telemetry collector may be a provider of a telemetry service 207 ( Figure 2 ) instances of any collector. The Probe Telemetry Collector 2301 can support multiple plugins to collect telemetry from different sources. Fig.23 In the example shown, the probe telemetry collector 2301 includes a Netrounds collector 2362 , a first custom probe collector 2304 , and a second custom probe collector 2306 .

[0092] The detection telemetry transformation 2307 can transform the data collected from the corresponding telemetry collector 2301. The detection telemetry transformation can be Figure 2 2302. The data from different probe collectors may arrive in different formats. As a result, the corresponding data transformer plug-in converts the input data in the input format into a standard telemetry format. For example, Netrounds transformer 2308 is used to transform raw data collected by Netrounds collector 2302. Similarly, custom transformers 2310 and 2312 are used to transform probe data collected by custom probe collectors 2304 and 2306, respectively.

[0093] Data aggregation 2314 is a pipeline stage responsible for aggregating probe module data at different granularities. For example, data aggregation 2314 can aggregate probe telemetry data at each compute node, which aggregates probe telemetry data for all probe modules configured on a given computer node. Similarly, probe telemetry data across all servers under a top of rack (ToR) switch is aggregated to provide the health of the network from a ToR perspective. At a higher granularity, probe telemetry data across a group of ToR switches is combined to generate a ToR group baseline from a network perspective, such as latency 95%, loss 95%, delay change.

[0094] Anomaly detection 2316 uses machine learning and rule-based methods to determine any abnormal behavior in the KPIs determined in the data aggregation layer 2314 of the pipeline. For example, if the latency and loss curves of the network deviate significantly from the baseline behavior, the service generates an anomaly signal. Anomaly detection 2316 is triggered for data aggregated at different granularities, such as servers, server groups under leaf node (ToR) switches, etc. Anomaly detection 2316 can be Figure 2 Implementation of anomaly detector 236.

[0095] Mapping anomalies from network to application 2318 ("Anomaly Mapping 2 Application 2318") creates a snapshot of the application and its mapping on the underlying infrastructure. Once the end-to-end health of the network is determined at different granularities, the framework system proposes a graph-based approach for mapping network anomalies to applications that may be affected due to anomalies in end-to-end behavior in specific areas of the network. This requires creating a snapshot of the application and its mapping on the underlying infrastructure in the form of a knowledge graph. Once the knowledge graph is built, the query set on the knowledge graph will be able to determine the nodes in the application layer that are affected by any anomalies in the underlying network layer.

[0096] Fig.24 is a conceptual diagram illustrating an example of mapping an exception for an application. Fig.24 In the example shown, anomalies detected using telemetry data from a probe module PR1 associated with Application 1 are mapped to Application 1 service S1 , and the service uses the association of probe module PR1 with Application 1 to invoke edges S1 - S3 and S1 - S4 .

[0097] Fig.26 is a flow diagram illustrating example operation of a network system in performing aspects of application-aware active measurement techniques. Fig.21The illustrated network system may determine a topology map for a network, wherein the topology map represents multiple logical layers of the network (2600). A topology service 2110 may receive an identifier associated with an application that utilizes a network for communication (2602). The topology service 2110 may determine, based on the topology map, a subgraph of the topology map based on the position of a node in the topology map that represents a computing node that is a host of the application (2604). The probe generation service 202 may determine, based on the subgraph, a set of one or more probe modules to measure performance metrics associated with network communications associated with the application (2606), and for each of the one or more probe modules, generate configuration data corresponding to a type of the probe module (2608). The probe generator service 202 may output configuration data corresponding to the probe module to the probe module (2610).

[0098] Telemetry service 207 includes cross-domain Figure 1 Multiple layers of a networked system of system 2 collect telemetry services from detection modules. As used herein, a layer may refer to a set of components (real or virtual) that use resources and services provided by another set of components at a different layer. Where a first component uses resources or services of a second component, the first component may be said to be dependent on the second component. For example, an application at the application layer may use resources and services of computing nodes at the computing node layer. The computing node layer may in turn use resources and services provided by network devices at the network layer, and so on. Thus, applications at the application layer depend on computing nodes at the computing node layer, and in turn, the computing node layer may depend on physical network devices at the network physical layer (e.g., Figure 1 The computing node 10 is shown in the example.

[0099] In some aspects, the telemetry service 207 includes a telemetry collector 216, a network telemetry collector 212, and a device / structure telemetry collector 214. The telemetry collector 216 can collect application telemetry 208 such as application performance data. Application telemetry can be collected using a service mesh, Istio, etc. The telemetry collector 216 can also collect compute / container group telemetry 210. The compute / container group telemetry 210 can include Kubernetes container group and node performance telemetry data, such as central processor unit (CPU) usage, memory usage, network statistics, etc. In some aspects, the telemetry collector 212 can be implemented using a Prometheus monitoring system combined with a Thanos high availability and data storage system. Both Prometheus and Thanos are open source components.

[0100] The network telemetry collector 212 can collect telemetry data that measures end-to-end network performance. For example, the network telemetry collector 212 can collect network performance measurements obtained or created by the probe module 19.

[0101] The device / structure telemetry collector 214 can collect telemetry data from network devices (e.g., network device 18) in the network structure of the network system 2.

[0102] The event stream engine 218 can receive telemetry data collected by the telemetry service 207 (such as the telemetry collector 216, the network telemetry collector 212, the device telemetry collector 214, and other telemetry collectors), and can store the data as time series data in the time series database (TSDB) 238. The data received from the telemetry service 207 can be in different formats, depending on the source of the telemetry data and the collector used to collect the telemetry data. In some aspects, the data transformer 232 applies data transformations to the input data to normalize the data and / or put the data in a standardized format. In some aspects, the data is transformed into an open TSDB format. In some aspects, the event stream engine 218 can be implemented at least in part using a Kafka stream processing system.

[0103] The event stream engine can apply alert rules 234 to the incoming telemetry data. Alert rules 234 can be a set of one or more rules that indicate that the telemetry value is abnormal, such as outside of an expected range, above or below a limit, etc.

[0104] Anomaly detector 236 can apply machine learning (ML) model 241 to the time series of telemetry data from TSDB 238 to determine anomalies associated with the time series of telemetry data. In some aspects, ML model 241 can be trained using supervised or unsupervised learning techniques to discover KPIs in telemetry data that are related to, for example, Figure 1 The relationships between abnormal conditions within one or more layers of the multi-layer network system of system 2 can span multiple layers of the network model. For example, a KPI associated with a network device can be related to a KPI of an application that utilizes the network device to communicate with another service. As an example, a KPI indicating a bottleneck at a network device or a CPU overload on a host device can affect a response time KPI of an application running on the host and / or utilizing the network device for communicating with another endpoint in the network.

[0105] When a fault (e.g., an abnormal condition) is detected using an alert rule or using an ML model 241, anomaly detector 236 can invoke fault locator 240. In some aspects, fault locator 240 includes knowledge graph generator 220, causal graph generator 222, graph pruner 224, and ranking service 225. In some aspects, knowledge graph generator 220, causal graph generator 222, graph pruner 224, and ranking service 225 can be executed as part of a root cause analysis pipeline. Figure 3An example of a root cause analysis pipeline is shown in .

[0106] The knowledge graph generator 220 can generate dependencies between different applications and infrastructure entities of the network system 2 based on the data in the TSDB 238 and the relationships found in the ML model 241. The causal graph generator 222 generates another graph data on top of the knowledge graph generated by the knowledge graph generator 220 to form a causal graph. The causal graph captures the causal relationships between different performance KPIs and abnormal conditions. The graph pruner 224 prunes the knowledge graph and the causal graph to determine a subset of the graph to be used in root cause location. The ranking service 225 ranks the nodes in the causal graph and indicates the nodes that may be the root cause of the observed anomaly in the order of the likelihood of the node causing the anomaly.

[0107] The root cause analysis framework system 200 may also include a graph analysis service 226 and a user interface (UI) 228. The graph analysis system 226 may receive graph queries on the knowledge and causal graphs for further investigation and verification of fault location results. In some aspects, the graph analysis system 226 may be used to perform alarm propagation path analysis, graph change analysis, fault injection analysis, or applied to network mapping analysis. Figures 14 to 16 , Fig.17A , Fig. 17B , Figure 8 and FIG. 19A to FIG. 19C Further details of the graph analysis system 226 are provided.

[0108] Alarm propagation path analysis

[0109] The goal of alarm propagation path analysis is to determine the propagation path of an alarm through different layers in a knowledge graph that can explain different symptoms (e.g., anomalies) observed in dependent layers.

[0110] Fig.14 1 is a conceptual diagram of an example knowledge graph illustrating anomaly alerts detected at multiple layers in the graph. The knowledge graph generator 220 may create an example knowledge graph 1400 including multiple first nodes (e.g., layer 1 node 1 and node 6) in a network system referenced in the time series data. The first node may represent an element residing in one or more layers of multiple layers (e.g., layers 1 to 5) associated with the network system, wherein an edge between two nodes of the multiple nodes indicates a communication or computational dependency between the two nodes.

[0111] In this example, the graph analysis service 226 performs an analysis of the graph topology to determine the propagation path of an anomaly alert that can explain the symptom alert detected at node 6. In some aspects, static analysis includes top-down analysis and / or bottom-up analysis. For example, the analysis framework system 200 can be configured to analyze the graph topology in a top-down or bottom-up manner as described herein. Top-down analysis of the graph can be useful for determining why an application is executing slower than expected. If a compute node or network node detects an anomaly, bottom-up analysis can be useful for determining which applications may be affected.

[0112] The graph analysis service 226 may receive a graph analysis request, which may include one or more of a request to determine a fault propagation path, a request to determine the impact of a simulated fault, or a request to determine an application to a network path. The graph analysis service 226 may determine a response to the graph analysis request and output the response to the graph analysis request.

[0113] Fig.15 It is a graphic Fig.14 Concept diagram of top-down analysis of the knowledge graph. Fig.15 A top-down analysis of Figure 2 In top-down analysis, the graph analysis service 226 performs a graph analysis query on the graph starting from the dependent node where the anomaly was detected, for which an alert can be raised. Fig.15 In the example shown, node 6 is a dependent node for which an alarm propagation analysis is performed. For example, node 6 may be a node representing an application to which an alarm regarding application performance has been issued. Figure 2 The anomaly detector 236 or other monitoring system may issue an alarm in response to a potential anomaly in the network system.

[0114] exist Fig.15 In the illustrated example, the top-down analysis performed by the framework system 200, for example, determined two possible paths (indicated by dashed lines 1504 and 1506) by which the anomaly may have propagated from a node in layer 5 to a node in layer 1. Once the possible paths are determined, the graph pruner 224 can perform further pruning of the candidate paths by selecting only those paths with intermediate nodes where anomalies have been detected. In this case, path 1504 is a more likely path for anomaly propagation because node 3 in the path also shows an anomaly associated with its KPI (key performance indicator) metric.

[0115] In this instance, the graph analysis request includes a request to determine a fault propagation path. In response to the request to determine the fault propagation path, the graph analysis service 226 can determine a first node and a second node that are each experiencing a fault based on the time series data stored in the TSDB 238, and the first node depends on the second node. The graph analysis service 226 can traverse the knowledge graph 1400 to determine one or more paths from the first node to the second node through the knowledge graph, and select a path (e.g., path 1504) in one or more paths (e.g., paths 1504 and 1506) in which an intermediate node (e.g., node 3) in the path between the first node and the second node is experiencing a fault as the fault propagation path. The graph analysis service 226 can output the fault propagation path (e.g., via UI 228).

[0116] In this example of top-down analysis, the request may identify a first node (e.g., node 6), and the graph analysis service 226 traverses the knowledge graph 1400 from the first node (e.g., node 6) to the second node (e.g., node 5). Fig.16 In the bottom-up analysis shown in the example of , the request includes an identifier of a second node (e.g., node 5), and the graph analysis system 226 can traverse the knowledge graph 1400 from the second node (e.g., node 5) to the first node (e.g., node 6).

[0117] Fig.16 It is a graphic Fig.14 A conceptual diagram of the bottom-up analysis of the knowledge graph. Fig.16 In the example shown, the graph analysis service 226 receives a request to perform a bottom-up analysis, and the alarm propagation analysis begins at node 5, which other layer nodes depend on. The graph analysis service 226 traverses the knowledge graph to find possible paths from node 5 to node 6 in the graph. Note that in this example, the output of the graph analysis service 226 using both the top-down and bottom-up approaches is the same, where paths 1604 and 1606 are determined through which alarm propagation can occur, and are consistent with Fig.15 1504 and 1506 are the same. Similar to the top-down approach described above, the graph pruner module 224 performs further pruning of candidate paths by selecting only those paths with intermediate nodes for which anomalies have also been detected. In this case, path 1604 is a more likely path for anomaly propagation because the intermediate nodes in the path (e.g., node 3) also show anomalies related to its KPI metric.

[0118] Fig.19A is a conceptual diagram illustrating the API exposed by the graph analysis service. The API for the graph analysis service 226 can be used to initiate alert propagation analysis for a graph and to retrieve paths through which observed anomalies may propagate to the application layer.

[0119] Graph change analysis can determine changes in the application to the network topology over time. The knowledge graph generator 220 generates a knowledge graph for a given monitoring window that captures the relationship between the application and its underlying infrastructure components during the time window. The knowledge graph also captures all alerts related to the performance metrics for each node in the knowledge graph. When two such graphs are created for adjacent time windows, the graph analysis service 226 can perform graph change analysis in response to a request to answer the following example questions:

[0120] 1. How does the application layer evolve in two adjacent time windows?

[0121] 2. How does the underlying computational layer evolve between two adjacent time windows?

[0122] 3. How does the network health evolve between two adjacent time windows?

[0123] 4. How many new nodes / edges are added / deleted between two monitoring time windows?

[0124] 5. How do alerts evolve / change between two monitoring time windows across all layers in the graph?

[0125] Fig.17A and 17B is a conceptual diagram illustrating two knowledge graphs generated by the knowledge graph generator 220, showing how the graphs evolve between two time windows. Fig.17A and Fig. 17B The illustrated example shows a per-layer analysis of what has changed between two graphs generated on two different monitoring windows. Fig.17A represents an example knowledge graph for the time range T1 to T2, and Fig. 17B The graph of represents an example knowledge graph for the time range T2 to T3. Graph analysis of the two graphs shows the following changes between time T1 to T2 and time T2 to T3:

[0126] Layer 1 (Application):

[0127] Nodes added: 13

[0128] Nodes deleted: None

[0129] Edges added: 13-6

[0130] Edges Deleted: None

[0131] Alerts added: 13

[0132] Alerts Resolved: None

[0133] Layer 2 (application instance container group layer):

[0134] Nodes added: 14

[0135] Nodes deleted: None

[0136] Sides added: 13-14

[0137] Edges Deleted: None

[0138] Alerts added: None

[0139] Alerts Resolved: None

[0140] Layer 3 (Computer Layer):

[0141] Nodes added: None

[0142] Nodes deleted: None

[0143] Edges added: None

[0144] Edges Deleted: None

[0145] Alerts added: None

[0146] Alerts resolved: 3

[0147] Layer 4 (Network Detection Layer)

[0148] Nodes added: None

[0149] Nodes deleted: 9

[0150] Edges added: None

[0151] Edges deleted: 8-9

[0152] Alerts added: 11

[0153] Alerts Resolved: None

[0154] Layer 5 (network device layer):

[0155] Nodes added: None

[0156] Nodes deleted: 10

[0157] Edges added: None

[0158] Edges deleted: 9-10, 9-5

[0159] Alerts added: 12

[0160] Alerts Resolved: None

[0161] In this example, the graph analysis request includes a request to determine changes in a graph, wherein the request identifies a first knowledge graph 1700A from a first time (e.g., T1 to T2) and a second knowledge graph 1700B from a second time later than the first time (e.g., T2 to T3). The graph analysis service 226 may compare the nodes and edges in the layer with the nodes and edges of the corresponding layer in the knowledge graph 1700B for each layer in the knowledge graph 1700A, and determine one or more differences (e.g., listed above) between the layer of the knowledge graph 1700A and the corresponding layer of the knowledge graph 1700B based on the comparison. The graph analysis service 226 may output (e.g., via UI 228) one or more differences between the layer of the knowledge graph 1700A and the corresponding layer of the knowledge graph 1700B. As shown above, these differences may include one or more of added nodes, deleted nodes, added edges, and / or deleted edges.

[0162] Fig.19B is a conceptual diagram illustrating a set of APIs exposed by the graph analysis service 226 that can be used to initiate graph change analysis and query the results of graph change analysis.

[0163] This use case of graph analytics simulates fault injection in a specific layer in the knowledge graph and determines the impact of a catastrophic failure of a node on the application layer. For example, the service helps answer questions like:

[0164] 1. What will be the impact of a compute node failure on the application layer?

[0165] 2. What is the impact of an increase in loss or latency between two servers on an application? The graph analysis service 226 first simulates fault injection in the knowledge graph. Different faults can be injected on nodes, depending on the node type. Once the faults are injected, a graph traversal is performed to determine upstream applications that may be affected by the injection of a specific fault at a lower layer in the graph. The current example includes two types of faults: compute layer faults and network layer faults.

[0166] When a compute node failure such as a power failure is injected, then this can result in re-provisioning of application nodes on different compute nodes. A graph traversal of application nodes before the actual failure can help determine which applications will be affected whenever a power failure of a particular node occurs. This analysis is useful in assigning a risk score to each node in the compute layer based on its impact on the application layer.

[0167] When a network layer fault is injected based on a request to the graph analysis service 226, such as a request to inject a latency spike or packet loss on the network between two server pairs, the graph analysis service 226 traverses the knowledge graph to determine the applications hosted on the affected servers. This allows the determination of a risk score for the network layer, which is a function of how many application instances will be affected by the fault injection.

[0168] Node failure risk score evaluation

[0169] The graph analysis service 226 can calculate risk scores for different injected faults based on the impact of the fault on the application layer. m and I m Instances are hosted on a set of compute nodes. If a failure occurs in the compute layer or the network layer, a subset of service instances, i.e., F m The instance is served by m The risk score for a given fault is then calculated as follows:

[0170]

[0171] The risk score varies between 0 and 1, where the larger the score, the more impact a given failure will have on applications and their instances. A score of 1 indicates that all applications and their corresponding instances will be impacted by a given node failure.

[0172] In this way, the graph analysis service 226 can receive a graph analysis request including a request to determine the impact of a simulated failure. The graph analysis service 226 can also receive an indication of a first node in the knowledge graph that simulates a failure (possibly via the graph analysis request and / or via input entered through the UI 228). The graph analysis service 226 can then traverse the knowledge graph to determine one or more paths through the knowledge graph from the first node to one or more second nodes, each of the one or more second nodes representing a service in a plurality of services in the network system. The graph analysis service 226 can then determine a risk score based on the number of the one or more second nodes in the one or more paths and the total number of instances of the plurality of services, and output the risk score as a response to the graph analysis request.

[0173] Applying a network mapping service may determine the network region over which a given microservice-to-microservice communication is occurring. Fig.18 is a conceptual diagram illustrating an example knowledge graph including a microservices-based application. Fig.18In the example shown, microservices S1 and S3 communicate with each other. The network mapping service is applied to perform a top-down graph traversal to determine candidate network endpoints on which communication between S1 and S3 occurs. As a first step, the graph analysis service 226 determines the instances of the caller and callee services. In this case, microservice S1 has two instances, hosted in container groups P on compute nodes H1 and H3 (where "H" stands for "host"), respectively. 1,1 and P 1,2 Similarly, the called service S3 has two instances, hosted by container groups P on compute nodes H2 and H4 respectively. 3,1 and P 3,2 Next, the graph analysis service 226 determines the network nodes responsible for communication from the computing node set [H1, H3] to the computing node set [H2, H4]. Fig.18 As seen in the figure, network detection nodes PR1 and PR2 are a set of detection modules configured between the two groups of computing nodes. The detection modules PR1 and PR2 can analyze the network traffic between the computing nodes [H1, H3] and [H2, H4] to determine the network endpoints that are serving the communication network traffic between S1 and S3. This analysis can be useful in determining the source of an anomaly in which the communication between S1 and S3 is slower than expected. For example, if all network devices in the communication path are operating normally, the analysis can rule out network components. Troubleshooting can then focus on other components in the communication path (e.g., hosts H1 and H2), which may be the cause of slower than expected performance.

[0174] In other words, the graph analysis service 226 may receive a graph analysis request that includes a request to determine an application to a network path and an identifier of a first application (S1) and a second application (S3). The graph analysis service 226 may traverse the knowledge graph from a first node representing the first application to a second pattern representing the second application to determine a path from the first node to the second node. The path may include one or more computing nodes and one or more network nodes. The graph analysis service 226 may output (e.g., via the UI 228) identifiers corresponding to the one or more computing nodes and the one or more network nodes.

[0175] Fig.19C is a conceptual diagram illustrating an API for performing application-to-network mapping analysis.

[0176] Fig. 20 It is shown Figure 22000. The knowledge graph generator 220 may receive time series data (stored to the TSDB 238) including measurements of one or more performance indicators from a network device in a network system. The knowledge graph generator 220 may create a knowledge graph (e.g., a plurality of first nodes in the network system referenced in the time series data) based on the time series data. Fig.14 The knowledge graph 1400) (2002) shown in the example of .

[0177] The graph analysis service 226 may receive a graph analysis request (2004), which may include one or more of a request to determine a fault propagation path, a request to determine an impact of a simulated fault, or a request to determine an application to a network path. The graph analysis service 226 may determine a response to the graph analysis request (2006) and output the response to the graph analysis request (2008).

[0178] Return to Figure 2 , user interface module (UI) 228 can generate data indicating various user interface screens that graphically depict the results of the fault location (e.g., ranking list 230) and the graph analysis. UI module 228 can output data indicating the various user interface screens, such as for display by a separate display device. UI module 228 can also output data indicating a graphical user interface element requesting input for display. The input can be, for example, a graph query to be provided to graph analysis service 226.

[0179] As described above, some or all of the root cause analysis framework system 200 may be implemented using containers. For example, the root cause analysis framework system 200 may include a container platform 219. In some aspects, the container platform 219 may be a Kubernetes container platform. Kubernetes is a container management platform that provides portability across public and private clouds, each of which may provide a virtualized infrastructure to the risk analysis framework system. In some aspects, Figure 2 Each of the components illustrated in can be a containerized component. In some aspects, Figure 2 Each of the components illustrated in may be an on-premises component that is not containerized. In some aspects, Figure 2 Some of the components in the may be containerized, while others are on-premises. For example, some telemetry services may be on-premises components that may or may not be containerized. For example, network telemetry collector 212 and device / structure telemetry collector 214 may be located within a customer's network (e.g., on-premises).

[0180] Figure 32 shows a root cause analysis pipeline according to various aspects of the present disclosure. In some aspects, the root cause analysis pipeline 300 includes a knowledge graph generator 220, a causal graph generator 222, a graph pruner 224, and a ranking service 225. Figure 2 Cross-layer telemetry is collected in the form of time series data across multiple layers of the infrastructure from applications to the network (e.g. Figure 1 2) The telemetry includes performance metrics at different layers that reflect the health of different components of the network system. Telemetry can be collected and analyzed in real time to automatically detect and troubleshoot anomalies and provide the root cause of anomalies in the network system.

[0181] When a performance anomaly is detected for a given application, processing by the root cause analysis pipeline 300 may be initiated. For example, the anomaly detector 236 may identify the anomaly using the ML model 241 and provide information about the anomaly to the root cause analysis pipeline 300 using the cross-layer telemetry 304. As an example, the cross-layer telemetry data 304 may include application layer telemetry data collected by any of the telemetry collectors 216, the network telemetry data collectors 212, and / or the device / fabric telemetry collectors 214, compute / container group layer telemetry data 608, network layer telemetry data, device layer telemetry data, fabric layer telemetry data, etc. Additionally, or alternatively, rule-based anomaly detection may be used to detect anomalies and provide the anomalies to the root cause analysis pipeline 300.

[0182] After detecting an application performance problem, the knowledge graph generator 220 initiates the creation of a knowledge graph from real-time telemetry data collected during the time period when the application anomaly was detected. As an example, if an application anomaly is detected at time T, the knowledge graph generator 220 can generate a knowledge graph for the entire infrastructure from application to network using cross-layer telemetry 304 for the past N time periods. Where N can be 5 minutes, 15 minutes, 30 minutes, etc.

[0183] Telemetry is parsed for each layer to determine the nodes used for each layer and their relationships to adjacent nodes. In general, a node can represent any entity in a network system, whether physical, virtual, or software. As an example, a node can represent devices such as network devices, computing devices, virtual devices (e.g., virtual machines, virtual routers, VRFs, etc.), applications, services, microservices, etc. For example, from the response time telemetry for a microservice, the caller and callee services can be identified from the labels present in the telemetry. Similarly, if the application is hosted on a Kubernetes platform, the relationship between a microservice and its multiple instances can be determined from container group-level telemetry.

[0184] The knowledge graph generated by the knowledge graph generator 220 may include multiple layers, such as an application layer, a container group instance layer, a computing node layer, a network detection layer, and / or a network structure layer. Other layers in the knowledge graph may be defined and added, depending on the environment. As an example, the root cause analysis framework system may be used to model different components within a router / switch, where the dependencies of ingress ports, ingress queues, structure ports, egress ports, and egress queues may be represented as a knowledge graph. The root cause analysis framework system 200 may be extended to such an environment.

[0185] Fig. 6A An example knowledge graph that may be generated by the knowledge graph generator 220 is shown. Fig. 6A The examples shown in include nodes at the microservice layer (“S” nodes), container group layer (“P” nodes), computer layer (“H” nodes), detection layer (“PR” nodes), and network fabric layer (“T” nodes). Fig. 6A Also shown are relationships that have been discovered, for example, through analysis of TSDB 238 (in Fig. 6A In the example of , it is represented as a directed graph edge and is depicted as a one-way arrow. Fig. 6A In the example shown, the anomaly detector 236 has been at the nodes S5, S9 and S 10 Anomalies are detected at node H4 (where "H" represents host) at the computing layer (where "S" represents service) and at node H4 (where "H" represents host).

[0186] Return to Figure 3 , the graph pruner 224 can prune the knowledge graph into a smaller set of nodes based on the anomaly. In some aspects, pruning is performed so that only nodes within a threshold distance of the node that exhibits the anomaly are selected. As used herein, the distance between two nodes is the number of edges between the nodes in the graph. In some aspects, the threshold distance is one (1), resulting in the selection of the node that experienced the anomaly and its immediate neighbor nodes. Pruning the graph can produce a smaller graph on which to perform further root cause analysis.

[0187] Figure 6B An example pruned knowledge graph is shown that may be generated by the knowledge graph generator 220 and pruned by the graph pruner 224. In this example, the threshold distance is assumed to be two.

[0188] The knowledge graph generator 220 may expose an application programming interface (API) for creating and processing a knowledge graph. Fig.11 An example set of APIs for an example knowledge graph generator is shown.

[0189] Return to Figure 3, the causal graph generator 222 can create a causal graph based on the pruned knowledge graph. Each node in the pruned knowledge graph has a different set of KPIs. The causal graph generator 222 can use the KPIs to add nodes to the pruned knowledge graph. The causal graph can be generated in various ways.

[0190] Use the Granger causality algorithm to create causal diagrams based on time series data analysis.

[0191] The algorithm operates as follows:

[0192] 1) Determine a KPI for each node in the knowledge graph and create a KPI node with an edge between the KPI and the corresponding node in the knowledge graph. Fig. 7A is a diagram showing the KPI node with the added Fig. 6A An example diagram of the knowledge graph.

[0193] 2) Discover the causal relationship between KPI metrics of adjacent nodes in the knowledge graph. This helps to avoid a large number of calculations for all possible combinations of KPI nodes in the pruned knowledge graph. Intuition is that KPIs of adjacent nodes in the knowledge graph are more likely to have causal relationships. This avoids running causal analysis on nodes that are multiple levels away from the node. The radius of the analysis is determined by controlling the parameter M, where M=1 by default, i.e., only direct neighbors are considered for causal analysis. After all possible KPI pairs are determined, the Granger causality method is used to determine the causal relationship between the two KPIs. Figure 7B It is shown Fig. 7A An example diagram of the cause-and-effect relationships between the KPI metrics shown in FIG.

[0194] 3) After Granger causality is determined, edges are then created between KPI nodes. The initial weights of the edges are determined by measuring the correlation between two KPIs using the Pearson correlation coefficient. Before performing page ranking on the causal graph, the direction of the edges is reversed and the weights are normalized by the number of out-degrees of the KPI nodes so that the sum of the weights of all out-edges is 1. This weight represents the probability of selecting an out-edge during a random walk in the page ranking algorithm. Figure 7C It shows that Figure 7B An example graph of weighted edges created by .

[0195] As will now be described, weights for edges may be determined in a variety of ways.

[0196] Normalized correlation coefficient based on time series data

[0197] In this case, as a first step, the Pearson correlation coefficient between the KPIs of adjacent nodes in the causal graph is measured. Thereafter, the outgoing edge weights are normalized with respect to their correlation coefficient as follows:

[0198] If a node Ni has outgoing edges E1, E2…Ek, where Cj is the Pearson correlation coefficient of edge Ej. Then the weight of edge Ej = Cj / ΣCi

[0199] Edge weight allocation based on abnormal status of neighboring nodes

[0200] The following operations can be performed to determine edge weights based on the abnormality status of neighboring nodes:

[0201] 1) Initialize the weight of each edge to 1 / K, where K is the outgoing edge.

[0202] For edge e in K

[0203] W e =1 / K

[0204] 2) If node Ni has outgoing edges E1, E2, ... Ek, where Aj is a flag indicating whether the node j corresponding to edge Ej is an anomaly. Let L be the set of nodes with anomalies, and M be the set of nodes without any anomalies. In this case, the weights of all edges are adjusted as follows:

[0205] For an edge e'm corresponding to a non-anomalous node in M:

[0206] We'm=We'm / (L+1) / / Reduce the weight of non-abnormal node edges

[0207] For an edge eL corresponding to a node in L:

[0208] WeL=WeL+w / / Increase the weight of abnormal edges proportionally

[0209] FIG. 8A to FIG. 8C is an example illustrating the above content. Fig. 8A is an example graph showing an example graph with initial weights for edges. In this example, node A1 has four neighbor nodes A2 to A5, with 4 outgoing edges that need weights determined. In this example, nodes A3 and A5 are outliers. In the current example, all outliers are treated equally. Fig. 8A As shown, initially, the edges are equally weighted. Each edge from A4 to a neighboring node is initially equally weighted to 1 / 4.

[0210] Figure 8B and 8C The weighting process based on step 2 above is shown. At the end of this step, higher weights should be assigned to the edges towards the abnormal node. As a result, both the A3 and A5 node edges will be assigned equal weights. Subsequently, differentiated weights can be assigned to the abnormal edges based on the criticality of the abnormality.

[0211] Note that during each iteration of the weights of the edges towards the outlier nodes, the weights of the edges towards the outlier nodes are increased by decreasing the weights of the outgoing edges towards the non-outlier nodes. In some aspects, the sum of the weights of all outgoing edges is always 1.

[0212] Creating causal diagrams using temporal analysis of events

[0213] Different from the original time series analysis, event-based analysis can be used to determine the causal relationship between different KPIs and thus generate a causal graph. An event is defined as an anomaly observed in a KPI metric, where the KPI metric can be a continuous time series performance data or log events generated by different components.

[0214] In this approach, the relationship between the alarm rankings is discovered over multiple iterations of historical data. The strength of the relationship between two adjacent events based on their timestamps depends on how often they occurred in a given period. Note that there can be two types of dependencies between events:

[0215] 1. Transient Dependency: In this scenario, the dependency between events is temporary in nature and valid only for a given time window. If the frequency of occurrence is below a threshold, the dependency is marked as transient.

[0216] 2. Persistent dependencies: In this scenario, the dependencies between events are persistent and independent of the monitoring time window.

[0217] The output of this method gives a causal graph with weights determined based on transient or persistent dependencies between any two KPI events. A cutoff in the weights can be used to decide what is considered a transient or persistent dependency between alerts. Transient dependencies are based on KPI events that occur infrequently or with regularity. Persistent dependencies are based on KPI events that occur repeatedly over a period of time.

[0218] There are two stages to creating a causal graph using temporal analysis of events. In the first stage, an ordering of events based on their occurrence is determined. Then the frequency with which two events are adjacent across multiple observations is determined. If a tentative causal relationship exists between the two measures, the direction of the causal relationship is determined based on the order of events in which the events occur more frequently.

[0219] Fig.9A An example causal graph at the first stage is shown, where transient and persistent dependencies are discovered on sets of events for different KPIs in the knowledge graph.

[0220] Fig. 9B 2 shows the graph after the graph pruner 224 has pruned the nodes related to the transient relationship events, leaving only the persistent relationships between the events. Fig.9A Example cause-effect diagram for .

[0221] Fig. 9C shows a graph with weights based on the strength of persistent relationships between events. Fig. 9B Example cause-effect diagram for .

[0222] FIG. 7A to FIG. 7C , FIG. 8A to FIG. 8C and 9A to 9C Cause-and-effect diagrams created based on different methods are shown. The root cause analysis framework system disclosed herein is extensible and can be combined with different mechanisms for generating cause-and-effect diagrams. The cause-and-effect diagram generator 222 can expose an application programming interface (API) for creating and processing knowledge graphs. For example, after generating a cause-and-effect diagram, the API can allow domain experts to optionally audit and modify cause-and-effect relationships based on the domain knowledge of the system. Fig.12 An example API set for an example causal graph generator is shown in .

[0223] Return to Figure 3 After the causal graph generator 222 has generated the causal graph, the ranking service 225 can analyze the graph and rank the most likely root causes of the anomaly. In some aspects, the ranking service 225 uses a "page rank" algorithm to rank the nodes in the causal graph. In the page ranking algorithm, the importance of a web page increases if other important web pages point to a given web page. A similar analogy is used to rank nodes in a causal graph, where a node may receive a higher score if it is pointed to as a root cause by other nodes.

[0224] There are two modes for using the page ranking algorithm. In the first mode, the root cause can be determined for all service level anomalies observed in the cause-effect graph. In the second mode, a personalized page ranking algorithm can be used that focuses on performing root cause analysis on a selected set of services or infrastructure components.

[0225] In the above presentation about FIG. 7A to FIG. 7B In the example above, the PageRank algorithm could produce the following possible root causes for the ranking:

[0226] 1.Host H4 CPU - Rank 1

[0227] 2. Detection module PR4 delay - rank 2

[0228] 3. Detection module PR4 lost - rank 3

[0229] Fig.10 A visual representation of an example causal graph associated with the above example is shown. Fig.13 Shown from Figure 2 and Figure 3 Example page ranking output from the root cause analysis pipeline described in .

[0230] Techniques for performing root cause analysis on distributed microservices-based applications deployed in a data center network are described. The techniques of this disclosure may be applicable to other environments.

[0231] Figure 4 is a block diagram illustrating an example network node in accordance with techniques described in this disclosure. Figure 4 The network node 400 may represent a network device 18, a server hosting service node 10, and Figure 1 A server or other computing device associated with the application 30, or Figure 1 and Figure 2 The root cause analysis framework system 28, any one of the servers or computing devices associated with 200.

[0232] In this example, the network node 400 includes a communication interface 402 (e.g., an Ethernet interface), a processor 406, an input / output 408 (e.g., a display, a button, a keyboard, a keypad, a touch screen, a mouse, etc.), and a memory 412 coupled together via a bus 414, and the various elements can exchange data and information on the bus 414. The communication interface 402 couples the network node 400 to a network such as an enterprise network. Although only one interface is shown by way of example, it should be recognized by those skilled in the art that a network node can and usually does have multiple communication interfaces. The communication interface 402 includes a receiver (RX) 420, via which the network node 400 (e.g., a server) can receive data and information. The communication interface 402 includes a transmitter (TX) 422, via which the network node 400 (e.g., a server) can send data and information.

[0233] The memory 412 stores an executable operating system 440 and may store software applications 432, detection modules 444, and / or cloud-based framework services 446 in various configurations. For example, the network node 400 may be configured to act as Figure 1 The application 432 may be a server that is part of the cloud-based service 26. In such a configuration, the application 432 may be Figure 1 If the techniques described herein are used to monitor the performance of software application 432, detection module 444 may be Figure 1 Implementation of detection module 19.

[0234] The network node 400 may be configured as a server that is part of a cloud-based analysis framework 28. In such a configuration, the memory 412 may store one or more cloud-based analysis framework services 446. The cloud-based analysis framework services 446 may be Figure 2Any one of the detection service 201, telemetry service 207, event stream engine 218, fault locator 240, graph analysis service 226 and / or anomaly detector 236.

[0235] Figure 5 is a flow chart illustrating an example operation of a root cause analysis framework system according to one or more techniques of the present disclosure. The root cause analysis framework system may receive time series data including measurements of one or more performance indicators from a device of a network system (505). Next, the root cause analysis framework system may determine one or more anomalies in the performance of the network system based on the time series data (510). Next, the root cause analysis framework system may create a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in the network system referenced in the time series, the first nodes representing elements residing at one or more network service layers in a plurality of network service layers associated with the network system (515). Next, the root cause analysis framework system may determine a causal graph in response to detecting one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with a previous time series of measurements, wherein the causal graph includes a plurality of second nodes associated with the performance indicators, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more of the first nodes and one or more of the second nodes in the first node, the second node being associated with an element residing at a different layer of the plurality of layers (520). Next, the root cause analysis framework system may determine a weight for each of the edges in the causal graph (525). Next, the root cause analysis framework system may determine one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph (530). Next, the root cause analysis framework system may determine a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph (535). Next, the root cause analysis framework system may output at least a portion of the ranking (540).

[0236] In this manner, one or more aspects of the technology may implement the following examples.

[0237] Example 1A. A method includes: receiving, by an analysis framework system, time series data including measurements of one or more performance indicators from a device of a network system; determining, by the analysis framework system and based on the time series data, one or more anomalies in the performance of the network system; creating a knowledge graph based on the time series data, the knowledge graph including first nodes in the network system referenced in the time series, the first nodes representing elements residing at one or more network service layers of a plurality of network service layers associated with the network system; determining, by the analysis framework system in response to detecting the one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with previous time series data, a causal graph, wherein the causal graph includes a second node associated with the performance indicator, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more first nodes in the first node and one or more second nodes in the second node, the second node associated with elements residing at different network service layers of the plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0238] Example 2A. The method of Example 1A, wherein outputting includes outputting at least a portion of the ranking for display.

[0239] Example 3A. The method of any combination of Examples 1A and 2A, further comprising: in response to determining one or more anomalies and before determining the causal graph, pruning the knowledge graph to remove a first node, the first node being at a distance greater than a threshold distance from a first node of the knowledge graph, the knowledge graph being associated with the one or more anomalies, wherein determining the causal graph comprises determining the causal graph based on the pruned knowledge graph.

[0240] Example 4A. The method of any combination of Examples 1A to 3A, wherein the machine learning model includes a first machine learning model, and wherein determining the one or more anomalies includes determining the one or more anomalies based on passing the time series data through a second machine learning model trained on previous time series data.

[0241] Example 5A. The method of any combination of Examples 1A-4A, wherein determining one or more anomalies comprises determining one or more anomalies based on static rules.

[0242] Example 6A. The method of any combination of Examples 1A-5A, wherein the network services layer includes an application layer, a compute layer, a container group layer, a device layer, and a fabric layer.

[0243] Example 7A. The method of any combination of Examples 1A-6A, wherein determining the causal graph comprises determining the causal graph using a Granger causal algorithm.

[0244] Example 8A. The method of any combination of Examples 1A-7A, wherein determining weights of edges in the causal graph comprises determining a Pearson coefficient for the edges.

[0245] Example 9A. The method of any combination of Examples 1A-8A, wherein determining a weight of an edge in the causal graph comprises determining the weight based on an abnormal state of a neighbor node of a first node associated with the edge.

[0246] Example 10A. The method of any combination of Examples 1A-9A, wherein determining a weight of an edge in the causal graph comprises determining the weight of the edge based on a count of persistent events associated with the edge.

[0247] Example 11A. A computing system comprising: a memory configured to store time series data including measurements of one or more performance indicators; and a processing circuit system configured to execute an analysis framework system, the analysis framework system configured to: determine one or more anomalies in performance of a network system based on the time series data; create a knowledge graph based on the time series data, the knowledge graph including a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; in response to detecting one or more anomalies, and based on the knowledge graph and a machine learning (ML) model trained with previous time series data, determine A causal graph, wherein the causal graph includes a second node associated with a performance indicator, wherein an edge between a first node and a second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include edges between one or more of the first nodes and one or more of the second nodes, the second nodes being associated with elements residing at different network service layers of a plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0248] Example 12A. The computing system of Example 11A, wherein the processing circuitry is configured to output the at least a portion of the ranking for display.

[0249] Example 13A. The computing system of any combination of Examples 11A-12A, wherein the processing circuitry is further configured to, in response to determining the one or more anomalies and before determining the causal graph, prune the knowledge graph to remove a first node that is greater than a threshold distance from a first node of the knowledge graph that is associated with the one or more anomalies; and wherein the processing circuitry is configured to determine the causal graph based on the pruned knowledge graph.

[0250] Example 14A. The computing system of any combination of Examples 11A-13A, wherein the machine learning model comprises a first machine learning model, and wherein the processing circuitry is configured to determine one or more anomalies based on passing the time series data through a second machine learning model trained on prior time series data.

[0251] Example 15A. The computing system of any combination of Examples 11A-14A, wherein the processing circuitry is configured to determine the one or more anomalies based on static rules.

[0252] Example 16A. The computing system of any combination of Examples 11A to 15A, wherein the network services layer includes an application layer, a compute layer, a container group layer, a device layer, and a fabric layer.

[0253] Example 17A. The computing system of any combination of Examples 11A-16A, wherein the processing circuitry is configured to determine the causal graph using a Granger causal algorithm.

[0254] Example 18A. The computing system of any combination of Examples 11A-17A, wherein the processing circuitry is configured to determine a Pearson coefficient for the edge.

[0255] Example 19A. The computing system of any combination of Examples 11A-18A, wherein the processing circuitry is configured to determine the weight based on an abnormality status of a neighbor node of the first node associated with the edge.

[0256] Example 20A. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to execute a framework analysis system, wherein the framework analysis system is configured to: receive time series data including measurements of one or more performance indicators; determine one or more anomalies in performance of a network system based on the time series data; create a knowledge graph based on the time series data, the knowledge graph including a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; in response to detecting the one or more anomalies, and based on the knowledge graph and a machine learning (ML) trained with previous time series data, determine one or more anomalies in performance of a network system; ) model to determine a causal graph, wherein the causal graph includes a second node associated with a performance indicator, wherein an edge between a first node and a second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more of the first nodes and one or more of the second nodes, the second node being associated with an element residing at a different network service layer of a plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; and outputting at least a portion of the ranking.

[0257] Example 1B. A method, comprising: receiving, by an analysis framework system, time series data including measurements of one or more performance indicators from a device of a network system; creating, by the analysis framework system, a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in the network system referenced in the time series, the first nodes representing elements residing at one or more layers of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates a communication or computational dependency between the two nodes; receiving, by a graph analysis service of the analysis framework system, a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine the impact of a simulated fault, or a request to determine an application to a network path; determining, by the graph analysis service, a response to the graph analysis request; and outputting, by the analysis framework working system, a response to the graph analysis request.

[0258] Example 2B. The method of Example 1B, wherein the graph analysis request includes a request to determine a fault propagation path, the method further comprising: determining, by the analysis framework system and based on time series data, a first node and a second node each experiencing a fault, the first node being dependent on the second node; traversing the knowledge graph to determine one or more paths from the first node to the second node through the knowledge graph; and selecting a path in the one or more paths in which an intermediate node in the path between the first node and the second node is experiencing a fault as the fault propagation path, wherein the output response includes outputting the fault propagation path.

[0259] Example 3B. The method of Example 2B, wherein the request includes an identification of a first node, and wherein traversing the knowledge graph includes traversing the knowledge graph from the first node to a second node.

[0260] Example 4B. The method of Example 2B, wherein the request identifies a second node, and wherein traversing the knowledge graph includes traversing the knowledge graph from the second node to the first node.

[0261] Example 5B. The method of any combination of Examples 1B to 4B, wherein the graph analysis request includes a request to determine changes in a graph, wherein the request identifies a first knowledge graph from a first time and a second knowledge graph from a second time that is later than the first time, wherein the method further includes: for each layer in the first knowledge graph, comparing the nodes and edges in the layer with the nodes and edges of the corresponding layer in the second knowledge graph, and determining one or more differences between the layer of the first knowledge graph and the corresponding layer of the second knowledge graph based on the comparison, wherein outputting a response to the graph analysis request includes outputting one or more differences between the layer of the first knowledge graph and the corresponding layer of the second knowledge graph.

[0262] Example 6B. The method of Example 5B, wherein the one or more differences include an added node, a deleted node, an added edge, or a deleted edge.

[0263] Example 7B. The method of any combination of Examples 1B to 6B, wherein the request includes a request to determine an impact of a simulated failure, the method further comprising: receiving an indication of a first node in a knowledge graph that simulates the failure; traversing the knowledge graph to determine one or more paths through the knowledge graph from the first node to one or more second nodes, each of the one or more second nodes representing a service in a plurality of services in a network system; and determining a risk score based on a number of the one or more second nodes in the one or more paths and a total number of instances of the plurality of services, wherein outputting a response to the graph analysis request includes outputting the risk score.

[0264] Example 8B. The method of Example 7B, wherein the first node represents a computing node in the network system.

[0265] Example 9B. The method of Example 7B, wherein the first node represents a network device in the network system.

[0266] Example 10B. The method of any combination of Examples 1B to 9B, wherein the graph analysis request includes a request to determine an application to a network path and wherein the graph analysis request includes identification of a first application and a second application, the method further comprising: traversing the knowledge graph from a first node representing the first application to a second pattern representing the second application to determine a path from the first node to the second node. The path includes one or more compute nodes and one or more network nodes, wherein outputting a result of the graph analysis request includes outputting identifiers corresponding to the one or more compute nodes and the one or more network nodes.

[0267] Example 11B. The method of any combination of Examples 1B to 10B, wherein outputting comprises outputting at least a portion of the ranking for display.

[0268] Example 12B. A computing device, comprising: a processing circuit system configured to execute an analysis framework system; and a memory configured to store time series data including measurements of one or more performance indicators, wherein the analysis framework system is configured to: create a knowledge graph based on the time series data, the knowledge graph including a plurality of first nodes in a network system referenced in the time series, the first nodes representing elements residing at one or more layers of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates a communication or computational dependency between the two nodes; cause a graph analysis service of the analysis framework system to receive a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine the impact of a simulated fault, or a request to determine an application to a network path; cause the graph analysis service to determine a response to the graph analysis request; and output a response to the graph analysis request.

[0269] Example 13B. The computing device of Example 12B, wherein the graph analysis request includes a request to determine a fault propagation path, and the analysis framework system is further configured to determine a first node and a second node that are each experiencing a fault based on the time series data, the first node being dependent on the second node; traverse the knowledge graph to determine one or more paths from the first node to the second node through the knowledge graph; and select a path in the one or more paths in which an intermediate node in the path between the first node and the second node is experiencing a fault as the fault propagation path; and output the fault propagation path as a response.

[0270] Example 14B. The computing device of Example 13B, wherein the request includes an identification of a first node, and wherein the analysis framework system is configured to traverse the knowledge graph from the first node to the second node.

[0271] Example 15B. The computing device of Example 13B, wherein the request identifies a second node, and wherein the analysis framework system is configured to traverse the knowledge graph from the second node to the first node.

[0272] Example 16B. A computing device of any combination of Examples 12B to 15B, wherein the graph analysis request comprises a request to determine changes in a graph, wherein the request identifies a first knowledge graph from a first time and a second knowledge graph from a second time that is later than the first time, wherein the analysis framework system is further configured to: for each layer in the first knowledge graph, compare the nodes and edges in the layer with the nodes and edges of the corresponding layer in the second knowledge graph; determine one or more differences between the layer of the first knowledge graph and the corresponding layer of the second knowledge graph based on the comparison; and output one or more differences between the layer of the first knowledge graph and the corresponding layer of the second knowledge graph in response.

[0273] Example 17B. The computing device of Example 16B, wherein the one or more differences include an added node, a deleted node, an added edge, or a deleted edge.

[0274] Example 18B. The computing device of any combination of Examples 12B to 17B, wherein the request comprises a request to determine an impact of a simulated failure, the analysis framework system being further configured to: receive an indication of a first node in a knowledge graph that simulates a failure; traverse the knowledge graph to determine one or more paths through the knowledge graph from the first node to one or more second nodes, each of the one or more second nodes representing a service in a plurality of services in a network system; determine a risk score based on a number of the one or more second nodes in the one or more paths and a total number of instances of the plurality of services; and output the risk score in response.

[0275] Example 19B. The computing device of Example 18B, wherein the first node represents a computing node in a network system.

[0276] Example 20B. The computing device of Example 18B, wherein the first node represents a network device in the network system.

[0277] Example 21B. The computing device of any combination of Examples 12B to 20B, wherein the graph analysis request includes a request to determine an application to network path and wherein the graph analysis request includes identification of a first application and a second application, the analysis framework system is further configured to: traverse the knowledge graph from a first node representing the first application to a second pattern representing the second application to determine a path from the first node to the second node. The path includes one or more computing nodes and one or more network nodes; and output identifiers corresponding to the one or more computing nodes and the one or more network nodes as a response.

[0278] Example 22B. The computing device of any combination of Examples 12B to 21B, wherein the analysis framework system is configured to output at least a portion of the ranking for display.

[0279] Example 23B. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed, causing one or more processors to execute a framework analysis system, the framework analysis system being configured to: receive time series data including measurements of one or more performance indicators; create a knowledge graph based on the time series data, the knowledge graph including a first node in a network system referenced in the time series, the first node representing an element residing at one or more layers of a plurality of layers associated with the network system, wherein an edge between two of the plurality of nodes indicates a communication or computational dependency between the two nodes; cause a graph analysis service of the analysis framework system to receive a graph analysis request, the graph analysis request including a request to determine a fault propagation path, a request to determine a change in a graph, a request to determine an impact of a simulated fault, or a request to determine an application to a network path; cause the graph analysis service to determine a response to the graph analysis request; and output the response to the graph analysis request

[0280] Example 1C. A method includes: determining, by a network system, a topology map for a network, the topology map representing multiple logical layers of the network; receiving, by the network system, an identifier associated with an application that communicates using the network; determining, by the network system and based on the topology map, a subgraph of the topology map based on a location in the topology map of a node representing a computing node that is a host of the application; determining, by the network system and based on the subgraph, a set of one or more probe modules to measure performance metrics associated with network communications associated with the application; generating, by the network system, for each of the one or more probe modules, configuration data corresponding to a type of the probe module; and outputting, by the network system and to the probe module, the configuration data corresponding to the probe module.

[0281] Example 2C. The method of Example 1C, wherein determining the subgraph comprises determining the subgraph based on nodes representing compute nodes hosting the application and one or more second nodes of a topology map associated with a network infrastructure used by the application to communicate on a network represented by the network topology.

[0282] Example 3C. The method of any combination of Examples 1C to 2C, wherein determining the set of one or more probe modules comprises determining a set of probe modules for nodes represented at a plurality of layers in a plurality of layers of the network system.

[0283] Example 4C. The method of any combination of Examples 1C to 3C, wherein the plurality of layers includes one or more of an underlying network layer, an overlay network layer, and a containerized platform cluster layer.

[0284] Example 5C. The method of any combination of Examples 1C to 4C, further comprising receiving telemetry data from the detection module according to the configuration.

[0285] Example 6C The method of Example 5C, further comprising converting telemetry data received from the detection module from a format associated with the type of detection module to a standard format.

[0286] Example 7C. The method of any combination of Examples 5C and 6C, further comprising: aggregating telemetry data at a first granularity corresponding to a single node of the topology map; aggregating telemetry data at a second granularity corresponding to a group of nodes of the topology map; generating first baseline data based on the telemetry data at the first granularity; and generating second baseline data based on the telemetry data at the second granularity.

[0287] Example 8C. The method of any combination of Examples 1C to 7C, further comprising: storing, by the network system, data associating the detection module to the application; and in response to detecting the anomaly based on telemetry data provided by the detection module, identifying the application as affected by the anomaly based on the data associating the detection module to the application.

[0288] Example 9C. The method of any combination of Examples 1C to 8C, wherein the probing module is configured to measure network performance between two hosts used by the application.

[0289] Example 10C. A network system, comprising: a processing circuit system configured to determine a topology map for a network, the topology map representing multiple logical layers of the network; and a memory configured to store the topology map, wherein the processing circuit system is further configured to: receive an identifier associated with an application that communicates using the network; determine, based on the topology map, a subgraph of the topology map based on the position of nodes representing computing nodes that are hosts of the application in the topology map; determine, based on the subgraph, a set of one or more probe modules to measure performance metrics associated with network communications associated with the application; generate, for each of the one or more probe modules, configuration data corresponding to a type of the probe module; and output the configuration data corresponding to the probe module to the probe module.

[0290] Example 11C. The network system of Example 10C, wherein the processing circuit is configured to determine a subgraph based on a node representing a computing node hosting an application and one or more second nodes of a topology graph associated with a network infrastructure used by the application to communicate on a network represented by the network topology.

[0291] Example 12C. The network system of any combination of Examples 10C and 11C, wherein the processing circuit is configured to determine a set of detection modules for nodes represented at a plurality of layers in the plurality of layers of the network system.

[0292] Example 13C. The network system of any combination of Examples 10C to 12C, wherein the plurality of layers includes one or more of an underlying network layer, an overlay network layer, and a containerized platform cluster layer.

[0293] Example 14C. The network system of any combination of Examples 10C to 13C, wherein the processing circuitry is further configured to receive telemetry data from the detection module according to the configuration.

[0294] Example 15C. The network system of Example 14C, wherein the processing circuitry is further configured to convert telemetry data received from the detection module from a format associated with the type of detection module to a standard format.

[0295] Example 16C. The network system of any combination of Examples 14C and 15C, wherein the processing circuit system is further configured to: aggregate telemetry data at a first granularity corresponding to a single node of the topology map; aggregate telemetry data at a second granularity corresponding to a group of nodes of the topology map; generate first baseline data based on the telemetry data at the first granularity; and generate second baseline data based on the telemetry data at the second granularity.

[0296] Example 17C. The network system of any combination of Examples 10C to 16C, wherein the processing circuit system is further configured to: store data associating the detection module to the application; and in response to detecting the anomaly based on telemetry data provided by the detection module, identify the application as affected by the anomaly based on the data associating the detection module to the application.

[0297] Example 18C. The network system of any combination of Examples 10C to 17C, wherein the probe module is configured to measure network performance between two hosts used by the application.

[0298] Example 19C. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to: determine a topology map for a network, the topology map representing multiple logical layers of the network; receive an identifier associated with an application that communicates using the network; determine, based on the topology map, a subgraph of the topology map based on a location in the topology map of a node representing a computing node that is a host for the application; determine, based on the subgraph, a set of one or more probe modules to measure a performance metric associated with network communications associated with the application; generate, for each of the one or more probe modules, configuration data corresponding to a type of the probe module; and output, to the probe module, the configuration data corresponding to the probe module.

[0299] The techniques described herein are implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices or other hardware devices. In some cases, the various features of the electronic circuit system may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0300] If implemented in hardware, the present disclosure may be directed to an apparatus such as a processor or an integrated circuit device (such as an integrated circuit chip or chipset). Alternatively or additionally, if implemented in software or firmware, the technology may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store these instructions for execution by a processor.

[0301] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0302] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier wave or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache memory).

[0303] The code or instructions may be software and / or firmware executed by a processing circuit system that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuit systems. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the present disclosure may be provided within software modules or hardware modules.

Claims

1. A method comprising: receiving, by the analysis framework system, time series data including measurements of one or more performance indicators from a device of the network system; determining, by the analysis framework system and based on the time series data, one or more anomalies in the performance of the network system; creating a knowledge graph based on the time series data, the knowledge graph comprising a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; determining, by the analytics framework system in response to detecting the one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with prior time series data, a causal graph, wherein the causal graph includes a second node associated with the performance indicator, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more first nodes of the first nodes and one or more second nodes of the second nodes, the second nodes being associated with elements residing at different network service layers of the plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; as well as At least a portion of the ranking is output. The method of claim 1 , wherein outputting comprises outputting the at least a portion of the ranking for display.

3. The method according to claim 1, further comprising: In response to determining the one or more anomalies and before determining the causal graph, pruning the knowledge graph to remove a first node of the knowledge graph that is greater than a threshold distance from the first node associated with the one or more anomalies, Wherein determining the causal graph comprises determining the causal graph based on the pruned knowledge graph.

4. The method according to claim 1, wherein the machine learning model comprises a first machine learning model, and Wherein determining the one or more anomalies comprises: The one or more anomalies are determined based on passing the time series data through a second machine learning model trained on the prior time series data. The method of claim 1 , wherein determining the one or more anomalies comprises determining the one or more anomalies based on static rules.

6. The method according to claim 1, wherein the network service layer includes an application layer, a computing layer, a container group layer, a device layer and a structure layer.

7. The method of claim 1, wherein determining the causal graph comprises determining the causal graph using a Granger causal algorithm. The method of claim 1 , wherein determining the weight of the edge in the causal graph comprises determining a Pearson coefficient for the edge.

9. The method of claim 1 , wherein determining the weights of the edges in the causal graph comprises: The weight is determined based on abnormal states of neighboring nodes of the first node associated with the edge.

10. The method of claim 1, wherein determining the weights of the edges in the causal graph comprises: The weight of the edge is determined based on a count of persistent events associated with the edge.

11. A computing system comprising: a memory configured to store time series data including measurements of one or more performance indicators; as well as a processing circuit system configured to execute an analysis framework system configured to: determining one or more anomalies in the performance of the network system based on the time series data; creating a knowledge graph based on the time series data, the knowledge graph comprising a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; determining a causal graph in response to detecting the one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with prior time series data, wherein the causal graph includes a second node associated with the performance indicator, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more first nodes of the first nodes and one or more second nodes of the second nodes, the second nodes being associated with elements residing at different network service layers of the plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; as well as At least a portion of the ranking is output. 12 . The computing system of claim 11 , wherein the processing circuitry is configured to output the at least a portion of the ranking for display.

13. The computing system according to claim 11, wherein the processing circuit system is further configured to: in response to determining the one or more anomalies and before determining the causal graph, prune the knowledge graph to remove a first node of the knowledge graph that is greater than a threshold distance from the first node associated with the one or more anomalies, and Wherein the processing circuit system is configured to determine the causal graph based on the pruned knowledge graph.

14. The computing system according to claim 11, wherein the machine learning model comprises a first machine learning model, and Wherein the processing circuit system is configured to: determine the one or more anomalies based on passing the time series data through a second machine learning model trained on the previous time series data. 15 . The computing system of claim 11 , wherein the processing circuitry is configured to determine the one or more anomalies based on static rules.

16. The computing system of claim 11, wherein the network service layer comprises an application layer, a computing layer, a container group layer, a device layer, and a fabric layer.

17. The computing system of claim 11, wherein the processing circuitry is configured to determine the causal graph using a Granger causal algorithm.

18. The computing system of claim 11, wherein the processing circuitry is configured to determine a Pearson coefficient for the edge.

19. The computing system of claim 11, wherein the processing circuitry is configured to determine the weight based on abnormal states of neighboring nodes of a first node associated with the edge.

20. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors to execute a framework analysis system, wherein the framework analysis system is configured to: receiving time series data including measurements of one or more performance indicators; determining one or more anomalies in the performance of the network system based on the time series data; creating a knowledge graph based on the time series data, the knowledge graph comprising a first node in the network system referenced in the time series, the first node representing an element residing at one or more network service layers of a plurality of network service layers associated with the network system; determining a causal graph in response to detecting the one or more anomalies and based on the knowledge graph and a machine learning (ML) model trained with prior time series data, wherein the causal graph includes a second node associated with the performance indicator, wherein an edge between the first node and the second node indicates a relationship between the first node and the second node, and wherein the knowledge graph and the causal graph each include an edge between one or more first nodes of the first nodes and one or more second nodes of the second nodes, the second nodes being associated with elements residing at different network service layers of the plurality of network service layers; determining a weight for each of the edges in the causal graph; determining one or more candidate root causes associated with the one or more anomalies based on the edges in the causal graph; determining a ranking of the one or more candidate root causes based on the weights of the edges in the causal graph; as well as At least a portion of the ranking is output.

Images