Method, controller and computer readable medium for computer network

Abstract

The present disclosure relates to methods, controllers, and computer-readable media for computer networks. The present disclosure describes techniques for network monitoring and fault location. For example, the controller includes one or more processors operably coupled to a memory, the processor being configured to: receive a first set of one or more quality of experience (QoE) metrics measured by a first probe passing through a first path including one or more links; receive a second set of one or more QoE metrics measured by a second probe passing through a second path including one or more links; determine that the first path has an anomaly based on the first set of one or more QoE metrics; determine that the second path has an anomaly based on the second set of one or more QoE metrics; and in response to determining that the first path and the second path have an anomaly, determine that the intersection between the first path and the second path is the root cause of the anomaly based on the metric type and the link type.

Description

Technical Field

[0001] The present disclosure relates to computer networks. Background Art

[0002] Over the past few decades, the Internet has grown exponentially from a small network consisting of a few nodes to a pervasive global network serving more than 1 billion users. Today, individual users are no longer limited to network sessions where they conduct some voice and / or data downloads over the network. Instead, the services used by users vary widely from multimedia games, audio and video streaming, network services, voice over IP (VoIP), etc. With the penetration of new technologies, such as the Internet of Things (IoT) and the increased utilization of M2M (machine-to-machine) communications, the network services and software applications that a given user may require have also changed from a few sessions to multiple sessions with concurrent flows. This number is growing rapidly as users are increasingly running multiple applications, services, and transactions simultaneously. The increase in the number and variety of user sessions and packet flows has brought network performance challenges such as latency, delay, and jitter to network service providers. Summary of the invention

[0003] In general, the present disclosure describes techniques for providing network monitoring and fault location. For example, a controller such as a software defined network (SDN) controller implementing the techniques described herein can detect anomalies within the network by sending probe packets through the network to measure quality of experience (QoE) metrics such as latency, jitter, packet loss, and any other metrics.

[0004] In some examples, the controller can evaluate traffic flows to build a network topology and use the topology to create an enhanced detection plan implemented in the topology, such as by trimming the topology to stop some detection processes that the controller determines to be redundant (e.g., sending detection packets on links shared with multiple paths), thereby reducing the total traffic in the network. For example, instead of instructing each node (e.g., computing device) in the network to send a detection packet to measure the QoE metric of each link connected to the node device and / or each other node in the network, the controller can use the topology to select a minimum number of paths that cover all links and configure a minimum number of nodes on the paths to send detection packets to measure the QoE metric.

[0005] The controller can use QoE metrics to identify anomalies in the network. Some monitoring mechanisms, such as white-box monitoring and black-box monitoring, cannot locate the root cause of anomalies in the network (e.g., a specific link or node) without intensive processing (e.g., by sending probes to every node in the network), are prone to errors, or require additional manual troubleshooting.

[0006] According to the techniques described herein, a controller can detect one or more anomalies based on a QoE metric and perform fault location of a node or link that causes one or more anomalies. In one example, the controller can evaluate the QoE metric relative to a calculated baseline value to detect anomalies (e.g., determine a deviation from a baseline value). In response to determining that multiple detection paths have anomalies, the controller can determine the intersection of detection paths with anomalies (e.g., overlapping links), and determine that the intersection of the detection paths is the root cause of the anomaly. In some examples, the controller can determine the type of QoE metric being measured, and / or the type of link (e.g., a wired or wireless link). In these examples, if the metric is a loss metric, the controller can determine that the wireless link of the path is the root cause of the loss anomaly.

[0007] The techniques described herein may provide one or more technical advantages. For example, by implementing the techniques described herein, a node sends fewer probe packets to detect anomalies while still determining the root cause of the anomaly. In this way, processor usage, memory usage, and / or power consumption may be reduced without the need for manual troubleshooting to determine the root cause of the anomaly.

[0008] In one example of the technology described herein, a method is described that includes: receiving, by a controller of a network including a plurality of node devices, a first set of one or more quality of experience (QoE) metrics measured by a first probe traversing a first path including one or more links; receiving, by the controller, a second set of one or more QoE metrics measured by a second probe traversing a second path including one or more links; determining, by the controller, that the first path has an anomaly based on the first set of one or more QoE metrics; determining, by the controller, that the second path has an anomaly based on the second set of one or more QoE metrics; determining, by the controller, metric types of the first set of one or more QoE metrics and the second set of one or more QoE metrics, and link types of one or more links of the first path and one or more links of the second path; and determining, by the controller, in response to determining that the first path and the second path have an anomaly, based on the metric type and the link type, that an intersection between the first path and the second path is a root cause of the anomaly.

[0009] In another example of the technology described herein, a controller for a network including multiple node devices is described. The controller includes one or more processors operably coupled to a memory, wherein the one or more processors are configured to: receive a first set of one or more quality of experience (QoE) metrics measured by a first probe passing through a first path including one or more links; receive a second set of one or more QoE metrics measured by a second probe passing through a second path including one or more links; determine that the first path has an anomaly based on the first set of one or more QoE metrics; determine that the second path has an anomaly based on the second set of one or more QoE metrics; determine the metric type of the first set of one or more QoE metrics and the second set of one or more QoE metrics, and the link type of one or more links of the first path and one or more links of the second path; and in response to determining that the first path and the second path have an anomaly, determine that the intersection between the first path and the second path is the root cause of the anomaly based on the metric type and the link type.

[0010] In another example of the technology described herein, a non-transitory computer-readable storage medium is described that includes instructions that cause one or more programmable processors of a controller to: receive a first set of one or more quality of experience (QoE) metrics measured by a first probe traversing a first path including one or more links; receive a second set of one or more QoE metrics measured by a second probe traversing a second path including one or more links; determine that the first path has an anomaly based on the first set of one or more QoE metrics; determine that the second path has an anomaly based on the second set of one or more QoE metrics; determine metric types for the first set of one or more QoE metrics and the second set of one or more QoE metrics, and link types for one or more links of the first path and one or more links of the second path; and in response to determining that the first path and the second path have an anomaly, determine that an intersection between the first path and the second path is a root cause of the anomaly based on the metric type and the link type.

[0011] The details of one or more examples will be set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram illustrating an example network system that provides network monitoring and fault localization according to the techniques described in this disclosure.

[0013] Figure 2 is a block diagram illustrating an example software-defined wide area network in accordance with one or more techniques described in this disclosure.

[0014] Figure 3is a block diagram illustrating an example of fault localization according to techniques described in this disclosure.

[0015] Figure 4 is a block diagram illustrating an example controller configured to perform one or more functions to perform network monitoring and fault location according to the techniques described in this disclosure.

[0016] Figure 5 is a flow chart illustrating example operations for fault localization according to techniques described in this disclosure.

[0017] Figure 6 is a block diagram illustrating an example of a distributed controller according to techniques described in this disclosure. DETAILED DESCRIPTION

[0018] Figure 1 is a block diagram illustrating an example network system 2 ("network system 2") that provides network monitoring and fault location according to the techniques described herein. As an example, network system 2 may represent a software-defined network system.

[0019] The network system 2 may operate as a private network to provide packet-based network services to the user device 16. That is, the network system 2 provides authentication and establishment of network access for the user device 16 so that the user device can begin exchanging data packets with the public network 12, which may be an internal or external packet-based network, such as the Internet.

[0020] exist Figure 1 In the example of , the network system 2 includes an access network 6, which provides a connection to a public network 12 through a service provider software defined wide area network 7 (hereinafter referred to as "SD-WAN 7") and a router 8. The SD-WAN 7 and the public network 12 provide packet-based services that can be requested and used by user devices 16. As an example, the SD-WAN 7 and / or the public network 12 can provide bulk data transmission, voice over Internet Protocol (VoIP), Internet Protocol Television (IPTV), Short Message Service (SMS), Wireless Application Protocol (WAP) services, or customer-specific application services. The public network 12 can include, for example, a local area network (LAN), a wide area network (WAN), the Internet, a virtual LAN (VLAN), an enterprise LAN, a third layer virtual private network (VPN), an Internet Protocol (IP) intranet operated by a service provider operating the access network 6, an enterprise IP network, or some combination thereof. In various examples, the public network 12 is connected to a public WAN, the Internet, or other networks. The public network 12 implements one or more packet data protocols, such as IP (IPv4 and / or IPv6), X.25, or Point-to-Point Protocol (PPP), to enable packet-based delivery of public network 12 services.

[0021] Generally, user devices 16 are connected to gateway router 8 via access network 6 to receive connections for user services of applications hosted by public network 12 or router 8. Users may represent, for example, enterprises, residential users, or mobile users. User devices 16 may be, for example, personal computers, laptops, or other types of computing devices located behind customer equipment (CE) 11, which may provide local routing and switching functions. Each user device 16 may run various software applications, such as word processing and other office support software, web browsing software, software supporting voice calls, video games, video conferencing, and email, etc. For example, user devices 16 may be various network-enabled devices, commonly referred to as "Internet of Things" (IoT) devices, such as cameras, sensors (S), televisions, appliances, etc. In addition, user devices 16 may include mobile devices that access data services of network system 2 via radio access network (RAN) 4. Example mobile user devices include mobile phones, notebook or desktop computers with, for example, wireless cards, wireless-enabled netbooks, tablet computers, video game devices, pagers, smart phones, personal data assistants (PDAs), etc.

[0022] The network service provider operates or, in some cases, leases elements of the access network 6 (e.g., network equipment) to provide packet transport between the user devices 16 and the router 8. The access network 6 represents a network that aggregates data traffic from one or more user devices 16 for transport to or from the service provider's SD-WAN 7. The access network 6 includes network nodes that execute communication protocols to transmit control and user data to facilitate communications between the user devices 16 and the router 8. The access network 6 may include a broadband access network, a wireless LAN, a public switched telephone network (PSTN), a customer premises equipment (CPE) network, or other types of access networks, and may include or otherwise provide connectivity to a cellular access network, such as a radio access network (RAN), such as RAN 4. Examples include networks that conform to the fifth generation (5G) mobile network, the universal mobile telecommunications system (UMTS) architecture, an evolution of UMTS known as long term evolution (LTE), mobile IP standardized by the Internet Engineering Task Force (IETF), and networks that conform to other standards proposed by the Third Generation Partnership Project (3GPP), the Third Generation Partnership Project 2 (3GGP / 2), and the WiMAX Forum.

[0023] The SD-WAN device 18 may be a customer edge (CE) router, a provider edge (PE) router, or other network device that provides connectivity between the access network 6 and the SD-WAN 7. The SD-WAN 7 provides packet-based connectivity to user devices 16 connected to the access network 6 for access to a public network 12 (e.g., the Internet). The SD-WAN 7 may represent a public network owned and operated by a service provider to interconnect multiple networks that may include the access network 6. In some examples, the SD-WAN 7 may implement multi-protocol label switching (MPLS) forwarding, and in such a case may be referred to as an MPLS network or MPLS backbone. In some cases, the SD-WAN 7 represents multiple interconnected autonomous systems, such as the Internet, providing services from one or more service providers. The public network 12 may represent the Internet. The public network 12 may represent an edge network coupled to the SD-WAN 7 via a transit network 22 and one or more network devices (e.g., a customer edge device such as a customer edge switch or router). The public network 12 may include a data center. Figure 1 In the example of , the router 8 can exchange packets with the service node 10 via the virtual network 20 , and the router 8 can forward the packets to the public network 12 via the transit network 22 .

[0024] In the example of the network system 2 including a wired / broadband access network, the router 8 may represent a broadband network gateway (BNG), a broadband remote access server (BRAS), an MPLS PE router, a core router or gateway, or a cable modem termination system (CMTS). In the example of the network system 2 including a cellular access network as the access network 6, the router 8 may represent a mobile gateway, such as a gateway general packet radio service (GPRS) service node (GGSN), an access gateway (aGW), or a packet data network (PDN) gateway (PGW). In other examples, the functions described for the router 8 may be implemented in a switch, a service card, or other network element or component. In some examples, the router 8 itself may be a service node.

[0025] A network service provider that manages at least a portion of the network system 2 typically provides services to users associated with devices (e.g., user devices 16) that access the network system 2. The services provided may include, for example, traditional Internet access, VoIP, video and multimedia services, and security services. As described above with respect to the SD-WAN 7, the SD-WAN 7 can support multiple types of access network infrastructures that are connected to the service provider network access gateway to provide access to the services provided. In some cases, the network system may include user devices 16 connected to multiple different access networks 6 having different architectures.

[0026] In general, any one or more user devices 16 may request authorization and data services by sending a session request to a gateway device, such as an SD-WAN device 18 or router 8. The SD-WAN device 18 may then access a central server (not shown), such as an authentication, authorization, and accounting (AAA) server, to authenticate one of the user devices 16 requesting network access. Once authenticated, any user device 16 may send user data traffic to the SD-WAN 7 to access and receive services provided by the public network 12, and such packets may traverse the router 8 as part of at least one packet flow. In some examples, the SD-WAN device 18 may forward all authenticated user traffic to the public network 12, and the router 8 may apply the service 15 and / or direct specific user traffic to the data center 9 if the user traffic requires services on the service node 10. Applications to be applied to user traffic (e.g., service applications) may be hosted on the service node 10.

[0027] For example, when forwarding user traffic, the router 8 may direct a single user packet flow through a service 15 executed on one or more service cards installed within the router 8. In addition, or alternatively, the network system 2 includes a data center 9 having a collection of service nodes 10, which provide an execution environment for most virtualized network services. In some examples, each service node 10 represents a service instance. Each service node 10 can apply one or more services to a traffic flow. In this way, the router 8 can direct the user packet flow through a defined set of services provided by the service nodes 10. That is, in some examples, each user packet flow can be forwarded through a specific ordered set of services provided by the service nodes 10, each ordered set being referred to herein as a "service chain." As an example, service 15 and / or service node 10 may apply stateful firewall (SFW) and security services, deep packet inspection (DPI), carrier-grade network address translation (CGNAT), traffic destination function (TDF) services, media (voice / video) optimization, Internet Protocol Security (IPSec) / Virtual Private Network (VPN) services, Hypertext Transfer Protocol (HTTP) filtering, packet flow counting, accounting, billing and / or load balancing, or other types of services applied to network traffic.

[0028] exist Figure 1 In the example of , a user packet flow can be directed along a service chain that includes service 15 and / or any of the services applied by service node 10. Once processed at the end node of the service chain, i.e., the last service is applied to the packet flow along a particular service path, the traffic can be directed to the public network 12.

[0029] While a "service chain" defines one or more services applied in a specific order to provide a composite service of an application to a packet flow bound to the service chain, a "service tunnel" or "service path" refers to the logical and / or physical path taken by a packet flow processed by a service chain, and the forwarding state of the packet flow forwarded according to the service chain sequence. Each service chain can be associated with a corresponding service tunnel, and the packet flow associated with each user device 16 flows along the service tunnel according to the service profile associated with the corresponding user. For example, a given user can be associated with a specific service profile, which is in turn mapped to a service tunnel associated with a specific service chain. Similarly, another user can be associated with a different service profile, which is in turn mapped to a service tunnel associated with a different service chain. In some examples, after the SD-WAN device 18 has authenticated the user and established an access session, the SD-WAN device 18 or router 8 can guide the packet flow for the user along the appropriate service tunnel, so that the data center 9 applies the necessary ordered services for the given user. In some examples, the SDN controller 14 can also provide a forwarding rule set to the SD-WAN device 18 or router 8 for managing the forwarding path. In some examples, SDN controller 14 manages forwarding paths through all elements in data center 9 starting from router 8 .

[0030] In some examples, the service nodes 10 may implement service chaining using internally configured forwarding state that directs packets of a packet flow along the service chain for processing according to an identified set of service nodes 10. Such forwarding state may specify tunnel interfaces between service nodes 10 using network tunnels such as IP or Generic Routing Encapsulation (GRE) tunnels, Network Virtualization using GRE (NVGRE), or by using VLANs, Virtual Extensible LANs (VXLANs), MPLS technologies, etc. In some cases, real or virtual switches, routers, or other network elements interconnecting the service nodes 10 may be configured to direct packet flows to the service nodes 10 according to the service chain.

[0031] exist Figure 1In the example of , the network system 2 includes a software defined network (SDN) and a network function virtualization (NFV) architecture. The SDN controller 14 may provide a controller for configuring and managing the routing and switching infrastructure of the network system 2. Although some examples described herein relate to an SD-WAN device 18 that performs the edge-based routing techniques described herein, the SDN controller 14 may also perform these techniques for the network system 2. The NFV coordinator device 13 may provide a high-level coordinator for configuring and managing the virtualization of network services into the service nodes 10 of the data center 9. In some cases, the SDN controller 14 manages the deployment of virtual machines (VMs) within the operating environment of the data center 9. For example, the SDN controller 14 may interact with the router 8 to specify service chain information. For example, the service chain information provided by the SDN controller 14 may specify any combination and ordering of services provided by the service node 10, business engineering information for tunneling or otherwise transmitting packet flows along a service path, rate limiting, service type (TOS) marking, or packet classifiers that specify criteria for matching packet flows to specific service chains. Further example details of an SDN controller are described in PCT International Patent Application PCT / US13 / 44378 filed on June 5, 2013, the entire contents of which are incorporated herein by reference.

[0032] Although shown as part of the data center 9, the service node 10 can be a network device coupled by one or more switches or virtual switches of the SD-WAN 7. In one example, each service node 10 can run as a virtual machine in a virtual computing environment. In addition, the computing environment can include a scalable cluster of general-purpose computing devices, such as servers based on x86 processors. As another example, the service node 10 may include a combination of general-purpose computing devices and dedicated devices. As a virtualized network service, the individual network services provided by the service node 10 can be scaled by virtualizing memory allocation, processor utilization, storage, and network policies, just as in a modern data center, and can also be scaled horizontally by adding additional load balancing VMs. In other examples, the service node 10 can be a gateway device or other router. In further examples, the functionality described for each service node 10 can be implemented in a switch, a service card, or another network element or component.

[0033] As described herein, network elements within the network system 2 may perform application data monitoring using various application quality of experience (QoE) metric functions. Example QoE metric functions include active or synthetic detection performed by nodes such as end hosts, underlying network nodes, virtual nodes, or other types of nodes. For example, a network device located at the edge of an access network or data center network (e.g., SD-WAN device 18 and router 8) may output a detection packet that is transmitted through an overlay tunnel to a virtual router in the data center and returned to the network device or another network device. In response to receiving the detection packet, one or more network devices, virtual routers, and applications along the forwarding path of the detection packet may provide detection report information to the SDN controller 14, which may compile and analyze the detection report information and present to the customer reporting information about latency, jitter, packet loss, and any problems in the data center as it may be related to the SLA.

[0034] In some examples, the metrics carried by the probe packets may include one or more of a timestamp for sending or receiving a test packet, an error estimate for sending or receiving a test packet, a sequence number for sending a test packet, a time to live (TTL) value for a test packet, a keep alive packet data unit (PDU), and / or a count of service packets, bytes, or users. One-way and two-way network performance measurements may include keep alive or path connectivity, round trip time (RTT), path delay, packet jitter, packet reordering, packet loss, service latency measurements, or service load measurements based on received metrics.

[0035] Depending on the type of node, the SDN controller 14 can configure the node with an appropriate probe packet type, such as Internet Control Message Protocol (ICMP), User Datagram Protocol (UDP), Transmission Control Protocol (TCP), Real-time Performance Monitoring (RPM), Two-Way Active Measurement Protocol (TWAMP), IP Service Level Agreement (IP SLA), Bidirectional Forwarding Detection (BFD), etc. For example, the network system 2 may include a source host device in a data center 9, which routes packets in an upstream path to a core switch through an IPinIP tunnel, and the core switch routes the internal IP packets in a downstream path to a destination host. In this example, the SDN controller 14 may configure the end hosts in the network system 2 to send probes using, for example, ICMP, UDP, and / or TCP to measure the QoE metrics of the end-to-end path between the end hosts. Alternatively or additionally, the network system 2 may include a network device (e.g., a physical switch) in an underlying network (e.g., edge to edge in a WAN network, or a top of rack (ToR) switch to ToR in a data center environment). In this example, the SDN controller 14 can configure the underlying network nodes (e.g., routers 8) in the network system 2 to send probes using RPM, TWAMP, IP SLA, and / or BFD to measure the QoE metrics of the underlying network. Alternatively or additionally, the network system 2 can include virtual nodes (e.g., virtual routers) in the overlay network. In this example, the SDN controller 14 can configure the virtual nodes in the network system 2 to use locally supported probes (e.g., ICMP, UDP, and / or TCP) to measure the QoE metrics of the overlay network.

[0036] In some examples, network system 2 may include multiple tenants. In this example, each tenant may have a different topology, each topology having different detection requirements. For example, network system 2 may include a first tenant having an end host device that sends probes using TCP, a second tenant having an end host device that sends probes using UDP, a third tenant having an underlying network device that sends probes using TWAMP, a fourth tenant having an underlying network device that sends probes using RPM, and so on.

[0037] In this example, the SDN controller 14 may implement a distributed probe selection algorithm to generate an optimized probe plan. The SDN controller 14 may generate a probe plan based on each tenant policy. For example, each tenant of the network system 2 may have a different probe policy. A first tenant may include a policy to measure QoE metrics between source and target end hosts, while a second tenant may have a different policy to measure the loss of WAN links in the SD-WAN 7. As described above, depending on the type of node selected, the probe plan may specify an appropriate type of probe (e.g., ICMP, UDP, TWAMP, etc.).

[0038] The SDN controller 14 can configure the detection plan based on the detection budget of each tenant. For example, the detection plan can specify the number of probes to be sent (e.g., the probe count), the frequency of sending probes, the interval for sending probes, etc. The frequency of sending probes can be determined based on whether the detection path includes wired and / or wireless links. For example, if the path to be detected includes a wireless link, the detection plan can specify that the probes are sent at a higher frequency because the possibility of loss of the wireless link is higher. The number of probes and the interval for sending probes can be based on the current load of the probes in the network and the probe business generated by each node. For example, the SDN controller 14 can configure the upper limit of the probes to be sent and / or the upper limit of the probe business sent by each node.

[0039] Additional examples of generating optimized probe plans are described in U.S. Patent Application No. 16 / 354,402, filed on March 15, 2019, “PLANNING AND MANAGINGNETWORK PROBES USING CENTRALIZED CONTROLLER,” which is incorporated herein by reference in its entirety. Additional examples of end-to-end monitoring of overlay networks are described in U.S. Patent No. 9,787,559, filed on March 28, 2014, “END-TO-END MONITORING OF OVERLAY NETWORKS PROVIDING VIRTUALIZED NETWORK SERVICES,” which is incorporated herein by reference in its entirety.

[0040] In some examples, the SDN controller 14 can implement edge-based routing to more accurately measure various QoE metrics in the network while limiting the number of probes entering the system. For example, the SDN controller 14 can coordinate and map the detection process across the network system 2. For example, for a network system 2 that includes multiple different node devices, the SDN controller 14 can build a topology representation for each tenant of the network system 2. The topology representation can take any form (e.g., a matrix, database, vertex, graph, text, or any other data structure) that provides an indication of each node device of a specific tenant and an indication of each link of multiple links connected to the node device. Figure 1 In the example of , the SDN controller 14 can create a topology representation of any number of networks described in the network system 2. For example, the topology representation can be a CE 11 in the access network 6, a user device 16, one or more instances of an SD-WAN device 18 in the SD-WAN 7, one or more virtual nodes in a virtualized environment in the data center 9, or any combination thereof in an SDN system (e.g., network system 2). The SDN controller 14 can determine the routing of the network from the routing table of the network device.

[0041] Based on the topological representation, the SDN controller 14 can determine the optimal probe allocation. For example, the SDN controller 14 can use the topological representation of each tenant of the network system to determine the minimum number of paths covering all links and select the minimum number of nodes within the path to send the probe packet. For example, the SDN controller 14 can apply an optimization algorithm (e.g., a heuristic greedy algorithm) to the topological representation to calculate the probe path and select the nodes within the path to send the probe packet. An example of applying the greedy algorithm to the path matrix is ​​shown below and described:

[0042] Table I. Path Matrix

[0043]

[0044] In the above example, the SDN controller 14 may apply a greedy algorithm to determine the minimum number of paths covering Link 1, Link 2, and Link 3. In this example, sending probes on Path 1 and Path 2 will cover all links in the network. The SDN controller 14 may determine the minimum number of nodes within Path 1 and Path 2 to send the probe packet.

[0045] Alternatively or additionally, nodes may be selected to send probes based on a static configuration of a particular node (e.g., a user configuration), or nodes may be selected based on a mixture of static configuration and heuristic algorithms. Nodes selected to send probe packets may be referred to herein as "probe generators," and nodes selected to respond to probe packets may be referred to herein as "probe responders." Probe generators may perform active or synthetic probe functions, while nodes that are not configured as probe generators do not send probe packets. In this way, only selected nodes are configured to send probe packets.

[0046] In some examples, the SDN controller 14 may use a probe intent data model to configure a probe generator and / or a probe responder. For example, the SDN controller 14 may convert a high-level probe intent data model (e.g., a network or data model based on user intent) that defines the probe intent at a high level (e.g., more generally) into a low-level probe configuration data model that defines a platform-specific configuration for configuring a specific endpoint (e.g., an end host, an underlying network device, a virtual node) to use an appropriate probe packet. For example, a user may define a high-level probe intent data model to detect latency anomalies in a WAN link within the SD-WAN 7. In this example, the SDN controller 14 may identify the node type as an underlying network device (e.g., an SD-WAN device 18), and may convert the high-level probe intent data model into a low-level probe configuration data model to configure the SD-WAN device 18 to send a probe using TWAMP, for example, to measure latency metrics of WAN links within the SD-WAN 7. As another example, a user may define a high-level probe intent data model to detect anomalies in a data center 9 where virtual machines are deployed. In this example, for example, the SDN controller 14 may identify the node type as a virtual node (eg, a virtual router) and may convert the high-level probe intent data model into a low-level probe configuration data model to configure the virtual router to send probes using ICMP.

[0047] The low-level probe configuration data model may specify, for example, the network address (eg, IP address) of the probe generator and the network address of the probe responder. The low-level probe configuration data model may also specify the number of probes to be sent, the interval at which probes are sent, and / or the frequency at which probes are sent.

[0048] The selected node configured as a probe generator sends probes on various paths in the network. In some examples, the probe packets can be generated and analyzed by various components of the node, such as an agent of an end host and / or a virtual node. The SDN controller 14 can receive one or more QoE metrics from the selected node and can use the one or more QoE metrics to determine whether there are any problems (e.g., anomalies) with the paths in the network.

[0049] Existing monitoring mechanisms, such as white-box monitoring and black-box monitoring, are unable to perform fault localization to pinpoint the link or node that is the root cause of the problem. For example, white-box monitoring is used to monitor applications running on servers. However, in white-box monitoring, each network node is queried for its performance metrics, which is inefficient because the statistics of a node may be prone to errors or unavailable during its current faulty state. Similarly, black-box monitoring sends end-to-end probes that cannot identify the specific link or node that is the root cause of the problem and requires additional manual troubleshooting to pinpoint the source of the problem.

[0050] According to the techniques described in this disclosure, the SDN controller 14 can detect anomalies along the detection path and perform fault location to pinpoint the node or link that caused the anomaly. Figure 1 In the example of , a user may specify a high-level probe intent data model to detect latency anomalies between a router 8 and a user device 16. In this example, the SDN controller 14 may configure the router 8 to send a probe 31 on a path 30 to measure the latency from the router 8 to the mobile user device 16. Similarly, the SDN controller 14 may configure the router 8 to send a probe 33 on a path 32 to measure the latency from the router 8 to another user device 16. Likewise, the SDN controller 14 may configure the router 8 to send a probe 35 on a path 34 to measure the latency from the router 8 to another user device 16.

[0051] The SDN controller 14 may receive latency metrics measured from the probes 31, 33, and 35, and evaluate the latency metrics using the calculated baseline latency values ​​to detect anomalies in the probe paths. In some examples, the SDN controller 14 receives QoE metrics for each tenant of the network system 2, aggregates the QoE metrics for each tenant to calculate a set of QoE metrics, and evaluates the set of QoE metrics using the calculated baseline values.

[0052] The baseline value can be calculated based on historical data of the QoE metric and can be dynamically updated as the QoE metric is updated. In other examples, the baseline value can be statically configured. For example, the baseline latency value can be calculated based on latency metrics previously measured on paths 30, 32, and 34 or statically configured by a user. The calculated baseline value can be a threshold, a range, or any value to determine whether the metric meets the baseline value or whether the metric deviates from the baseline value (i.e., an anomaly).

[0053] If the SDN controller 14 determines that the multiple paths have an anomaly, the SDN controller 14 may determine whether the multiple paths have intersections, such as any overlapping links. If the multiple paths have at least one overlapping link, the SDN controller 14 determines whether the at least one overlapping link is the root cause of the anomaly. The determination of the root cause may depend on the type of QoE metric measured and / or whether the links of the path include wireless links.

[0054] For example, the SDN controller 14 may determine that each of the paths 32 and 34 has a latency anomaly, and that the paths 32 and 34 have at least one overlapping link, such as a WAN link in the SD-WAN 7. In this example, if the latency metrics of each of the paths 32 and 34 deviate from the baseline latency value, the SDN controller 14 may determine that the overlapping WAN link may be the root cause of the anomaly.

[0055] In some examples, when determining the root cause of the anomaly, the SDN controller 14 may determine whether the path includes a wired or wireless link. For example, path 30 may include a wireless link, and path 32 may include a wired link. In this example, the SDN controller 14 may receive a first loss metric for path 30 and a second loss metric for path 32, and determine that each of paths 30 and 32 includes an anomaly. In this example, the SDN controller 14 determines that path 30 includes a wireless link. If the first loss metric for path 30 deviates from the second loss metric for path 32, the SDN controller 14 may determine that the wireless link is a candidate for the root cause of the anomaly because the wireless link is more likely to drop packets and may therefore be the cause of packet loss. The SDN controller 14 may then configure an edge node to the wireless link to send additional probes, such as probe 37, on the wireless link to determine whether the wireless link is actually the root cause of the anomaly.

[0056] Figure 2 is a block diagram illustrating an exemplary SD-WAN 207 in accordance with one or more techniques described in this disclosure. Figure 2 SD-WAN 207 in can represent Figure 1 An example of SD-WAN 7 in Figure 2In the example of , the SD-WAN 207 includes three different WAN links: a first WAN link 240 coupling the SD-WAN device 218 to a multi-protocol layer switching (MPLS) network 250, a second WAN link 242 coupling the SD-WAN device 218 to the Internet 252, and a third WAN link 244 coupling the SD-WAN device 218 to a mobile network 254 (e.g., LTE / 5G). In other examples, the SD-WAN 207 may include any number of links of any suitable type for transmitting data flows between the client side (e.g., the client device 216 and the SD-WAN device 218) and the application side (e.g., the SD-WAN device 256 and the application server 258).

[0057] exist Figure 2 In the example of , the SDN controller 214 can configure any type of node, such as the client device 216, the SD-WAN device 218, the SD-WAN device 256, and / or the application server 258, to send appropriate types of probe packets to measure QoE metrics. In this example, the SDN controller 214 can configure the client device 216 to be able to send probes 231 and 233 along the end-to-end path between the client device 216 and the application server 258. For example, the client device 216 can represent an end host. In this example, the SDN controller 214 can configure the client device 216 to send appropriate types of probes (e.g., ICMP echo, UDP ping, or TCP ping) to measure QoE metrics along the end-to-end path to the application server 258.

[0058] In this example, the SDN controller 214 can detect anomalies measured by the probes 231, 233 along the path and perform fault location to pinpoint the node or link causing the anomaly. In this example, the SDN controller 214 can configure the client device 216 to send the probe 231 to the application server 258 on a first end-to-end path using the WAN link in the SD-WAN 207, and configure the client device 216 to send the probe 233 to the application server 258 on a second end-to-end path also using the WAN link in the SD-WAN 207. For example, the probe 231, such as a TCP ping, is sent on a first end-to-end path including the WAN link 242 coupling the SD-WAN device 218 to the Internet 252, and the probe 233 is sent on a second end-to-end path also including the WAN link 242. In this example, the SDN controller 214 can receive the QoE metrics measured from the probes 231 and 233, and evaluate the latency metrics using the calculated baseline latency value to detect anomalies in the probe path.

[0059] The SDN controller 214 may determine that each path has a latency anomaly and that the paths have crossings, such as WAN link 242 in the SD-WAN 207. In this example, if the latency metric for each path deviates from the baseline latency value, the SDN controller 214 may determine that the overlapping WAN link 242 may be the root cause of the latency anomaly.

[0060] As follows Figure 3 As further described in, in some examples, when determining the root cause of the anomaly, the SDN controller 214 can determine the type of metric being measured and whether the path includes a wired or wireless link. For example, one of the paths measured by the probes 231 and 233 may include a wireless link (e.g., coupling the SD-WAN device 218 to the client device 216). In this example, the SDN controller 214 can receive the loss metric and determine that one of the paths includes a wireless link. The SDN controller 214 can determine that the wireless link between the SD-WAN device 218 and the client device 216 is a candidate for the root cause of the anomaly (in addition to or in lieu of the WAN link 242). In this example, the SDN controller 214 can then configure an edge node to the wireless link, e.g., the SD-WAN device 218, to send additional probes on the wireless link to determine whether the wireless link is actually the root cause of the anomaly.

[0061] Figure 3 is a conceptual diagram illustrating example operations of fault localization according to techniques described in this disclosure. Figure 3 The system includes nodes 302A and 302B (collectively referred to as "nodes 302") and user devices 304A to 304D (collectively referred to as "user devices 304"). Node 302 may represent Figure 1 Any node in the network system 2, such as a terminal host, an underlying network device and / or a virtual node. User device 304 may represent Figure 1 Any user device 16. Figure 3 In the example of FIG. 3 , user equipment 304A and 304B are connected to node 302B via wireless link 306 , respectively. User equipment 304C and 304D are connected to node 302B via wired link 308 , respectively.

[0062] In this example, the controller 320 (eg, Figure 1304D) can configure node 302A to send probes along the path to user device 304. For example, node 302A sends probe 310 to measure one or more QoE metrics of the end-to-end path from node 302A to user device 304A. In this example, probe 310 can measure loss metrics and / or latency metrics. Similarly, node 302A sends probe 312 to measure one or more QoE metrics of the end-to-end path from node 302A to user device 304D. In this example, probe 312 can also measure loss metrics and / or latency metrics.

[0063] As an example, the SDN controller 320 receives a latency metric from each of the probes 310 and 312 and determines whether the path measured by the probes 310 and 312 has an anomaly. As an example, the SDN controller 320 may calculate a baseline latency value of 40 milliseconds (ms). If the probe 310 measures a latency of 100 ms and the probe 312 measures a latency of 120 ms, the SDN controller 320 may determine that each path has an anomaly because the measured latency deviates from the baseline latency value. In response to determining that the path measured by the probes 310 and 312 has an anomaly, the SDN controller 320 may determine the intersection of the path (e.g., overlapping links). In this example, link 322 is shared between the paths. The SDN controller 320 then determines whether link 322 is the root cause of the latency anomaly. In this example, if the latency metrics from the probes 310 and 312 deviate from the baseline latency value, the SDN controller 320 determines that the shared link 322 is the root cause of the latency anomaly. In some examples, the SDN controller 320 can determine whether the measured metric deviates from a baseline value based on a threshold value. The threshold value can be user-defined or configured through machine learning.

[0064] As another example, the controller 320 receives loss metrics from the probes 310 and 312 and determines whether there is an anomaly in the path measured by the probes 310 and 312. As an example, the SDN controller 320 may calculate a baseline loss value. If both the probe 310 and the probe 312 measure a loss that deviates from the baseline loss value, then assuming that the metric is a loss metric, the SDN controller 320 may determine whether the path measured by the probes 310 and 312 includes a wireless link. Since the wireless link is a possible cause of packet loss, the SDN controller 320 may determine that the wireless link (e.g., the wireless link 306 connecting the node 302B and the user device 304A) is a possible root cause of the loss anomaly (in addition to or in lieu of the shared link 322). In order to determine whether the wireless link is the root cause of the packet loss, the SDN controller may configure the edge computing device, such as the node 302B, to send additional probe packets, such as the probe packet 314, to measure the loss metric on the wireless link to determine whether the wireless link is the root cause of the loss anomaly.

[0065] Figure 4 is a block diagram illustrating an exemplary SDN controller configured to implement the techniques described herein. Figure 4 In the example of , the SDN controller 460 creates a topological representation of the network contained or managed by the SDN controller 460 to effectively create an overall process for probing various links within the network. The SDN controller 460 can operate as a network service controller for a service provider network. Figure 4 In the example shown, the SDN controller 460 includes a management unit 465 , a control unit 464 for controlling the operation of the SDN controller 460 , and a network interface 466 for exchanging packets with network devices through an inbound link 467 and an outbound link 468 .

[0066] In some examples, control unit 464 and / or management unit 465 may be implemented as one or more processes executed on one or more virtual machines of one or more physical computing devices. That is, although generally shown and described as being executed on a single SDN controller 460, aspects of each of these units may be delegated to or distributed on other computing devices.

[0067] Each of the control unit 464 and / or the management unit 465 may include one or more processors (not shown) that execute software instructions, such as a processor that defines software or a computer program stored to a computer-readable storage medium (not shown), such as a non-transitory computer-readable medium including a storage device (e.g., a disk drive or an optical drive) or a memory (e.g., flash memory or random access memory) or any other type of volatile or non-volatile memory that stores instructions to cause the one or more processors to perform the techniques described herein. Alternatively or additionally, each of the control unit 464 and / or the management unit 465 may include dedicated hardware, such as one or more integrated circuits, one or more application specific integrated circuits (ASICs), one or more application specific processors (ASSPs), one or more field programmable gate arrays (FPGAs), or any combination of one or more of the foregoing examples of dedicated hardware, for performing the techniques described herein. Figure 4 The architecture of the SDN controller 460 shown in FIG. 4 is shown for example purposes only and should not be limited to this architecture. In other examples, the SDN controller 460 can be implemented in various ways, such as software only, hardware only, or a combination of software and hardware.

[0068] The management unit 465 may include a management layer of the SDN controller 460, and the control unit 464 may include a control layer of the SDN controller 460. The management unit 465 includes an analysis unit 461 and a configuration unit 462. The analysis unit 461 may obtain information from the SD-WAN system (e.g., Figure 1 physical and / or virtual network elements (e.g., Figure 1 The information is captured by the gateway, service node 10 or each data center 9) and analyzed for use in managing the network services provided by the service provider. The information may include statistics, logs, events and errors.

[0069] Configuration unit 462 stores configuration information of network elements within the SD-WAN system. In some examples, the configuration information includes a virtual network configuration. Configuration unit 462 can convert a high-level data model of an expected virtual network configuration into a low-level data model for interacting with network elements.

[0070] The control unit 464 of the SDN controller 460 implements a centralized control plane for the SD-WAN system that is responsible for maintaining a constantly changing network state. The control unit 464 interacts with network elements within the SD-WAN system to maintain a consistent network state for all network elements. The control unit 464 provides an operating environment for a command line interface daemon 475 (“CLI 475”), which provides an interface through which an administrator or other management entity can modify the configuration of the SDN controller 460 using text-based commands. The control unit 464 also provides an operating environment for several protocols 470, including the Border Gateway Protocol (BGP) 472 and the Extensible Messaging and Presence Protocol (XMPP) 474, such as Figure 4 These commands can change which QoE metrics are measured, how the SDN controller 460 builds the network topology, and whether the SDN controller 460 optimizes the distribution of the detection processes performed on the node devices. User configuration (e.g., intent-based configuration) can also configure the probe itself, providing instructions on whether the probe is an active probe, a passive probe, or a combination thereof.

[0071] In some examples, control unit 464 uses XMPP 474 to communicate with network elements (e.g., Figure 1 The control unit 464 may communicate with a router 8 within the network system 2, a user device 16, or a service node 10 of a data center 9. Virtual network routing data, statistics collection, logs, and configuration information may be sent as an extensible markup language (XML) document according to XMPP 474 for communication between the SDN controller 460 and network elements. The control unit 464 may also communicate with one or both of the analysis unit 461 and the configuration unit 462 of the SDN controller 460 using XMPP 474.

[0072] The control unit 464 also includes a device manager 471, a topology unit 476, a detection management unit 477, a topology database 478, and a metric database 479, which enables the control unit 464 to build a topological representation of nodes, links, and detection processes to develop a complete detection plan across the network topology. The SDN controller 460 can be a cloud controller for the network, coordinating and mapping the detection process across the network. For example, the topology unit 476 can build a topological representation of the network for a network including multiple node devices (e.g., user devices and service nodes). The topological representation can take any form (e.g., a matrix, a database, a graph, a text, or any other data structure) that provides an indication of each node device and an indication of each link of multiple links, where each link connects two node devices. The topological representation can be stored in the topological database 478.

[0073] In some examples, topology unit 476 can convert a high-level data model associated with a network topology into a data model suitable for communicating with network elements or devices (e.g., Figures 1 to 3 In some cases, topology unit 476 can receive high-level data models (e.g., network or data models based on user intent) from a coordination engine and / or administrator via network interface 466. These models may be associated with network topology. Topology unit 476 can use these models and intents to build a topology representation for storage in topology database 478.

[0074] When constructing a topological representation of a network, the topology unit 476 may monitor one or more probe packets sent on each of a plurality of links. For example, when the technology described herein is first applied to a network, each node device may send a probe packet through each node device's respective link. However, the topology unit 476 may not inherently possess a data structure that describes which node devices are in the network and which other node devices each node device is linked to. By monitoring the various probe packets sent across the network, the topology unit 476 can fill in a data structure, such as an adjacency matrix, with indications of which node devices are in the network and which other node devices each node device is sending probe packets to. Since the network can be configured so that each node device performs a probe function on each link connected to the node device, the monitoring function will provide the topology unit 476 with a complete representation of the various connections within the network, and the topology unit 476 can use graph theory to derive a topological representation of the various node devices and links within the network.

[0075] In order to optimize the efficiency of the entire system described herein, the detection management unit 477 can determine the minimum number of paths covering all links and the minimum number of nodes within the path to send detection packets based on the topological representation of the network. For example, when selecting a node device for detection on a corresponding link, the detection management unit 477 can execute an optimization algorithm (e.g., a greedy algorithm).

[0076] Although the above functions can be part of the initialization techniques at the beginning of the implementation of the techniques described herein, these techniques can be repeated when new node devices enter the network. SD-WAN is dynamic in nature, and node devices can enter or leave the network at any time. This means that links are created and removed from the system throughout the existence of the SD-WAN. In this way, the topology unit 476 and the detection management unit 477 can be configured to repeat the process described herein whenever a new node device enters the system or whenever a current node device leaves the system.

[0077] In response to selecting the selected node device to measure one or more QoE metrics of the path, the detection management unit 477 can receive one or more QoE metrics of the path from the selected node device. The detection management unit 477 can store the QoE metrics in the metric database 479 so that the QoE metrics can be referenced in further extrapolation and estimation for other links and / or paths.

[0078] The device manager 471 can generate vendor-independent device information based on input provided by the topology database 478 and the metric database 479. The vendor-independent device information can also be referred to as device abstract configuration information. The vendor-independent device information is independent, per-device configuration information for each individual network device in the network. In some examples, the vendor-independent device information can include extensible markup language (XML) schema or yet another next generation (YANG) schema information.

[0079] The SDN controller 460 can then utilize the device manager 471 to implement one or more transformation processes to transform the vendor-independent device information into vendor-specific device information. The vendor-specific device information may also be referred to as vendor-specific or specific device configuration information. Each individual network device in the network may have both vendor-independent device information and vendor-specific device information. In some examples, the vendor-specific device information may be customized using, for example, Jinja2 templates for each different vendor.

[0080] As a result, the device manager 471 may be configured to Figure 4 The illustrated approach generates both vendor-independent device information and vendor-specific device information. The vendor-independent device information may include vendor-independent device configuration and / or command information (e.g., input via CLI 475 or other management interface), and the vendor-specific device information may also include vendor-specific device configuration and / or command information (e.g., sent via network interface 466 and / or via protocol 470). Figure 4The illustrated SDN controller 460 may send vendor-specific device information to a single network device for which the vendor-specific device information has been generated. For example, the SDN controller 460 may provide the vendor-specific device configuration information to the network device via a configuration interface, and may provide the vendor-specific device command information to the network device via a command interface. In this example, the vendor-specific device configuration information and the vendor-specific device command information may each be included in the vendor-specific device information, which is transformed from the vendor-independent device information. Additional information regarding translating high-level configuration instructions into low-level device configurations may be found in U.S. patent application serial number 15 / 198,657, filed on June 30, 2016, entitled “TRANSLATING HIGH-LEVEL CONFIGURATION INSTRUCTIONS TO LOW-LEVEL DEVICE CONFIGURATION,” which is incorporated herein by reference.

[0081] These techniques can enable the probe management unit 477 to create a dynamic representation of QoE metrics in the topology database 478 and the metrics database 479 for each link and each possible connection in all directions. For example, the probe management unit 477 can store the received measurement values ​​for each link in the metrics database 479. Then, for each extrapolation / estimate indicated in the topology database 478 (e.g., for each reverse direction of the received QoE metric, for each logical path that is also connected by two node devices connected by a link, and for each complex path that is not the shortest path between two node devices directly connected by a link), the probe management unit 477 can store the corresponding estimated QoE metric in the metrics database 479. When the probe management unit 477 receives new metrics, it can update this information. With this information, whenever the SDN controller 460 must make a routing decision for an application service or a service between nodes, the SDN controller 460 can refer to the topology database 478 and the QoE metrics stored in the metrics database 479 to make routing decisions for services within the network.

[0082] The SDN controller 460 is one example instance of a controller configured to perform the techniques described in this disclosure. The SDN controller 460 may include additional units that perform various functions, such as an alarm unit (not shown) that warns a user.

[0083] According to the techniques described in the present disclosure, the SDN controller 460 includes a fault location unit 480 to pinpoint nodes or links that cause one or more anomalies. For example, the SDN controller 460 can receive QoE metrics from selected nodes in the network through the network interface 466. The fault location unit 480 can evaluate the QoE metrics with calculated baseline values ​​to detect anomalies in the probe path. As an example, the calculated baseline value can be calculated based on historical data of the QoE metrics, such as the QoE metrics stored in the metrics database 479, and can be dynamically updated as the QoE metrics are updated. In other examples, a user can use the CLI 475 to statically configure the calculated baseline value.

[0084] The fault location unit 480 can evaluate the received QoE metrics with the calculated baseline value to detect anomalies in the detection path. If the fault location unit 480 determines that the multiple paths have anomalies, the fault location unit 480 can determine whether the multiple paths have intersections, such as any overlapping links. For example, the fault location unit 480 can determine from the topology database 478 whether the detection path has intersections (e.g., overlapping links). In response to determining that the detection path includes overlapping links, the fault location unit 480 determines whether at least one overlapping link is the root cause of the anomaly.

[0085] For example, the fault localization unit 480 may determine whether the QoE metric of the measured path deviates from the baseline value. If each QoE metric of the measured path deviates from the baseline value, the fault localization unit 480 determines that the overlapping link is the root cause of the abnormality.

[0086] In some examples, when determining the root cause of the anomaly, the fault location unit 480 can determine whether the path includes a wired or wireless link. The fault location unit 480 can determine that the QoE metric is a loss metric. In this example, the fault location unit 480 can determine from the topology database 478 whether the probed path includes a wireless link (e.g., whether the link uses a wireless protocol). If the fault location unit 480 determines that the path includes a loss anomaly, then assuming that the metric is a loss metric, the fault location unit 480 can determine whether the path measured by the probe includes a wireless link. Since wireless links are a possible cause of packet loss, the fault location unit 480 can determine whether the wireless link (e.g., the connection) is lost. Figure 3 The wireless link 306 between the node 302B and the user equipment 304A of the edge computing device 400 is a possible root cause of the loss anomaly (supplemented or replaced by the shared link 322). In order to determine whether the wireless link is the root cause of the packet loss, the fault location unit 480 can configure the edge computing device to the wireless link to send additional detection packets on the wireless link to determine whether the wireless link is the root cause of the loss anomaly.

[0087] If the SDN controller 14 determines that the multiple paths have an anomaly, the SDN controller 14 may determine whether the multiple paths have intersections, such as any overlapping links. If the multiple paths have at least one overlapping link, the SDN controller 14 determines whether the at least one overlapping link is the root cause of the anomaly. The determination of the root cause may depend on the type of QoE metric measured and / or whether the links of the path include wireless links.

[0088] For example, the SDN controller 14 may determine that each of the paths 32 and 34 has a latency anomaly, and that the paths 32 and 34 have at least one overlapping link, e.g., a WAN link in the SD-WAN 7. In this example, if the latency metric of each of the paths 32 and 34 deviates from the baseline latency value, the SDN controller 14 may determine that the overlapping WAN link may be the root cause of the anomaly.

[0089] In some examples, when determining the root cause of the anomaly, the SDN controller 14 may determine whether the path includes a wired or wireless link. For example, path 30 may include a wireless link and path 32 may include a wired link. In this example, the SDN controller 14 may receive a first loss metric for path 30 and a second loss metric for path 32, and determine that each of paths 30 and 32 includes an anomaly. In this example, the SDN controller 14 determines that path 30 includes a wireless link. If the first loss metric for path 30 deviates from the second loss metric for path 32, the SDN controller 14 may determine that the wireless link is a candidate for the root cause of the anomaly because the wireless link is more likely to drop packets, and therefore the wireless link may be the cause of the packet loss. The SDN controller 14 may then configure an edge node to the wireless link to send additional probes, such as probe 37, on the wireless link to determine whether the wireless link is actually the root cause of the anomaly.

[0090] Figure 5 is a flow chart illustrating an example technique of a software defined wide area network system according to the techniques of the present disclosure, the software defined wide area network system performing a probe allocation function based on a topological representation of the network. For example, the example operation is about Figure 3 The SDN controller 320 is described, but may represent any SDN controller described in the present disclosure. The following are the steps of the process, but other examples of processes performed in the technology of the present disclosure may include additional steps or may not include some of the steps listed below.

[0091] According to the techniques described herein, the SDN controller 320 receives a first set of one or more QoE metrics measured by a first probe that traverses a first path that includes one or more links (502). For example, the SDN controller 320 may configure the node 302A to send a probe 310 on a first end-to-end path between the node 302A and the user device 304A to measure QoE metrics (e.g., latency and loss). The SDN controller 320 receives latency metrics and loss metrics for the path between the node 302A and the user device 304A. Similarly, the SDN controller 320 may configure the node 302A to send a probe 312 on a second end-to-end path between the node 302A and the user device 304D to measure QoE metrics (e.g., latency and loss).

[0092] The SDN controller 320 receives a second set of one or more QoE metrics measured by a second probe that traverses a second path that includes one or more links (504). For example, the SDN controller 320 may configure the node 302A to send the probe 312 on a second end-to-end path between the node 302A and the user device 304D to measure QoE metrics (e.g., latency and loss). The SDN controller 320 receives latency metrics and loss metrics for the path between the node 302A and the user device 304D.

[0093] The SDN controller 320 determines that the first path has an anomaly based on the first set of QoE metrics (506), and determines that the second path has an anomaly based on the second set of QoE metrics (508). For example, the SDN controller 320 may calculate a baseline latency value of 40 milliseconds (ms). If the probe 310 measures a latency of 100 ms, and the probe 312 measures a latency of 120 ms, the SDN controller 320 may determine that each path has an anomaly because the measured latency deviates from the baseline latency value.

[0094] In response to determining that the first path has an anomaly and the second path has an anomaly, the SDN controller 320 determines a crossing between the first path and the second path 510. For example, in response to determining that the latency metrics measured by the probes 310 and 312 each deviate from a baseline latency value, the SDN controller 320 determines that the crossing between the paths (e.g., the overlapping link 322) is the root cause of the latency anomaly.

[0095] In some examples, when determining the root cause of the anomaly, the SDN controller 320 may determine whether the path includes a wired or wireless link. As an example, the SDN controller 320 may calculate a baseline loss value. If both the probe 310 and the probe 312 measure a loss that deviates from the baseline loss value, the SDN controller 320 may determine whether the path measured by the probes 310 and 312 includes a wireless link, assuming that the metric is a loss metric. Since the wireless link is a possible cause of packet loss, the SDN controller 320 may determine that the wireless link (e.g., the wireless link 306 connecting the node 302B and the user device 304A) is a possible root cause of the loss anomaly (in addition to or in lieu of the shared link 322). In order to determine whether the wireless link is the root cause of the packet loss, the SDN controller 320 may configure an edge computing device, such as the node 302B, to send additional probe packets, such as the probe packet 314, to measure the loss metric on the wireless link to determine whether the wireless link is the root cause of the loss anomaly.

[0096] Figure 6 is a block diagram illustrating an example of a distributed controller according to the techniques described in this disclosure. Figure 6 In the example of Figure 1 Controller 14 or any controller of multiple network devices.

[0097] Figure 6 An example edge computing architecture is shown that includes different types of edge nodes from a core network 602 to a customer device, each with different metric (e.g., latency) requirements. Depending on the type of edge node, the link can be wired or wireless. In this example, customer edge nodes 614A through 614N (collectively, “customer edge nodes 614”) can have a wireless connection with user devices 616A through 616N (collectively, “user devices 616”). For example, customer edge nodes 614 can represent edge devices of a 5G network or a telephone company (“telco”) network. In a 5G network, the last hop can include a wireless link to a 5G-enabled device (e.g., user device 616).

[0098] Different levels of edge nodes have different latency requirements. In this example, there is a path with a wired link between the core node to the node in the far edge network (e.g., far edge network 606A), and then there is a wireless link between the base station and the customer edge. The nodes in the near edge network (e.g., 604A) can be responsible for managing the edge nodes in a given geographic area. In large-scale networks (e.g., near edge nodes and customer edge nodes are on the order of 1000s and millions), a distributed detection controller (e.g., controller 600) will be used to configure the detectors of different nodes as detection generators. In Figure 6 In the example of , detection controller instances can be instantiated at edge nodes of different levels, and each instance of the detection controller is responsible for configuring the detectors for its downstream and upstream network paths. In this example, detection controller instances, such as near edge detection controller 610A, can be instantiated at near edge network 604A, near edge detection controller 610N can be instantiated at near edge network 604N, and far edge detection controller 612 can be instantiated at far edge network 606A.

[0099] For scalability purposes, a detection controller instance at the "near edge", such as near edge detection controllers 610A to 610N, can be responsible for configuring detectors for paths to "far edge" nodes in the same area. Similarly, a controller instance at the "far edge", such as far edge detector controller 612, can be responsible for configuring detectors to customer edge nodes and end devices. Analysis of the detection results for root causes can be performed in a distributed manner at different edge nodes. For example, a "far edge" site can perform a root cause analysis on any failure in the downstream network for which it is responsible. These partial results can be made available to higher-level controller instances, such as controller instances at the "near edge" and "core" nodes, for further aggregation of root cause analysis.

[0100] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. The various features described as modules, units, or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices or other hardware devices. In some cases, various features of electronic circuits may be implemented as one or more integrated circuit devices, such as an integrated circuit chip or chipset.

[0101] If implemented in hardware, the present disclosure may be directed to an apparatus such as a processor or an integrated circuit device, such as an integrated circuit chip or chipset. Alternatively or additionally, if implemented in software or firmware, the techniques may be implemented at least in part by a computer-readable data storage medium including instructions that, when executed, cause a processor to perform one or more of the above methods. For example, a computer-readable data storage medium may store such instructions that are executed by a processor.

[0102] The computer-readable medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0103] In some examples, computer-readable storage media may include non-transient media. The term "non-transient" may mean that the storage medium is not contained in a carrier or propagating signal. In some examples, non-transient storage media may store data that changes over time (e.g., in RAM or cache).

[0104] The code or instructions may be software and / or firmware executed by a processing circuit that includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Furthermore, in some aspects, the functionality described in the present disclosure may be provided in software modules or hardware modules.

[0105] Various examples have been described. These and other examples are within the scope of the following claims.

Claims

1. A method for a computer network, the method comprising: Receiving, by a controller of a network including a plurality of node devices, a first set of one or more quality of experience (QoE) metrics measured by a first probe traversing a first path including one or more links; receiving, by the controller, a second set of one or more QoE metrics measured by a second probe traversing a second path including one or more links; Determining, by the controller, that the first path has an abnormality based on the first set of one or more QoE metrics; determining, by the controller, that the second path has an abnormality based on the second set of one or more QoE metrics; determining, by the controller, metric types of the first set of one or more QoE metrics and the second set of one or more QoE metrics, and link types of one or more links of the first path and one or more links of the second path; and In response to determining that the first path and the second path have an anomaly, determining, by the controller, based on the metric type and the link type, whether a crossing between the first path and the second path is a root cause of the anomaly; Wherein, determining whether the intersection is the root cause of the anomaly includes: In response to determining that the metric type is a loss metric and one or more links of the first path or one or more links of the second path include a wireless link, determining the wireless link as a candidate for the root cause of the anomaly.

2. The method according to claim 1, wherein: The crossing includes overlapping links between one or more links of the first path and one or more links of the second path.

3. The method according to any one of claims 1 to 2, in, Determining whether the first path has an anomaly includes evaluating the first set of one or more QoE metrics relative to a baseline value, and Wherein, determining whether the second path has an anomaly includes evaluating the second set of one or more QoE metrics relative to the baseline value.

4. The method according to claim 3, wherein: Determining that the intersection is the root cause of the anomaly includes: It is determined that the first set of one or more QoE metrics and the second set of one or more QoE metrics each deviate from the baseline value.

5. The method according to claim 3, wherein: The baseline value is calculated based on one or more historical QoE metrics of the first path.

6. The method according to claim 3, wherein: The baseline value is statically configured.

7. The method according to claim 1, further comprising: A third probe is sent by the controller over the wireless link to measure a QoE metric of the wireless link.

8. A controller for a network, the network comprising a plurality of node devices, the controller comprising: One or more processors operably coupled to the memory, wherein the one or more processors are configured to: receiving a first set of one or more quality of experience (QoE) metrics measured by a first probe traversing a first path including one or more links; receiving a second set of one or more QoE metrics measured by a second probe traversing a second path including the one or more links; determining that the first path has an anomaly based on the first set of one or more QoE metrics; determining that the second path has an anomaly based on the second set of one or more QoE metrics; determining metric types of the first set of one or more QoE metrics and the second set of one or more QoE metrics, and link types of one or more links of the first path and one or more links of the second path; and In response to determining that the first path and the second path have an anomaly, determining whether a crossing between the first path and the second path is a root cause of the anomaly based on the metric type and the link type; Wherein, in order to determine whether the crossing is the root cause of the anomaly, the one or more processors are configured to: In response to determining that the metric type is a loss metric and one or more links of the first path and one or more links of the second path include a wireless link, determining the wireless link as a candidate for the root cause of the anomaly. 9 . The controller of claim 8 , wherein the intersection comprises overlapping links between one or more links of the first path and one or more links of the second path.

10. A controller according to any one of claims 8 to 9, in, To determine whether the first path has an anomaly, the one or more processors are configured to evaluate the first set of one or more QoE metrics using a baseline value, and To determine whether the second path has an anomaly, the one or more processors are configured to evaluate the second set of one or more QoE metrics using the baseline value.

11. The controller according to claim 10, wherein: To determine that the crossing is the root cause of the anomaly, the one or more processors are further configured to: It is determined that the first set of one or more QoE metrics and the second set of one or more QoE metrics each deviate from the baseline value.

12. The controller according to claim 10, wherein: The baseline value is calculated based on one or more historical QoE metrics of the first path.

13. The controller according to claim 10, wherein the network comprises a first edge device and a second edge device, and Wherein an instance of the controller is assigned to the first edge device to configure the first probe to measure the first set of one or more QoE metrics, and an instance of the controller is assigned to the second edge device to configure the second probe to measure the second set of one or more QoE metrics.

14. The controller according to claim 8, wherein: The one or more processors are further configured to: A third probe is sent over the wireless link to measure a QoE metric of the wireless link.

15. A non-transitory computer readable medium comprising instructions for causing one or more programmable processors of a controller to: receiving a first set of one or more quality of experience (QoE) metrics measured by a first probe traversing a first path including one or more links; receiving a second set of one or more QoE metrics measured by a second probe traversing a second path including the one or more links; determining that the first path has an anomaly based on the first set of one or more QoE metrics; determining that the second path has an anomaly based on the second set of one or more QoE metrics; determining metric types of the first set of one or more QoE metrics and the second set of one or more QoE metrics, and link types of one or more links of the first path and one or more links of the second path; and In response to determining that the first path and the second path have an anomaly, determining whether a crossing between the first path and the second path is a root cause of the anomaly based on the metric type and the link type; Wherein, in order to determine whether the intersection between the first path and the second path is the root cause of the exception, the one or more programmable processors are configured to determine the wireless link as a candidate for the root cause of the exception in response to determining that the metric type is a lost metric and one or more links of the first path and one or more links of the second path include a wireless link.

16. The non-transitory computer readable medium of claim 15, wherein: The one or more programmable processors are further configured to: A third probe is sent over the wireless link to measure a QoE metric of the wireless link.

17. The non-transitory computer readable medium according to any one of claims 15 to 16, wherein: To determine that the crossing is the root cause of the anomaly, the one or more programmable processors are further configured to: It is determined that the first set of one or more QoE metrics and the second set of one or more QoE metrics each deviate from a baseline value.

Images