Automatic application-based multi-path routing for SD-WAN services

Abstract

The invention discloses automatic application-based multi-path routing for SD-WAN (Secure Digital-Wide Area Network) services. Example network devices, systems, and methods are disclosed. In one example, a network device includes a memory configured to store information associated with one or more service level agreements (SLAs) of an application in a software defined wide area network (SD-WAN); and an application-based multipath routing (AMR) module including processing circuitry. The AMR module is configured to identify one or more applications for the AMR based on criteria, where each of the criteria is associated with a corresponding attribute of the application. The AMR module is configured to determine a violation of one of the SLAs on each WAN link associated with a first application of the identified one or more applications. The AMR module is configured to apply AMR to the first application in response to determining the violation.

Description

[0001] This application is a divisional application of application number 202211097066.8 filed on September 8, 2022, entitled "Automatic Application-Based Multipath Routing for SD-WAN Services", the entire contents of which are incorporated herein by reference.

[0002] This application claims priority to U.S. Patent Application No. 17 / 473,561, filed September 13, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to computer networks. Background Technology

[0004] A computer network is a collection of interconnected computing devices that can exchange data and share resources. In packet-based networks such as the Internet, computing devices transmit data by dividing it into variable-length chunks called packets, which are individually routed across the network from source devices to destination devices. The destination device extracts data from the packets and reassembles the data into its original form.

[0005] Network providers and enterprises can use Software-Defined Networking (SD-WAN) in a wide area network (WAN) to manage network connectivity between distributed locations, such as remote branch offices, central offices, or data centers. SD-WAN extends SDN, enabling enterprises to quickly and efficiently create connections over a WAN, which may include the Internet or other transport networks offering various WAN connectivity types, such as Multiprotocol Label Switching (MPLS) based connections, mobile network connections (e.g., 3G, LTE, 5G), Asymmetric Digital Subscriber Line (ADSL), etc. This connection is often referred to as a "WAN link" or more simply a "link." SD-WAN is considered a connectivity solution that leverages the aforementioned or other WAN connectivity types, utilizing WAN links as overlays on top of traditional WAN access.

[0006] SD-WAN services enable users such as enterprises to utilize WAN links to meet business and customer needs. In an SD-WAN environment, low-priority traffic can use lower-cost internet-based WAN links, while more important traffic can be transmitted via higher-quality WAN links, such as those provided by MPLS networks. WAN link usage can also be allocated on an application-by-application basis. With SD-WAN solutions, enterprise customers can mix and match cost optimization with Service Level Agreement (SLA) requirements to their needs. Users may expect their applications to experience a connection with an acceptable level of quality, often referred to as Quality of Experience (QoE). QoE can be measured based on various link performance metrics, including latency, delay (inter-frame gap), jitter, packet loss, and / or throughput (e.g., bandwidth). Users can define the expected level of one or more metrics of QoE (such as SLAs) in their service contracts with service providers. SLA metrics are typically user-configurable values, derived through trial and error or benchmarking environments against user experience or actual best-in-class application metrics. Summary of the Invention

[0007] Generally, this disclosure describes techniques for automatically identifying which applications should apply application-based multipath routing (AMR) when delivering application traffic over SD-WAN, and applying AMR to the identified applications when one or more conditions occur. According to the techniques of this disclosure, network devices can apply AMR to identified applications even if AMR was not pre-provisioned for the identified applications.

[0008] For example, network devices can use standards that can be associated with corresponding attributes of an application. In some examples, each standard in the standards can have a corresponding weight. These standards can be predetermined by, for example, the network device manufacturer, or can be provisioned by the network operator, administrator, or customer of the SD-WAN service. In some examples, the standards and / or the weights of the standards can be dynamically and automatically adapted based on network conditions and / or SD-WAN deployment, allowing the standards and / or weights to change without operator input. Network devices can use standards to identify which applications should be eligible for AMR. Then, when a breach occurs of the Service Level Agreement (SLA) for each WAN link associated with an application eligible for AMR, the network device can apply AMR, even if no AMR was prepared for that application. For example, the network device can apply AMR to an application by creating a copy of the application data packet and sending copies of the application data packet over two or more WAN links.

[0009] Manually determining which applications should be provisioned for AMR and provisioning such applications is a complex and tedious process, as thousands of applications may be providing traffic on the network. If AMR is pre-provisioned for some applications, and new applications that are critical to QoE are added to the network without AMR being pre-provisioned for them, the QoE of the new applications will be compromised if SLAs are not met on all WAN links associated with the new applications. By automatically identifying which applications should be eligible candidates for AMR and applying AMR to them even if it has not yet been pre-provisioned, network devices can save network administrators countless hours analyzing applications to determine which applications should be provisioned for AMR, while also reducing the impact on QoE when SLAs are violated. In this way, the example techniques provide a technical solution to the technical problems related to determining which applications should have AMR enabled and applying AMR to them even if it has not been pre-provisioned, and combine these techniques in a practical application to identify applications and apply AMR to them.

[0010] In one example, a network device includes a memory and an application-based multipath routing (AMR) module. The memory is configured to store information associated with one or more Service Level Agreements (SLAs) for applications used in a software-defined wide area network (SD-WAN). The AMR module includes processing circuitry and is configured to: identify one or more applications for the AMR based on criteria, wherein each criterion is associated with a corresponding attribute of the application; determine a violation of one of the SLAs on each WAN link associated with a first application among the identified one or more applications; and apply the AMR to the first application in response to the determination of the violation.

[0011] In one example, a method includes: identifying one or more applications for application-based multipath routing (AMR) in a software-defined wide area network (SD-WAN) based on processing circuitry and criteria, wherein each criterion in the criteria is associated with a corresponding attribute of the application; determining, by processing circuitry, a violation of the Service Level Agreement (SLA) for the first application on each WAN link associated with the first application among the identified one or more applications; and applying AMR to the first application by processing circuitry in response to determining the violation.

[0012] In one example, a non-transitory computer-readable storage medium includes instructions that, when executed, cause processing circuitry to: identify one or more applications for application-based multipath routing (AMR) in a software-defined wide area network (SD-WAN) based on criteria, wherein each criterion in the criteria is associated with a corresponding attribute of the application; determine a violation of the Service Level Agreement (SLA) for the first application on each WAN link associated with the first application among the identified one or more applications; and, in response to determining the violation, apply AMR to the first application.

[0013] Details of one or more examples are set forth in the accompanying drawings and the following description. Other features, objects, and advantages will become apparent from the specification, drawings, and claims. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating an example software-defined wide area network (SD-WAN) system implemented in a network according to the technology of this disclosure.

[0015] Figure 2 A block diagram of an example SD-WAN edge device is shown in more detail based on the techniques described in this disclosure.

[0016] Figure 3 This is a table diagram showing an example of application priority sorting.

[0017] Figure 4 This is a table showing examples of when AMR is applied.

[0018] Figure 5 This is a flowchart illustrating examples of automatic identification of applications for AMR and AMR technology applications.

[0019] Figure 6 This is a flowchart illustrating another example of automatic identification of applications for AMR and AMR technology applications.

[0020] Throughout the text and accompanying drawings, the same reference numerals denote the same parts. Detailed Implementation

[0021] Figure 1This is a block diagram illustrating an example software-defined wide area network (SD-WAN) system implemented in a network according to the techniques disclosed herein. SD-WAN system 100 includes transport networks 110A to 110N (collectively referred to as "transport network 110") for connecting sites attached to the transport network and for transmitting network traffic between such attached sites. One or more service providers may deploy transport network 110; therefore, transport network 110 may alternatively be referred to as a "service provider network." Sites attached to the service provider network may be referred to as "subscriber sites." As used herein, the terms "subscriber," "customer," and "tenant" are used interchangeably. SD-WAN system 100 may be configured to implement the techniques disclosed herein to identify applications eligible for Access to Mobile Registries (AMRs) and apply AMRs to such applications.

[0022] Service providers use SD-WAN system 100 to provide SD-WAN service 101 to their subscribers or organizations authorized by those subscribers. These subscribers or organizations may include, for example, cloud providers, cloud networks, and subscriber partners. SD-WAN service 101 provides a virtual overlay network that enables application-aware, coordinated connections to deliver IP packets between sites associated with the subscriber, based on policies. Service providers may offer multiple SD-WAN services.

[0023] SD-WAN system 100 includes a service coordinator 102, an SD-WAN controller 104, and multiple SD-WAN edge devices 108A to 108C (hereinafter referred to as "SD-WAN edges," and collectively as "SD-WAN edges 108") that implement SD-WAN service 101. The SD-WAN edges 108 are interconnected via a transport network 110. Control and ownership of the service coordinator 102, SD-WAN controller 104, SD-WAN edges 108, and transport network 110 may be distributed among one or more service providers, subscribers, enterprises, or other organizations. However, the SD-WAN service provider uses all these components to provide SD-WAN service 101. The SD-WAN service provider may be an enterprise, a network / internet service provider, a cloud provider, or other entity.

[0024] Generally, Service Coordinator 102 manages SD-WAN services. Service Coordinator 102 can control, enforce, configure, monitor usage, guarantee, analyze, protect, modify, reconfigure, and apply policies to SD-WAN services. Service Coordinator 102 can establish application-based forwarding on transport network 110 based on security policies, Quality of Service (QoS) policies, QoE policies, and / or business or intent-based policies. Service Coordinator 102 may include or represent a Network Service Coordinator (NSO). Service coordinator 102 is aware of the resources of SD-WAN system 100 and can perform, for example: tenant site and service management; end-to-end traffic coordination, visibility, and monitoring; physical network function (PNF) and / or virtual network function (VNF) management; policy and SLA management (PSLAM) for implementing SD-WAN functions; route management for managing routing operations, including creating virtual private networks, enabling routes on SD-WAN edge 108, and interfacing with route reflectors and routers; telemetry services, which provide an interface used by fault monitoring and execution monitoring systems to collect service check results from telemetry agents; and network activation functions for equipment provisioning. At least some of the above functions can be performed by individual or integrated components of SD-WAN controller 104.

[0025] SD-WAN controller 104 may include or represent a Network Services Controller (NSC). Generally, the Services Coordinator 102 interacts with SD-WAN controller 104 to manage SD-WAN edge 108, thereby creating and operating end-to-end SD-WAN management services between SD-WAN edges 108 via transport network 110. SD-WAN controller 104 can provide topology and SD-WAN edge 108 lifecycle management functions. For example, SD-WAN controller 104 provides PNF / VNF management for SD-WAN edge 108 managed by services coordinator 102. For example, SD-WAN controller 104 can configure network configurations for SD-WAN edge 108, configure policies on SD-WAN edge 108, etc. SD-WAN controller 104 can monitor the status and performance data of SD-WAN edge 108 and WAN links 142A-A to 142N-N (collectively referred to as "WAN links 142") and provide this information to services coordinator 102. In other words, the SD-WAN controller 104 can communicate with the SD-WAN edge 108 to determine the operational status of the WAN link 142 on the transport network 110 and obtain QoS / QoE performance metrics for the WAN link 142. As described in further detail, the SD-WAN system 100 can modify traffic patterns based on the WAN link performance metrics to better meet the SLA requirements of the SD-WAN services in the SD-WAN system 100.

[0026] In various examples of the SD-WAN system 100, the service coordinator 102 and the SD-WAN controller 104 may, for example, be combined to form a single service coordination platform with separate service coordination and domain coordination layers, which may be deployed as a separate device or apparatus, or each service coordination platform may be distributed among one or more components executed on one or more servers deployed in one or more locations. The service coordinator 102 may be a scalable and cloud-deployable platform. For example, a service provider of the SD-WAN service in the SD-WAN system 100 may deploy the service coordinator 102 to a provider site or a public, private, or hybrid cloud. Thus, the operations and functions attributable to the service coordinator 102 in this disclosure may be performed by a separate SD-WAN controller 104, and vice versa. In some example architectures, aspects of service coordination and SD-WAN control may also be distributed from the service coordinator 102 and the SD-WAN controller 104, respectively, at the SD-WAN edge 108.

[0027] Administrators and applications can interact with the service coordinator 102 using northbound interfaces, such as RESTful interfaces (web-based REST APIs), command-line interfaces, portal or graphical user interfaces, web-based user interfaces, or other interfaces of the service coordinator 102. Figure 1 (Not shown in the image). Service coordinator 102 can communicate with SD-WAN controller 104 via a southbound interface, which can be the northbound interface of the SD-WAN controller, such as a RESTful interface, command-line interface, graphical user interface, or other interface of service coordinator 102. Figure 1 (Not shown in the image).

[0028] Network link 140 connects SD-WAN edge 108 to transport network 110. Network link 140 and transport network 110 constitute the underlying network of SD-WAN service 101 and provide underlying connectivity between paired SD-WAN edges 108. For example, transport networks 110A and 110N provide separate underlying connectivity between SD-WAN edges 108A and SD-WAN edge 108C. Figure 1 (Not shown in the image). The underlying connection can be public or private, and can provide network services such as Label Switched Path (LSP), Ethernet and IP services, public internet services, or other services that implement overlay WAN links. The cost of using the underlying connection can be a flat rate or based on usage. Each underlying connection can have bandwidth limitations, performance metrics (e.g., latency, loss, jitter, etc.). SD-WAN service 101 can be deployed using underlying connections based on multiple different types of network services. Figure 1In the example, for instance, the underlying connection from SD-WAN edge 108A to SD-WAN edge 108C via transport network 110A could be an LSP for IP-VPN, while the underlying connection from SD-WAN edge 108A to SD-WAN edge 108C via transport network 110N could be an IPSec tunnel over the Internet. This diversity can be advantageous for SD-WAN services by promoting redundancy and by providing differentiated service capabilities to match cost / performance with application requirements / SLAs for different traffic using SD-WAN services. For example, SD-WAN edge 108A could route low-cost traffic over the Internet while simultaneously routing traffic for applications requiring low latency (e.g., IP telephony) via an LSP. The underlying connection can be created and / or managed by the SD-WAN service provider or SD-WAN service 101 subscriber, who will notify the service coordinator 102 of the underlying connection. The service coordinator 102 obtains link data for WAN link 142, including bandwidth limitations of WAN link 142 (if any). Service coordinator 102 may obtain link data from SD-WAN controller 104, receive configuration data containing link data, or obtain link data from another network controller or from SD-WAN edge 108. WAN links 142 are described and shown as bidirectional, but each of WAN links 142 may represent two separate WAN links, one in each direction.

[0029] The SD-WAN system 100 illustrates multiple sites associated with subscribers of the SD-WAN service provider 101 and attached to subscriber-facing interfaces of the SD-WAN edge 108. These sites may be referred to as subscriber sites, and they constitute a subscriber network because the SD-WAN service 101 interconnects multiple sites to form a single network. Figure 1 The SD-WAN system 100 in the example includes sites 106A to 106B, and may optionally include any one of site 106C, hub 112, cloud 114, or cloud service 116. In some cases, the “subscriber” and the SD-WAN provider are the same entity, such as in the case of an enterprise deploying and managing the SD-WAN system 100.

[0030] Each of sites 106A through 106C refers to a subscriber location and can represent, for example, a branch office, private cloud, on-premises branch, enterprise hub, or cloud branch. Provider hub 112 represents a multi-tenant hub device located at a point of presence (PoP) on the service provider network. Provider hub 112 can terminate overlay tunnels for overlay networks, which can be of various types, such as MPLS based on generic route encapsulation (MPLSoGRE), MPLSoGRE based on IPSec (MPLSoGREoIPsec), and MPLS based on User Datagram Protocol (MPLSoUDP) tunnels. Provider hub 112 can be a hub in a hub-and-spoke architecture used in some example deployments of SD-WAN service 101.

[0031] Cloud 114 refers to public, private, or hybrid cloud infrastructure. Cloud 114 can be a virtual private cloud within a public cloud. Cloud service 116 is a resource or higher-level service provided by a cloud service provider to a subscriber via SD-WAN service 101. Cloud service 116 can be, for example, Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), Storage as a Service, or other types of cloud services. Cloud service 116 can be provided by the infrastructure of cloud 114.

[0032] Internet 118 refers to web and / or internet connectivity services provided via the web. In this example, the SD-WAN edge 108B includes an internet breakout 120, and application flows are distributed to the internet breakout 120 according to policies.

[0033] Each of the SD-WAN edges 108 includes physical network functions or virtual network functions for implementing the SD-WAN service 101. In various examples, each of the SD-WAN edges 108 may be one or more VNFs or PNFs, for example, located within any of a service provider data center, provider hub, client, or cloud provider endpoint. Each of the SD-WAN edges 108 may be a router, a security device such as a firewall, a gateway, a WAN acceleration device, a switch, a cloud router, a virtual gateway, a cloud virtual gateway, an SD-WAN device, or other devices that implement aspects of the SD-WAN service 101.

[0034] In various examples, each of the SD-WAN edges 108 can be an on-premises branch, which is a PNF placed at a user branch site in a hub-and-spoke or full-mesh topology; a cloud branch, which is a VNF in a subscriber's Virtual Private Cloud (VPC) (or equivalent) located in a public cloud; a PNF or VNF located in a service provider's cloud operating as a hub to establish tunnels with the branch site (the hub is multi-tenant, i.e., shared between multiple sites using virtual routing and forwarding instances configured on multiple sites); or a PNF or VNF located in the enterprise, operating as an enterprise hub to provide additional hub-like functionality to ordinary branch sites (e.g., as an anchor point for branches created by a Dynamic Virtual Private Network (VPN), providing a central exit option within the enterprise, hosting data center divisions, importing routing protocols to create dynamic LAN segments, and meshing with other enterprise hubs belonging to the same tenant / subscriber). Each of the SD-WAN edges can be located at any of the sites 106, hubs 112, clouds 114, or cloud services 116.

[0035] SD-WAN edge 108 is logically located at the boundary between provider SD-WAN service 101 and subscriber network. SD-WAN edge 108 has a network-side interface for underlying connectivity and a subscriber-side interface for communicating with subscriber network. As described above, SD-WAN edge 108 can have multiple paths to each other (different underlying connections). For example, in a hub-and-spoke deployment, SD-WAN edge 108A has multiple paths, each via a different one of the transport networks 110, to SD-WAN edge 108C at hub 112. The interfaces of SD-WAN edge 108 can be primarily used for underlying connectivity of user data traffic, but the interfaces can also be used for management (operation, maintenance, and management (OAM)) traffic, such as sending performance metrics to service coordinator 102 and receiving policies, device configurations, and other configuration data from service coordinator 102.

[0036] Service coordinator 102 can provide and establish overlay tunnels between SD-WAN edges 108 to implement the SD-WAN service topology 101. Figure 1 In the example, any of the WAN links 142 can be implemented using point-to-point overlay tunnels, for example, for a virtual private network. Overlay tunnels inherit the performance characteristics of the underlying connection. Overlay tunnels can be encrypted or unencrypted. The SD-WAN edge 108 can use any of a variety of encapsulation types, such as MPLS, MPLSoGRE, IP-in-IP, MPLSoUDP, MPLSoGREoIPSec, IPSec, and GRE, to implement overlay tunnels.

[0037] SD-WAN edge 108 uses WAN link 142 to send application traffic to other SD-WAN edges 108 via SD-WAN service 101. WAN link 142 typically, but not necessarily, traverses the different underlying connections between SD-WAN edges 108. N WAN links 142A-A to 142A-N connect SD-WAN edges 108A and SD-WAN edge 108C. Figure 1 In the example, each of WAN links 142A-A to 142A-N traverses a different one of transport networks 110. Similarly, N WAN links 142N-A to 142N-N connect SD-WAN edges 108B and SD-WAN edge 108C, each via a different one of transport networks 110. In a full mesh topology (not shown), additional WAN links would connect SD-WAN edges 108A and 108B. WAN link 142 may also be referred to as an "overlay connection," "virtual connection," "tunnel virtual connection," "SD-WAN link," or other terms describing WAN links used to implement SD-WAN services.

[0038] According to some aspects of the invention, the SD-WAN edge 108B can identify one or more applications in a Software-Defined Wide Area Network (SD-WAN) for AMR based on standards, wherein each standard in the standards is associated with a corresponding attribute of the application. In some examples, the standards have weights. The SD-WAN edge 108B can determine a violation of one of the SLAs on each WAN link associated with a first application among the one or more applications. The SD-WAN edge 108B can apply AMR to the first application in response to determining the violation.

[0039] In some examples, the SD-WAN edge 108 can be configured to identify applications for AMR, such as those associated with application traffic forwarded through the SD-WAN edge 108. For example, the SD-WAN edge 108B can be configured to identify applications that are relatively important for QoE (also referred to herein as relatively important applications). In some examples, the SD-WAN edge 108B can be configured to store criteria, each of which is associated with a corresponding attribute of an application, and in some examples, each criterion has a corresponding weight. The SD-WAN edge 108B can then identify one or more applications for AMR based on these criteria. For example, the SD-WAN edge 108B can compare an application's attributes to the criteria, and if one or more of an application's attributes match any of the criteria, the SD-WAN edge 108B can identify the application as a candidate for AMR or as eligible for AMR. If none of the application's attributes match any criteria, the SD-WAN edge 108B can identify the application as not a candidate for AMR or as ineligible for AMR. In this way, the SD-WAN edge 108B can use criteria to automatically determine whether an application is a candidate for AMR.

[0040] The SD-WAN edge 108B can monitor each WAN link associated with an application identified for AMR and determine SLA violations on each WAN link associated with a first application identified as eligible for AMR. In response to a violation determination, the SD-WAN edge 108B can apply AMR to the first application. This can be done even if AMR is not pre-provisioned for the first application. That is, the first application is not explicitly pre-defined as an application and AMR implementation in the SD-WAN edge 108B's configuration data.

[0041] When applying AMR to the first application, the SD-WAN edge 108B can replicate the application data packets of the first application on the SD-WAN edge 108B and can forward a copy of each application data packet on each of two or more WAN links. For example, the SD-WAN edge 108B can replicate the application data packets of the first application and forward the copies of the application data packets to the SD-WAN edge 108C via both WAN links 142N-A and 142N-N. In this way, the SD-WAN edge 108C can receive multiple copies of the same application data packets. When the SD-WAN edge 108C receives multiple copies of the same application data packets from the SD-WAN edge 108B, the SD-WAN edge 108C can retain the first arriving application data packet and delete or ignore the second arriving application data packet. In this way, the application's QoE may not be affected as much as if only a single WAN link were used to carry application traffic.

[0042] In some examples, the SD-WAN system 100 may not have sufficient resources to apply AMR to applications identified as eligible for AMR. Therefore, in some examples, the SD-WAN edge 108B can be configured to determine whether available resources are sufficient to support AMR for a first application, and further apply AMR based on the availability of sufficient resources. In some examples, each of the multiple applications identified for AMR may experience an SLA violation on each WAN link associated with that application. Thus, in some examples, the SD-WAN edge 108B can be configured to prioritize applications identified for AMR. For example, as described above, a weight can be assigned to each standard in the criteria, which can be used to determine priority for AMR among applications identified as eligible for AMR. The SD-WAN edge 108B can determine the priority of each application identified for AMR based on the cumulative weight of each standard having corresponding attributes satisfied by the application. In this way, if there are sufficient resources to apply AMR to one or more of the identified applications, but not enough resources to apply AMR to all of the identified applications, the SD-WAN edge 108B can use the priority of each of the identified applications to determine which applications should receive AMR. For example, the SD-WAN edge 108B can apply AMR to higher-priority applications and prohibit AMR from being applied to lower-priority applications. As another example, the SD-WAN edge 108B can apply AMR to additional applications in priority order until resource usage meets a threshold. In some examples, the SD-WAN edge 108B can reserve a portion of available resources or network resources for other purposes, such as new applications. In this example, the SD-WAN edge 108B can avoid applying AMR to one or more lower-priority applications, even if there are sufficient resources to apply AMR to one or more lower-priority applications.

[0043] Service coordinator 102 can deploy SD-WAN service 101 using SD-WAN controller 104 in various architectural topologies, including mesh and hub-and-spoke. A mesh topology is one where traffic can flow directly from any site 106 to another site 106. In a dynamic mesh, SD-WAN edge 108 stores resources for implementing a full mesh topology. All sites in the full mesh network are included in the topology, but site-to-site VPN is only enabled when traffic exceeds a user-defined threshold (called the dynamic VPN threshold). Sites in a mesh topology can include site 106, cloud 114, and / or cloud service 116. In a hub-and-spoke topology, all traffic passes through hub 112, more specifically, through SD-WAN edge 108C deployed at provider hub 112. By default, traffic to the Internet also flows through provider hub 112. In a hub-and-spoke topology, network services (e.g., firewalls or other security services) may be applied at the central hub 112 location, allowing all network traffic for SD-WAN service 101 to be handled using the network services of a single site. The SD-WAN service 101 can have a combination of full-mesh and hub-and-spoke regional topology, which uses one or more regional hubs to connect multiple branches to a wider mesh.

[0044] In some examples, the SD-WAN controller 104 includes a route reflector (not shown) to facilitate routing in the SD-WAN service 101. The route reflector forms an overlay Border Gateway Protocol (BGP) session with the SD-WAN edge 108 to receive, insert, and reflect routes.

[0045] SD-WAN edge 108 receives inbound network traffic from the corresponding subscriber site and applies SD-WAN service 101 to forward the network traffic to another SD-WAN edge 108 via one of WAN links 142. SD-WAN edge 108 receives network traffic on WAN link 142 and applies SD-WAN service 101 to, for example, forward the network traffic via one of WAN links 142 to another SD-WAN edge 108 (where the SD-WAN edge is the hub) or forward it to the destination subscriber site.

[0046] To apply SD-WAN service 101, SD-WAN edge 108 processes network traffic based on routing information, policy information, performance data, and service characteristics of WAN link 142. This information can be derived at least in part from the performance, bandwidth constraints, and behavior of the underlying connection. For example, SD-WAN edge 108 uses dynamic path selection to route network traffic to different WAN links 142 to attempt to meet the QoS / QoE requirements defined in the SLA and configured for SD-WAN service 101 in SD-WAN edge 108, or to bypass faulty WAN links for routing. For example, SD-WAN edge 108A might select WAN link 142A-A as a low-latency MPLS path (in this example) for VoIP traffic, while selecting WAN link 142A-N as a low-cost, broadband internet connection for file transfer / storage traffic. SD-WAN edge 108 can also apply traffic shaping. The terms "link selection" and "path selection" refer to the same operation of selecting WAN links for an application and are used interchangeably.

[0047] SD-WAN edge 108 processes and forwards received network traffic from SD-WAN service 101 based on policy and configuration data, routing information, and current network conditions including underlying connectivity performance characteristics from service coordinator 102. In some examples, service coordinator 102 may push SLA parameters, path selection parameters, and related configurations to SD-WAN edge 108, and SD-WAN edge 108 may monitor SLA violations on links and apply AMR to applications identified by AMR, or switch applications to a different one of WAN links 142. SD-WAN edge 108 can thus implement the data plane functionality of SD-WAN service 101 at the underlying connectivity, including, in this example, applying AMR to applications for QoE and switching applications to a different WAN link 142. If an SLA violation is detected in one of the SD-WAN edges 108, the SD-WAN edge may or may not apply AMR, as described throughout this disclosure, but may still report to service coordinator 102 and send log messages describing the SLA violation and the selected WAN link. The SD-WAN edge 108 can also summarize, optionally average, and report SLA metrics for WAN link 142 in log messages sent to the service coordinator 102. In some examples, the service coordinator 102 can receive SLA metrics from the SD-WAN edge 108, determine that an application's SLA has been violated, and perform path selection to choose a new WAN link 142 for the violating application. The service coordinator 102 can then configure one or more of the SD-WAN edges 108 to switch application traffic for that application onto the new WAN link. SLA metric analysis, SLA assessment, path selection, and link switching are all performed by the SD-WAN system 100, but different examples of the SD-WAN system 100 may have different control plane functionalities distributed between the service coordinator 102 and the SD-WAN edge 108 than those just described. However, the following description focuses primarily on the SD-WAN edge 108.

[0048] The SD-WAN edge 108 can forward traffic based on application flows. Application flow packets can be identified using packet characteristics such as Layer 3 and Layer 4 (e.g., TCP, UDP) header fields (e.g., source / destination Layer 3 address, source / destination port, protocol), deep packet inspection (DPI), or other flow identification techniques used to map packets to applications (or more specifically, application flows). Application flows can include packets for multiple different applications or application sessions, and a single application can be segmented across multiple application flows (e.g., separate video and audio streams for a video conferencing application).

[0049] SLAs can specify applicable application flows and can include policies for forwarding application flows. The SD-WAN edge 108 can identify application flows and apply appropriate policies to determine how to forward them. For example, the SD-WAN edge 108 can use application-specific QoE and advanced policy-based routing (APBR) to identify application flows and specify paths for them by associating the SLA profile with the routing instance on which the application flow is to be sent. The routing instance can be a Virtual Router and Forwarder Instance (VRF) configured with an interface for WAN link 142.

[0050] QoE aims to improve the user experience at the application layer by monitoring Service Level Parameters (SLA) compliance and application traffic, and by facilitating the placement of application data on SLA-compliant WAN links 142 (or the WAN link with the highest SLA compliance available), or by promoting the use of applications identified for AMR. For example, the SD-WAN edge 108 and the service coordinator 102 can monitor application traffic for SLA compliance. In some examples, the SD-WAN edge 108 (independently or as instructed by the service coordinator 102) can move application traffic from WAN links 142 that do not meet SLA requirements to one of the WAN links 142 that do meet SLA requirements, or apply AMR for applications identified for AMR when all WAN links associated with the application fail to meet SLA requirements.

[0051] To monitor SLA compliance on links that send application traffic, the service coordinator 102 can cause the SD-WAN edge 108 to send embedded probes (in some cases, along with the already sent application traffic) along WAN links 142. These embedded probes may be referred to as “passive probes”. To identify the best available WAN link 142 for application use if an active WAN link fails to meet SLA standards, the service coordinator 102 monitors and collects SLA compliance data for other available WAN links 142 for SD-WAN service 101. Probes sent by the service coordinator 102 through other WAN links 142 to check SLA compliance may be referred to as “active probes”. Active probes are performed based on probe parameters provided by the subscriber in some cases. Active and passive probes measure end-to-end analysis for WAN links 142. The data collected through active and passive probes is used to monitor the network to find sources of failure or congestion. If a violation is detected for any application or a group of applications (“application group”), the service coordinator 102 evaluates synthetic probe metrics to determine the desirable, and in some cases best, WAN link 142 that meets the SLA. As used herein, the reference to an application may refer to a single application or any group of applications.

[0052] Configuring the service coordinator 102 to enable the SD-WAN system 100 to apply QoE to the SD-WAN service 101 may involve configuring multiple profiles of various profile types, allowing users to parameterize QoE for various applications / application groups with traffic transported by the SD-WAN service 101. Profiles typically include human-readable text defining one or more parameters for a function, or associating profiles with other profiles to parameterize higher-level functions. In various examples, the service coordinator 102 may provide various configuration schemes for QoE parameterization of the SD-WAN service 101.

[0053] Subscribers or network administrators can interact with Service Coordinator 102 to create SLA profiles for applications, referred to herein as “application SLA profiles” or simply “SLA profiles”. SLA profiles can include SLA configuration data such as traffic type profiles, indications of whether local breakout is enabled, path preferences (e.g., preferred WAN link for WAN link 142 or indications of WAN link type (e.g., MPLS, Internet, etc.), indications of whether failover is allowed when an active WAN link has an SLA violation of its SLA profile, and failover criteria (e.g., violation of any SLA parameter or violation of all SLA parameters required to trigger failover). In some examples, network administrators can interact with Service Coordinator and / or SD-WAN Edge 108 to provision AMRs for selected applications or customize the criteria and / or weights used by SD-WAN Edge 108 to identify and prioritize applications for AMRs.

[0054] SLA parameters can be included in an SLA metrics profile, which is associated with or part of the SLA profile. The service coordinator 102 and the SD-WAN edge 108 can use SLA parameters to evaluate the SLA of the WAN link 142. SLA parameters can include parameters such as throughput, latency, jitter, jitter type, packet loss, round-trip delay, or other performance metrics of traffic (which are related to and correspond to the performance metrics of the WAN link carrying that traffic). Throughput can refer to the amount of data sent upstream or received downstream of a site over a period of time. Latency refers to the time it takes for packets to travel from one specified point to another. Packet loss can be specified as the percentage of packets dropped by the network to manage congestion. Jitter is the difference between the maximum and minimum round-trip time of a packet.

[0055] The SLA configuration file can also specify SLA sampling parameters and rate limiting parameters. Sampling parameters can include session sampling percentage, SLA violation count, sampling period, and switchover cooldown period. Session sampling percentage can be used to specify the percentage of sessions for which the service coordinator should run passive probes. The SLA violation count is used to specify the number of SLA violations after which the service coordinator should switch to a different one of the WAN links 142, or consider the SLA violated when determining whether to apply AMR to a given application. The sampling period can be used to specify the sampling period for counting SLA violations. The switchover cooldown period can be used to specify a waiting period after which WAN link switching should occur if the active link returns to online after a failure, or after which AMR should be stopped for a given application. This parameter helps prevent frequent switching of traffic between active and standby WAN links 142, or frequent switching between AMR mode and normal mode.

[0056] Rate limiting parameters can include maximum uplink rate, maximum uplink burst size, maximum downlink rate, maximum downlink burst size, and loss priority. Maximum uplink rate specifies the maximum uplink rate for all applications associated with the SLA profile. Maximum uplink burst size specifies the maximum uplink burst size for all applications associated with the SLA profile. Maximum downlink rate specifies the maximum downlink rate for all applications associated with the SLA profile. Maximum downlink burst size specifies the maximum downlink burst size for all applications associated with the SLA profile. Loss priority allows you to select a loss priority that determines whether packets are dropped or retained during network congestion. The probability of a packet being dropped by the network is higher or lower based on the loss priority value.

[0057] SLA rules can be used to specify application SLA profiles, which include all the information needed to measure SLAs and identify whether any SLA violations have occurred. SLA rules can include a complete probe profile, a time period for the application profile, preferred SLA configurations, and other SLA parameters mentioned above (e.g., SLA sample parameters, rate limiting parameters, metric profiles). SLA rules are associated with an application or application group and become its SLA profile. In other words, the SLA profile for an application can be a specific SLA rule (e.g., "SLA3"), as configured in Service Coordinator 102. In some cases, SLA rules can be associated in this way by associating them with APBR rules that match the identified application or application group. As described above, in some examples, Service Coordinator 102 can push SLA parameters, path selection parameters, routing information, routing and interface data, and related configurations to SD-WAN Edge 108, and SD-WAN Edge 108 monitors links for SLA violations and can apply AMR to applications identified for AMR, or switch the application to a different WAN link in WAN Link 142.

[0058] An SLA violation occurs when the performance of a link falls below the acceptable level specified by the SLA. To attempt to meet the SLA, the SD-WAN system 100 can monitor the network for sources of failure or congestion. If the SD-WAN system 100 determines that an SLA violation has occurred, it can determine an alternative path to select the best WAN link 142 that meets the SLA. Similarly, the SD-WAN edge 108 can monitor for SLA violations. If one of the SD-WAN edges 108 determines that an SLA violation or breach has occurred on all WAN links associated with an application identified for AMR, the SD-WAN edge can apply AMR to that application.

[0059] Coverage paths include WAN links 142 used to send application traffic. SD-WAN system 100 can assign applications to specific WAN links 142 based on the SLA metric of WAN link 142. Destination groups are groups of multiple coverage paths terminating at a destination.

[0060] Typically, the service coordinator 102 configures the SD-WAN edge 108 to recognize application traffic for applications, and the service coordinator 102 specifies the path for specific traffic by associating an SLA profile with a routing instance through which the SD-WAN edge 108 sends application traffic to satisfy the rules of the APBR profile.

[0061] APBR implements application-based routing through the service coordinator 102 that manages the SD-WAN edge 108. The APBR profile specifies the types of traffic to be matched, for example, by listing one or more applications or application groups. The APBR profile can include multiple APBR rules, each specifying one or more applications or application groups. If network traffic matches a specified application, the rule is considered a match. SLA rules can be associated with APBR rules to specify how matching traffic should be processed for QoE. APBR rules can also specify routing instances that the SD-WAN edge 108 uses to route traffic that matches the APBR rules. Routing instances can have interfaces for one or more WAN links 142. The service coordinator 102 configures the SD-WAN edge 108 with the APBR profile (or configuration data derived from it) to enable the SD-WAN edge 108 to use APBR to implement SD-WAN service 101 according to the APBR profile.

[0062] In some examples, the SD-WAN edge 108 (e.g., SD-WAN edge 108A) processes packets received on an interface to identify the application targeting that packet. The SD-WAN edge 108A may apply an APBR profile to attempt to match the application with the APBR rules within it. If no matching APBR rule is found, the SD-WAN edge 108A forwards the packet normally. However, if a matching APBR rule is found, the SD-WAN edge 108A uses the routing instance specified in the APBR rule to route the packet.

[0063] A routing instance has interfaces associated with one or more links used by the routing instance to send and receive data. A routing instance configured in the SD-WAN edge 108 and associated with APBR rules has interfaces for WAN link 142 to send and receive application traffic. These interfaces can be interfaces used for underlying connectivity.

[0064] SD-WAN edge 108 can route traffic using different links based on link preferences determined using SLA rule 122. In some cases, service coordinator 102 determines application performance on WAN links 142 by calculating scores based on latency, round-trip time, jitter, packet loss, and / or other factors. Based on the corresponding scores of one or more WAN links 142, if the performance of the current link is below the acceptable level specified by one of SLA rules 122, service coordinator 102 and SD-WAN edge 108 can divert application traffic to an alternative WAN link of SD-WAN service 101. In some cases, the new WAN link is the WAN link that best meets the SLA requirements, as determined by the score. As already mentioned, service coordinator 102 can use probes to measure and monitor application performance on WAN links 142.

[0065] In some examples, multiple WAN links 142 can satisfy the application's SLA requirements. The SD-WAN system 100 can select from these multiple WAN links 142 that match the user-configured link preferences. These preferences can be based at least in part on the link type and link priority of the WAN links 142. For example, for SD-WAN edge 108A, the SD-WAN system 100 can select one of the WAN links 142A-A to 142A-N reaching SD-WAN edge 108C that matches the preferred link type (e.g., MPLS). If multiple such WAN links 142 with this preference exist, the WAN link with the highest priority is selected. If no priority or link type preference is configured, a random path or a default path is selected. If no WAN link 142 that satisfies the SLA requirements is available, the best available WAN link based on the highest SLA score and link type preference is selected, provided strict affinity is configured. If multiple WAN links 142 that satisfy the SLA requirements are available, the one with the highest priority is selected. One or more of the WAN links 142 can be configured with priorities, which can be expressed as integer values ​​representing priorities in the configuration. The service coordinator 102 prefers higher-priority WAN links 142 over lower-priority WAN links 142.

[0066] In the service coordinator 102, users can configure link types (e.g., IP or MPLS) and set priorities for the application's WAN link 142. For example, a user can define an APBR profile with WAN link 142 and configure WAN link 142 using link type / priority.

[0067] By associating APBR rules for a specified application or application group with the APBR profile, the service coordinator 102 and the SD-WAN edge 108 implement link preferences at the application or application group level to achieve SD-WAN service 101. Users can also specify link type preferences in SLA rules, and in some cases, specify link type affinity. SLA rules are appended to APBR rules to associate preferences with the applications specified in the APBR rules.

[0068] Based on the APBR profile, the SD-WAN edge 108 matches network traffic to the applications and application groups specified in the associated APBR rules, and may, for example, forward traffic to the down-hop address specified in the static route and the route instance of the APBR rule also associated with the APBR profile. The SD-WAN system 100 may assign application traffic to specific paths / links based on the configured link type and WAN link 142 preferences, and in some cases, the specified link type affinity used in the SLA rules (as described above).

[0069] For the preferred link type, link type affinity can be strict or loose (optionally the default setting). For strict affinity, SD-WAN system 100 always selects a WAN link of the preferred link type. For loose affinity, if no WAN link 142 that meets the SLA and belongs to the preferred link type exists, the service coordinator 102 selects a link that does not have the preferred link type but meets the SLA in other respects.

[0070] Service coordinator 102 implements the SD-WAN policy intents of SD-WAN service 101 to facilitate better utilization of WAN link 142 and efficient distribution of application traffic. Subscribers can set advanced SD-WAN policies that include one or more SD-WAN policy intents. Each SD-WAN policy intent may have the following parameters: source, destination, and SLA profile. A source is one or more source endpoints selected from a list of sites, site groups, departments, or combinations thereof. The SD-WAN policy intent applies to the selected source endpoints. A destination is a destination endpoint selected from a list of applications and predefined or custom application groups. The SD-WAN policy intent applies to the selected destination endpoint. Applications can be defined using, for example, network information (e.g., source or destination prefixes), by protocol, or by application name. An SLA profile can be defined as described above, having SLA parameters to be applied to the policy intent for which the SLA profile is set.

[0071] SLA rule 122 specifies one or more applications. As used herein, this or other association between an application and the SLA parameters used for that application implies that the application has an SLA (or SLA rule). Violation of an SLA parameter constitutes a violation of the application's SLA / SLA rule.

[0072] SD-WAN system 100 can determine the available bandwidth of one of WAN links 142 (e.g., WAN link 142A-A) in various ways. For example, SD-WAN system 100 can obtain link data indicating the available bandwidth or total bandwidth of WAN link 142A-A. SD-WAN controller 104 can provide the link data to service controller 102, which may be partially obtained from SD-WAN edge 108. The link data may be configuration data for the underlying connectivity of WAN link 142A-A. SD-WAN system 100 can sum the bandwidth required by applications placed on WAN link 142A-A and calculate the difference between the total bandwidth of WAN link 142A-A and the sum of the bandwidth required by applications placed on WAN link 142A-A as the available bandwidth.

[0073] By identifying applications for AMR based on standards, the SD-WAN edge 108B eliminates the need for network administrators to determine and manually provision AMR applications. This reduces the possibility of human error and enables faster AMR implementation, resulting in better QoE, especially for more critical applications or those that meet specific application profiles prioritized by the administrator for automatic AMR. Application profiles can be defined by the administrator using standards, and in some cases, by combining standards with their weights.

[0074] Although primarily described as being performed by SD-WAN edge 108B, the techniques described herein for automatically identifying applications eligible for AMR can be performed by other SD-WAN edges 108 and in part by SD-WAN controller 104. For example, SD-WAN controller 104 can receive telemetry data indicating the conditions of WAN link 142 and the attributes of the application being processed by SD-WAN service 101, determine that the application is eligible for AMR, and direct one or more of the SD-WAN edges 108 to apply the AMR.

[0075] Figure 2 A block diagram of an example SD-WAN edge device is shown in more detail according to the technology described in this disclosure. SD-WAN edge device 308 (“SD-WAN edge 308”) may represent Figure 1Any of the SD-WAN edges 108. SD-WAN edge 308 is a computing device and may represent a PNF or VNF. SD-WAN edge 308 may include one or more physical or virtual servers configured to perform one or more VNFs to perform SD-WAN edge operations.

[0076] In this example, the SD-WAN edge 308 includes a bus 342 for coupling hardware components of the hardware environment. Bus 342 couples a network interface card (NIC) 330, a storage disk 346, and one or more microprocessors 310 (hereinafter referred to as "microprocessor 310"). In some cases, the front-side bus may couple the microprocessor 310 and a memory device 344. In some examples, bus 342 may couple the memory device 344, the microprocessor 310, and the NIC 330. Bus 342 may represent a Peripheral Component Interface (PCI) Fast (PCIe) bus. In some examples, a Direct Memory Access (DMA) controller may control DMA transfers between components coupled to bus 342. In some instances, components coupled to bus 342 control DMA transfers between components coupled to bus 342.

[0077] The microprocessor 310 may include one or more processors, each processor including an independent execution unit to execute instructions conforming to an instruction set architecture, which are stored in a storage medium. The execution unit may be implemented as a separate integrated circuit (IC) or may be combined within one or more multi-core processors (or "many-core" processors), with each processor implemented using a single IC (i.e., chip multiprocessor).

[0078] Disk 346 represents a computer-readable storage medium, which includes volatile and / or non-volatile, removable and / or non-removable media implemented in any method or technique for storing information such as processor-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, random access memory (RAM), read-only memory (ROM), EEPROM, flash memory, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disc storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by the microprocessor 310.

[0079] Main memory 344 includes one or more computer-readable storage media, which may include random access memory (RAM), such as various forms of dynamic RAM (DRAM), for example, DDR2 / DDR3 SDRAM, or static RAM (SRAM), flash memory, or any other form of fixed or removable storage medium that can be used to carry or store required program code and program data in the form of instructions or data structures and is accessible by a computer. Main memory 344 provides a physical address space consisting of addressable memory locations.

[0080] Network interface card (NIC) 330 includes one or more interfaces 332 configured to exchange data packets using links of the underlying physical network. Interface 332 may include a port interface card with one or more network ports. NIC 330 may also include on-card memory, for example, for storing data packets. Direct memory access transfers between NIC 330 and other devices coupled to bus 342 can read from / write to the NIC memory. Interface 332 may be an interface for underlying connections between SD-WAN edge 308 and one or more other SD-WAN edges configured for the WAN links of SD-WAN module 306.

[0081] Memory 344, NIC 330, storage disk 346, and microprocessor 310 provide an operating environment for a software stack, which includes an operating system kernel 314 executing in kernel space. Kernel 314 can represent, for example, Linux, Berkeley Software Distribution (BSD), other Unix variant kernels, or a Windows server operating system kernel available from Microsoft Corp. In some cases, the operating system can execute a hypervisor and one or more virtual machines managed by the hypervisor. Example hypervisors include kernel-based virtual machines (KVM) for Linux kernels, Xen and ESXi provided by VMware, and Windows Hyper-V provided by Microsoft, as well as other open-source and proprietary hypervisors. The term hypervisor can include a virtual machine manager (VMM). The operating system including kernel 314 provides an execution environment for one or more processes in user space 345. Kernel 314 includes a physical drive 325 using NIC 330.

[0082] The hardware environment and kernel 314 provide a user space 345 operating environment for the SD-WAN edge 308 module, including routing processing 328, configuration interface 374, and SD-WAN module 306. Configuration interface 374 enables the SD-WAN controller 104 or the operator to configure the SD-WAN edge 308. Configuration interface 374 can provide a NETCONF interface, Simple Network Management Protocol (SNMP), command-line interface, RESTful interface, remote procedure call, or other interfaces, allowing remote devices to configure the SD-WAN edge 308 using configuration information stored in configuration database 375. Configuration information may include, for example, SLA rules 322 that partially define the operation of the WAN link switching module 350 for the SD-WAN module 306, routes, and virtual routing and forwarding instances (VRFs) configured with interfaces for WAN links, specifying link types (IP, MPLS, mobile, etc.), priorities, maximum bandwidth, encapsulation information, overlay tunnel types, and / or other link characteristics.

[0083] The routing process 328 executes routing protocols to exchange routing information (e.g., routes) with other network devices and uses the routing information collected in the routing table 316 to select the active route to each destination, which is the route used by the SD-WAN edge 308 to forward incoming packets to the destination. To route traffic from a source host to a destination host via the SD-WAN edge 308, the SD-WAN edge 308 learns the path the packets should take. These active routes are inserted into the forwarding table 318 of the SD-WAN edge 308 and used by the forwarding plane hardware for packet forwarding. For example, the routing process 328 may generate the forwarding table 318 in the form of a radix or other lookup tree to map packet information (e.g., header information with destination information and / or label stacks) to the next hop and ultimately to interface 332 for output. In some examples, the SD-WAN edge 308 may have a physically branched control plane and data plane, where a switching control card manages one or more packet forwarding line cards, each with one or more high-speed packet processors.

[0084] SD-WAN edge 308 executes SD-WAN module 306 to implement SD-WAN services, such as... Figure 1The SD-WAN service 101. SD-WAN module 306 enables SD-WAN edge 308 to forward traffic based on application flows. SD-WAN module 306 uses packet characteristics to identify packets of different application flows. Once an application is identified using initial packets, information about the traffic used to identify the application session can be stored in a flow table for faster processing. WAN link switching module 350 selects a WAN link to allocate to the application based on routing information, policy information, performance data, and the service characteristics of the WAN link used for the SD-WAN service implemented by SD-WAN module 306. SD-WAN module 306 can program forwarding table 318 using the WAN link selected for the application, flow table data, or other data used to map application traffic to the selected WAN link.

[0085] The SD-WAN edge 308 executes the AMR module 320. The AMR module 320 may include SLA rule 322, standard (C) 323, and AMR priority (AP) 324.

[0086] According to the technology of the present invention, the AMR module 320 can be configured to be used in the SD-WAN system 100 ( Figure 1 Based on standard 323, AMR identifies one or more applications, where each standard in standard 323 is associated with a corresponding attribute of the application, and each standard has a weight associated with it. AMR module 320 can be configured to determine a violation of one of the SLAs on each WAN link associated with a first application among the one or more applications, and in response to determining the violation, apply AMR to the first application. Therefore, network devices such as the SD-WAN edge 308 using AMR module 320 can automatically identify applications for which AMR should be applied and apply AMR to those applications.

[0087] AMR module 320 can maintain a dynamic list, table, or other data structure (e.g., in the priorities of AMR priority 324) for prioritizing applications based on application attributes or characteristics and / or on-site traffic patterns in the network. AMR module 320 can continuously analyze traffic in the network and, for example, construct a list of applications that expect or require AMR based on standard 323. Once AMR module 320 detects an SLA violation for any application that is relatively important for QoE (e.g., an application on the AMR application list in AMR priority 324), AMR module 320 can apply AMR to that application, thereby ensuring QoE or enhancing QoE for those applications that are relatively important for QoE. In some examples, AMR module 320 can also continuously monitor the SLAs of these identified applications for further remediation, such as disabling AMR when the SLA is met again on at least one link associated with the application.

[0088] AMR module 320 can dynamically identify relatively important applications and apply AMR to an application when an associated SLA is violated on all existing WAN links associated with that application. AMR module 320 can also reverse this action (e.g., restore the application's previous operation) when one or more WAN links recover. For example, AMR module 320 can identify relatively important applications and prioritize them based on standard 323, storing these priorities in AMR priorities 324. AMR module 320 can monitor SLA 322 and apply AMR to applications that violate the SLA on all associated WAN links. For example, AMR module 320 can monitor all WAN links and determine if sufficient resources (CPU and bandwidth) are available for the identified applications. In some examples, AMR module 320 can further apply AMR to applications based on their priority.

[0089] For example, AMR module 320 can identify relatively important applications based on standard 323, and in some examples, prioritize these applications. In some examples, standard 323 may include pre-defined or default standards programmed by the manufacturer of the SD-WAN edge 308. For example, pre-defined or default standards may specify attributes or characteristics of applications, which may be a set of applications with common attributes or characteristics. In some examples, multiple default standards may be stored in standard 323. For example, different sets of standards may exist for different deployment environments, such as one for deployment in a banking network, one for deployment in an office network, and so on. In some examples, some or all standards in one set of standards may differ from standards in another set of standards. For standards in more than one set of standards, the standards may have associated weights that are different or the same across different sets of standards. For example, in one set of standards, the weight for a bank might be 10, while in another set, the weight for a bank might be 2.

[0090] Table 1 below is an example of the default criteria for criterion 323, which the AMR module 320 can use to identify potential applications for AMR. In this example, each criterion is assigned a weight within a scale of 1 to 10, where 10 is the most important for applications using AMR (typically applications that expect a high QoE); however, any scale can be used.

[0091] Table 1 - Example Default Standards for AMR Recognition Applications

[0092] In the example in Table 1, AMR module 320 can assign a weight of 10 to RTP-based applications, a weight of 9 to office-related applications (e.g., word processors, spreadsheet applications, presentation applications, etc.), and a weight of 8 to applications with small packet sizes (< 256 bytes). While some criteria and weights are described in Table 1, these are illustrated by way of example and any criteria or weights may be used according to the technology of the present invention.

[0093] In some examples, network administrators may choose to modify the default criteria and / or associated weights of criterion 323 used for AMR identification applications, or, in the absence of a default criterion, create their own criteria and / or weights based on their own usage. In some examples, if a network administrator modifies the default criteria or weights, this can override the default criteria or weights. For example, AMR module 320 can override the weights in the default criteria or criterion 323 with the modified criteria or weights. In other examples, the modified criteria or weights can be stored as a separate set of criteria that can be activated later, and the original default criteria can be deactivated but remain stored in criterion 323. Table 2 shows examples of custom criteria created or modified by network administrators.

[0094] Table 2 - Examples of Custom Standards for AMR Identification Applications

[0095] In the examples in Table 2, AMR module 320 can assign weight 4 to RTP-based applications, weight 5 to office-related applications, weight 8 to applications with small packet sizes (e.g., < 256 bytes), weight 9 to banking and / or trading (e.g., trading stocks, bonds, or other instruments) applications, weight 8 to streaming applications, and weight 10 to applications with Differential Code Points of Service (DSCP) marking. While some criteria and weights are described in Table 2, these are described as examples and any criteria or weights can be used.

[0096] For example, AMR module 320 can monitor SD-WAN system 100 ( Figure 1 The network has different applications with traffic and compares each application against each criterion of standard 323. In some examples, when an SLA violation occurs, there may not be enough network resources to apply AMR to each relatively important application. To address this example, AMR module 320 can determine the priority of each application identified for AMR based on the weights given for each specific criterion met by the application. AMR module 320 can store the corresponding priorities in AMR priority 324.

[0097] Figure 3 This is a table diagram illustrating an example of prioritizing applications based on standards. Figure 3 In the example, the standards are depicted in the top row, with each standard having an associated weight shown in the second row. For example, RTP is a standard with a weight of 4. Various applications, such as the first video conferencing application, are listed in the left column. An "X" in the table indicates that the application conforms to a specified standard. For example, the first video conferencing application meets the following standards: RTP, office-related, packet size less than 256 bytes, and streaming media. The absence of an "X" in the table indicates that the application does not conform to a standard. For example, the first video conferencing application does not meet the banking / transaction or DSCP tagging standards. The sum of the weights for each standard met by each application is also shown. For example, the total weight for the first video conferencing application is 25. AMR Module 320 ( Figure 2 The cumulative weight of each criterion, corresponding to the attributes of each application for AMR identification, can be determined. The AMR module 320 can use the cumulative weight of each application to determine the priority of each application. Figure 3 In the example, the first video conferencing application has the highest weight (25) and is therefore assigned the highest priority (shown as 1).

[0098] The AMR module 320 can determine the sum of the weights of the met criteria to determine the priority of each of the identified applications. For example, as Figure 3As shown, applications with application traffic on the SD-WAN system 100 can include a first video conferencing application, a second video conferencing application, an email application, a stock trading application, a banking application, an education application, a video application, and an FTP application. The first video conferencing application meets the standards for RTP applications, office applications, packets with less than 256 bytes, and streaming media applications. The AMR module 320 can sum the weights associated with each of the standards met by the first video conferencing application to obtain a total weight of 25. In this example, weight 25 is the highest weight, which causes the AMR module 320 to assign the highest priority (priority 1) to the first video conferencing application. The AMR module 320 can similarly sum the weights associated with each identified application. Thus, in this example, the second video conferencing application, the stock trading application, and the banking application each have a total weight of 17, making these applications priority 2. The email and education applications in this example have a total weight of 5, making these applications priority 3. In this example, the video application and the FTP application do not meet any standards (shown as mismatch). In this way, AMR module 320 can identify video and FTP applications as not requiring AMR or not worth consuming the resources associated with applying AMR. For example, AMR module 320 can assign priority 0 to the application to indicate that AMR should not be applied to these applications even if the associated SLA for any of these applications is violated on all associated WAN links. Thus, even if all WAN links associated with the video or FTP application fail, AMR module 320 can still avoid applying AMR to that application. In some examples, AMR module 320 can store a table, list, or other data structure or entry indicating the priority of each application in AMR priority 324.

[0099] AMR module 320 can proactively monitor applications with a priority assigned in AMR priority 324 for any SLA violation or breach. In some examples, AMR module 320 may not monitor any SLA violation or breach for any application assigned priority 0. In other examples, AMR module 320 may also monitor any SLA violation or breach for applications assigned priority 0.

[0100] SLA metrics can include jitter, latency, packet loss, bandwidth, or other SLA metrics, and can be customized for each application. When an application identified in AMR priority 324 fails to meet its SLA on available WAN links, AMR module 320 can detect this and apply AMR to that application. In some examples, AMR module 320 can determine whether available resources, such as central processing unit (CPU) and link bandwidth, are sufficient, and apply AMR to the application only if sufficient resources are available. For example, once AMR module 320 determines that SLA violations have occurred on all associated links for a particular application, AMR module 320 can perform a resource check before applying AMR to that application. As part of the resource check, system parameters such as CPU and link bandwidth can be defined and evaluated to avoid congestion and overload conditions in the SD-WAN system 100. In some examples, AMR module 320 can apply modifications to available resources when determining availability, such as 90%, 80%, or 70% of link bandwidth, or use predetermined thresholds to reserve some resources for other purposes, such as application traffic from new applications.

[0101] In some examples, if an SLA violation occurs for more than one application, the AMR module 320 can further determine whether to apply AMR based on the priority assigned to the applications. For example, if both the first and second video conferencing applications experience SLA violations on all WAN links, and there are only sufficient resources available to apply AMR to one of the two applications, the AMR module 320 can apply AMR to the first video conferencing application but not to the second video conferencing application because the first video conferencing application has a higher priority than the second video conferencing application.

[0102] Figure 4 This is a tabular diagram showing an example of applying AMR. Figure 4 In the example, the first video conferencing application (with priority 1—the highest priority) experiences SLA violations on both WAN link 1 and WAN link 2, as indicated by the "X" in the WAN link 1 and WAN link 2 columns. Thus, the AMR module 320 can apply AMR to the first video conferencing application. Other applications identified for AMR may only experience SLA violations on either WAN link 1 or WAN link 2, as indicated by the "X" in the corresponding column. Therefore, the AMR module 320 may not apply AMR to those applications, even if those applications have been identified as eligible for AMR. In this example, the video application and the FTP application are not eligible for AMR because they do not meet any criteria (indicated as mismatch).

[0103] AMR module 320 can monitor and identify SLA violations, perform resource checks, and apply AMR to the first video conferencing application based on the identified SLA violations and the availability of sufficient resources to support the AMR for the first video conferencing application. AMR module 320 can continue to monitor the SLA status and disable AMR for the application when its SLA is met again. In this way, AMR module 320 can release network resources supporting the AMR for the first video conferencing application, as these resources are no longer needed to meet the SLA for the first video conferencing application.

[0104] Figure 5 This is a flowchart illustrating examples of automatic application identification and application of AMR technology for AMR. Network equipment manufacturers, network administrators, or AMR module 320 can define criteria (400) for identifying important applications. For example, network equipment manufacturers, network administrators, or AMR module 320 can define criteria and assign weights to each criterion. Each criterion in the criteria can be associated with a corresponding attribute of an application, such as RTP applications, office-related applications, applications with small packet sizes, banking / transaction applications, streaming media applications, applications with DSCP tags, etc. In some examples, the criteria are predetermined. In some examples, the criteria and / or weights are dynamic, and AMR module 320 can autonomously (i.e., without input from network operators, customers, or other agents) modify the criteria and / or weights based on the type of application traffic being experienced at a given time, network resources, network conditions, deployment topology, deployment environment, or the service type of the network operator or SD-WAN service customer. For example, AMR module 320 can add new criteria to the criteria or remove criteria from the criteria. For example, in a banking network deployment, AMR module 320 can automatically remove one or more standards that are less critical to banking operations, or add banking standards if they are not already included in the default or pre-defined standards. In some examples, instead of modifying standards, or in addition to modifying standards, AMR module 320 can modify one or more weights associated with a standard. For example, AMR module 320 can increase or decrease the weights associated with a standard, or add weights for new standards. For example, in a banking network deployment, AMR module 320 can increase the weights associated with banking standards and / or the weights associated with small packet sizes. In some examples, at least some of the above operations can be performed by the SD-WAN controller, which then configures SD-WAN module 306 with the weights of the standards and / or standard 323.

[0105] AMR module 320 can determine whether any criteria are met for an application (402). For example, AMR module 320 can compare the criteria with the application's attributes to determine whether any of the criteria are met for the application. If no criterion is met for the application (from the "No" path of box 402), AMR module 320 can ignore the application for AMR (406). For example, AMR module 320 can determine that the application does not require AMR, or that the cost of applying AMR to the application outweighs the benefits of applying AMR to that particular application. In some examples, AMR module 320 can assign the application a priority of 0.

[0106] If any criterion is met for the application (from the "Yes" path in box 402), the AMR module 320 can prioritize the application (404). For example, the AMR module 320 can sum the weights associated with each criterion met by the application to obtain a total weight. The AMR module 320 can compare the total weight of the application with the total weights of other applications to determine the application's priority. For example, if the total weight associated with the current application is the highest among all applications, the AMR module 320 can assign the current application the highest priority, such as priority 1. If the total weight associated with the current application is the lowest among all applications, the AMR module 320 can assign the current application the lowest priority among those applications identified as eligible for AMR.

[0107] AMR module 320 can monitor applications and / or associated WAN links to detect any violations of application-related SLAs (408). For example, AMR module 320 can monitor the performance of WAN links associated with applications (e.g., bandwidth, jitter, latency, etc.).

[0108] AMR module 320 can determine whether an SLA has been violated and whether resources are sufficient to apply the AMR to the application (410). For example, AMR module 320 can compare performance with SLA rule 322 to determine if an SLA has been violated. AMR module 320 can determine resource availability and compare resource availability with the expected resource consumption for applying the AMR to the application to determine whether resources are sufficient. In some examples, when determining whether resources are sufficient to apply the AMR to the application, AMR module 320 can apply a modification to the amount of available resources, such as a percentage, to reserve a predetermined amount of network resources for other purposes, such as handling traffic from a new application.

[0109] If AMR module 320 determines that an SLA has been violated and there are sufficient resources to apply AMR to the application (from the "Yes" path in box 410), then AMR module 320 can apply AMR to the application (412). For example, AMR module 320 can copy application packets and can do so on more than two WAN links (e.g., Figure 1 On each WAN link (links 142N-A and 142N-N), a copy of each application packet is forwarded to the receiving network device. If the AMR module 320 determines that the SLA is not violated, resources are insufficient, or the SLA is not violated and resources are insufficient, the AMR module 320 may not apply the AMR and return to monitoring the application and / or WAN link (408). In some examples, if the SLA is violated, resources are insufficient, and the application has a higher priority than another application to which AMR has been applied, the AMR module 320 may stop applying AMR to the lower priority application, and if resources are sufficient to apply AMR to the higher priority application, the AMR module 320 may apply AMR to the higher priority application.

[0110] Figure 6 This is a flowchart illustrating an automatic AMR identification application and another example of applying AMR technology. For example, AMR module 320 ( Figure 2 AMR can identify one or more applications in SD-WAN based on standards, where each standard in the standard is associated with a corresponding attribute of the application (500). For example, AMR module 320 can associate one or more applications with standard 323 ( Figure 2 The AMR module 320 compares the applications to identify one or more applications for which an AMR may be applied. For example, the AMR module 320 may identify an application as eligible for an AMR based on the application having at least one attribute that matches at least one criterion. The AMR module 320 may also identify an application as ineligible for an AMR based on the application not having at least one attribute that matches at least one criterion.

[0111] AMR module 320 can determine an SLA violation (502) of the first application on each WAN link associated with the first application among the identified one or more applications. For example, AMR module 320 can determine that the performance (e.g., bandwidth, latency, jitter, or other performance metrics) of each WAN link associated with the first application is below the requirements of the SLA associated with the first application. In some examples, each WAN link may violate the SLA in the same way, for example, each WAN link may violate the SLA due to low bandwidth. In some examples, each WAN link may violate the SLA in different ways, for example, one WAN link may violate the SLA due to low bandwidth while another WAN link may violate the SLA due to high latency. In some examples, some WAN links may violate the SLA in the same way, while one or more WAN links may violate the SLA in different ways.

[0112] AMR module 320 may apply AMR (504) to the first application in response to a determined violation. For example, AMR module 320 may copy the application packets of the first application and forward a copy of each application packet on each of at least two WAN links.

[0113] In some examples, AMR was not pre-configured for the first application. For instance, the network administrator did not manually configure AMR for the first application.

[0114] In some examples, the standards are predefined and include at least one of RTP, office-related, size, bank, transaction, or DSCP tags. In some examples, the AMR module 320 can autonomously modify the standards. In some examples, each criterion in the standards has an associated weight, and the AMR module 320 can autonomously modify one or more weights associated with the standards.

[0115] In some examples, AMR module 320 can determine that the available resources at SD-WAN edge 308 are sufficient to support AMR for a first application, wherein applying AMR to the first application is based on the premise that the available resources are sufficient to support AMR for the first application.

[0116] In some examples, one or more identified applications include a second application. In some examples, each criterion in the criteria has an associated weight. In some examples, the AMR module 320 can determine a first priority for the first application and a second priority for the second application based on the criteria, with the first priority being higher than the second priority. In some examples, the AMR module 320 can identify a violation of one or more SLAs on each WAN link associated with the second application. In this example, the AMR module 320 can determine that available resources are insufficient to support AMR for both the first and second applications. In this example, the AMR module 320 can apply AMR to the first application based on the first priority being higher than the second priority, and avoid applying AMR to the second application.

[0117] In some examples, the first priority is based on the cumulative weight of each criterion having attributes corresponding to the first application, and the second priority is based on the cumulative weight of each criterion having attributes corresponding to the second application.

[0118] In some examples, the AMR module 320 is configured to send copies of data packets associated with the first application via each of two or more WAN links.

[0119] In some instances, the techniques of this invention are performed by the SD-WAN edge 308. In some examples, the identification of one or more applications for AMR and the determination of SLA violations are performed by the SD-WAN controller 104 ( Figure 1 The SD-WAN controller 104 executes the configuration data so that the SD-WAN edge 308 applies AMR to the first application.

[0120] The technology of this invention has several potential advantages over other technologies. For example, the technology of this invention provides dynamic identification of relatively important applications. The technology of this invention facilitates the on-demand activation and deactivation of AMR for a given application. The technology disclosed herein can provide better application QoE for relatively important applications. The technology of this invention can provide better management and efficient utilization of WAN links and system resources. The technology of this invention can reduce the burden on network administrators in determining whether AMR should be provisioned for a given application and in manually provisioning AMR for such applications.

[0121] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof. Various features described as modules, units, or components can be implemented together in an integrated logic device or individually as discrete but interoperable logic devices or other hardware devices. In some cases, various features of an electronic circuit can be implemented as one or more integrated circuit devices, such as integrated circuit chips or chipsets.

[0122] If implemented in hardware, the invention is applicable to devices such as processors or integrated circuit devices such as integrated circuit chips or chipsets. Alternatively, if implemented in software or firmware, the technology can be implemented at least in part by a computer-readable data storage medium comprising instructions that, when executed, cause the processor to perform one or more of the methods described above. For example, the computer-readable data storage medium may store the instructions for execution by the processor.

[0123] Computer-readable media can form part of a computer program product, which may include packaging material. Computer-readable media may include computer data storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic or optical data storage media, etc. In some examples, the article of manufacture may include one or more computer-readable storage media.

[0124] In some examples, computer-readable storage media may include non-transitory media. The term "non-transitory" can indicate that the storage medium is not contained in a carrier or propagating signal. In some examples, non-transitory storage media may store data that may change over time (e.g., in RAM or cache).

[0125] The code or instructions can be software and / or firmware executed by processing circuitry, which includes one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Therefore, the term "processor" as used herein can refer to any of the foregoing structures or any other structure suitable for implementing the techniques described herein. Additionally, in some aspects, the functionality described in this invention can be provided within software or hardware modules.

Claims

1. A network device, comprising: The storage is configured to store information associated with one or more Service Level Agreements (SLAs) for applications in Software-Defined Wide Area Networks (SD-WAN). as well as Application-based multipath routing (AMR) module, including processing circuitry, is configured to: Based on standards, application-based multipath routing (AMR) is used to identify one or more applications, wherein each standard in the standards is associated with a corresponding attribute of the application. Determine a violation of one of the Service Level Agreements (SLAs) on each WAN link associated with the first application in one or more identified applications; and In response to determining the violation, an application-based multipath routing (AMR) is applied to the first application.

2. The network device according to claim 1, wherein, There is no application-based multipath routing (AMR) pre-configured for the first application.

3. The network device according to claim 1, wherein, The standard is predetermined, and the standard includes at least one of Real-time Transport Protocol (RTP), Office-related, Size, Banking, Transaction, or Differential Service Code Point (DSCP) marking.

4. The network device according to claim 1, wherein, The application-based multipath routing (AMR) module is also configured to autonomously modify the standard.

5. The network device according to claim 1, wherein, Each of the standards has an associated weight, and the application-based multipath routing (AMR) module is also configured to autonomously modify one or more weights associated with the standard.

6. The network device according to any one of claims 1 to 5, wherein, The application-based multipath routing (AMR) module is further configured to: determine that the available resources of the network device are sufficient to support the application-based multipath routing (AMR) for the first application. The application of application-based multipath routing (AMR) to the first application is based on the premise that the available resources are sufficient to support application-based multipath routing (AMR) for the first application.

7. The network device according to any one of claims 1 to 5, wherein, The identified one or more applications include a second application, wherein each of the criteria has an associated weight, and wherein the application-based multipath routing (AMR) module is further configured to: Based on the aforementioned standard, a first priority of the first application and a second priority of the second application are determined, wherein the first priority is higher than the second priority; Determine a violation of one or more Service Level Agreements (SLAs) on each WAN link associated with the second application; It was determined that the available resources were insufficient to support application-based multipath routing (AMR) for both the first and second applications; and Based on the fact that the first priority is higher than the second priority, application-based multipath routing (AMR) is applied to the first application, and application-based multipath routing (AMR) is avoided from being applied to the second application.

8. The network device according to claim 7, wherein, The first priority is based on the cumulative weight of each standard having attributes corresponding to the first application, and the second priority is based on the cumulative weight of each standard having attributes corresponding to the second application.

9. The network device according to any one of claims 1 to 5, wherein, To apply application-based multipath routing (AMR) to the first application, the application-based multipath routing (AMR) module is configured as follows: A copy of the data packet associated with the first application is transmitted via each of two or more WAN links.

10. A method of operating a network device, comprising: The processing circuitry identifies one or more applications for application-based multipath routing (AMR) in Software-Defined Wide Area Network (SD-WAN) based on standards, wherein each standard in the standards is associated with a corresponding attribute of the application. The processing circuitry determines, via each WAN link, a violation of the Service Level Agreement (SLA) for the first application associated with one or more identified applications; and The processing circuitry, in response to determining the violation, applies application-based multipath routing (AMR) to the first application.

Images