Dynamic determinism adjustment across bandwidth of multiple hubs with adaptive per-tunnel quality of service (QoS)

Abstract

According to some embodiments, a system includes one or more processors and one or more computer-readable non-transitory storage media, the one or more computer-readable non-transitory storage media including instructions that, when executed by the one or more processors, cause one or more components of the system to perform operations. These operations include sending data from a hub to a spoke and receiving feedback at the hub from the spoke. The feedback is based on at least one of the occurrence of a congestion state detected by the spoke or bandwidth utilization. These operations also include adjusting the shaper rate of an adaptive Quality of Service (QoS) shaper based at least in part on the feedback received from the spoke.

Description

[0001] Related applications

[0002] This application claims priority to International Application No. PCT / CN2020 / 109400, filed on August 14, 2020. Technical Field

[0003] This disclosure generally relates to adjusting bandwidth in a computer network, and more specifically to the dynamic deterministic adjustment of bandwidth across multiple hubs having adaptive per-tunnel quality of service (QoS). Background Technology

[0004] A computer network can refer to a set of computing components (e.g., gateways, routers, terminal devices, etc.) that communicate through network connections. Computer networks can support many applications and services. As an example, computer networks can support cloud-based services such as Software as a Service (SaaS) and Infrastructure as a Service (IaaS). Attached Figure Description

[0005] Figure 1A , Figure 1B and Figure 1C Each example shows a network where bandwidth is shared among multiple hub routers.

[0006] Figure 2 An example of visualizing adaptive QoS history in the management plane is shown.

[0007] Figure 3A and Figure 3B Each example demonstrates the use of adaptive QoS history to recommend or dynamically implement configuration changes.

[0008] Figure 4 An example architecture for SD-WAN is shown.

[0009] Figure 5 An example of a computer system is shown.

[0010] Figure 6 An example of a method for adjusting the shaper rate based at least in part on feedback from one or more remote routers is shown. Detailed Implementation

[0011] Overview

[0012] Various aspects of the invention are set forth in the independent claims, and preferred features are set forth in the dependent claims. A feature of one aspect may be applied alone or in combination with other aspects in any aspect.

[0013] According to an embodiment, a system includes one or more processors and one or more computer-readable non-transitory storage media, the computer-readable non-transitory storage media including instructions that, when executed by the one or more processors, cause one or more components of the system to perform operations. The operations include sending data from a hub to a spoke and receiving feedback at the hub from the spoke. The feedback is based on at least one of the occurrence of a congestion state detected by the spoke or bandwidth utilization. The operations further include adjusting the shaper rate of an adaptive Quality of Service (QoS) shaper based at least in part on the feedback received from the spoke.

[0014] According to another embodiment, a method includes sending data from a hub to a spoke and receiving feedback at the hub from the spoke. The feedback is based on at least one of the occurrence of a congestion state detected by the spoke or bandwidth utilization. The method further includes adjusting the shaper rate of an adaptive QoS shaper based at least in part on the feedback received from the spoke.

[0015] According to another embodiment, one or more computer-readable non-transitory storage media may contain instructions that, when executed by a processor, cause to perform an operation including sending data from a hub to a spoke and receiving feedback at the hub from the spoke. The feedback is based on at least one of the occurrence of a congestion state detected by the spoke or bandwidth utilization. The operation further includes adjusting the shaper rate of an adaptive QoS shaper based at least in part on the feedback received from the spoke.

[0016] The technical advantages of certain embodiments of this disclosure may include one or more of the following. The systems and methods described herein can allow efficient bandwidth sharing among hubs in a full-mesh network or other multi-hub network. For example, some embodiments can allow fair bandwidth sharing among hubs based on feedback regarding the congestion state and bandwidth utilization of each remote hub. In some embodiments, the hub can selectively adjust the current shaper rate based on desired throughput along with WAN loss measurements. Some embodiments can reduce or eliminate resonance in adaptive behavior and / or shaper rate oscillations.

[0017] Other technical advantages will be apparent to those skilled in the art from the following figures, description, and claims. Furthermore, while specific advantages have been listed above, various embodiments may include all, some, or none of the listed advantages.

[0018] Example Implementation

[0019] Shapers can be applied as part of a QoS policy to services in a software-defined wide area network (SD-WAN). Previously, static shapers were used as part of egress QoS policies. These static shapers were configured based on the service provider's bandwidth availability and did not change over time. Therefore, static shapers would not reflect the actual available Internet bandwidth. To address this issue, adaptive QoS features can be introduced into SD-WAN. Examples of adaptive QoS techniques are described in U.S. Patent Application No. 16 / 891,624, filed June 3, 2020, entitled "Systems and Methods for Adapting a WAN egress shaper rate," and U.S. Patent Application No. 16 / 666,904, filed October 29, 2019, entitled "Congestion avoidance with Adaptive QoS policy enforcement from SD-WAN Controller in SD-WAN Networks."

[0020] Some implementations of adaptive QoS features enable WAN interface shapers and per-tunnel shapers at the enterprise edge to adapt to available WAN bandwidth in the upstream and downstream, respectively. This bandwidth adaptation capability allows for control of differentiated drop at the enterprise edge and reduces or prevents packet drop in the network core.

[0021] Adaptive QoS can be used in data centers and branch sites with hub-spoke topologies. For example, adaptive QoS can be configured on the hub-spoke WAN interface, and upstream and downstream adaptive QoS can be enabled independently. Upstream adaptation operates on the hub-to-hub per-tunnel QoS shaper. Downstream adaptation operates on the hub-to-hub per-tunnel QoS shaper, which requires hub-to-hub per-tunnel QoS to be enabled.

[0022] Figure 1A An example deployment of adaptive QoS in SD-WAN is shown. Specifically, Figure 1A This illustrates how adaptive QoS works for a use case with two hubs (hub BR1 and hub BR2), where both hubs use sequence-based loss measurement and a distributed loss-based QoS shaper that adapts to the spokes (remote site 1) between the hubs and the spoke tunnel. Figure 1AIn this setup, hubs BR1 and BR2 operate independently based on WAN loss measurements and feedback. Each hub independently adjusts its own shaper rate based on the percentage of loss per adaptation cycle. For multi-hub BR deployments, two requirements are typically considered:

[0023] • QoS fairness among multiple senders used for bandwidth sharing during remote site congestion; and

[0024] • Dynamic bandwidth adjustment based on the weights and priorities of multiple hubs.

[0025] Some adaptive QoS implementations work well with two hubs by properly configuring minimum, maximum, and default bandwidth parameters to ensure QoS fairness with a 50% minimum bandwidth guarantee. However, ensuring QoS fairness and dynamic bandwidth adjustment among multiple hubs based on weights becomes more challenging when multiple hubs or multiple data center deployments are involved (such as full mesh deployments or other deployments with more than two hubs).

[0026] Figure 1B This illustrates potential issues with adaptive QoS when more than two hub routers are present. Figure 1B In the example, the bandwidth parameters used for adaptive QoS are configured on the downstream WAN interface to the hub as a default of 100 Mbps, a minimum of 25 Mbps, and a maximum of 100 Mbps. These adaptive QoS parameters are advertised on the SD-WAN architecture (e.g., hub routers Hub BR1, Hub BR2, Hub BR3, and Hub BR4). The hub routers can then use the adaptive QoS parameters for downstream adaptive QoS functions. Using such a configuration profile, each hub router can achieve a minimum guarantee of 25 Mbps during congestion. This can present certain challenges. For example, the hub routers may not be able to share bandwidth fairly during congestion, and different hub routers may have different bandwidth consumption. For these hub routers, this can cause adaptive behavior and shaper rate oscillations to resonate with up / down (UP / DOWN) respectively. Therefore, relying on such QoS adaptive lower bound parameters to ensure QoS fairness does not work effectively for more than two hub routers (e.g., in multi-hub / data center use cases).

[0027] Certain embodiments of this disclosure propose solutions to this problem. For example, some embodiments propose extending adaptive QoS path monitoring (Path-Mon) measurements to provide feedback on the congestion status and bandwidth utilization of each remote hub (BR) at remote branch sites, allowing the hub BR to selectively adjust the current shaper rate based on expected throughput along with WAN loss measurements. Additionally, some embodiments introduce per-tunnel-level bandwidth / throughput measurements with congestion monitoring on the receiver side. The spoke side can selectively provide feedback on bandwidth adjustment to the sender side using Path Monitoring Type-Length-Value (TLV) parameters to complement the WAN shaper adaptation algorithm and account for remote WAN congestion. Therefore, multiple hub senders (e.g., BR1-BR4) can adjust the shaper rate in a coordinated manner for each monitoring interval based on bandwidth utilization and congestion status across the entire SD-WAN architecture from multiple senders.

[0028] In some embodiments, adaptive QoS includes a path monitor function module that enables proactive monitoring and synchronization of sender / receiver sequences, as well as transmit throughput and bandwidth per SD-WAN tunnel in each monitoring interval. Using these path monitoring metrics, the sender side implements a loss-based QoS shaper adaptation algorithm and adaptively and independently adjusts the shaper rate for each monitoring interval. A new field can be introduced into the path monitoring TLV metric. This new field indicates the expected throughput the receiver expects to receive from the sender side. The “Expected Throughput” field (default “0” for older versions) can be used as part of the QoS shaper adaptation algorithm to facilitate additional shaper rate bandwidth adjustments as well as existing loss-based methods.

[0029] • If no local loss and WAN loss are detected (within a specific threshold), there is no need to adjust the QoS shaper rate.

[0030] Otherwise, if the “expected throughput” is greater than (or equal to) the current QoS shaper rate, then there is no need to adjust the QoS shaper rate.

[0031] Otherwise, if WAN loss is detected (exceeding a certain threshold), the QoS shaper rate is adjusted downwards.

[0032] Otherwise, if the "expected throughput" is less than the current QoS shaper rate, the QoS shaper rate needs to be adjusted downwards.

[0033] Otherwise, increase the QoS shaper rate.

[0034] The following provides a detailed explanation of how it works at a branch site, which can selectively provide remote feedback on bandwidth adjustments and existing WAN loss-based measurement methods by introducing bandwidth / throughput monitoring with “expected throughput” feedback on the receive side to the hub site via existing bidirectional forwarding detection paths that monitor the TLV. Hubs with current shaper rates below “expected throughput” should not reduce their shaper rates, even if WAN loss has been detected. Hubs with current shaper rates above “expected throughput” should directly reduce their shaper rates to expected throughput based on WAN loss conditions.

[0035] Figure 1C An example is provided. On the spoke side, the “Desired Throughput” for each Remote Hub Transport Location (TLOC) session can be adjusted to handle the weight and priority of each TLOC, and the “Desired Throughput” feedback for each hub can be adjusted based on the weight of each TLOC session (which can be published via the Operations and Maintenance Platform (OMP) on the SDWAN architecture). Example steps for calculating “Desired Throughput” are illustrated below:

[0036] 1. Enable weighted adaptive QoS bandwidth policy on the spoke TLOC;

[0037] 2. Each hub BR enables tunnel encapsulation with existing weight attributes for each TLOC and publishes it via OMP;

[0038] 3. Monitor the downstream throughput (per tunnel) from the receiver-side hub to the spoke session;

[0039] 4. During the end of path monitoring, calculate the "expected throughput" using the following operations:

[0040] a. Iterate through all hub sessions whose throughput meets the expected throughput: if (session throughput <= total bandwidth * weight / total weight), then "expected throughput" = session throughput;

[0041] b. Iterate through all hub sessions whose throughput does not meet the expected throughput: If (session throughput > total bandwidth * weight / total weight (session_throughput > total_bandwidth * weight / total_weight)), then "Expected Throughput" = remaining throughput * weight / remaining total weight (throughput_left * weight / total_weight_left); and

[0042] Summarizing some of the embodiments described above, this disclosure proposes a novel method that enables hub senders in an SD-WAN comprising multiple hub senders to more effectively adjust shaper rates based on feedback from remote branch sites regarding bandwidth utilization and congestion status. This distributed approach leads to deterministic adaptive QoS shaper behavior and QoS fairness across multiple hub BRs, along with existing WAN loss-based measurement mechanisms. The method can also provide bandwidth sharing across multiple hubs based on weights and priorities. In some embodiments, the path monitoring module can be extended to support monitoring ingress bandwidth / throughput on the receiver side based on each TLOC, and the desired throughput can be derived by taking into account the weights of the hub BRs. Feedback of the “desired throughput” from the BFD path monitoring TLV to the hub BRs is used to extend the distributed loss-based QoS shaper adaptation algorithm, allowing multiple hub senders to coordinately adjust shaper rates for each adaptation interval.

[0043] Historical visualization and intelligent configuration recommendations for adaptive QoS

[0044] Some embodiments may provide historical visualization and / or intelligent configuration recommendations for adaptive QoS. Some embodiments provide visualization of adaptive QoS history (e.g., QoS shaper rate adaptation history) on the management plane. As an example, a manager (such as Cisco's vManage) may be used to provide visualization of adaptive QoS history on the management plane. Alternatively, visualization of adaptive QoS history may also be visible on edge devices (such as Cisco's cEdge), for example, via a command-line interface (CLI).

[0045] In some embodiments, adaptive QoS can be configured via a management plane. As an example, initial values ​​for minimum (MIN) bandwidth, maximum (MAX) bandwidth, and default bandwidth can be used to establish adaptive QoS on a specific interface. The adaptive QoS values ​​can be configurable, allowing different customers or different sites to use different values. As an example, some embodiments update values ​​based on network performance and / or provide a user interface that allows users to select a range of values. Some embodiments can apply rules based on the relationship between one value and another, such as Min = 50% of Max, and Default = 80-90% of Max bandwidth. Some embodiments can recommend values ​​to optimize the performance of adaptive QoS features.

[0046] Some embodiments of this disclosure propose an adaptive QoS history management plane visualization, through Speedtest's automatic bandwidth configuration, and / or an intelligent approach that recommends configuration revisions based on historical alerts.

[0047] Currently, cEdge stores the latest 8 adapt histories for each session (downstream of the hub) and WAN interface (upstream of the spokes). The adapt history can be viewed via the CLI, for example:

[0048] HUB# shows the platform software sdwan qos adapted to the historical tunnel 1 10.1.14.14

[0049] SDWAN OMP Session

[0050] Color: lte

[0051] Destination (DEST) TLOC: 172.16.255.14

[0052] Destination IP: 10.1.14.14

[0053] Destination port: 12346

[0054] Encapsulation: IPSec

[0055] Adaptive QoS History:

[0056]

[0057] SPOKE# indicates the platform software sdwan qos adaptation history GigabitEthernet 0 / 0 / 4

[0058] SDWAN upstream adaptive QoS

[0059] Interface: GigabitEthernet0 / 0 / 4

[0060] Adaptive QoS History:

[0061]

[0062]

[0063] In some embodiments of this disclosure, the edge router may periodically output adaptation history to the management plane (vManage), and vManage may make the adaptation history visible. Figure 2 This example demonstrates how the management plane visualizes historical information alongside detailed local and WAN attrition data. Figure 2 In the figure, local loss and WAN loss are shown in alternating bars, such that for each time period, local loss is shown on the left and WAN loss on the right. For example, for the first time period, local loss = 0 and WAN loss = 0; for the second time period, local loss = 0 and WAN loss = 0; for the third time period, local loss = 3.5 and WAN loss = 0; for the fourth time period, local loss = 1.8 and WAN loss = 2.0, and so on. The “Shaper Rate” line shows whether the shaper rate increases, decreases, or remains the same for the adaptive time period, depending on the local loss and WAN loss values. The example in Figure 3 shows that the shaper rate remains at its default value after the first and second time periods because there is no local loss or WAN loss. The shaper rate increases after the third time period because there is local loss (3.5) but no WAN loss (0) during the third time period. The shaper rate decreases after the fourth time period because there is local loss (1.8) and WAN loss (2.0) during the fourth time period. The shaper rate continues to adapt according to local loss and WAN loss in subsequent time periods.

[0064] The most straightforward way to configure a default bandwidth value is to directly use the contracted rate with the service provider (SP). Considering variations in configuration size and active bandwidth, when adaptive QoS is enabled, speed testing can be used to detect upstream and downstream active bandwidth, and the detected bandwidth can be directly used as the default value. Once the minimum, maximum, and default values ​​are configured, the edge router will adapt the QoS shaper rate based on path monitor metrics and an adaptation algorithm. Depending on whether the user-configured values ​​are suitable for reality, intelligent configuration decisions (or recommendations) can be generated as follows:

[0065] • If the QoS shaper rate maintains maximum bandwidth during certain adaptation periods and there is no WAN loss, but there is local loss or high traffic throughput, the maximum value can be dynamically increased (or an alert can be sent to the customer to recommend increasing the maximum value). Figure 3AAn example is shown in which the QoS shaper rate maintains maximum bandwidth for five adaptation time periods (i.e., time periods 2, 3, 4, 5, and 6).

[0066] • If the QoS shaper rate maintains minimum bandwidth during certain adaptation periods and WAN loss still exists, the minimum can be dynamically reduced (or an alert can be sent to the customer to recommend reducing the minimum). Figure 3A An example is shown in which the QoS shaper rate maintains minimum bandwidth for five adaptation time periods (i.e., time periods 10, 11, 12, 13, and 14).

[0067] The management plane's fully adaptive QoS history can be used to discover daily, weekly, monthly, and yearly patterns in WAN available bandwidth. Figure 3B The weekly adaptation history is shown as an example. Based on the regularity of the adaptation history, the management plane can recommend intelligent configurations or even dynamically adjust parameters.

[0068] Figure 4 An example architecture for SD-WAN is shown. In a particular embodiment, network 400 may be an SD-WAN, which is a virtual overlay network based on tunnels carrying services between multiple sites on one or more underlying networks 430. Figure 4The example shown illustrates two sites: a first site 410 and a second site 420. Each site may include one or more user equipments, such as user equipment 411 of the first site 410 and user equipment 421 of the second site 420. Each site may be connected to network 400 via one or more WAN edge routers. For example, the first site 410 is connected to network 400 via edge router 401, while the second site 420 is connected to network 400 via edge router 402. In some embodiments, each edge router 401 and 402 may be configured as a hub router that can receive data from multiple hub routers (e.g., BR1, BR2, BR3, and / or BR4 discussed above). Sites connected to network 400 may have data plane connectivity to each other site via Internet Protocol Security (IPSec) tunnels. Edge routers 401 and 402 may have data plane connectivity on an underlying network 430. The underlying network 430 may include Multiprotocol Label Switching (MPLS), the Internet, and cellular networks. The SD-WAN control plane may include a controller 405, which can maintain centralized routing tables and routing policies to program the forwarding behavior of the data plane. The controller 405 can maintain direct control plane connections to each edge router. The controller 405 can provide, maintain, and secure the entire overlay network. The SD-WAN network 400 may also include a management / orchestration plane 407. Although this disclosure describes SD-WAN networks in a particular manner, this disclosure contemplates any suitable form of SD-WAN network.

[0069] Figure 5 An example computer system 500 is shown. In some embodiments, regarding... Figures 1A-1C Any of the hubs, routers, or remote sites described herein can be implemented using computer system 500. In some embodiments, computer system 500 can be used for implementation. Figure 4 One or more components of network 400. In a particular embodiment, one or more computer systems 500 perform one or more steps of one or more methods described or illustrated herein, such as regarding Figure 6 One or more steps of the described methods. In a particular embodiment, one or more computer systems 500 provide the functionality described or illustrated herein. In a particular embodiment, software running on one or more computer systems 500 performs one or more steps of the one or more methods described or illustrated herein, or provides the functionality described or illustrated herein. A particular embodiment includes one or more portions of one or more computer systems 500. Here, references to computer systems may include computing devices and vice versa, where appropriate. Furthermore, references to computer systems may include one or more computer systems, where appropriate.

[0070] This disclosure contemplates any suitable number of computer systems 500. This disclosure contemplates computer systems 500 in any suitable physical form. By way of example and not limitation, computer system 500 may be an embedded computer system, a system-on-a-chip (SOC), a single-board computer system (SBC) (e.g., a computer-on-module (COM) or system-on-module (SOM)), a desktop computer system, a laptop or notebook computer system, an interactive kiosk, a mainframe, a computer system grid, a mobile phone, a personal digital assistant (PDA), a server, a tablet computer system, an augmented / virtual reality device, or a combination of two or more of these. Where appropriate, computer system 500 may include one or more computer systems 500; may be single or distributed; may span multiple locations; may span multiple machines; may span multiple data centers; or may reside in a cloud that may include one or more cloud components in one or more networks. Where appropriate, one or more computer systems 500 may perform one or more steps of one or more methods described or illustrated herein without substantial spatial or temporal limitations. By way of example and not limitation, one or more computer systems 500 may perform one or more steps of one or more methods described or illustrated herein in real time or in batch mode. Where appropriate, one or more computer systems 500 may perform one or more steps of one or more methods described or illustrated herein at different times or in different locations.

[0071] In a particular embodiment, computer system 500 includes processing circuitry (e.g., processor 502), memory 504, storage device 506, input / output (I / O) interface 508, communication interface 510, and bus 512. While this disclosure describes and illustrates a particular computer system having a particular number of components in a particular arrangement, this disclosure contemplates any suitable computer system having any suitable number of components in any suitable arrangement.

[0072] In a particular embodiment, processor 502 includes hardware for executing instructions, such as those constituting a computer program. As an example, and not a limitation, of executing instructions, processor 502 may retrieve (or fetch) instructions from internal registers, internal caches, memory 504, or storage device 506; decode and execute them; and then write one or more results to internal registers, internal caches, memory 504, or storage device 506. In a particular embodiment, processor 502 may include one or more internal caches for data, instructions, or addresses. This disclosure contemplates that processor 502 may include any suitable number of suitable internal caches where appropriate. As an example, and not a limitation, processor 502 may include one or more instruction caches, one or more data caches, and one or more translation back buffers (TLBs). Instructions in the instruction cache may be copies of instructions in memory 504 or storage device 506, and the instruction cache may accelerate the retrieval of those instructions by processor 502. The data in the data cache may be a copy of data in memory 504 or storage device 506 for operation by instructions executed at processor 502; the result of a previous instruction executed at processor 502, for access by a subsequent instruction executed at processor 502 or for writing to memory 504 or storage device 506; or other suitable data. The data cache can accelerate read or write operations of processor 502. The TLB can accelerate virtual address translation of processor 502. In a particular embodiment, processor 502 may include one or more internal registers for data, instructions, or addresses. This disclosure contemplates that processor 502 may include any suitable number of suitable internal registers where appropriate. Where appropriate, processor 502 may include one or more arithmetic logic units (ALUs); be a multi-core processor; or include one or more processors 502. Although this disclosure describes and illustrates a particular processor, this disclosure contemplates any suitable processor.

[0073] In some embodiments, processor 502 may include one or more modules to facilitate the functionality described herein. As an example, processor 502 may include a feedback module X configured to receive feedback from one or more remote routers and / or an adaptive QoS module Y configured to adjust the shaper rate based at least in part on the feedback.

[0074] In a particular embodiment, memory 504 includes main memory for storing instructions to be executed by processor 502 or data to be operated by processor 502. By way of example and not limitation, computer system 500 may load instructions from storage device 506 or another source (e.g., another computer system 500) into memory 504. Processor 502 may then load instructions from memory 504 into internal registers or internal caches. To execute an instruction, processor 502 may retrieve the instruction from an internal register or internal cache and decode it. During or after instruction execution, processor 502 may write one or more results (which may be intermediate or final) into internal registers or internal caches. Processor 502 may then write one or more of these results into memory 504. In a particular embodiment, processor 502 executes only the instructions in one or more internal registers or internal caches or memory 504 (as opposed to storage device 506 or elsewhere) and operates only on the data in one or more internal registers or internal caches or memory 504 (as opposed to storage device 506 or elsewhere). One or more memory buses (each of which may include an address bus and a data bus) may couple processor 502 to memory 504. Bus 512 may include one or more memory buses as described below. In a particular embodiment, one or more memory management units (MMUs) reside between processor 502 and memory 504 and facilitate access to memory 504 requested by processor 502. In a particular embodiment, memory 504 includes random access memory (RAM). Where appropriate, the RAM may be volatile memory. Where appropriate, the RAM may be dynamic RAM (DRAM) or static RAM (SRAM). Furthermore, where appropriate, the RAM may be single-port or multi-port RAM. This disclosure contemplates any suitable RAM. Where appropriate, memory 504 may include one or more memories 504. Although this disclosure describes and illustrates specific memories, this disclosure contemplates any suitable memories.

[0075] In a particular embodiment, storage device 506 includes a mass storage device for data or instructions. By way of example and not limitation, storage device 506 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk drive, magneto-optical disk drive, magnetic tape drive, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, storage device 506 may include removable or non-removable (or fixed) media. Where appropriate, storage device 506 may be internal or external to computer system 500. In a particular embodiment, storage device 506 is a non-volatile solid-state memory. In a particular embodiment, storage device 506 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmable ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically variable ROM (EAROM), or flash memory, or a combination of two or more of these. This disclosure contemplates mass storage device 506 in any suitable physical form. Where appropriate, storage device 506 may include one or more storage device control units that facilitate communication between processor 502 and storage device 506. Where appropriate, storage device 506 may include one or more storage devices 506. Although this disclosure describes and illustrates specific storage devices, any suitable storage device is contemplated.

[0076] In a particular embodiment, I / O interface 508 includes hardware, software, or both that provide one or more interfaces for communication between computer system 500 and one or more I / O devices. Where appropriate, computer system 500 may include one or more of these I / O devices. One or more of these I / O devices can enable communication between a person and computer system 500. By way of example and not limitation, I / O devices may include a keyboard, keypad, microphone, monitor, mouse, printer, scanner, speaker, still camera, pen, input pad, touchscreen, trackball, camera, another suitable I / O device, or a combination of two or more of these. I / O devices may include one or more sensors. This disclosure contemplates any suitable I / O devices and any suitable I / O interface 508 for them. Where appropriate, I / O interface 508 may include one or more device or software drivers that enable processor 502 to drive one or more of these I / O devices. Where appropriate, I / O interface 508 may include one or more I / O interfaces 508. Although this disclosure describes and illustrates specific I / O interfaces, this disclosure contemplates any suitable I / O interface.

[0077] In a particular embodiment, communication interface 510 includes hardware, software, or both that provide one or more interfaces for communication (e.g., packet-based communication) between computer system 500 and one or more other computer systems 500 or one or more networks. By way of example and not limitation, communication interface 510 may include a network interface controller (NIC) or network adapter for communicating with Ethernet or other wired-based networks, or may include a wireless NIC (WNIC) or wireless adapter for communicating with wireless networks (e.g., Wi-Fi networks). This disclosure contemplates any suitable network and any suitable communication interface 510 for use therewith. By way of example and not limitation, computer system 500 may communicate with one or more portions of an ad hoc network, personal area network (PAN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), or the Internet, or a combination of two or more of these. One or more portions of one or more of these networks may be wired or wireless. As an example, computer system 500 may communicate with a wireless PAN (WPAN) (e.g., Bluetooth WPAN), a Wi-Fi network, a Wi-Fi Max network, a cellular telephone network (e.g., a Global System for Mobile Communications (GSM) network, a Long Term Evolution (LTE) network, or a 5G network), or other suitable wireless networks, or a combination of two or more of these networks. Where appropriate, computer system 500 may include any suitable communication interface 510 for any of these networks. Where appropriate, communication interface 510 may include one or more communication interfaces 510. Although specific communication interfaces are described and illustrated in this disclosure, any suitable communication interface is contemplated in this disclosure.

[0078] In a particular embodiment, bus 512 includes hardware, software, or both, that couple components of computer system 500 to each other. By way of example and not limitation, bus 512 may include Accelerated Graphics Port (AGP) or other graphics buses, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), HyperTransport (HT) interconnect, Industry Standard Architecture (ISA) bus, Infinite Bandwidth (INFINIBAND) interconnect, Low Pin Count (LPC) bus, memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCIe) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus, or another suitable bus or a combination of two or more of these buses. Where appropriate, bus 512 may include one or more buses 512. Although this disclosure describes and illustrates specific buses, this disclosure contemplates any suitable bus or interconnect.

[0079] In this document, one or more computer-readable non-transitory storage media may include one or more semiconductor-based or other integrated circuits (ICs) (e.g., field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs)), hard disk drives (HDDs), hybrid hard disk drives (HHDs), optical disks, optical disk drives (ODDs), magneto-optical disks, magneto-optical drives, floppy disks, floppy disk drives (FDDs), magnetic tape, solid-state drives (SSDs), RAM drives, secure digital cards or drives, any other suitable computer-readable non-transitory storage media, or any suitable combination of two or more of these. Where appropriate, computer-readable non-transitory storage media may be volatile, non-volatile, or a combination of volatile and non-volatile.

[0080] Figure 6 An example of a method 600 for adjusting the shaper rate at a hub router based at least in part on feedback from one or more remote (spoke) routers is shown. In some embodiments, method 600 may be performed by a system such as BR1, BR2, BR3, or BR4 described above. In some embodiments, the system may include one or more processors and one or more computer-readable nontransitory storage media. The one or more computer-readable nontransitory storage media include instructions that, when executed, cause one or more components to perform operations including the steps of method 600.

[0081] Method 600 begins at step 602 by sending data from a hub (e.g., BR1) to a spoke (e.g., remote site 1). The data is associated with a session (e.g., a hub-to-spoke session). Method 600 proceeds to step 604, where the hub receives feedback from the spoke. This feedback is based on at least one of the occurrence of a congestion state detected by the spoke or bandwidth utilization. As an example, the spoke can determine the feedback based on bandwidth utilization by monitoring the downstream throughput of the session on the path monitoring receiver side (based on each tunnel), and can provide the feedback via the path monitor TLV parameters of the BFD protocol. In some embodiments, the feedback may include a new field (e.g., “expected throughput” discussed above) contained in the structure definition of each subsequence namespace (SNS) path_monTLV.

[0082] Method 600 proceeds to step 606, where the hub adjusts the shaper rate of the adaptive QoS shaper, such as an egress shaper associated with a session and used by the hub to shape traffic sent from the hub to the spokes. In some embodiments, adjusting the shaper rate may include determining the shaper rate for the current adaptive interval and / or changing the shaper rate (increasing or decreasing the shaper rate) if the shaper rate determined for the current adaptive interval is different from the shaper rate used during a previous adaptive interval. In step 604, the shaper rate is adjusted at least in part based on feedback received from the spokes.

[0083] As an example, adjusting the shaper rate in response to feedback indicating that the session's bandwidth utilization exceeds a threshold during congestion includes reducing the shaper rate: the spoke has detected that the session's bandwidth utilization exceeds a threshold. As another example, adjusting the shaper rate in response to feedback indicating that the session's bandwidth utilization exceeds a threshold during congestion includes not reducing the shaper rate (e.g., the shaper rate for the current adaptive interval can remain the same as the shaper rate for the previous adaptive interval): the spoke has detected that the session's bandwidth utilization is below a threshold during congestion. In some embodiments, congestion occurs when the spoke detects that the WAN loss has exceeded a WAN loss threshold.

[0084] In some embodiments, bandwidth utilization is considered to exceed a session bandwidth utilization threshold when the session throughput exceeds the session's expected throughput. Similarly, bandwidth utilization is considered to be below a session bandwidth utilization threshold when the session throughput is less than the session's expected throughput. Expected throughput may be based on a weighting factor associated with the session and the total available bandwidth on the spokes. Expected throughput may be indicated as feedback or may be used to calculate feedback. For example, as discussed above:

[0085] • If (session_throughput <= total_bandwidth * weight / total_weight)), then “Expected Throughput” = session_throughput;

[0086] If (session_throughput > total_bandwidth * weight / total_weight), then "Expected Throughput" = Remaining Throughput * weight / Remaining Total Weight; and

[0087] Therefore, according to an embodiment, feedback based on bandwidth utilization may include expected throughput or other information related to or indicating bandwidth utilization.

[0088] As described above, the desired throughput can be based on a weighting factor associated with a session. The weighting factor can be configured to facilitate bandwidth sharing among multiple hubs. For example, the weighting factor can be configured differently for different sessions and / or different hubs depending on service patterns, QoS requirements, etc. In some embodiments, the hub can indicate the weighting factor to the spokes. As an example, the hub can pre-publish the weighting factor so that it can be used by the spokes to provide feedback. Some embodiments publish the weighting factor across the SD-WAN architecture via OMP.

[0089] Method 600 then proceeds to step 608, where additional data associated with the session is sent. The additional data is sent from the hub to the spokes according to the adjusted shaper rate of step 606.

[0090] In some embodiments, the shaper rate used for a session can initially be based on a default bandwidth and can be limited based on a maximum and minimum bandwidth. Bandwidth parameters can be used to configure bandwidth parameters (default bandwidth, maximum bandwidth, and minimum bandwidth). In some embodiments, one or more of the bandwidth parameters used by the adaptive QoS shaper can be dynamically adjusted based on real-time performance. As an example:

[0091] • When the shaper rate is already set to the current maximum bandwidth for a first predetermined period without WAN loss (but with local loss or high throughput), the maximum bandwidth can be increased; and / or

[0092] • When the shaper rate has been set to the current minimum bandwidth for the second predetermined period and there is WAN loss, the minimum bandwidth can be reduced.

[0093] In some embodiments, WAN loss detection may be based on a threshold. For example, if the WAN loss measurement is below the threshold, a "no WAN loss" condition can be detected, while if the WAN loss measurement is above the threshold, a "WAN loss" condition can be detected. The threshold can be set to any suitable value (e.g., 0, 1, 2…n). The first predetermined period for increasing maximum bandwidth can be the same as or different from the second predetermined period for decreasing minimum bandwidth. Figure 3A An example is shown where the predetermined time period is set to 5 relative to each of the maximum and minimum bandwidths. Therefore, it is possible to base this at least in part on five adaptation time periods (e.g., Figure 3AThe QoS shaper rate (as shown in time periods 2, 3, 4, 5, and 6) is used to dynamically increase the maximum bandwidth while maintaining the current maximum bandwidth. Similarly, the maximum bandwidth can be dynamically increased at least in part based on the QoS shaper rate during the five adaptation time periods (e.g., time periods 2, 3, 4, 5, and 6). Figure 3A (As shown in time periods 10, 11, 12, 13 and 14) The QoS shaper rate is maintained at the current minimum bandwidth to dynamically reduce the minimum bandwidth.

[0094] Alternatively, some embodiments may dynamically adjust one or more bandwidth parameters (the maximum bandwidth, minimum bandwidth, and / or default bandwidth used by the adaptive QoS shaper) based on historical performance patterns. Figure 3B An example is shown where certain days of the week exhibit a pattern of using higher bandwidth (e.g., Monday) while other days of the week exhibit a pattern of using lower bandwidth (e.g., Thursday). Based on this pattern, some embodiments may dynamically adjust the bandwidth parameter upwards on Mondays and downwards on Thursdays.

[0095] In some embodiments, the spokes can provide reversibility to support Figure 6 The hub provides the functionality described above. For example, the spokes may include the router of the aforementioned remote site 1. The router may include one or more processors and one or more computer-readable non-transitory storage media. The one or more computer-readable non-transitory storage media include instructions that, when executed, cause one or more components to perform operations including the steps of the method performed by the spokes.

[0096] In some embodiments, the method performed by the hub includes: receiving data associated with a session from a hub; determining feedback based on at least one of the occurrence of a congestion state detected by the hub or bandwidth utilization; and providing the feedback to the hub. The hub can determine the feedback in any suitable manner, such as by determining the expected throughput, as described above. For example, the hub can provide the “expected throughput” to the hub via a path monitor (TLV). In some embodiments, the hub monitors the bandwidth utilization of multiple sessions, where each session can be associated with any of a plurality of hubs. As an example, the hub can monitor a first session between the hub and hub BR1, a second session between the hub and hub BR2, a third session between the hub and hub BR3, and / or a fourth session between the hub and hub. For each session, the hub weights the monitored bandwidth utilization based on a weighting factor configured for that session and provides feedback to the hub associated with that session based on the bandwidth utilization. Different sessions can be configured with different weighting factors, for example, based on service patterns, QoS requirements, etc. In this way, the spokes facilitate the sharing of bandwidth between hubs.

[0097] In this document, "or" is inclusive rather than exclusive, unless otherwise expressly indicated or the context otherwise indicates. Therefore, "A or B" in this document means "A, B, or both," unless otherwise expressly indicated or the context otherwise indicates. Furthermore, "and" is both joint and separate, unless otherwise expressly indicated or the context otherwise indicates. Therefore, "A and B" in this document means "A and B, jointly or separately," unless otherwise expressly indicated or the context otherwise indicates.

[0098] The scope of this disclosure includes all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments described or illustrated herein that will be understood by those skilled in the art. The scope of this disclosure is not limited to the exemplary embodiments described or illustrated herein. Furthermore, although this disclosure describes and illustrates various embodiments including specific components, elements, features, functions, operations, or steps, any embodiment in these embodiments may include any combination or arrangement of any components, elements, features, functions, operations, or steps described or illustrated anywhere herein that will be understood by those skilled in the art. Additionally, references to a device or system or component of a device or system adapted to, arranged to, capable of, configured to, enabled, operable, or operable to perform a particular function include that device, system, or component, whether or not it or the particular function is activated, turned on, or unlocked, provided that the device, system, or component is so adapted to, arranged to, capable of, configured to, enabled, operable, or operable. Furthermore, while this disclosure describes or illustrates specific embodiments to provide particular advantages, those embodiments may not provide these advantages, or may provide some or all of these advantages.

[0099] The embodiments disclosed herein are merely examples, and the scope of this disclosure is not limited thereto. Specific embodiments may include all, some, or none of the components, elements, features, functions, operations, or steps of the embodiments disclosed herein. Embodiments according to this disclosure specifically relate to a method, storage medium, system, and computer program product, wherein any feature mentioned in one category (e.g., a method) may also be applied to another category (e.g., a system). The various features disclosed herein may be combined or separated in any suitable manner.

Claims

1. A system for network communication, the system comprising: One or more processors; as well as One or more computer-readable non-transitory storage media, the one or more computer-readable non-transitory storage media including instructions, which, when executed by the one or more processors, cause one or more components of the system to perform operations, the operations including: Data from multiple hubs is sent to the spokes, where the data is associated with a session; Feedback is received at the hub from the spokes, the feedback being based on both the occurrence of a congestion state detected by the spokes and bandwidth utilization, wherein the feedback indicates that the bandwidth utilization of the session has exceeded a threshold when the session throughput of the session exceeds the expected throughput of the session, the expected throughput being based on a weighting factor associated with the session and the total bandwidth available to the spokes, and wherein the congestion state occurs when the spokes detect WAN loss exceeding a wide area network (WAN) loss threshold; and In response to feedback indicating that the spoke has detected that the bandwidth utilization of the session has exceeded a threshold during the congestion state, the hub reduces the shaper rate of the adaptive quality of service (QoS) shaper associated with the session, the shaper rate being at least partially based on the feedback received from the spoke.

2. The system of claim 1, wherein the operation further includes: The weighting factor is indicated to the spokes.

3. The system as described in claim 1 or 2, wherein the operation further includes: In response to the feedback indicating the following, adjusting the shaper rate includes not reducing the shaper rate: the spokes have detected that the bandwidth utilization of the session is below a threshold during the congestion state.

4. The system as described in claim 1 or 2, wherein, The feedback is received via the Path Monitoring Type-Length-Value (TLV) parameter of the Bidirectional Forwarding Detection (BFD) protocol.

5. The system as described in claim 1 or 2, wherein the operation further comprises: The value of the bandwidth parameter used by the adaptive QoS shaper is dynamically adjusted based on real-time performance, wherein the bandwidth parameter corresponds to the maximum bandwidth, minimum bandwidth, or default bandwidth.

6. The system of claim 5, wherein, Dynamically adjusting the value of the bandwidth parameter includes at least one of the following: When the shaper rate has been set to increase the maximum bandwidth when there is no WAN loss for a first predetermined period of time, the maximum bandwidth is increased; and When the shaper rate has been set to the current minimum bandwidth for a second predetermined period and there is WAN loss, the minimum bandwidth is reduced.

7. The system of claim 1 or 2, wherein the operation further comprises: The bandwidth parameters used by the adaptive QoS shaper are dynamically adjusted based on historical performance patterns, wherein the bandwidth parameters correspond to the maximum bandwidth, minimum bandwidth, or default bandwidth.

8. A method for network communication, the method comprising: Data from multiple hubs is sent to the spokes, where the data is associated with a session; Feedback is received at the hub from the spokes, the feedback being based on both the occurrence of a congestion state detected by the spokes and bandwidth utilization, wherein the feedback indicates that the bandwidth utilization of the session has exceeded a threshold when the session throughput of the session exceeds the expected throughput of the session, the expected throughput being based on a weighting factor associated with the session and the total bandwidth available to the spokes, and wherein the congestion state occurs when the spokes detect WAN loss exceeding a wide area network (WAN) loss threshold; and In response to feedback indicating that the spoke has detected that the bandwidth utilization of the session has exceeded a threshold during the congestion state, the hub reduces the shaper rate of the adaptive quality of service (QoS) shaper associated with the session, the shaper rate being at least partially based on the feedback received from the spoke.

9. The method of claim 8, further comprising: In response to the feedback indicating that the shaper rate is not reduced when the spokes have detected that the bandwidth utilization of the session is below the threshold during the congestion state.

10. The method of claim 9, wherein: When the session throughput of the session is lower than the expected throughput of the session, the feedback indicates that the bandwidth utilization of the session is lower than the threshold.

11. The method of any one of claims 8 to 10, further comprising: The maximum bandwidth associated with the adaptive QoS shaper is dynamically increased based on real-time performance. When the shaper rate has been set to the current maximum bandwidth for a first predetermined period of time without WAN loss, the maximum bandwidth is increased. as well as The minimum bandwidth associated with the adaptive QoS shaper is dynamically reduced based on real-time performance. The minimum bandwidth is reduced when the shaper rate has been set to the current minimum bandwidth for a second predetermined period and there is WAN loss.

12. The method of any one of claims 8 to 10, further comprising: The bandwidth parameters used by the adaptive QoS shaper are dynamically adjusted based on historical performance patterns, wherein the bandwidth parameters correspond to the maximum bandwidth, minimum bandwidth, or default bandwidth.

13. One or more computer-readable non-transitory storage media, the one or more computer-readable non-transitory storage media comprising instructions that, when executed by a processor, cause to perform an operation, the operation comprising: Data from multiple hubs is sent to the spokes, where the data is associated with a session; The hub receives feedback from the spokes, the feedback being based on both the occurrence of a congestion state detected by the spokes and bandwidth utilization, wherein the feedback indicates that the bandwidth utilization of the session has exceeded a threshold when the session throughput of the session exceeds the expected throughput of the session, the expected throughput being based on a weighting factor associated with the session and the total bandwidth available to the spokes, and wherein the congestion state occurs when the spokes detect WAN loss exceeding a wide area network (WAN) loss threshold; In response to feedback indicating that the spokes have detected that the bandwidth utilization of the session has exceeded a threshold during the congestion state, the hub reduces the shaper rate of the adaptive Quality of Service (QoS) shaper used by the hub to shape the traffic associated with the session, the shaper rate being at least partially based on the feedback received from the spokes. as well as Additional data associated with the session is sent according to the adjusted shaper rate.

14. One or more computer-readable non-transitory storage media as described in claim 13, wherein, The operation further includes: in response to the feedback indicating that the shaper rate is not reduced when the spokes have detected that the bandwidth utilization of the session is below the threshold during the congestion state.

15. One or more computer-readable non-transitory storage media as described in claim 14, wherein: When the session throughput of the session is lower than the expected throughput of the session, the feedback indicates that the bandwidth utilization of the session is lower than the threshold.

16. One or more computer-readable non-transitory storage media as claimed in any one of claims 13 to 15, wherein the operation further comprises: The maximum bandwidth associated with the adaptive QoS shaper is dynamically increased based on real-time performance. When the shaper rate has been set to the current maximum bandwidth for a first predetermined period of time without WAN loss, the maximum bandwidth is increased. as well as The minimum bandwidth associated with the adaptive QoS shaper is dynamically reduced based on real-time performance. The minimum bandwidth is reduced when the shaper rate has been set to the current minimum bandwidth for a second predetermined period and there is WAN loss.

17. One or more computer-readable non-transitory storage media as claimed in any one of claims 13 to 15, wherein the operation further comprises: The bandwidth parameters used by the adaptive QoS shaper are dynamically adjusted based on historical performance patterns, wherein the bandwidth parameters correspond to the maximum bandwidth, minimum bandwidth, or default bandwidth.

18. An apparatus for network communication, comprising: A means for sending data from a hub among multiple hubs to the spokes, wherein the data is associated with a session; A means for receiving feedback from the spokes at the hub, the feedback being based on at least one of the occurrence of a congestion state detected by the spokes or bandwidth utilization, wherein the feedback indicates that the bandwidth utilization of the session has exceeded a threshold when the session throughput of the session exceeds the expected throughput of the session, the expected throughput being based on a weighting factor associated with the session and the total bandwidth available to the spokes, and wherein the congestion state occurs when the spokes detect WAN loss exceeding a wide area network (WAN) loss threshold; and A means for reducing the shaper rate of an adaptive quality of service (QoS) shaper associated with the session by the hub in response to feedback indicating that the spoke has detected that the bandwidth utilization of the session has exceeded a threshold during the congestion state, the shaper rate being at least partially based on the feedback received from the spoke.

19. The apparatus of claim 18, further comprising: A means for responding to feedback indicating that the following does not reduce the rate of the shaper: the spokes have detected that the bandwidth utilization of the session is below the threshold during the congestion state.

20. The apparatus of claim 18, wherein, When the session throughput of the session is lower than the expected throughput of the session, the feedback indicates that the bandwidth utilization of the session is lower than the threshold.

21. The apparatus of any one of claims 18 to 20, further comprising: A means for dynamically increasing the maximum bandwidth associated with the adaptive QoS shaper based on real-time performance, wherein the maximum bandwidth is increased when the shaper rate has been set to reach the current maximum bandwidth for a first predetermined period of time without WAN loss; and A means for dynamically reducing the minimum bandwidth associated with the adaptive QoS shaper based on real-time performance, wherein the minimum bandwidth is reduced when the shaper rate has been set to the current minimum bandwidth for a second predetermined period and there is WAN loss.

22. The apparatus of any one of claims 18 to 20, further comprising: A means for dynamically adjusting the bandwidth parameters used by the adaptive QoS shaper based on historical performance patterns, wherein the bandwidth parameters correspond to maximum bandwidth, minimum bandwidth, or default bandwidth.

23. A computer program product comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method as claimed in any one of claims 8 to 12.

24. A computer-readable medium comprising instructions that, when executed by a computer, cause the computer to perform the steps of the method as claimed in any one of claims 8 to 12.

Images