Dynamic client balancing between branch gateways

Abstract

One comprehensive system: a processor (600a, 600b, 600c) and a non-transitory storage medium (608a, 608b, 608c) that stores instructions (610a, 610b, 610c) which, when executed on the processor, cause the system to: determines that a network traffic load of a first branch gateway, BG, (104a, 604a) of a local network, LAN, (102) is different from a network traffic load of a second BG (104b, 604b) of the LAN; determines that a first network traffic load skew between the first BG and the second BG is greater than a first threshold; selects a first virtual local area network, VLAN, from a set of VLANs (114a, 114b, 114c, 114d) of the LAN, setting up a first virtual router for the first VLAN and a second virtual router for a second VLAN from the set of VLANs, with network traffic from client devices of the first VLAN, sent to the first virtual router, being routed through the first BG, based on the first BG being a master BG for the first VLAN according to a configuration of the first virtual router; and sends a reconfiguration message to at least one of the first BG and the second BG, which changes the configuration of the first virtual router set up for the first VLAN, whereby by changing the configuration of the first virtual router the second BG is reassigned as the master BG for the first VLAN, whereby after the change in the configuration of the first virtual router the network traffic from client devices of the first VLAN is henceforth routed via the second BG.

Description

BACKGROUND

[0001] A wide area network (WAN) can span multiple network locations (e.g., geographic, logical). WAN locations are interconnected, allowing devices at one location to access resources at another. In some topologies, many services and resources are installed at core locations (e.g., data centers, headquarters), while numerous branch offices (e.g., regional offices, retail stores) connect client devices (e.g., laptops, smartphones, Internet of Things devices) to the WAN. These types of topologies are often used by businesses to configure their corporate networks.

[0002] Each network location has its own local area network (LAN), which is connected to the LANs of the other locations to form the wide area network (WAN). Network infrastructure, such as switches and routers, is used to route network traffic through each LAN, across the WAN as a whole, and between the WAN and the internet. Each network location's LAN is connected to the wider network (e.g., the WAN, the internet) via a gateway router. Branch gateways (BGs) connect branch locations to the wider network, and headend gateways (also known as virtual internet gateways) connect core locations to the wider network.

[0003] Wide Area Networks (WANs) are often implemented using software-defined long-range network (SD-WAN) technology. An SD-WAN decouples (logically or physically) the control aspects of switching and routing from the physical routing of network traffic. In some SD-WAN implementations, each gateway (gateway groups and headend gateways) controls specific aspects of routing for its corresponding LAN, but a network orchestrator controls overall switching and routing across the WAN.

[0004] US 2005 / 0025179A1 refers to methods and devices for controlling the distribution of traffic flow through a gateway using multiple gateway devices acting as virtual routers.

[0005] The present invention is defined by independent claims 1, 9 and 15. Embodiments are the subject of the respective dependent claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] For a more comprehensive understanding of the present disclosure, examples according to the various features described herein are easier to understand by reference to the following detailed description in conjunction with the accompanying drawings, where the same reference symbols denote the same structural elements and where: Fig. 1 An exemplary SD-WAN illustrates which includes a branch LAN connected to a main office LAN; Fig. 2A to 2B simplified representations of the exemplary SD-WAN of the Fig. 1 are, which include the operation of the branch gateways of the branch LAN; Fig. 3A to 3B simplified representations of the exemplary SD-WAN of the Fig. 1 are, which include the operation of the branch gateways and infrastructure devices of the branch LAN; Fig. 4 is a flowchart that illustrates an exemplary procedure for balancing network load across branch gateways of an SD-WAN; Fig. 5 is a flowchart which illustrates another exemplary procedure for balancing network load across branch gateways of an SD-WAN; Fig. Figure 6 illustrates an exemplary simplified SD-WAN, which includes a network orchestrator and branch gateways; Certain examples exhibit features that are present in addition to, or instead of, the features illustrated in the figures described above. In some figures, certain labels may have been omitted for the sake of clarity. DETAILED DESCRIPTION

[0007] In a software-defined long-range network (SD-WAN), gateways between local area networks (LANs) and the wider SD-WAN can be single points of failure for segments of the SD-WAN. For example, a single branch gateway (BG) serving a large branch LAN can disrupt SD-WAN access for many client devices (e.g., laptops, mobile phones, servers, IoT devices) within the branch LAN and prevent client devices throughout the SD-WAN from accessing services provided by that branch LAN. To mitigate the impact of a single point of failure, a high-availability (HA) configuration can employ multiple redundant gateways to maintain network functionality if one gateway fails.

[0008] When BGs are used in an HA configuration, they are usually deployed in an "active-standby" configuration, with one of the BGs ("active") routing traffic between its branch LAN and LANs of other locations in the SD-WAN, and another of the BGs ("standby") remaining idle while the active BG is functioning correctly.

[0009] In some other HA deployments, gateways are used in an "active-active" configuration, with each gateway routing a portion of the traffic between the branch office LAN and LANs at other points in the SD-WAN. Certain client devices may be assigned to a first gateway of the branch office LAN, and other client devices may be assigned to a second gateway of the branch office LAN. In many such deployments, default gateway parameters of each client device's network configuration can be configured for its corresponding assigned gateway.

[0010] In some such deployments, client devices are organized into virtual local area networks (VLANs), and each VLAN is assigned to a corresponding business group (BG). For example, a branch office LAN with VLANs 1 through 10 might assign even-numbered VLANs (2, 4, 6, 8, 10) to a first BG of the branch office LAN and odd-numbered VLANs (1, 3, 5, 7, 9) to a second BG of the branch office LAN. Often, these VLAN assignments required a significant amount of manual configuration by a network administrator to ensure that traffic from client devices in a given VLAN was routed through the branch office LAN to the correct BG.

[0011] In some HA deployments, a Virtual Router Redundancy Protocol (VRRP) can be used to reduce the amount of configuration required across multiple devices in the branch LAN. Essentially, VRRP creates a virtual router (in this case, a virtual branch gateway) with an IP address that client devices use as the default gateway in their network configurations. Then, based on the configurations of the gateways, packets sent to the virtual router are routed through a master gateway (which is the active gateway in active-standby configurations). To establish virtual routing in an active-active configuration, multiple virtual routers (also known as VRRP instances) can be created on a per-VLAN basis, and each VLAN is assigned a master gateway.Similar to what was mentioned above, the VLANs can be assigned to the BGs in a ring distribution format (even-numbered VLANs to a first BG, odd-numbered VLANs to a second BG), or according to a method that is appropriate in the respective context.

[0012] Although this rudimentary allocation of VLANs to BGs can roughly balance the network traffic load across the branch office LAN's BGs, it does not guarantee that each VLAN will contain the same number of client devices, nor that all client devices will have similar network usage requirements and structures. For example, VLAN 2 might contain several edge computing devices that send and receive large amounts of network traffic, while VLAN 7 might contain several personal mobile phones that only occasionally send and receive more than a negligible amount of network traffic. In a topology where even-numbered VLANs are distributed to a first BG and odd-numbered VLANs to a second BG, the first BG might be congested (partly due to devices in VLAN 2), and the second BG might have excess capacity (partly due to devices in VLAN 7).

[0013] In the present disclosure, such imbalances are captured, and VRRP instances for specific VLANs are reconfigured so that traffic for those specific VLANs is routed through a different BG. Extending the preceding example, the VRRP instances for VLANs 4 and 6 can be reconfigured so that the master BG for these VRRP instances is the second BG. Then, the network traffic load can be more evenly distributed between the master BG and the second BG. In some situations, the unbalanced load between the BGs may be due to a temporary phenomenon (e.g., a group of client devices in a particular VLAN is temporarily experiencing heavy activity from certain network traffic). In such situations, the VLAN-to-BG mappings can be readjusted to a baseline configuration after the temporary phenomenon has ended.

[0014] Certain network infrastructure devices (e.g., switches, access points) in the branch office LAN may have the ability to adjust device VLAN assignments when they are notified that they are operating in an HA network. For example, when notified of the network's HA topology, the network infrastructure devices can assign each successively added client device, following a ring topology, to a VLAN from a pool of VLANs. This prevents an imbalance of assigned clients across VLANs, which could lead to problems due to unbalanced load across business groups.

[0015] Each gateway (BG) can include packet services beyond simply routing packets. For example, BGs can perform content classification and deep packet analysis services. While a BG may have the bandwidth required to simply route throughcoming network traffic, it may lack the resources to run content classification and deep packet analysis services on that traffic. Network traffic load for a BG considers the load placed on all of the BG's resources, including those providing packet services.

[0016] Although the network infrastructure devices are informed of the network's high-availability (HA) topology, individual clients may still generate more load on the business group (BG) than others. In such situations, even if clients have been assigned in a ring distribution (or other suitable configuration), certain VLANs may still have a disproportionate impact on the network traffic load of a particular BG. In such situations, commands can be issued to the network infrastructure devices to attempt to mitigate the load imbalance in particularly large VLANs. For example, network infrastructure devices can be instructed to temporarily halt the addition of new clients to a specific VLAN. As another example, network infrastructure devices can be instructed to move certain existing clients from one VLAN to another.

[0017] In some examples, the network traffic load for each business unit (BG) can be determined on every corresponding BG. In other examples, network traffic load information is forwarded to a network orchestrator, which calculates network traffic load factors for each BG. In still other examples, one of the BGs collects network traffic load information from all BGs in the branch LAN and calculates network traffic load factors for each BG. Once network traffic load factors have been calculated for each BG, a network traffic load skew is determined. If the skew is greater than a certain threshold, corrective actions are implemented, as described in the preceding paragraphs, to reduce the skew and balance the load across the BGs.Depending on the specific network topology, corrective actions can be determined and executed by the network orchestrator, a master BG, each BG, or a combination thereof. In some examples, the BGs may include network controllers which, although logically separate from the data path that carries components of the respective BGs, are located together in the same BG device. In some of these examples, branch LAN management may be reserved for these network controllers. In other examples, branch LAN management may be organized by an SD-WAN network orchestrator in conjunction with the BG network controllers.

[0018] The features of this disclosure improve the performance of SD-WANs by redirecting network traffic to utilize available bandwidth. The resulting improved performance can reduce jitter, latency, and other signs of degrading network efficiency, as well as improve the quality of service for time-sensitive applications such as Voice over IP.

[0019] Fig. Figure 1 illustrates an example SD-WAN comprising a branch LAN connected to a core LAN. The SD-WAN 100 is a long-range network designed using software-defined networking principles, such as separating routing control decisions from network infrastructure devices that perform the actual routing and switching of packets within the SD-WAN 100. The SD-WAN includes a branch LAN 102, a core LAN 108, and a network orchestrator 110. The branch LAN 102, the core LAN 108, and the network orchestrator 110 are interconnected via the Internet 106 for communication. The branch LAN 102 includes branch gateways (BGs) 104 (including 104a, 104b), infrastructure devices 112 (including 112a, 112b), VLANs 114 (including 114a, 114b, 114c) and virtual gateways 116.VLANs 114 each include client devices, such as edge computing servers, user devices (e.g., laptops, desktops, mobile devices), interface devices (e.g., point-of-sale devices, kiosks), Internet of Things (IoT) devices, etc.

[0020] In some examples, each VLAN 114 is assigned to a specific infrastructure device 112, so only client devices of infrastructure device 112 are assigned to the VLANs 114 of infrastructure device 112. In other examples, each VLAN 112 is independent of the infrastructure devices 112, and client devices are assigned to a VLAN 114 based on a criterion or combination of criteria. Some example criteria include selecting a VLAN for each successive client device using a ring distribution format, assigning a VLAN based on network role (e.g., VLAN of a personal employee device, point-of-sale VLAN), or assigning a VLAN based on expected network bandwidth usage (e.g., users with high network bandwidth are split across different VLANs).

[0021] Client devices in branch LAN 102 can communicate with each other via infrastructure device 112 (including those not shown but included in the branch LAN 102 cloud). Client devices in branch LAN 102 can also communicate with devices outside of branch LAN 102. All traffic sent and received between a client device in branch LAN 102 and a device outside of branch LAN 102 passes through a BG 104. The SD-WAN 100 includes two BGs, 104a and 104b, in branch LAN 102. BGs 104a and 104b can serve multiple purposes, including high availability (HA), such as failover without interruption if one of the BGs fails, and load balancing.The gateways (BGs) can be configured to enable a Virtual Router Redundancy Protocol (VRRP), which is a protocol for implementing high availability (HA) across two routers (or branch gateways). VRRP creates a virtual gateway (116) that is assigned a virtual IP address (VIP address). Client devices configure their default gateway in their network settings to point to the VIP, and packets are forwarded to the correct BG (104) based on the VRRP configuration. Once configured, a VRRP instance designates which of the BGs (104) is the master. The master routes all traffic sent to the VIP. For example, if a VRRP instance is set up between BG (104a) and BG (104b), the VIP (192.168.100.4) is assigned to the instance, and BG (104a) is designated as the master. For all client devices where 192.168.100.If 4 is configured as the default gateway, traffic from the BG 104a is routed through the SD-WAN 100.

[0022] In many SD-WANs, redundant gateways configured with a VRRP instance are deployed in an "active-standby" mode. The active gateway (e.g., the VRRP master) handles all incoming and outgoing traffic for the entire branch LAN, while the standby gateway remains idle until the active gateway (or its connections to the wider network or the branch LAN) fails. This meets the high-availability (HA) requirements of the SD-WAN but is inefficient in utilizing gateway resources. A network administrator could disable VRRP and manually configure each client device to point to gateway 104a or 104b, but this solution is not scalable for larger deployments and does not adapt to changing network conditions.

[0023] In some SD-WANs, redundant gateways (BGs) can be used in an "active-active" mode, with each BG handling a portion of the inbound and outbound traffic for the branch LAN. However, a VRRP instance cannot perform load balancing between its master gateway and its backup gateway. All traffic sent to a VRRP instance's VIP is routed to the instance's master gateway. Alternatively, to enable "active-active" mode, multiple concurrent VRRP instances can create multiple virtual gateways, and each VLAN can be assigned a different VIP as its default gateway.For example, VLAN 114a can point to a first VIP that routes network traffic through BG 104a, VLAN 114b can point to a second VIP that routes network traffic through BG 104b, VLAN 114c can point to a third VIP that routes network traffic through BG 104c, and VLAN 114d can point to a fourth VIP that routes network traffic through BG 104d. In some examples, the VLANs are created, numbered, and populated sequentially, with even-numbered VLANs assigned to a first BG (104a) and odd-numbered VLANs assigned to a second BG (104b).

[0024] Although these multiple VRRP instances shift a large portion of the overall network traffic load from one BG to another, it is still possible for the load on BG 104a to be greater than the load on gateway 104b, or vice versa. Not all client devices utilize network bandwidth in the same way. For example, edge computing devices may use larger amounts of bandwidth on a generally constant basis to synchronize data with computing devices in other LANs, such as the core LAN 108. Employee laptops, on the other hand, may make occasional data transactions with other LANs but use bandwidth more consistently to access jobs and services on the Internet 106. Point-of-sale devices and kiosks, as another example, may primarily use bandwidth in the branch LAN and only occasionally access bandwidth in the wider network.Thus, even though the VLANs have been distributed between BG 104a and BG 104b, network performance may deteriorate if devices in a particular VLAN use a disproportionate amount of bandwidth.

[0025] Each BG 104 monitors the load on its hardware and logical resources due to the network traffic routed by the respective BG 104 and occasionally sends network traffic load information to the Network Orchestrator 110. In some examples, the BG 104 sends the raw load information to the Network Orchestrator 110, which calculates network traffic load factors from this information. In other examples, the BG 104 calculate network traffic load factors based on the network traffic load information and send these factors to the Network Orchestrator 110. The Network Orchestrator 110 then compares the network traffic load factors for each BG 104 to a network traffic load threshold (either a universal threshold or an individual threshold for each BG 104).For example, if the network traffic load factor for BG 104a is greater than the network traffic load threshold, then the network orchestrator 110 can begin correcting the congestion on BG 104a. Next, the network orchestrator 110 can calculate a network traffic load skew between BG 104a and BG 104b. The network traffic load skew represents the relative load on each BG 104. For example, if all VLANs are assigned to BG 104a and BG 104b is in idle mode (active-standby configuration), the network traffic load skew between BG 104a and BG 104b will be quite high. On the other hand, if the VLANs are evenly distributed between BGs 104a and 104b, and the client devices in each VLAN have approximately the same usage as the client devices in the other VLANs, the network traffic load skew is low.Network Orchestrator 110 can compare the network traffic skew between BG 104a and BG 104b against a skew threshold. If the skew exceeds the threshold, Network Orchestrator 110 will correct the congestion on BG 104a.

[0026] In some examples, the Network Orchestrator 110 can determine the network traffic load factor based on the corresponding number of devices in each VLAN. For example, BG 104a can be considered congested if the number of devices in the VLANs assigned to BG 104a exceeds a certain threshold.

[0027] The network orchestrator 110 may have several tools available to correct the congestion of BG 104a. Although these tools are described as having the network orchestrator 110 issue reconfiguration instructions to various network infrastructure devices of the SD-WAN 100, in some examples the network orchestrator 110 sends instructions to a network controller (resident on the branch gateway 104a or 104b) of the branch LAN 102, and the network controller then issues a separate instruction to the respective network infrastructure devices.

[0028] Such a tool reassigns the master BG to a VRRP instance. For example, network orchestrator 110 might determine that the network traffic loads of BG 104a and BG 104b would be better balanced by moving traffic from VLAN 114a from BG 104a to BG 104b. Network orchestrator 110 then issues an instruction to BG 104a and BG 104b to modify the VRRP instance associated with VLAN 114a to change the master BG from BG 104a to BG 104b. Later, when the network orchestrator 110 determines that the network conditions that necessitated moving VLAN 114a from BG 104a to BG 104b have ended, the network orchestrator 110 can send an instruction to BG 104a and BG 104b to change the VRRP instance associated with VLAN 114a to change the master BG from BG 104b to BG 104a.

[0029] Another such tool modifies how client devices are assigned to VLANs by certain infrastructure devices 112 that support receiving such instructions. Certain access points and switches can support receiving messages indicating that the BGs 104 are operating in an HA configuration, and can further support receiving messages that cause the infrastructure device 112 to change the procedure for assigning new client devices to VLANs and / or reassigning existing client devices to different VLANs. For example, the network orchestrator 110 can instruct the infrastructure device 112b to no longer assign new client devices to VLAN 114c because VLAN 114c is consuming a disproportionately large amount of the bandwidth of BG 104a.As another example, network orchestrator 110 can instruct infrastructure device 112a to move a kiosk device from VLAN 114a to VLAN 114b because the kiosk device, in combination with other kiosk devices, is consuming a disproportionately large amount of bandwidth from BG 104a. In yet another example, network orchestrator 110 can notify infrastructure devices 112 that BGs 104a and 104b are operating in an HA configuration, and infrastructure devices 112 can then adjust their client assignments to VLAN 114 based on this notification.

[0030] Although in Fig. Since Figure 1 (and the other figures of this disclosure) depicts a specific network topology with a specific number of LANs, a specific number of BGs, a specific number of network infrastructure devices, a specific number of client devices, and components of the SD-WAN 100 in a specific configuration, it is clear to the person skilled in the art that the SD-WAN 100 can be configured in one of many network topologies with any number of LANs and devices and still conform to the features of the present disclosure. For example, the present disclosure provides for any number of BGs in a LAN, any number of branch LANs, and any number of core LANs, and it provides for the network orchestrator to be located in any part of the SD-WAN 100.

[0031] Fig. 2A to 2B are simplified representations of the exemplary SD-WAN of the Fig. 1, which include the operation of the branch gateways of the branch LAN.

[0032] Fig. Figure 2A illustrates SD-WAN 100 in a simplified form, where VLANs 114 are unbalanced in the amount of network traffic load they generate. VLAN 114a generates an excessively large amount of network traffic load compared to the rest of VLANs 114. VLAN 114b generates less network traffic load compared to the rest of VLANs 114. VLANs 114c and 114d generate moderate amounts of network traffic load. As network traffic passes through BGs 104a and 104b, the network traffic load is measured. In some examples, each BG 104 applies an algorithm to determine a network traffic load factor from various attributes of the network traffic load information collected by the BGs 104. Each BG 104 then compares its corresponding network traffic load factors with network traffic load thresholds 202.For example, BG 104a has a network traffic load factor that is greater than the network traffic load threshold 202a, and BG 104b has a network traffic load factor that is lower than the network traffic load threshold 202b.

[0033] Since BG 104a has a network traffic load factor greater than the threshold 202a, BGs 104 can determine a network traffic load skew between BG 104a and BG 104b. This skew is compared to a load skew threshold to determine whether a correction of the BG 104 overload can be achieved. For example, if BG 104a were overloaded and BG 104b were only slightly less overloaded (resulting in a low skew), shifting the load from one BG to another can only shift the overload from BG 104a to BG 104b. However, if BG 104a is overloaded and BG 104b has significant available bandwidth (resulting in a strong skew), shifting the load from BG 104a to BG 104b can improve network performance.Since the network traffic load skew between BG 104a and BG 104b is high (not shown), the SD-WAN 100 corrects the overload of BG 104a.

[0034] In Fig. In example 2B, VLAN 114b was assigned to BG 104b. In some examples, BG 104 changed the VRRP instance assigned to VLAN 114b, so that traffic sent and received by devices on VLAN 114b was subsequently routed through BG 104b. As a result, the network traffic load factor for BG 104a became lower than the network traffic load threshold 202a, and the network traffic load factor for BG 104b did not exceed the network traffic load threshold 202b. Although VLAN 114a contributed more to the network traffic load of BG 104a, moving VLAN 114a to BG 104b would have overloaded BG 104b. Instead, all of VLAN 114 were analyzed and it was selected that VLAN 114b be moved to BG 104b, since the network traffic load contribution of VLAN 114b did not cause BG 104b to become overloaded.In some examples, a network traffic load factor is calculated for each VLAN 114, and based on predicted network traffic load factors for each BG 104, when the selected VLAN 114b is reassigned, a VLAN 114b is chosen for reassignment. In some examples, the predicted network traffic load factors for each BG 104 are compared with the corresponding network traffic load thresholds 202.

[0035] Fig. 3A to 3B are simplified representations of the exemplary SD-WAN of the Fig. 1, which include the operation of the branch gateways and infrastructure devices of the branch LAN.

[0036] Fig. Figure 3A illustrates an example SD-WAN comprising VLAN 114 with assigned client devices. For example, VLAN 114a might include point-of-sale devices and kiosks, VLAN 114b might include employee laptop computers, and VLANs 114c through 114d might include edge computing devices and desktop computers. VLANs 114a and 114c might be assigned to BG 104a, and VLANs 114b and 114d might be assigned to BG 104b. The network traffic load factor for BG 104a is greater than the network traffic load threshold 202a, so BG 104a is in a congested state. Although the BG 104a may not show any performance degradation from being in an overloaded state, new devices can be assigned to VLANs, which will prevent the BG 104a from being burdened with additional network traffic.

[0037] Fig. Figure 3B illustrates the assignment of a new device connected to infrastructure device 112b for communication. The new device is prevented from being assigned to VLAN 114c because network traffic for VLAN 114c is routed through BG 104a. Instead, the new device is assigned to VLAN 114d, and network traffic for the new device is routed through BG 104b.

[0038] Fig. Figure 4 is a flowchart illustrating an example of Procedure 400 for balancing network load across branch gateways of an SD-WAN. In some examples, Procedure 400 is executed by a network orchestrator of the SD-WAN. In other examples, Procedure 400 is executed by a branch gateway of the SD-WAN.

[0039] Block 402 collects network traffic load information for a first branch gateway and a second branch gateway. In some examples, the network traffic load information consists of a set of device counts for each VLAN assigned to the first gateway and a set of device counts for each VLAN assigned to the second gateway. In other examples, the network traffic load information may include branch gateway usage information, such as packet buffer usage, CPU usage, packet service resource usage, and so on.

[0040] Block 404 calculates network traffic load factors for the first and second branch gateways, based on the corresponding network traffic load information collected in block 402. In some examples, the network traffic load factor is the sum of the set of device counts for each VLAN assigned to the respective gateway. In other examples, the network traffic load factor is determined using a weighting algorithm to weight different branch gateway usage measurements for each branch gateway.

[0041] Block 406 determines whether the network traffic load factor of the first BG is greater than the network traffic load factor of the second BG. If so, procedure 400 proceeds to block 408. If not, procedure 400 terminates. In some examples, the network load factor of the first BG is also compared to a network load factor threshold, so the procedure does not continue if the network load of the first BG is not above a certain critical level.

[0042] Block 408 determines whether the network traffic skewness between the first and second BG exceeds a threshold. In some examples, the network traffic skewness is the difference between the network traffic skewness factor of the first BG and the network traffic skewness factor of the second BG. For instance, if both the first and second BG have high network traffic loads, the network traffic skewness may be low, even though the determination in Block 406 is "Yes".

[0043] Block 410 selects the first VLAN from a set of virtual local area networks (VLANs) within a local area network (LAN). The first VLAN can be selected based on network traffic load characteristics (e.g., number of users, latency, jitter, QoS requirements, etc.). It can also be selected because it is assigned to the first building block (BG). In some examples, the first VLAN might be selected because moving it from the first BG to the second BG could resolve excessive BG resource utilization.

[0044] Block 412 sends a message to the first and second Business Gateways (or alternatively, to an assigned lead Business Gateway), assigning the master role for the first VLAN to the second Business Gateway. In some examples, the message is sent from a network orchestrator on a cloud device to the LAN's Business Gateways. The message may be a VRRP reconfiguration message, which causes the VRRP instance for the first VLAN to be reconfigured with different costs, making the second Business Gateway the master of the VRRP instance.

[0045] Fig. Figure 5 is a flowchart illustrating another exemplary method for balancing network load across branch gateways of an SD-WAN. In some examples, Method 500 is executed by a network orchestrator of an SD-WAN.

[0046] Block 502 collects network traffic load information for a first branch gateway and a second branch gateway. In some examples, the network traffic load information consists of a set of device counts for each VLAN assigned to the first gateway and a set of device counts for each VLAN assigned to the second gateway. In other examples, the network traffic load information may include branch gateway usage information, such as packet buffer usage, CPU usage, packet service resource usage, and so on.

[0047] Block 504 calculates network traffic load factors for the first and second branch gateways, based on the corresponding network traffic load information collected in block 502. In some examples, the network traffic load factor is the sum of the set of device counts for each VLAN assigned to the respective gateway. In other examples, the network traffic load factor is determined using a weighting algorithm to weight different branch gateway usage measurements for each branch gateway.

[0048] Block 506 determines whether the network traffic load factor of the first BG is higher than the network traffic load factor of the second BG. If so, procedure 500 proceeds to block 508. If not, procedure 500 terminates. In some examples, the network load factor of the first BG is also compared to a network load factor threshold, so the procedure does not continue if the network load of the first BG is not above a certain critical level.

[0049] Block 508 determines whether the network traffic skewness between the first and second BG exceeds a threshold. In some examples, the network traffic skewness is the difference between the network traffic skewness factor of the first BG and the network traffic skewness factor of the second BG. For instance, if both the first and second BG have high network traffic loads, the network traffic skewness may be low, even though the determination in Block 506 is "Yes".

[0050] Block 510 selects the first VLAN from a group of virtual local area networks (VLANs) within a local area network (LAN). The first VLAN can be selected based on network traffic load characteristics (e.g., number of users, latency, jitter, QoS requirements, etc.). It can also be selected because it is assigned to the first building block (BG). In some cases, the first VLAN might be selected because moving it from the first BG to the second BG could resolve excessive BG resource utilization.

[0051] Block 512 sends a message to the first and second Business Gateways (or alternatively, to an assigned lead Business Gateway), assigning the master role for the first VLAN to the second Business Gateway. In some examples, the message is sent from a network orchestrator on a cloud device to the LAN's Business Gateways. The message may be a VRRP reconfiguration message, which causes the VRRP instance for the first VLAN to be reconfigured with different costs, making the second Business Gateway the master of the VRRP instance.

[0052] Block 514 calculates updated network traffic load factors for the first and second branch gateways, based on updated network traffic load information. In some examples, the updated network traffic load factor is the sum of the set of device counts for each VLAN assigned to the respective gateway. In other examples, the updated network traffic load factor is determined using a weighting algorithm to weight different branch gateway usage measurements for each branch gateway.

[0053] Block 516 determines whether the updated network traffic load factor of the first BG is greater than the updated network traffic load factor of the second BG. If so, procedure 500 proceeds to block 518. If not, procedure 500 terminates. In some examples, the updated network load factor of the first BG is also compared to a network load factor threshold, so the procedure does not continue if the updated network load of the first BG is not above a certain critical level.

[0054] Block 518 determines whether a second network traffic skew between the first and second BG is greater than a threshold. In some examples, the second network traffic skew is the difference between the updated network traffic skew factor of the first BG and the updated network traffic skew factor of the second BG. For example, if both the first and second BG have high network traffic loads, the second network traffic skew may be low, even though the determination in Block 516 is "Yes".

[0055] Block 520 sends a message to the first and second Business Gateways (or alternatively, to an assigned lead Business Gateway), assigning the master role for the first VLAN to the first Business Gateway. In some examples, the message is sent from a network orchestrator on a cloud device to the LAN's Business Gateways. The message may be a VRRP reconfiguration message, which causes the VRRP instance for the first VLAN to be reconfigured with different costs, making the first Business Gateway the master of the VRRP instance.

[0056] Fig. Figure 6 illustrates a simplified example SD-WAN comprising a network orchestrator and branch gateways. The SD-WAN 600 includes a network orchestrator 602, which is connected via the Internet 612 to branch gateways 604a to 604b. The network orchestrator 602 includes the processor 606a and the memory 608a, which contains instructions 610a. Branch gateway 604a includes the processor 606b and the memory 608b, which contains instructions 610b. Branch gateway 604b includes the processor 606c and the memory 608c, which contains instructions 610c.

[0057] Network Orchestrator 602 comprises instructions 610a in memory 608a, which, when executed by processor 606a, cause Network Orchestrator 602 to perform specific actions. For example, instructions 610a might include instructions for monitoring an initial set of load parameters for branch gateway 604a, comprising a utilization factor for a deep packet inspection service provided by BG 604a and a utilization factor for a content classification service of BG 604a. In some examples, the utilization factors for the services are weighted hardware utilization measures for each service (e.g., CPU utilization, network interface card utilization, etc.).

[0058] The 610a commands may also include commands to monitor a second set of load parameters for the branch gateway 604b, comprising a utilization factor for a deep packet inspection service provided by BG 604b and a utilization factor for a content classification service of BG 604b. In some examples, the service utilization factors are weighted hardware utilization metrics for each service (e.g., CPU utilization, network interface card utilization, etc.).

[0059] The 610a instructions can also include instructions for periodically calculating a network traffic load for BG 604a based on the first set of load parameters and a network traffic load for BG 604b based on the second set of load parameters. The network traffic loads for BGs 604a to 604b can be load factors (i.e., numerical indicators of the load in each BG 604).

[0060] The 610a commands can also be commands to update a network traffic skew, which indicates the difference in traffic loads between BG 604a and BG 604b. In some examples, the network traffic skew is the numerical difference between the traffic loads. The greater the difference in load between BG 604a and BG 604b, the greater the network traffic skew.

[0061] The 610a commands can also include commands to determine that the network traffic load for BG 604a exceeds a congestion threshold. Even if the loads between BG 604a and BG 604b are unbalanced, there is no need to improve performance by redistributing clients across BGs 604 if neither BG 604 is congested. Therefore, if BG 604a is not congested, the network orchestrator 602 does not need to attempt to reduce the load on BG 604a by moving some clients to BG 604b.

[0062] The 610a commands can also include commands to determine that the updated network traffic load skew is greater than a skew threshold. Even if BG 604a is congested, BG g04b may also be congested, and there will be no performance improvement from redistributing clients across BGs 604. Therefore, if the updated network traffic load skew is less than the skew threshold, the network orchestrator 602 does not need to attempt to reduce the load on BG 604a by moving some clients to BG 604b.

[0063] Commands 610a can also include commands to send a message to BGs 604, which can forward the message to a switch, access point (AP), or other infrastructure device connected to communicate with a client device in a first VLAN. The first VLAN can be configured to route traffic through BG 604a. The message can instruct the infrastructure device to reassign the client device to a second VLAN. The second VLAN can be configured to route traffic through BG 604b. Alternatively, the message can also instruct the infrastructure device to stop assigning new clients to the first VLAN, thus preventing an imbalance in network traffic load generated by the first VLAN compared to the second VLAN (and other VLANs).

[0064] The branch gateway 604a contains the instructions 610b in memory 608b, which, when executed by the processor 606b, cause the branch gateway 604a to perform certain actions. The instructions 610b may include commands to enable VRRP and to set up VRRP instances for each VLAN of the branch LAN. The instructions 610b may also include commands to compile load parameters, including the number of users per assigned VLAN, jitter, packet loss, and latency for data packets routed through the branch gateway 604a, hardware device usage (e.g., CPU usage, NIC usage, etc.) of the branch gateway 604a, and service usage for packet services (e.g., DPI, content classification, etc.) of the branch gateway 604a.Commands 610b can also include commands to change VRRP instances for a specific VLAN based on a message from network orchestrator 602, so that the master of the VRRP instance switches back and forth between BG 604a and BG 604b.

[0065] The branch gateway 604b contains the 610c instructions in memory 608c, which, when executed by the processor 606c, cause the branch gateway 604b to perform certain actions. The 610c instructions can include commands to enable VRRP and to set up VRRP instances for each VLAN of the branch LAN. The 610c instructions can also include commands to compile load parameters, including the number of users per assigned VLAN, jitter, packet loss, and latency for data packets routed through the 604b, hardware device usage (e.g., CPU usage, NIC usage, etc.) of the 604b, and packet service usage (e.g., DPI, content classification, etc.) of the 604b.The 610c commands can also include commands to change VRRP instances for a specific VLAN based on a message from the network orchestrator 602, so that the master of the VRRP instance jumps back and forth from BG 604a to BG 604b or from BG 604b to BG 604a.

[0066] Although Fig. Six devices are shown, configured with a single processor 606 and a memory 608, any suitable computing configuration is provided, including multiple processors, alternative data storage models, distributed computing, infrastructure as a service, etc.

[0067] Branch gateways are network infrastructure devices located at the edge of a branch LAN. Often, branch gateways are routers that provide connection points between the LAN and a wider network, either directly to other LANs within the WAN via dedicated network links (e.g., MPLS) or via the internet to other LANs within the WAN through links provided by an internet service provider. Many branch gateways can establish multiple upstream connections to the WAN, including connections to multiple other LAN locations and redundant upstream connections to a single other LAN location. Branch gateways often also include network control for the branch LAN. In such an example, a branch gateway used in an SD-WAN might include network control, which is logically partitioned by an embedded router.The network controller can control infrastructure devices of the branch LAN and can receive routing instructions from a network orchestrator.

[0068] A network orchestrator is a service (e.g., commands stored on a non-volatile, machine-readable medium and executed by a processing circuit system) that runs on a computing device, managing switching and routing across an SD-WAN. In some examples, the network orchestrator runs on a computing device in a core site LAN of the SD-WAN. In other examples, the network orchestrator runs on a cloud computing device. The network orchestrator can be provided to the SD-WAN as a service (as a Service, aaS). The network orchestrator aggregates network operational information from various network infrastructure devices within the SD-WAN, such as network traffic load information, network topology information, network usage information, and so on.The network orchestrator then sends instructions to various network infrastructure devices of the SD-WAN to change the network topology and network routing in order to achieve various goals regarding network efficiency and effectiveness.

[0069] A virtual local area network (VLAN) is a logical subdivision of a portion of a wide area network (WAN). A VLAN can be contained within a specific LAN of the WAN or it can span multiple LANs within the WAN. VLANs are implemented at Layer 2 of the OSI model (the data link layer) and, among other advantages, improve network configurability as the network size changes. VLAN-enabled infrastructure devices can assign VLANs on a per-port basis or can tag specific data frames with information that assigns the frames to their corresponding VLANs. VLANs can be used to group related devices, balance loads on specific network infrastructure devices, apply security and routing strategies on a broad scale, implement Quality of Service (QoS), and more.

[0070] A network infrastructure device (NET) is a device that receives network traffic and forwards it to a destination. NETs can include controllers, access points, switches, routers, bridges, and gateways, among other devices. Certain NETs can be SDN-enabled, allowing them to receive network instructions from a controller or orchestrator and adjust their operations based on those instructions. Some NETs perform packet services, such as application classification and deep packet inspection, on certain network traffic received at the NET. Some NETs monitor load parameters for various physical and logical resources of the NET and report this load information to a controller or orchestrator.

[0071] The features of the present disclosure can be implemented using a variety of special devices, which exhibit a variety of different technologies and properties. For example, features comprising instructions to be executed by a processing circuit system can store instructions in a cache memory of the processing circuit system, in random access memory (RAM), in a hard disk drive, in a removable disk (e.g., a CD-ROM), in a field-programmable gate array (FPGA), in read-only memory (ROM), or in any other non-volatile, computer-readable medium, as is applicable to the specific device and the specific exemplary implementation.As is evident to the person skilled in the art, the features of the present disclosure are not altered by technology, whether known or unknown, or by the characteristics of specific devices on which the features are realized. Any modifications or adaptations necessary to realize the features of the present disclosure on a specific device or in a specific example are obvious to the person skilled in the art.

[0072] Although the present revelation has been described in detail, it is understood that various changes, substitutions, and modifications may be made without departing from the idea and scope of the revelation. The use of the words "may" or "can" in relation to features of the revelation indicates that certain examples include the feature and certain other examples do not, as is appropriate in the given context. The use of the words "or" and "and" in relation to features of the revelation indicates that examples may include any combination of the listed features, as is appropriate in the given context.

[0073] Expressions and bracketed passages beginning with "e.g." or "i.e." are used solely to give examples for the purpose of clarification. The revelation is not intended to be limited by the examples given in these expressions and bracketed passages. The scope and understanding of the present revelation may include certain examples not revealed in such expressions and bracketed passages.

Claims

[1] A system, encompassing: a processor (600a, 600b, 600c) and a non-transitory storage medium (608a, 608b, 608c) that stores instructions (610a, 610b, 610c) which, when executed on the processor, cause the system to: determines that a network traffic load of a first branch gateway, BG, (104a, 604a) of a local network, LAN, (102) is different from a network traffic load of a second BG (104b, 604b) of the LAN; determines that a first network traffic load skew between the first BG and the second BG is greater than a first threshold; selects a first virtual local area network, VLAN, from a set of VLANs (114a, 114b, 114c, 114d) of the LAN, setting up a first virtual router for the first VLAN and a second virtual router for a second VLAN from the set of VLANs, with network traffic from client devices of the first VLAN, sent to the first virtual router, being routed through the first BG, based on the first BG being a master BG for the first VLAN according to a configuration of the first virtual router; and sends a reconfiguration message to at least one of the first BG and the second BG, which changes the configuration of the first virtual router set up for the first VLAN, whereby by changing the configuration of the first virtual router the second BG is reassigned as the master BG for the first VLAN, whereby after the change in the configuration of the first virtual router the network traffic from client devices of the first VLAN is henceforth routed via the second BG. [2] System according to claim 1, wherein the set of VLANs comprises a first subset of VLANs and a second subset of VLANs, and wherein for each VLAN of the first subset of VLANs a master role is assigned to the first BG and for each VLAN of the second subset of VLANs a master role is assigned to the second BG. [3] System according to claim 2, wherein the reconfiguration message moves the first VLAN from the first subset of VLANs to the subset of VLANs. [4] System according to claim 1, wherein the instructions, when executed on the processor, cause the system to: determines that an updated network traffic load of the first BG is lower than an updated network traffic load of the second BG; and In response to determining that the updated network traffic load of the first BG is less than the updated traffic load of the second BG, another reconfiguration message is sent to at least one of the first BG and the second BG, reassigning the first BG as the master BG for the first VLAN. [5] System according to claim 1, wherein the network traffic load of the first BG is based on hardware load factors that represent the use of the hardware in the first BG. [6] System according to claim 1, wherein the reconfiguration message changes the first virtual router by changing a master BG for the virtual router from the first BG to the second BG. [7] System according to claim 4, wherein the instructions, when executed on the processor, cause the system to: determined that a second network traffic load skew between the first BG and the second BG is less than a second threshold; and in response to determining that the updated network traffic load of the first BG is less than the updated network traffic load of the second BG, and determining that the second network traffic load skew between the first BG and the second BG is less than the second threshold, sending a further reconfiguration message. [8] System according to claim 1, wherein the first network traffic load skew indicates a difference in the traffic loads between the first BG and the second BG and the instructions, when executed on the processor, cause the system to: a first set of load parameters for the first BG is monitored, comprising a utilization factor for a first BG deep packet inspection service and a utilization factor for a first BG content classification service; a second set of load parameters for the second BG is monitored, comprising a utilization factor for a deep packet inspection service of the second BG and a utilization factor for a content classification service of the second BG; The network traffic load of the first BG is calculated based on the first set of load parameters, and the network traffic load of the second BG is calculated based on the second set of load parameters; and determined that the network traffic load of the first BG is greater than a load threshold (202a). [9] Procedures, comprehensive: Collect traffic congestion information for a first branch gateway, BG, (104a, 604a) and a second BG (104b, 604b); Calculate a network traffic load factor for the first BG and a network traffic load factor for the second BG based on the traffic load information; Determine that the network traffic load factor for the first BG is higher than the network traffic load factor for the second BG; Determine that an initial network traffic load skew between the first BG and the second BG is greater than an initial threshold; Selecting a first virtual local area network, VLAN, a set of VLANs (114a, 114b, 114c, 114d) of a local area network, LAN, (102), where a first virtual router is set up for the first VLAN and a second virtual router is set up for a second VLAN from the set of VLANs, where network traffic from client devices of the first VLAN, which is sent to the first virtual router, is routed through the first BG, based on the fact that the first BG is a master BG for the first VLAN according to a configuration of the first virtual router, and where the first VLAN is selected partly based on the number of client devices in the first VLAN; and Sending, to at least one of the first BG and the second BG, a reconfiguration message that changes the configuration of the first virtual router assigned to the first VLAN, wherein the change in the configuration of the first virtual router reassigns the second BG as the master BG for the first VLAN, wherein after the change in the configuration of the first virtual router, network traffic of the client devices of the first VLAN is routed through the second BG. [10] Method according to claim 9, wherein the first BG and the second BG are members of a Virtual Router Redundancy Protocol (VRRP) group and each VLAN of the set of VLANs is assigned to a corresponding VRRP instance which assigns a master BG to the respective VLAN. [11] The method of claim 9, further comprising: Calculate updated network traffic load factors for the first BG and the second BG based on updated traffic load information; Determine that an updated network traffic load factor for the first BG is lower than an updated network traffic load factor for the second BG; Determine that a second network traffic load skew between the first BG and the second BG is less than a second threshold; and In response to determining that the updated network traffic load factor for the first BG is lower than the updated network traffic load factor for the second BG, and determining that the second network traffic load skew between the first BG and the second BG is less than the second threshold, send, to at least one of the first BG and the second BG, a further reconfiguration message that reassigns the first BG as the master BG for the first VLAN. [12] The method of claim 9, further comprising: Calculate updated network traffic load factors for the first BG and the second BG based on respective updated network traffic load information; Determine that an updated network traffic load factor for the first BG is still higher than an updated network traffic load factor for the second BG; Determine that a second network traffic load skew between the first BG and the second BG is less than a second threshold; and In response to determining that the second network traffic load skew between the first BG and the second BG is less than the first threshold, send, to at least one of the first BG and the second BG, a further reconfiguration message that reassigns the first BG as the master BG for the first VLAN. [13] Method according to claim 9, wherein the collection of network traffic load information comprises the collection of a first set of load parameters for the first BG, comprising a utilization factor for a deep packet inspection service of the first BG and a utilization factor for a content classification service of the first BG, and the collection of a second set of load parameters for the second BG, comprising a utilization factor for a deep packet inspection service of the second BG and a utilization factor for a content classification service of the second BG, and wherein the calculation of the network traffic load factor for the first BG is based on the first set of load parameters and the calculation of the network traffic load factor for the second BG is based on the second set of load parameters. [14] Method according to claim 9, wherein sending the reconfiguration message to at least one of the first BG and the second BG comprises: Send the reconfiguration message to both the first BG and the second BG, or Sending the reconfiguration message to a leading BG, which is selected from the first BG and the second BG. [15] Non-transitory computer-readable medium (608a, 608b, 608c) comprising instructions (610a, 610b, 610c) which, when executed by a processing circuit system, cause a system to: a first set of load parameters for a first branch gateway, BG, (104a, 604a), comprising a utilization factor for a first BG deep packet inspection service and a utilization factor for a first BG content classification service; a second set of load parameters for a second BG (104b, 604b) is monitored, comprising a utilization factor for a deep packet inspection service of the second BG and a utilization factor for a content classification service of the second BG; A network traffic load of the first BG is calculated based on the first set of load parameters, and a network traffic load of the second BG is calculated based on the second set of load parameters; selects a first virtual local area network, VLAN, from a set of VLANs (114a, 114b, 114c, 114d), with network traffic from client devices of the first VLAN being routed through the first BG, based on the first BG being a master BG for the first VLAN; a network traffic load skew is determined, which indicates a difference in traffic loads between the first BG and the second BG; determines that the network traffic load of the first BG is higher than a first threshold (202a); determines that the network traffic load skew is greater than a second threshold; and sends a reconfiguration message to at least one of the first BG and the second BG, which changes a virtual router assigned to the first VLAN, whereby the change to the virtual router reassigns the second BG as the master BG for the first VLAN, wherein after the change to the virtual router the network traffic of the client devices of the first VLAN is routed through the second BG. [16] Non-transient computer-readable medium according to claim 15, wherein the first set of load parameters and the second set of load parameters comprise at least one of a number of users, jitter, number of lost packets or latency. [17] Non-transitory computer-readable medium according to claim 15, wherein the first BG and the second BG are each a gateway of a software-defined long-range network, SD-WAN, which connects a branch LAN (102) for communication with another point of an SD-WAN (100, 200, 300, 600). [18] Non-transitory computer-readable medium according to claim 15, wherein the reconfiguration message changes the virtual router by changing a master BG for the virtual router from the first BG to the second BG. [19] Non-transitory computer-readable medium according to claim 15, wherein the first BG and the second BG are members of a Virtual Router Redundancy Protocol (VRRP) group and each VLAN of the set of VLANs is assigned to a corresponding VRRP instance which assigns a master BG to the respective VLAN. [20] Non-transitory computer-readable medium according to claim 15, wherein the commands, when executed by the processing circuit system, cause the system to: determines that an updated network traffic load skew between the first BG and the second BG is less than the second threshold; and In response to determining that the updated network traffic load skew between the first BG and the second BG is less than the second threshold, another reconfiguration message is sent to at least one of the first BG and the second BG, assigning the first BG as the master BG for the first VLAN.

Images