Gateway device, method and storage medium for fast correction of WAN or LAN traffic loss

Abstract

Systems and methods are provided for monitoring network accessibility from gateway devices via physical uplinks and virtual uplinks. For example, before a WAN health check is completed, the system can mark the physical uplink as reachable and the virtual uplink as unreachable. The system can update the default gateway to the physical uplink and remove the virtual uplink, allowing the default gateway to use the physical uplink to maintain data transmission from the gateway device to the network via the physical uplink, and suspend or stop connections via the virtual uplink. The system can continue monitoring network accessibility from the gateway device via the virtual uplink by listening to tunnel heartbeats from the virtual uplink.

Description

[0001] Cross-reference to related applications

[0002] This application is filed based on three co-pending applications: U.S. Patent Application No. 18 / 161,304 (Case No.: 710230916US01; 61CT-346660), U.S. Patent Application No. 18 / 161,323 (Case No.: 710230916US02; 61CT-365375), and U.S. Patent Application No. 18 / 161,341 (Case No.: 710230916US03; 61CT-365376), the entire contents of which are incorporated herein by reference. Background Technology

[0003] WAN uplink solutions in Software-Defined Wide Area Networks (WANs) (SDWANs) can include physical uplinks and virtual uplinks. For example, if there are two Gateway Controllers (GWs), each with a physical uplink to an Internet Service Provider (ISP), each gateway can use the uplink of the other peer gateway as a virtual uplink by forming a Generic Routing Encapsulation (GRE) tunnel from the downstream Local Area Network (LAN). WAN health checks can be performed by a Health Check Manager (HCM) based on configured parameters.

[0004] Such a network topology may suffer packet latency and data loss on either the physical or virtual uplink. Typically, to accommodate WAN packet latency and loss, health check probes can be sent at ten-second intervals, with probe responses expected within a ten-second window. Over high-speed internet, responses can be received within milliseconds. If a response is received within milliseconds after the probe, but a complete WAN outage occurs immediately after receiving the response, the outage will not be detected until the next 10-second window following the next probe. If three probe retries are attempted, it could take up to 40 seconds to detect and acknowledge such an outage. Attached Figure Description

[0005] This disclosure will be described in detail with reference to the following figures, according to one or more different embodiments. The accompanying drawings are for illustrative purposes only and illustrate only typical or exemplary embodiments.

[0006] Figure 1 This is a diagram illustrating an example computing environment with at least two gateways in a branch having WAN and LAN communication channels, according to embodiments described herein;

[0007] Figure 2This is a diagram illustrating an illustrative process for establishing a virtual LAN uplink in a computing environment having at least two gateway controllers in a branch, according to an embodiment described herein;

[0008] Figure 3 This is a diagram illustrating an example of a topology using a LAN connection to remedy WAN service loss according to embodiments described herein;

[0009] Figure 4 This is based on the use of the embodiments described herein. Figure 3 An illustrative communication flow of the topology in the code;

[0010] Figure 5 This is a diagram illustrating an example of a topology using a WAN connection to remedy lost LAN services according to embodiments described herein;

[0011] Figure 6 This is a flowchart illustrating an example of a health check manager process according to an embodiment described herein;

[0012] Figure 7 This is a diagram illustrating an example of using a cluster manager to resolve cluster splitting issues and remediate LAN service loss topology according to embodiments described herein;

[0013] Figure 8 This is a flowchart illustrating an example of a virtual uplink tunnel heartbeat process according to an embodiment described herein;

[0014] Figure 9 This is a flowchart illustrating an example of a cluster manager process according to an embodiment described herein;

[0015] Figure 10 This is a flowchart illustrating an example of a cluster manager process according to an embodiment described herein;

[0016] Figure 11 This is a diagram illustrating an example of using data path monitoring to remedy a lost WAN or LAN service topology according to embodiments described herein;

[0017] Figure 12 An example process for identifying WAN or LAN service loss using data path monitoring according to the above embodiments is shown;

[0018] Figure 13 A marking for a probe according to the above embodiment is shown;

[0019] Figure 14 An illustrative example of a data service using flags according to the above embodiments is shown;

[0020] Figure 15These are example computing components that can be used to implement the various features of the embodiments described in this disclosure;

[0021] Figure 16 These are example computing components that can be used to implement the various features of the embodiments described in this disclosure;

[0022] Figure 17 These are example computing components that can be used to implement the various features of the embodiments described in this disclosure; and

[0023] Figure 18 These are example computing components that can be used to implement the various features of the embodiments described in this disclosure.

[0024] These accompanying drawings are not exhaustive and do not limit this disclosure to the precise form disclosed. Detailed Implementation

[0025] As described above, gateways typically perform WAN health checks by sending probes to the WAN IP on both the physical and virtual uplinks (e.g., by a health check manager). Responses can be received on both the physical and virtual uplinks, and WAN traffic can be balanced between the physical and virtual uplinks in each gateway (e.g., by a load balancing engine). However, this example of the technique is designed with the understanding that WAN health check timers are typically set high to accommodate packet loss and latency. For example, a typical WAN health check configuration might involve a 10-second probe interval and three retries (i.e., connection failure is only acknowledged after three consecutive probes at 10-second intervals). For instance, a first gateway might send a probe packet on the virtual uplink at 10:00:00 (hours:minutes:seconds). A successful response to the probe packet can be received within milliseconds. Then, the first gateway might lose connection to a second gateway (e.g., via the virtual uplink) at 10:00:02. The next probe might not be sent until 10:00:10. If the connectivity issue persists, the attempt may fail again, and two more attempts will be made at 10.00.20 and 10.00.30 before confirming the failure and rerouting the traffic. This means that the first gateway may lose WAN traffic on the virtual uplink for 38 seconds or longer (e.g., if the retry and frequency values ​​correspond to the illustrative values ​​provided in this document). As shown in the example, the 10-second frequency value can be a high-frequency value relative to other options.

[0026] In this context, this technology provides systems and methods for proactively checking WAN health to identify connectivity issues and for mitigating WAN service loss on virtual uplinks (e.g., caused by poor LAN network connectivity) by utilizing physical uplinks when virtual uplinks are down. As will be described in more detail below, this technology can utilize “heartbeat” probes used by the internal LAN network to perform these more frequent / proactive WAN health checks. In other words, the system of this technology can use LAN connections between devices to resolve WAN connectivity issues and minimize overall service loss. For example, such a system can use one or more LAN-related heartbeat probes to perform proactive LAN health checks, detecting network problems faster than existing WAN health checks. That is, by leveraging internal LAN network infrastructure already used for other purposes, the system of this technology can perform WAN health checks faster / more frequently than existing WAN health checks / WAN HCMs.

[0027] As described above, and because the examples of this technique are designed for illustrative purposes, WAN health check timers are typically set high to accommodate packet loss and latency. However, for internal LAN networks, packet loss and latency are negligible. In internal LAN networks, it is not necessary to set retry and frequency values ​​high. For example, a first-gateway WAN high availability (HA) virtual LAN (VLAN) (e.g., a virtual uplink tunnel) might send a probe per second, attempting to re-establish the connection three times. Repeated probes at 1-second intervals can be described as a "heartbeat." For example, a 1-second interval could be an example of a relatively low-frequency interval (e.g., some LAN-related heartbeat probes operate at millisecond frequencies).

[0028] In some examples of this technology, if a probe failure persists across multiple LAN-related heartbeats, the HCM procedure (e.g., a tunneled HCM procedure) can immediately notify the uplink manager to suspend the virtual uplink from the WAN uplink load balancer in the gateway. This suspension can be achieved by notifying the Fast Path Application (“FPAPPS”) procedure to remove any routes via the virtual uplink without waiting for the typical 38-40 second WAN health check timer to complete. The FPAPPS procedure can then handle removing and / or adding routes to the virtual uplink. Sessions can then only be initiated on the physical uplink to the first gateway accessible to the WAN due to the suspension of other routes.

[0029] In various examples, the system of this technology can mark the physical uplink as reachable and the virtual uplink as unreachable before WAN health checks are completed. The system can update the default gateway to the physical uplink and remove the virtual uplink, allowing the default gateway to use the physical uplink to maintain data transmission from the gateway device to the network via the physical uplink, and suspend or stop connections via the virtual uplink. The system can continue to monitor network accessibility from the gateway device via the virtual uplink by listening to tunnel heartbeats from the virtual uplink.

[0030] In some examples, the system of this technology can monitor network accessibility from gateway devices in a gateway device cluster. For example, once the gateway device cluster is formed, the system can initiate a cluster heartbeat process. When a response is received from the first gateway (in the gateway cluster) instead of the second gateway (in the gateway cluster) within a threshold time period, the system can notify the uplink about network problems. The uplink manager can be configured to suspend virtual uplinks as unreachable, suspend WAN health checks, update the default gateway to remove virtual uplinks, and utilize physical uplinks outside the gateway device cluster as the default connection to a first network accessible by the ISP via the first gateway device, while simultaneously maintaining data transmission from physical uplinks outside the gateway device cluster to the first network.

[0031] In some examples, the system of this technology can use a data session table to monitor network accessibility from a gateway device. For example, the system can determine a subset of shortest-interval heartbeats for at least two types of heartbeat traffic detected between a first gateway device and a second gateway device. The first gateway device is communicatively connected to the second gateway device via a LAN connection and a virtual uplink. The system can then determine a heartbeat response loss from the second gateway device to the first gateway device for two of the at least two types of heartbeat traffic in the subset of shortest-interval heartbeats. Determining a heartbeat response loss from the second gateway device to the first gateway device for two of the at least two types of heartbeat traffic in the subset of shortest-interval heartbeats can include monitoring forward and reverse session entries in the data session table to detect reverse session entries associated with the first and second gateway devices, and that two of the at least two types of heartbeat traffic in the subset of shortest-interval heartbeats are marked as out of sync with forward session entries. Upon determining a heartbeat response loss from the second gateway device to the first gateway device for two of the at least two types of heartbeat traffic in the subset of shortest-interval heartbeats, the system can suspend the virtual uplink between the first and second gateway devices.

[0032] Figure 1This is a diagram illustrating an example computing environment with at least two gateways in a branch having WAN and LAN communication channels, according to embodiments described herein. In Figure 100, a branch or site location 110 is provided. Branch 110 includes a set of gateways 112 (shown as first gateway 112A and second gateway 112B), access points (APs) 116 (shown as first AP 116A and second AP 116B), routers 120 (shown as first router 120A and second router 120B), and a switch 162, which may be located downstream of gateways 112. Internet service providers (ISPs) 130, 132 can be located via… Figure 1 One or more devices (including via a second AP 116B) provide client 114 with access to the Internet 170 or some other communication network. ISPs 130, 132 may constitute a WAN 160 controlled via SD-WAN and a second endpoint device, in which case the second endpoint device may be VPNC 154 of data center 150 (which receives traffic from traffic source 152).

[0033] Service flow 152 can be any data transmission addressed to one or more receivers / endpoints (e.g., streaming media, information transmission, etc.). Figure 1 In the example shown, only one service flow (i.e., service flow 152) is shown in data center source 150. However, in other examples, data center 150 may include any number of service flows, and branch 110 may include any number of service sources, receivers, etc.

[0034] Typically, a VPNC, such as VPNC 154, can refer to a hardware or software application used to connect a VPN. As shown in the figure, data center 150 includes VPNC 154. Therefore, VPNC 154 can be used to send data associated with traffic flow 152 to one or more branches, in this example, to the group of gateways 112 (such as those orchestrated by overlay tunnel orchestrator 142) of branch 110.

[0035] SD-WAN 140 can be a cloud-based SD-WAN technology platform that includes centralized services capable of performing orchestration operations within a given WAN (e.g., WAN160). Typically, SD-WAN orchestration can refer to multiple centralized management services that provide cloud-delivered WAN control and management. In some examples, SD-WAN 140 may include additional centralized network management services. Therefore, various sub-services can reside within SD-WAN 140. As shown in the figure, SD-WAN 140 includes an overlay tunnel orchestrator 142, which can be a sub-service of SD-WAN orchestrator 144, capable of orchestrating routes, orchestrating key exchanges, creating and managing clusters, etc., in addition to orchestrating tunnels.

[0036] Overlay tunnel orchestrator 142 can be a central management entity that orchestrates service routes between data center 150 and branch 110 via connection 146 (shown as a first connection 146a between overlay tunnel orchestrator 142 and VPNC 154, a second connection 146b between overlay tunnel orchestrator 142 and first gateway 112A, and a third connection 146c between overlay tunnel orchestrator 142 and first AP 116A). For example, overlay tunnel orchestrator 142 may locally include a tunnel manager process to form overlay tunnels, then generate messages through the overlay tunnels and send the messages to each endpoint (VPNC 154, first gateway 112A, and first AP 116A). Each endpoint may include a similar tunnel proxy process to receive and parse messages from overlay tunnel orchestrator 142 via the overlay tunnels. To accomplish this task, overlay tunnel orchestrator 142 can identify or manage various aspects of the network topology / configuration and the needs of the network endpoints. Based on this information, the overlay tunnel orchestrator 142 can then orchestrate appropriate routes between the VPNC and the gateway, as well as routes to the receivers of interest.

[0037] As a central management entity integrated within SD-WAN 140, Overlay Tunnel Orchestrator 142 can collect this information and make these determinations in a manner that reduces WAN bandwidth consumption. In other words, centralized decision-making within Overlay Tunnel Orchestrator 142 significantly reduces the number of communications / decision-making processes required for transmitting services (such as multicast services) within, for example, large-scale SD branch deployments. As mentioned above, under the decentralized approach used in the prior art, much of the aforementioned information would be transmitted between various network devices / nodes (e.g., routers, branch gateways, VPNCs) of the network responsible for transmitting services. Overlay Tunnel Orchestrator 142 can obtain certain network configuration / topology information from another service(s) of SD-WAN Orchestrator 144 and information related to the needs of network receivers / hosts from designated branch gateway leaders.

[0038] For example, SD-WAN orchestrator 144 may include a Container as a Service (CaaS) service (not shown), which can refer to a cloud-based service that provides an organization with a way to manage its virtualized applications, clusters, and containers. CaaS may include a container orchestration engine for running and maintaining the infrastructure between an organization's clusters. CaaS may also manage clusters associated with branch gateways, which may be referred to as BG clusters. Within a given BG cluster, there may be a cluster leader. As will be described in more detail below, the leader of a given BG cluster may (a) issue a request to overlay multicast orchestrator 142 to join or leave a multicast group; and (b) receive multicast traffic from one of the VPNCs located in data center 150. By leveraging existing cloud-based service infrastructure and features, examples of currently disclosed technologies can enhance multicast orchestration services without consuming significant additional WAN bandwidth, cloud resources, etc.

[0039] In some examples, the routing calculation engine in the overlay tunnel orchestrator 142 can calculate service routes based on the aforementioned source information (e.g., which VPNC is associated with a given multicast flow) and receiver information (which branch gateways are designated as leaders of a given multicast flow). In some examples, the routing calculation engine can learn to calculate the optimal route to reduce bandwidth consumption of WAN 160. For example, the overlay tunnel orchestrator 142 can employ artificial intelligence (AI) or machine learning to determine the overlay tunnel for multicast services between VPNCs and branch gateways (e.g., based on service requirements and historical data).

[0040] In other examples, overlay tunnel orchestrator 142 can utilize routes already calculated by SD-WAN 140 (and / or its sub-services). Existing SD-WAN services typically calculate routes for unicast traffic between VPNC and branch gateways. Therefore, overlay tunnel orchestrator 142 can orchestrate traffic through these pre-calculated routes.

[0041] It should be understood that in a given network, "underlying layer" or "underlying network" can refer to the physical connections of the network (e.g., Ethernet). In contrast, "overlay" or "overlay network" can refer to a logical network that uses virtualization to establish connections over the physical infrastructure of a network encapsulated using tunnels. In other words, an "overlay tunnel" can refer to a virtual link connecting nodes in the network. Here, an overlay tunnel can connect VPNC and branch gateways. Various protocols such as IPSec and GRE can be used to send network traffic through these overlay tunnels. Typically, the SD-WAN architecture shown can rely on overlay tunnels to connect the various branches of its network and other nodes.

[0042] As used in this article, branch 110 can refer to a physical location where one or more endpoint devices can connect to WAN 160. For example, a branch could be an organization's remote office, café / coffee shop, home office, etc. While in Figure 1 The example shown only depicts a single branch (branch 110 comprising a set of gateways 112), but large-scale SD branch deployments can include any number of branches. In some examples, these can be branches of a specific organization. In other examples, branches may not all be associated with a single organization. Although not shown, each branch can have its own local area network (LAN). Various network devices on a given branch (e.g., hosts (user-related devices), endpoint devices (such as branch gateways), and network devices (such as routers) can communicate with each other through the branch's LAN. A branch can have any number of endpoints. In this example, branch gateway 112 can receive services (such as multicast services) associated with service flow 152.

[0043] A given host (e.g., smartphone 114) can connect to AP 116, which communicates with gateway 112. As mentioned above, multiple gateways 112 can be deployed at branch 110 for load balancing and redundancy purposes. Therefore, a given host can connect to a given branch gateway based on factors such as path latency.

[0044] AP 116 can be configured to provide a communication channel between smartphone 114 and network 170 using various devices in branch 110, although it may not be the only communication point. Multiple APs are shown, although branch 110 may include one or more APs 116 with multiple controllers and / or multiple communication points. In some embodiments, AP 116 communicates with network 170 via switch 162, second gateway 112B, and second router 120B, or it may communicate with network 170 via switch 162, first gateway 112A, and first router 120A. In some examples, AP 116 may include a multi-band radio, such as a dual-band simultaneous radio that can operate with, for example, eight radio chains in the 5 GHz band and four radio chains in the 2.4 GHz band. For example, in operation, the 5 GHz radio may be converted into two logical radios, each using four radio chains. Channel switching may be prompted by, for example, radar detection, when channel utilization exceeds acceptable limits, or in response to some regulatory change affecting channel utilization or allocation. Other features of the general access point can also be implemented using one or more AP 116s.

[0045] Gateway 112 can refer to a network device (hardware or software) that transmits services between devices associated with branch 110 and other networks (e.g., AP 116 and switch 162). For example, Figure 1 The gateway 112 shown can transmit services between WAN 160 and various network devices (not shown) in its branches (e.g., other branch gateways, hosts, etc.).

[0046] Gateway 112 can also be configured to establish communication connections to various networks and detect / correct communication problems, such as... Figure 2 As shown. In this example, the gateway computing device 200 includes a processor 202, a memory 204, and a machine-readable storage device 206, which can implement machine-readable instructions for performing various actions within the network. For example, the machine-readable storage device 206 may include a physical uplink engine 210, a virtual uplink engine 220, a health check manager 230, a connectivity and routing component 240, a load balancing engine 250, and a cluster manager 260.

[0047] Physical uplink engine 210 is configured to manage the physical uplink between gateway 112 and another device. For example, the physical uplink may include a wired connection from first gateway 112A to first router 120A to ISP 130, and also manage the physical uplink from second gateway 112B to second router 120B to ISP 132. Connections via the physical uplink may be initiated by a Dynamic Host Configuration Protocol (DHCP) handshake (e.g., via Ethernet port 001 of first gateway 112A), followed by a broadcast process responded to by ISP 130, or a similar connection process (e.g., zero-touch provisioning). In this example, the first ISP connection will be associated with first gateway 112A, and the second ISP connection will be associated with second gateway 112B.

[0048] Virtual uplink engine 220 is configured to use virtual connections to manage virtual connections between gateway 112 and another device. For example, a virtual uplink may include virtual (e.g., wireless) connections from first gateway 112A to switch 162 to second gateway 112B to second router 120B to ISP 132, and also manages virtual uplinks from second gateway 112B to switch 162 to first gateway 112A to first router 120A to ISP 130. Connections via virtual uplinks can be managed by an orchestrator or a central processing unit (e.g., in...). Figure 1 (SD-WAN 140). In the event of a physical uplink failure, these additional connections can help provide network access via different communication paths.

[0049] In some examples, physical uplink engine 210 can access the network via a WAN connection, while virtual uplink engine 220 can access the network by forming a Generic Routing Encapsulation (GRE) tunnel from a downstream LAN. In other words, from the perspective of the first gateway 112A, the first ISP 130 is a physical uplink connection (via WAN), and the second ISP 132 is a virtual uplink connection (via LAN), both of which provide access to the network.

[0050] The Health Check Manager 230 (sometimes called "HCM") is configured to identify Internet Protocol (IP) routes and track probe requests and responses (interchangeably) from one or more devices. For example, in Figure 1 The health check manager 230, operating at the first gateway 112A, can send heartbeat requests or WAN health check probes (which can be used interchangeably) to devices within branch 110 (including the second gateway 112B and switch 162, as well as the first and second ISPs) to identify which devices are reachable and respond, or whether they are unreachable. When a device responds to a heartbeat request (e.g., within milliseconds of a response time via a virtual uplink connection between devices), the health check manager 230 can determine that these devices are functioning correctly (e.g., able to send and receive data packets) and that the network is healthy. The status of reachable or unreachable devices can be updated in a table, as shown herein.

[0051] Heartbeat requests or WAN health check probes can include various formats and protocols. For example, a heartbeat request or WAN health check probe can be an Internet Control Message Protocol (ICMP) ping transport or a User Datagram Protocol (UDP) ping transport. When a device is functioning normally and receives a heartbeat request or WAN health check probe, the device can be configured to automatically respond to the heartbeat request or WAN health check probe using the same protocol format.

[0052] If no heartbeat response is received within a threshold time period, the health check manager 230 can resend the heartbeat request at predetermined intervals. For example, the default time can be set to 10 seconds, such that a new heartbeat request is sent every 10 seconds for a total number of attempts (e.g., three attempts). In some examples, repeatedly attempting to receive a heartbeat response (without a response) can confirm that the device has a problem causing data loss (e.g., a WAN failure causing the device not to provide a heartbeat response) rather than an intermittent failure (e.g., intermittent ping loss that the device can recover from quickly). In this example, the total time period for confirming data loss is 30 seconds, or three attempts with a 10-second interval between each attempt.

[0053] When data loss is confirmed, the health check manager 230 can communicate with the uplink engine to disconnect the affected uplink (e.g., the device corresponding to the "unreachable" status identifier) ​​in response to the reported fault. For example, when the connection between the first gateway and the first ISP is affected, the physical uplink engine 210 can disconnect the physical uplink connection. In another example, when the connection between the first gateway and the second ISP is affected, the virtual uplink engine 220 can disconnect the virtual uplink connection.

[0054] By initiating a removal procedure for the virtual uplink connection, the health check manager 230 can perform various actions. For example, it can remove the default gateway from the DHCP table of the affected gateway 112. In another example, once the uplink connection is removed via the removal procedure, the health check manager 230 can communicate with the connectivity and routing component 240 to update the routing path used to send or receive data (e.g., using the FPAPPS procedure to identify the route used to send data, or changing the default uplink to the virtual uplink that maintains operation). In some examples, the removal procedure initiates the action of removing the default gateway for the virtual uplink (e.g., when the ISP associated with the virtual uplink is inaccessible and the ISP associated with the physical uplink is accessible). For example, the removal procedure may include restarting one or more specific procedures individually, or restarting the entire second gateway. In some examples, a single application can be rebooted without restarting the second gateway. Similar operations can be performed when a removal procedure is initiated for a physical uplink connection.

[0055] The load balancing engine 250 is configured to use one or more uplink availability to bootstrap data packets, including to a first ISP 130 or a second ISP 132. For example, for one physical uplink and one virtual uplink, the load balancing engine 250 can bootstrap data packets substantially equally between the two uplink options (e.g., physical uplink and virtual uplink).

[0056] An example of load balancing is shown in Table 1, which uses first gateway 112A and second gateway 112B as options for the default gateway for routing Internet services. As described herein, first gateway 112A can access the Internet 170 via a physical uplink through first ISP 130, and can also optionally access the Internet 170 via a virtual uplink through second ISP 132.

[0057]

[0058] Table 1

[0059] The load balancing engine 250 is also configured to identify latency or delay in one uplink option and redirect traffic to another uplink option that does not experience latency or delay. In some examples, when an uplink fails, the load balancing engine 250 is configured to redirect traffic to another uplink option that is still operational.

[0060] Cluster Manager 260 is configured to create device clusters, for example, including Figure 1 The first gateway 112A and the second gateway 112B are a cluster of gateway devices. For example, the cluster manager can associate these devices as a cluster to monitor operations, reallocate processing tasks, or perform other features such as reallocation among devices in the cluster. The cluster manager 260 is configured to recover devices from failure scenarios and reallocate tasks to other devices when a failed device reboots or performs other operations. In failure scenarios, when a hardware component fails, the cluster manager 260 detects and recovers the cluster by identifying another device in the cluster (e.g., the first gateway 112A and the second gateway 112B) or by creating a new component member on the infrastructure basis. The cluster manager 260 and the infrastructure can be designed with redundancy to prevent failure conditions.

[0061] Various problems may exist in this system, which can be effectively solved using the connections and procedures described in this paper.

[0062] Using LAN connections to resolve WAN service loss issues

[0063] Gateway 112 typically performs WAN health checks by sending probes to the WAN IP on both the physical and virtual uplinks (e.g., by...). Figure 2 The health check manager 230). Replies are received on both the physical and virtual uplinks, and WAN traffic is balanced between the physical and virtual uplinks in each gateway (e.g., by...). Figure 2(Load balancing engine 250). However, for the example configuration involving a 10-second probe interval and three retries, the connection failure is only confirmed after three consecutive probes. For example, the first gateway might send a probe packet on the virtual uplink at 10:00:00 (hours:minutes:seconds). A successful response to the probe packet can be received within milliseconds. Then, the first gateway might lose connection with the second gateway at 10:00:02 (e.g., via the virtual uplink). The next probe may not be sent until 10.00.10. If the connection problem persists, the attempt may fail again, and two more attempts will be made at 10.00.20 and 10.00.30 before confirming the failure and rerouting the traffic. This means that the first gateway could lose WAN traffic on the virtual uplink for 38 seconds or longer (e.g., if the retry and frequency values ​​correspond to the illustrative values ​​provided herein). As shown in the example, the 10-second frequency value can be a high-frequency value relative to other options.

[0064] WAN service loss on a virtual uplink (e.g., due to a poor LAN network) can be mitigated by actively checking LAN health to determine if connectivity issues exist and utilizing the physical uplink when the virtual uplink fails, as illustrated in this article. Typically, the WAN health check timer is set high to accommodate packet loss and latency. However, for internal LANs, packet loss and latency are negligible. In internal LANs, it is not necessary to set retry and frequency values ​​high. For example, a first-gateway WAN high availability (HA) virtual LAN (VLAN) (e.g., a virtual uplink tunnel) can send a probe every second, attempting to re-establish the connection three times. Repeated probes at 1-second intervals can be described as a “heartbeat.” For example, a 1-second interval could be an example of a relatively low-frequency interval (e.g., three seconds before or equal to three seconds). If the probe failure persists for three seconds, the tunnel HCM process can immediately notify the uplink manager to suspend the virtual uplink from the WAN uplink load balancer in the first gateway. This suspension can be achieved by notifying the Fast Path Application (“FPAPPS”) process to remove any routes via the virtual uplink without waiting for the typical 38-40 second WAN health check timer described above to complete. The FPAPPS process can be responsible for removing and / or adding routes to the virtual uplink. Then, because other routes are paused, sessions can only be initiated on the physical uplink that is the first gateway to the WAN that can be reached.

[0065] In some examples, illustrative systems can use LAN connections between devices to address WAN connectivity issues and minimize overall service loss. For instance, the system can use one or more tunnel HCM probes to perform proactive LAN health checks and detect network problems early, as WAN HCMs take significantly longer to detect issues than internal LANs already established for other purposes (through retry attempts and heartbeat probes).

[0066] exist Figures 3-4 The document provides illustrative examples of using LAN connections to address WAN service loss issues. For instance, for ease of explanation, Figure 3 The topology shown can be compared with Figure 1 The simplified versions of the devices in the example include a first ISP 130, a first gateway 112A, a second ISP 132, a second gateway 112B, a switch 162, and the Internet 170. Furthermore, various device implementation options are provided. For example, two gateways can communicatively connect to each ISP using either a physical uplink or a virtual uplink. In this example, four gateways can be used to direct services through the network to the Internet 170, including both physical and virtual uplinks.

[0067] As described herein, each gateway 112 may have physical uplinks 320, 322 to each ISP. This allows each gateway 112 to utilize the uplink of another peer gateway as a virtual uplink by forming a GRE tunnel from the downstream LAN network. As shown, the first gateway 112A may have a first virtual uplink 324. The second gateway 112B may have a second virtual uplink 326. The WAN HA feature can then configure two separate GRE tunnels 328, 330 between the first gateway 112A and the second gateway 112B to carry the traffic mean for the first virtual uplink 324 and the second virtual uplink 326. The first gateway 112A can use the uplink 326 of the second gateway 112B by treating GRE tunnel 330 as its own virtual uplink. Similarly, the second gateway 112B can use the uplink 324 of the first gateway 112A by using GRE tunnel 328 as its virtual uplink. The FPAPPS procedure can handle routing. The FPAPPS procedure can use the default gateway of the second gateway 112B in its routing table, as shown in Table 1. It will also combine... Figure 4 To describe Figure 3 The aforementioned components.

[0068] Figure 4 An example process for resolving WAN service loss issues using a LAN connection according to the above embodiments is shown. Figure 4The device shown can be used with Figure 3 The topology shown corresponds to this.

[0069] In box 400, as shown in 400A and 400B, the first gateway 112A can use the HCM procedure to monitor the accessibility of the first physical uplink 320 and the first virtual uplink 324. The HCM procedure can be performed by... Figure 2 The health check manager 230 shown is used to implement this.

[0070] In this example, the first gateway 112A monitors accessibility by sending an ICMP probe to the first ISP 130 on the first physical uplink 320 and to the switch 162 on the first virtual uplink 324, which in turn sends an ICMP probe to the second gateway 112B. The ICMP probe may eventually be delivered to the second gateway 112B (e.g., when attempting to access the second ISP 132). Accessibility can be monitored by sending probes via each uplink. For example, the first gateway 112A may send ICMP probes on the first physical uplink 320 at 10-second intervals with three retries. Therefore, three probes, each 10 seconds apart, may be sent before reachability determination. Since in this example, the connection to the second gateway 112B (via switch 162) represents the HA VLAN of the first virtual uplink 324, the ICMP probe arrives at the second gateway 112B within milliseconds when the second gateway 112B becomes available.

[0071] Other frequencies are also possible. For example, probes can be emitted at intervals of 8 seconds, 12 seconds, 20 seconds, or any other frequency. However, for HCM processes, the frequency is typically set high, around 10 seconds, to avoid delays and other issues. Additional retries are also possible. For example, two retries can be set, or five retries can be set. Other settings are also possible.

[0072] In box 410, the first gateway 112A can receive one or more responses and generate a response. For example, when an ICMP probe is issued at box 400, an ICMP response to the probe can be received at the first gateway 112A. In this example, there is a connectivity problem at the physical uplink 320 between the first ISP 130 and the first gateway 112A. Thus, the reception of the ICMP response can indicate that the virtual uplink 324 is reachable.

[0073] Since the connection to the second gateway 112B (via switch 162) represents the HAVLAN of the first virtual uplink 324, responses to ICMP probes from the second gateway 112B are also expected to be in milliseconds. When a response is received within a threshold time period (e.g., milliseconds), the first gateway 112A can use the second gateway 112B to continue the connection process to access the second ISP 132. When no responses are received within the threshold time period (e.g., milliseconds), the second gateway 112B is assumed to be unavailable. Based on the use of the HA VLAN representing the first virtual uplink 324, availability or unavailability can be determined within this shortened time frame.

[0074] In box 420, within the first gateway 112A, the health check manager can notify the uplink manager of the gateway status based on received responses. For example, receiving a response to an ICMP probe can indicate that the first virtual uplink 324 is available. The status of the virtual uplink 324 can then be marked as reachable. This reachable status can then be transmitted to the uplink manager.

[0075] In box 430, various operations can be performed based on the non-responsive first physical uplink 320. These operations can utilize WAN health checks via the first physical uplink 320, or alternative connections via a LAN to the Internet 170. The LAN connection can utilize, for example, a first virtual uplink 324, a second virtual uplink 326, and a second physical uplink 322 to access a second ISP 132, which also provides access to the Internet 170 (alongside the unreachable first ISP 130 via the first physical uplink 320). Illustrative examples illustrating the differences between the two approaches are provided herein.

[0076] For example, a WAN health check on the first physical uplink 320 can be performed by the HCM process based on configurable health check parameters. Configurable health check parameters may include frequency intervals and repetition parameters (e.g., the frequency can be configured to 10 seconds and the retry value to 3). The HCM process can use configurable health check parameters (e.g., a 10-second frequency interval or other high-frequency interval values ​​and three retry attempts) to send ICMP packets to the first ISP 130 via the first physical uplink 320 (which is unresponsive). If a WAN reachability failure occurs, the HCM process can repeat the health check, in this case, three times, sending two more ICMP packets at 10-second intervals. If the failure persists after the third attempt, the HCM process can acknowledge the WAN reachability failure via the first physical uplink 320. The HCM process can notify the uplink manager of the WAN reachability failure. The uplink manager can mark the uplink as unreachable in the event of a probe failure. Load can be balanced between physical and virtual uplinks based on WAN reachability. The example command-line interface ("CLI") output for the uplink maintained by the uplink manager is as follows:

[0077]

[0078] Table 2

[0079] An example of an uplink management table with health check parameters configured as described above is shown:

[0080]

[0081] Table 3

[0082] As described in this article, and for clarity, the health check interval can be set high, approximately 10 seconds, to accommodate WAN loss, jitter, latency, and other issues, thus avoiding unnecessary flapping of the WAN uplink. However, setting the health check interval to around 10 seconds can also present problems. Given the high frequency, if a failure occurs shortly after an ICMP probe is sent, a WAN reachability failure may not be acknowledged for nearly 40 seconds. This is because the next probe will be sent at 10-second intervals, and the failure will only be acknowledged after three failed attempts to reach the WAN destination (e.g., the first ISP 130). For example, if a virtual uplink fails, traffic originating on the virtual uplink will be lost for nearly forty seconds before being rerouted to the physical uplink. Failures can occur for several reasons. For example, a failure may occur if the second gateway is reloaded due to an image upgrade, reloaded due to high temperature, if the WAN HA tunnel is disconnected, or for many other reasons.

[0083] Specifically, in an example scenario, the first gateway may lose connection to the second gateway at time T. The second gateway may be unable to access the first gateway for various reasons. For example, load rebalancing could cause this inaccessibility. Other possibilities exist. Reloading could be due to a process crash, high temperature, or other reasons. Time T could occur 5 seconds after the most recent probe was sent. The most recently sent probe may have already received a response indicating WAN reachability, so the uplink manager is unaware of any failure at time T. If the probes were sent at 10-second intervals, another probe to check WAN reachability could be sent 5 seconds after time T. This probe might not receive a response indicating a WAN reachability failure. However, three attempts are required to confirm the failure. The first probe will be sent 15 seconds after time T, also without a response. The second probe will be sent 25 seconds after time T, also without a response. The last probe will be sent 35 seconds after time T, also without a response. This final attempt will confirm the WAN reachability failure. Only at this point, 35 seconds after the failure, will the uplink manager receive the failure notification. Therefore, service may be lost for 35 seconds before load rebalancing. Such loss could have a significant impact on critical applications.

[0084] Alternatively, the health check manager at the first gateway 112A can utilize the LAN connection instead of (or simultaneously) performing WAN health checks. In this example, when a connection failure with the first ISP 130 is identified, the first gateway 112A can automatically access the Internet 170 via the second ISP 132. Faster fault identification and appropriate load rerouting can significantly reduce service loss and allow for faster load rebalancing, preventing prolonged service loss. WAN health check timers are typically set to a higher frequency, approximately 10 seconds, as described above, to accommodate packet loss. However, the internal LAN does not cause the same packet loss and latency issues. Therefore, identifying the loss in the LAN network rather than the WAN network allows for more aggressive health checks. Faster identification of virtual uplink losses in the LAN network allows for load rebalancing, preventing the virtual uplink from being used for uplink load balancing, as using a virtual uplink would result in a failure.

[0085] In box 440, the first gateway 112A can run probes via the first virtual uplink 324 (e.g., at active intervals such as every 10 milliseconds). For example, the first gateway can use an HA VLAN to send probes via the first virtual uplink 324 at 1-second intervals, and subsequently perform one or more retries to form a virtual uplink tunnel “heartbeat.”

[0086] In box 450, the first gateway 112A can use this path to update the default gateway. For example, if the heartbeat receives a probe response from the probe request, a new default gateway can be set using the path to the second ISP 132 (via virtual uplink 326 to the second gateway 112B). When setting a new default gateway, the first gateway 112A can suspend or stop the WAN connection to the first ISP 130 via the first physical uplink 320, thereby removing the second gateway 112B as the default gateway (e.g., using FPAPPS), so that traffic is not sent via that connection. As shown in Table 3 (for the first gateway 112A), the reachability identifier of the second gateway 112B can also be changed from "reachable" to "unreachable". In some examples, WAN health checks can be suspended based on a second connection to the Internet 170 via a LAN identifier.

[0087] If the WAN HA heartbeat fails during the initial probe and any retry attempts, the tunnel HCM process can immediately notify the uplink manager processor. The uplink manager can simultaneously execute the health check manager process, as described in box 430. The uplink manager can listen for both the health check manager process and the virtual uplink tunnel heartbeat. As mentioned above, a period of up to 40 seconds may pass before the health check manager process can confirm the loss of the virtual uplink. However, the virtual uplink heartbeat tunnel can confirm the loss of the virtual uplink in as little as 4 seconds.

[0088] When the loss of the first virtual uplink 324 is confirmed, the tunnel HCM procedure can also notify the uplink manager.

[0089] In box 460, the first gateway 112A can send data to the second ISP 132 via a LAN connection. This allows the first gateway 112A to maintain data transmission from the first gateway 112A to the network (Internet 170) via the first virtual uplink 324. In some examples, the first gateway 112A can suspend or stop the connection via the first physical uplink 320 to remove data transmission from the first gateway 112A to the network (Internet 170) via the first physical uplink 320, while maintaining data transmission from the first gateway 112A to the network via the first virtual uplink 324. This can help reduce data transmission in the network when the system has identified an uplink or device failure.

[0090] The load on data transmission can then be rebalanced (e.g., using both available and reachable virtual uplinks and physical uplinks). Load rebalancing can help avoid overusing the first virtual uplink 324, which could cause all remaining network connections to fail because the physical uplink is unreachable and the virtual uplink is overloaded, resulting in losses along both uplinks.

[0091] In boxes 470 (shown as first box 470A to first ISP 130 and second box 470B to second ISP 132), the tunnel HCM process can continue listening for physical uplink tunnel heartbeats or other responses. When the tunnel HCM process detects that a heartbeat has been restored via the first physical uplink 320 with first gateway 112A, the tunnel HCM process can immediately notify the uplink manager. Detecting that a heartbeat has been restored indicates that the first physical uplink 320 is now reachable. The uplink manager can then notify the FPAPPS process to update the default gateway to the first physical uplink 320 (e.g., by adding it back to the default gateway; see Table 1 for the FPAPPS routing table). With multiple default gateways, via both physical and virtual uplinks, the traffic load can be rebalanced. Furthermore, the uplink health check manager process can be resumed for all paths (e.g., the first physical uplink 320 and the first virtual uplink 324).

[0092] In some examples, the second gateway 112B can also identify that the second virtual uplink 326 is disconnected and the first ISP 130 is unreachable (e.g., using both WAN heartbeat probes and virtual uplink probes). This operation can be performed simultaneously with a similar operation performed by the first gateway 112A. For example, the second gateway 112B can use the second physical uplink 322 to set a new default gateway, which can be a new default path to the second ISP 132. When setting a new default gateway, the second gateway 112B can suspend or stop the WAN connection to the first ISP 130 via the second virtual uplink 326, thereby removing the first gateway 112A as the default gateway (e.g., using the FPAPPS procedure), so that services are not sent through that connection. As shown in Table 3 (for the second gateway 112B), the reachability identifier of the first gateway 112A can also be changed from "reachable" to "unreachable".

[0093] Use WAN connection to solve LAN service loss issues

[0094] Similar to using a LAN connection to resolve WAN connection loss issues, this system can also use a WAN connection to resolve LAN or virtual uplink connection loss issues (including switch 162 or HA VLAN), or other intermittent device or connectivity problems. Figure 5 Combination Figures 1-4 The several communication processes shown provide an illustrative topology.

[0095] In this example, if a virtual uplink loss is detected, for example, by the first gateway 112A at the first virtual uplink 524, the uplink manager can immediately suspend virtual uplink transmissions via the first virtual uplink 528. In some examples, the default gateway for virtual uplink 524 is removed by the FPAPPS procedure. In some examples, the uplink manager can also immediately suspend and wait for a heartbeat response via the first virtual uplink 524. If a loss is detected using a virtual uplink tunnel heartbeat, the loss can be detected within 3-4 seconds, even if the default probe rate is set higher (e.g., 10 seconds) via the physical or WAN connection. The uplink manager can then immediately suspend the first virtual uplink 524 after confirming the loss via the virtual uplink tunnel heartbeat. The uplink manager does not need to wait for WAN probe results from the uplink health check manager procedure (which could take up to 40 seconds, for example, using the illustrative example discussed herein). This achieves a significant reduction in service loss.

[0096] Furthermore, the uplink manager can immediately notify the health check manager to suspend its regular WAN health check probes on virtual uplink 524 after a heartbeat confirmation of loss via the virtual uplink tunnel. After confirming the loss of virtual uplink 524, all sessions can be initiated only on the first physical uplink 520 of the first gateway. Even if the first virtual uplink 524 is unreachable, the first physical uplink 520 remains reachable from the Internet 170.

[0097] In some examples, the tunnel HCM process can continue listening for virtual uplink tunnel heartbeats. The tunnel HCM process can detect that the heartbeat has been restored in the first gateway 112A. Detecting that the heartbeat has been restored indicates that the first virtual uplink 524 is now reachable. Upon detecting the restoration of the heartbeat indicating that the first virtual uplink 524 is now reachable, the tunnel HCM process can immediately notify the uplink manager. The uplink manager can then notify the FPAPPS process to add the default gateway back to the first virtual uplink 524, as shown in Table 1 of this document. Using multiple default gateways, using both physical and virtual uplinks, allows for a rebalancing of the service load. Furthermore, the uplink health check manager process can be resumed.

[0098] Figure 6 An illustrative example is provided, which shows a virtual uplink problem, for example, when the second gateway 112B is removed, restarted, or reloaded.

[0099] In box 600, the virtual uplink is lost due to the reloading of the second gateway 112B, which can trigger the transmission of a virtual uplink tunnel heartbeat. This initial event 300 can include several events that cause the virtual uplink to be lost. For example, one initial event could be the reloading of the second gateway 112B.

[0100] In box 602, the reloading of the second gateway may cause the first gateway 112A to begin losing service. At time T0, the first gateway 112A may begin losing service. A virtual uplink tunnel heartbeat can be sent via the first virtual uplink 524 to determine if the virtual uplink 524 is available for service transmission.

[0101] In box 604, a virtual uplink tunnel heartbeat can be sent via a first virtual uplink 524. The heartbeat can be a probe (e.g., sent via a communication channel at 1-second intervals or other low-frequency intervals). The first probe can be sent at time = T0.

[0102] In box 606, one or more retry attempts can be initiated to emit the probe, including at 1-second intervals. In this illustration, retry attempts can be emitted at times T0+1 and T0+2. Three attempts can be emitted. An attempt can include an initial attempt and two retry attempts. Therefore, a total of three retry values ​​are possible, since a total of three probes are emitted. Other numbers of retries are also possible. For example, in one embodiment, a single probe can be emitted. In another embodiment, two probes can be emitted. In other embodiments, a total of four, five, or more probes can be emitted. Furthermore, a range of frequencies is also possible.

[0103] In box 608, a heartbeat loss notification can be sent to the uplink manager at time T0+2. This can be the same time as the final retry probe is sent. A heartbeat loss can mean that although a probe was sent, no response was received. Therefore, a heartbeat loss is recorded.

[0104] In box 610, the uplink manager can immediately suspend the virtual uplink service and immediately stop the health check manager probe. The uplink manager can suspend the virtual uplink service and HCM probe at time = T0+2. This can be the same time as issuing the final retry probe.

[0105] In box 612, all services can be sent through the physical uplink in the first gateway, instead of through an unreachable virtual uplink.

[0106] Once the link to the second gateway 112B is re-established with the first gateway 112A, the process can continue with the physical uplink. For example, in block 613, the process can determine whether the link to the second gateway 112B has been re-established.

[0107] In box 614, the HCM at the first gateway 112A can optionally issue a next scheduling probe. The next scheduling probe can be issued at time T1. The optional transmission of this HCM probe can depend on the re-establishment of the connection with the second gateway 112B, referred to as T1. Before the connection with the second gateway 112B is re-established, the HCM probe can be paused, suspended, stopped, or delayed, which can reduce data transmission over the network connection if one of the uplink connections is broken or if the device using the uplink to reach the ISP is unreachable.

[0108] In box 616, one or more retry attempts can optionally be issued at a frequency of 10 seconds. For example, a total of three probes can be issued. After the initial probe issued at time = T1, two additional probes can be issued at times = T1+10 and T1+20. However, if the virtual uplink tunnel heartbeat process as described herein is performed, the uplink manager can pause the HCM probes immediately at time = T0+2. This can happen even before the next scheduled HCM probe is issued, and certainly after the retry HCM probe is issued. Therefore, potential operations 614 and 616 can be omitted.

[0109] Use the cluster manager of the cluster gateway to resolve cluster splitting issues caused by LAN service loss.

[0110] In some examples, the first gateway 112A and the second gateway 112B can form a gateway device cluster via a cluster manager (e.g., within a cloud-based SD-WAN technology such as SD-WAN 140 or within each gateway 112). The cluster manager can add more communication engines when one or more uplinks or devices fail. A cluster can be a combination of multiple managed devices (e.g., gateway 112, etc.) that work together to provide high availability and ensure service continuity in the event of failure. Cluster manager procedures can be implemented to enable communication between gateways using cluster features that could originally be used for other purposes.

[0111] The device can communicate via multiple communication tunnels. For example, as described herein, the overlay tunnel orchestrator 142 of the SD-WAN 140 can communicate with VPNC 154 via a first overlay tunnel through a first connection 146a, the overlay tunnel orchestrator 142 can communicate with the first gateway 112A via a second overlay tunnel through a second connection 146b, and the overlay tunnel orchestrator 142 can communicate with the first AP 116A via a third overlay tunnel through a third connection 146c. In some examples, the overlay tunnel orchestrator 142 may include a tunnel manager procedure locally at each gateway 112 to form each overlay tunnel, then generate messages through the overlay tunnels and send the messages to each endpoint (VPNC 154, the first gateway 112A, and the first AP 116A). Each endpoint may include a similar tunnel proxy procedure to receive and parse messages from the overlay tunnel orchestrator 142 via the overlay tunnels.

[0112] For example, when a cluster is implemented between the first gateway 112A and the second gateway 112B, these devices can also communicate under the direction of the cluster leader selected by the cluster manager. The cluster manager, implemented at each gateway 112, can also identify other devices as cluster members. In this example, the first gateway 112A can be selected as the cluster leader, and the second gateway 112B can be selected as a cluster member. The cluster manager can also perform load balancing when both the first gateway 112A and the second gateway 112B are operational.

[0113] The cluster manager can also determine when HCM heartbeats are issued and processed by other cluster members and track any responses. In some examples, the cluster manager will generate a data packet map and send it to one or more APs 116 to identify accessible cluster members. When a new connection is established between AP 116 and client 114, AP 116 can identify where to issue data traffic based on the data packet map generated by the cluster manager (e.g., first gateway 112A).

[0114] When implementing a cluster, the illustrative system can use the functionalities that can be included in the cluster implementation to address WAN connectivity issues and minimize overall service loss. For example, the system can implement proactive cluster heartbeat checks to detect network problems early, because the WAN HCM takes significantly longer to detect (through retries and heartbeat probes) than a group heartbeat check already established between devices in the cluster (e.g., first gateway 112A and second gateway 112B) for other purposes.

[0115] Figure 7An illustrative topology is provided. In this example, gateway cluster 700 is formed by a cluster manager at SD-WAN 140 or other cloud-managed or locally managed cluster procedures (such as a cluster manager implemented at each gateway 112) together with a first gateway 112A and a second gateway 112B. The first gateway 112A, the second gateway 112B, and SD-WAN 140 are... Figure 1 It is described similarly in the context of the application and throughout the entire application.

[0116] Devices in the cluster can communicate via LAN connections implemented as overlay tunnels (e.g., a GRE tunnel via switch 162). When the gateway cluster 700 is formed, each device in the cluster can handle traffic as a high-availability (HA) feature. If one device fails, other devices can take action from their location, for example, by running heartbeats for peer devices in the cluster and sending instructions to a second device in the cluster. One or more cluster heartbeat packets can be sent through the same communication channel as the default gateway.

[0117] Figure 8 It shows the use of Figure 7 The topology shown uses the resulting device cluster as a method to address the cluster splitting problem of LAN service loss.

[0118] In box 800, SD-WAN 140 can push profiles and configuration settings to each device to generate gateway cluster 700, as shown in the figure. In box 800A, SD-WAN 140 sends a cluster profile to the first gateway 112A, and in box 800B, SD-WAN 140 sends a cluster profile to the second gateway 112B.

[0119] In box 805, the profiles and configuration settings are implemented by each cluster manager. As shown in the figure, in box 805A, the first gateway 112A and the second gateway 112B establish a gateway cluster 700, and in box 805B, the second gateway 112B and the first gateway 112A form a gateway cluster 700. Using these settings, gateway 112 can form a gateway cluster 700 including the first gateway 112A and the second gateway 112B.

[0120] As an illustrative example, various steps can occur when two gateways of gateway cluster 700 are configured as part of the same cluster profile. For instance, the cluster manager initiates a cluster heartbeat process to its peers. The cluster manager exchanges information about the gateways of gateway cluster 700 and their Media Access Control (MAC) addresses with each peer. The controller at gateway 112 with the highest platform capacity can be elected as the cluster leader, while the other gateways 112 can be elected as cluster members. If all gateways 112 have the same platform capacity, the gateway with the highest MAC address can be elected as the cluster leader, and the remaining gateways 112 can be elected as cluster members.

[0121] The cluster leader can generate settings that are sent to one or more APs, and these settings are also used to configure the APs. For example, these settings may include a bucket map instructing the APs how to forward client traffic (e.g., ...). Figure 1 AP 116 forwarding Figure 1 (The service of client 114). The bucket diagram can contain the IP address of each gateway participating in the gateway cluster 700.

[0122] In some examples, bucket graphs can be used to support client load balancing when configuring overlay tunnels (e.g., IPSec or GRE). For example, an overlay tunnel is formed between AP 116 and gateway 112. When client 114 connects to a communication channel (e.g., Wi-Fi), traffic from client 114 arrives at AP 116. AP 116 checks the bucket graph to identify the gateway 112 to which the traffic needs to be forwarded via the overlay tunnel. The client traffic can then reach the appropriate gateway based on the bucket graph entry. The receiving gateway can also perform load balancing to further forward its traffic to the Internet 170.

[0123] In box 810, gateway cluster 700 can be monitored by the cluster manager. For example, a cluster heartbeat probe can be sent to each device in cluster 700. When a device returns a response to the cluster manager, the device status (e.g., active, disconnected, reachable, unreachable, etc.) can be determined.

[0124] During the cluster heartbeat process, the first gateway 112A can run probes via the first virtual uplink 724 (e.g., at active intervals similar to every 10 milliseconds). For example, the first gateway 112A can use a high-availability (HA) virtual LAN (VLAN) to send probes via the first virtual uplink 724 at 1-second intervals, and can subsequently make one or more retries to constitute a virtual uplink tunnel "heartbeat".

[0125] In box 820, the first gateway 112A is reachable from the local cluster manager (or, if the cluster manager is implemented remotely), which is identified as a response to a probe request, but the second gateway 112B may be unreachable. In this example, the second gateway 112B may not send a response to the probe request. A lost response from the second gateway 112B could be due to an interruption event such as gateway reload, and could be detected from a cluster heartbeat failure. A cluster split can occur within milliseconds and is detected, at least in part, by the cluster manager actively sending probe requests (or cluster heartbeats) and listening for responses within milliseconds (e.g., 900 ms). A cluster split can immediately indicate a failure in the reachability of the LAN network. The cluster manager process can listen for cluster splits and can detect them within milliseconds of the split occurring.

[0126] In some examples, cluster heartbeat packets can be transmitted via the same downstream LAN switches described herein, including the first HA VLAN 728 or the second HA VLAN 730 of switch 162, similar to... Figure 3 The first HA VLAN is 328, and the second HA VLAN is 330. Running a cluster heartbeat in milliseconds ensures early detection and correction of problems to maintain high availability of LAN services. This cluster heartbeat process allows for rapid identification of outage events in one or more devices within the cluster 700.

[0127] In box 830, the cluster manager can detect network problems (e.g., cluster splits or other communication errors) and notify the uplink manager of the unaffected device, i.e., the first gateway 112A, by generating and sending error messages. The notification to the uplink manager can identify cluster splits affecting the second gateway 112B.

[0128] In box 832, various operations can be performed in parallel or serially based on the unresponsive first virtual uplink 724. For example, the uplink manager at the first gateway 112A may immediately suspend the virtual uplink (e.g., the LAN connection to the second gateway) due to unreachability. In some examples, the uplink manager at the first gateway 112A may instruct FPAPPS to remove the route to the second gateway 112B (e.g., via the virtual uplink).

[0129] In box 834, the uplink manager at first gateway 112A can immediately suspend probe transmissions to other devices in cluster 700. For example, the uplink manager can stop WAN health checks to second gateway 112B via first virtual uplink 724. In some examples, the uplink manager of first gateway 112A can notify the health check manager process to suspend its regular WAN health check probes to second gateway 112B based on cluster splits identified by overlay tunnels. WAN health checks can be sent on virtual uplink 724 identified as unresponsive. The notification instructing the suspension of this WAN health check can be issued immediately after the cluster manager confirms the connection loss. This saves resources because WAN details do not need to be handled in the event of a LAN failure. Other processes can also be started or stopped.

[0130] In box 840, the uplink manager at the first gateway 112A can update the default gateway to remove the second gateway 112B and utilize an alternative connection to the Internet 170. This suspension can be based on an unresponsive LAN connection to a second ISP 132 (using the established cluster 700) via the second gateway 112B, which also provides access to the Internet 170.

[0131] In some examples, the new default gateway can point to the first physical uplink 720. A pause in communication sent via the first virtual uplink 724 can be achieved by notifying the FPAPPS process to remove any routes via the virtual uplink tunnel 724 without waiting for the HCM process (e.g., using a WAN ICMP probe that includes a first attempt, waiting ten seconds, repeating three times, etc.). As shown in Table 3 (for the first gateway 112A), the reachability identifier of the second gateway 112B can also be changed from "reachable" to "unreachable". In some examples, WAN health checks can be paused based on a second connection to the Internet 170 identified via the LAN.

[0132] Using the default gateway, the first gateway 112A can continue to send communication with the first ISP 130 only on the first physical uplink 720, thereby minimizing expected service loss to the second ISP 132 via the first virtual uplink 724, which is unreachable due to cluster splitting. Communication and other operations enabled by the creation of cluster 700 can be stopped. By continuing to send communication via the first physical uplink 720, the uplink manager can perform a load balancing process with the remaining devices to avoid packet loss. The load can be rebalanced within milliseconds after a fault is detected. This millisecond-level load rebalancing significantly reduces service loss. In addition to listening for notifications from the cluster manager process, the uplink manager can also listen for virtual uplink tunnel heartbeats and uplink health check manager processes.

[0133] This process offers several advantages. For example, using cluster heartbeats to identify connectivity issues and halting WAN heartbeat transmissions helps to confirm faults more quickly and reroute services accordingly. This can help mitigate service loss and allow for faster rebalancing of service loads to prevent prolonged service loss. As discussed in this paper, WAN health check timers are typically set to a higher frequency, such as approximately 10 seconds as mentioned above, to accommodate packet loss. However, internal LAN networks do not present the same packet loss and latency issues. Therefore, identifying losses in the LAN network, rather than the WAN network, allows for more aggressive health checks. Faster identification of virtual uplink losses in the LAN network allows for load rebalancing, preventing the use of virtual uplinks for uplink load balancing, as using virtual uplinks would result in failures.

[0134] Although the process described herein is implemented from the first gateway 112A, a similar process can also be implemented from the second gateway 112B, which also identifies the second virtual uplink as disconnected and the first ISP 130 as inaccessible (e.g., using both WAN heartbeat probes and virtual uplink probes). This operation can be performed concurrently with a similar operation performed by the first gateway 112A. For example, the second gateway 112B can use a second physical uplink to set a new default gateway, which could be a new default path to the second ISP 132. When setting a new default gateway, the second gateway 112B can suspend or stop the WAN connection to the first ISP 130 via the second virtual uplink, thereby removing the first gateway 112A as the default gateway (e.g., using the FPAPPS procedure), so that no traffic is sent via that connection. As shown in Table 3 (for the second gateway 112B), the reachability identifier of the first gateway 112A can also be changed from "reachable" to "unreachable".

[0135] In box 850, the cluster manager can continue to attempt to re-establish the connection, for example, in box 850A when the cluster manager is inside the first gateway 112A, and in box 850B when the second gateway 112B uses the cluster heartbeat procedure. In other words, the WAN HCM heartbeat can be stopped, but the cluster heartbeat procedure can continue through periods of no response. The cluster health check procedure can be performed to send one or more probes over the working LAN connection based on retry intervals (e.g., in milliseconds), while the first gateway 112A in cluster 700 continues to send packet transmissions over the first physical uplink 720, regardless of the cluster's communication characteristics (e.g., the first virtual uplink 724). In some examples, the cluster manager at the first gateway 112A can notify the uplink manager after a detected cluster split and a persistent cluster split following a retry interval. In some examples, the cluster manager at the first gateway 112A can continue to listen to the second gateway 112B in response to the cluster heartbeat, based on time intervals configurable by one or more orchestrators and one or more retry attempts.

[0136] As an illustrative example, the cluster manager can send cluster heartbeat probes to the second gateway 112B at millisecond-level time intervals. The set time interval can be every 96 milliseconds. In one embodiment, the set time interval can range up to 500 milliseconds. Therefore, if a probe is sent at 100 milliseconds and no response is received within 600 milliseconds, the lack of a response can indicate a probe failure. A probe failure can identify a cluster split.

[0137] In box 860, the cluster manager can receive one or more responses from gateway 112, including responses from first gateway 112A (box 860A), and detect that second gateway 112B has successfully rejoined the communication network after a certain amount of time (box 860B). Detecting that second gateway 112B has rejoined can indicate that first virtual uplink 724 is now reachable (and update Table 3 from "unreachable" to "reachable").

[0138] In box 862, the cluster manager can detect that a heartbeat from a WAN health check has been restored via the first virtual uplink 724 with the second gateway 112B (outside of the cluster heartbeat process). The cluster manager can immediately notify the uplink manager of the first gateway 112A. Detecting that the heartbeat has been restored indicates that the first virtual uplink 724 is now reachable.

[0139] The uplink manager can instruct the FPAPPS process to add the default gateway back to the second gateway 112B via the first gateway 112A (using virtual uplink 724). With multiple default gateways, the service load can be rebalanced via both physical and virtual uplinks. Furthermore, the uplink health check manager process can be resumed for all paths (e.g., the first physical uplink 720 and the first virtual uplink 724). By adding additional data paths, the FPAPPS process can achieve load balancing by again issuing packets on both the first physical uplink 720 and the first virtual uplink 724 (e.g., via the second gateway 112B).

[0140] In box 870 (shown as first box 870A to first gateway 112A and second box 870B to second gateway 112B), the cluster manager can automatically recreate the cluster using first gateway 112A and second gateway 112B. For example, upon detecting that second gateway 112B has rejoined the communication network (indicating that virtual uplink 724 is now reachable), the cluster manager can also notify the uplink manager of first gateway 112A of the health status of second gateway 112B. The cluster manager can also instruct WAN health check probes to restart from first gateway 112A (used for virtual uplink 724) to the second ISP 132 via second gateway 112B.

[0141] Cluster 700 can recover. Cluster recovery indicates that virtual uplink 724 is reachable again. After this initial indication of virtual uplink reachability, the virtual uplink can be monitored for a longer period to ensure stability. Once stability is ensured, the uplink manager can be notified that the virtual uplink is reachable again. To confirm that the virtual uplink is reachable, the health check manager of the first gateway 112A can initiate a WAN health check probe for virtual uplink 724 in the next scheduled probe interval slot of physical uplink 720 to periodically realign the probe of virtual uplink 724 with physical uplink 720.

[0142] As described in this article, the Health Check Manager probe can be configured to run on both the physical uplink 720 and the virtual uplink 724 for each WAN health check probe. If the virtual uplink 724 is reachable, the Health Check Manager can receive ICMP responses to ICMP probes sent through the virtual uplink. The Health Check Manager then notifies the Uplink Manager that it has confirmed the virtual uplink is reachable again. Following the notification from the Health Check Manager, the Uplink Manager can instruct FPAPPS to add routes to the virtual uplink 724 again for load balancing because the virtual uplink is reachable.

[0143] Figure 9 An illustrative example is provided, for example, Figure 9 This demonstrates how the cluster manager responds to cluster splitting to instruct gateway services.

[0144] In box 900, a cluster manager procedure may be required to check the reachability of the virtual uplink due to an event. This initial event could include several events that caused the virtual uplink to be lost. For example, an initial event could be that the second gateway is reloaded.

[0145] In box 902, the result may appear. The result could be a cluster split. A cluster split can occur at time T0. Due to the initial event and its consequences, a cluster manager procedure may be needed to determine whether the virtual uplink can be used to send traffic.

[0146] In box 904, the cluster manager procedure can detect cluster splits. Cluster splits can occur at time = T0. Cluster splits can be detected within milliseconds after they occur. Therefore, for times in seconds, cluster splits can be detected at time = T0.

[0147] In box 906, the cluster split can be notified to the uplink manager at time T0. This can be the same time the cluster manager process detects the cluster split.

[0148] In box 908, the uplink manager can immediately suspend the virtual uplink service, and can do so at time = T. 0+2 Immediately stop the health check manager probe on the virtual uplink.

[0149] In box 910, all services can be sent through the physical uplink 720 in the first gateway 112A, instead of through the unreachable virtual uplink 724.

[0150] In box 911, the cluster manager can identify whether the virtual uplink to the second gateway 112B has been re-established. If so, the HCM probe can be scheduled or restarted because the first gateway 112A or another cluster manager can reach the second gateway 112B.

[0151] In box 912, the HCM process may optionally issue the next scheduling probe. The next scheduling probe may be issued at time T1.

[0152] In box 914, retry attempts can be issued at a frequency of 10 seconds. A total of three probes can be issued. After the initial probe is issued at time = T1, two additional probes can be issued at times = T1+10 and T1+20.

[0153] In some examples, in response to the cluster split identifier and the cluster manager's instruction to the HCM at the first gateway 112A not to perform this step, boxes 912 and 914 are instructed to be removed. For example, if the virtual uplink cluster manager procedure as described in the previous paragraph is executed, the uplink manager can immediately pause the HCM probe at time = T0 during box 908. This can happen before issuing the next scheduled HCM probe, as in box 912. This can also happen before issuing a retry HCM probe, as in box 914. Therefore, potential operations 912 and 914 can be omitted.

[0154] Figure 10 An example of a virtual uplink reachability confirmation process according to the above embodiments is shown. Figure 8 The illustrative topology provided can perform the operations described in this article.

[0155] In box 1000, the cluster can recover. Any of these events can indicate that the virtual uplink may become reachable again.

[0156] In box 1002, the virtual uplink 724 of the first gateway 112A can be monitored to ensure its stability.

[0157] In box 1004, the uplink manager can be notified of a stable LAN status. A stable LAN status can occur when both the virtual and physical uplinks are reachable.

[0158] In box 1006, the uplink manager can notify the health check manager to initiate a health check probe.

[0159] In box 1008, the health check manager can wait for the next time slot scheduled for probes of both the physical and virtual uplinks. Probes can be issued for both the physical and virtual uplinks at a set frequency. For example, probes can be issued every 10 seconds. If the virtual uplink is lost, the health check manager can pause virtual uplink probes but can continue issuing physical uplink probes. Therefore, when the health check manager resumes issuing virtual uplink probes, it can wait for the next scheduled probe for the physical uplink so that it can issue probes together.

[0160] In box 1010, after a probe is sent, the health check manager can receive probe responses (e.g., ICMP request and response probes) sent on virtual uplink 724. The responses confirm that virtual uplink 724 is reachable.

[0161] In box 1012, the health check manager can notify the uplink manager of any probe responses. A probe response can indicate that virtual uplink 724 is reachable and that traffic can be sent on virtual uplink 7240 without loss.

[0162] In box 1014, the uplink manager can remove the paused status designation of virtual uplink 724 and mark virtual uplink 724 as reachable. Service load can then be balanced between the physical uplink and the virtual uplink.

[0163] In some examples, the FPAPPS procedure can add the virtual uplink communication route to the second ISP 132 in Table 1 via the second gateway 112B, as shown in this document.

[0164] Use data path monitoring to identify WAN and LAN service loss.

[0165] Gateway 112 typically monitors the data path between two devices, including via a virtual uplink from first gateway 112A and second ISP 132, or via a physical uplink from first gateway 112A and first ISP 130. The FPAPPS procedure can monitor each data path and identify one or more routes used to send data. The FPAPPS procedure can be initiated by SD-WAN 140 via, for example, a connection between first gateway 112A and SD-WAN 140 (e.g., SD-WAN orchestrator 144), such as... Figure 11 As shown in the example topology.

[0166] For example, first gateway 112A may identify one or more flags associated with the data path between first gateway 112A and second gateway 112B, including second virtual uplink 1126. When a first problem is identified, the data path (second virtual uplink 1126) may be associated with a flag in the session table indicating that the problem has occurred. If the problem is resolved, the flag can be removed from the data path in the table. If the flag is identified a second time for the same data path, it can be determined that the data path has a problem, and traffic from first gateway 112A to second gateway 112B via second virtual uplink 1126 can be redirected to a different data path, such as first physical uplink 1120 to first ISP 130, or via an overlay tunnel when cluster 1100 is established between first gateway 112A and second gateway 112B to access the Internet 170 through second ISP 132.

[0167] For ease of explanation, Figure 12The first gateway 112A is shown as having a cluster manager 1202 and an uplink manager 1204, although these components can be implemented in any example of the gateway described herein, including Figure 7 topology and Figure 8 The process. For example, when a gateway receives cluster configuration settings sent from SD-WAN 140, both the first gateway 112A and the second gateway 112B can include a cluster manager (e.g., a client agent).

[0168] Figure 12 It shows the usage Figure 11 The above embodiment of the topology shown illustrates an example process for identifying WAN or LAN service loss using data path monitoring.

[0169] In box 1210, SD-WAN 140 can send cluster profiles and configuration settings to one or more gateways to create a gateway cluster 1100. Clustering may or may not be implemented to use the FPAPPS process described herein, but the process is independent of cluster formation. As shown, communication for forming the cluster can be sent from SD-WAN 140 to a first gateway 112A (box 1210A) and from SD-WAN 140 to a second gateway 112B (box 1210B). First gateway 112A and second gateway 112B can form the gateway cluster 1100 by receiving configuration settings from SD-WAN 140 (or other cloud-managed clustering processes) to form a cluster manager for each gateway. Devices in the cluster can communicate via LAN connections (e.g., via an IPSec or GRE tunnel of switch 162). When gateway cluster 1100 is formed, each device in the cluster can handle services as a high-availability (HA) feature. If one device fails, other devices can perform operations from its location, for example, by running heartbeats for peer devices in the cluster and sending instructions to a second device in the cluster. One or more cluster heartbeat packets can be sent through the same communication channel as the default gateway.

[0170] In box 1220, a cluster health check can be initiated. For example, when gateway cluster 1100 is established, first gateway 112A can initiate a cluster health check with other devices in the cluster (including second gateway 112B). During the cluster health check, first gateway 112A can send one or more cluster health check probes to second gateway 112B and expect to receive a response within a threshold time period. The cluster health check can wait for the threshold time period, which may be less than the expected time period for a response to the cluster health check, until the cluster health check can determine whether second gateway 112B is disconnected.

[0171] In box 1230, a WAN health check can be initiated. For example, first gateway 112A can initiate an HCM health check by sending one or more HCM probes to second gateway 112B and expecting a response within a threshold time period. The HCM health check can continue to make a predetermined number of retries until the HCM health check can determine whether second gateway 112B is disconnected.

[0172] In either of boxes 1220 or 1230, the first gateway 112A can monitor any packets originating on the data path between the first gateway 112A and the second gateway 112B, including LAN or WAN uplinks. A similar monitoring procedure can be implemented from the second gateway 112B to track the same data path (not shown). When monitoring the data path, any packets sent from the first gateway 112A to the switch 162 or the second gateway 112B via the virtual uplink 1124 can be associated with WAN services. Packets may include, for example, heartbeat probes, data packets, management packets, and other forms of data transmitted electronically. Response packets sent via the virtual uplink 1124 can also be tracked. If a response packet is seen in the same session, the FPAPPS procedure can identify the response packet and continue monitoring the data path.

[0173] During the WAN HCM health check in box 1230, heartbeat probes can be periodically sent to the second gateway 112B (e.g., IP 8.8.8.8). The default probe frequency can be 10 seconds, with a retrace period of 3 seconds. When a probe is sent, a data path session can be created in the uplink session table on the data path.

[0174] In some examples, the data path uplink session table may only maintain sessions connected to the Internet 170 via the WAN. The configuration manager may create forward and reverse session entries in the data path uplink session table at a known frequency (e.g., every 10 seconds).

[0175] In box 1240, the uplink manager 1204 can detect one or more flags in the data path uplink session table. For example, a forward session entry can be created every 10 seconds. If no reverse session is seen, a flag can be added to the data path uplink session table. In this illustration, a "Y" flag will be created and added to the table. However, since each forward session expects a reverse session as a response (e.g., corresponding to request and response probes), the "Y" flag should not be seen. The same process can be followed for each retry.

[0176] Various flags can be used to mark probes or other packets sent by the first gateway 112A, such as Figure 13As shown. One or more of these flags can be added to the packet header of a data path session. For example, "F" can indicate fast packet aging, "S" can indicate source, "N" can indicate destination, "D" can indicate rejection, "R" can indicate redirection, "Y" can indicate desynchronization, "H" can indicate high priority, "P" can indicate setting priority, "T" can indicate setting ToS, "C" can indicate client, "M" can indicate mirroring, "V" can indicate VoIP, "Q" can indicate real-time quality analysis, "u" (lowercase "U") can indicate upstream real-time quality analysis, "h" can indicate high value, "A" can indicate application firewall inspection, "I" can indicate deep inspection, and "U" (uppercase "U") can indicate local destination. “E” can be a media depth check, “G” can be a media signal, “r” can be a routing next hop, “h” can be a high value, “A” can be an application firewall check, “i” (lowercase “I”) can refer to a session classified on the first packet, “J” can refer to SD-WAN default probe statistics used as an alternative, “f” (uppercase “F”) can refer to FEC enabled for a session, “X” can refer to an SD-WAN anomaly, “x” (lowercase “x”) can refer to a transition, “B” can refer to permanent, “O” can refer to open flow, “L” can refer to logs, “o” can refer to an open flow configuration revision mismatch, and “Z” can refer to a session being redirected to IDPS.

[0177] exist Figure 14 The document provides an illustrative example of a data service. In this example, a source device (e.g., first gateway 112A) corresponding to IP address 172.33.192.1 can send packets to a destination device (second gateway 112B) corresponding to IP address 8.8.8.8. The FPAPPS procedure can monitor the transmissions on the data path between the devices and generate a table showing the transmissions marked according to flags.

[0178] For example, in the first example 1400, when a packet includes “FCI”, the packet may be associated with fast aging, may be a client device service, and may be subject to deep inspection. In the second illustration 1410, when a packet includes “FYI”, the packet may be associated with fast aging, may be associated with a client synced with a destination device, and may be subject to deep inspection.

[0179] When no probe response is received from the second gateway 112B within a threshold time period (e.g., as a reverse session in the session table) and the session is still active, the FPAPPS procedure can mark the probe with a flag (e.g., a "Y" flag) indicating that there is no synchronization between the first gateway 112A and the switch 160 via the virtual uplink 1124 or the second gateway 112B. Figure 14 The second figure 1410 shows that when the first gateway 112A sends a probe to the second gateway 112B, and the FPAPPS process indicates that the first gateway 112A has not received a probe response from the second gateway 112B, the next probe sent by the first gateway 112A to the second gateway 112B is marked with a "Y" flag.

[0180] exist Figure 12 In some examples of the uplink manager 1204, the FPAPPS procedure can create a counter to determine the number of flags identified in a session. If a "Y" flag is seen, the counter can be incremented for each "Y" flag that appears in all retry attempts.

[0181] As an illustrative example, if a packet is sent at 10:00:00, then all probe packets sent at 10:00:00 should not have a “Y” flag. If five probes are sent at 10:00:00, and if a “Y” flag appears for each of the five sessions, then the uplink manager 1204 can determine that no reverse packets have been received.

[0182] In box 1250, the uplink manager can cross-check details with the WAN HCM (e.g., when a counter exceeds a threshold). If the WAN HCM does not identify the same communication problem as the lost response, the uplink manager can determine that there is an error in the WAN health check process. During the WAN health check, if a lost probe response is not detected, an error message can be generated and sent.

[0183] In some examples, as described herein, WAN heartbeat probes sent during WAN health checks can be paused to remove additional and unnecessary communication from the network. Some heartbeat probes may include HCM heartbeats, which can be canceled because the connection to the device may have already been identified by a flag in the data path uplink session table.

[0184] In box 1260, the default gateway can be updated to remove the second gateway 112B, as described in this article.

[0185] In box 1270, uplink manager 1204 can receive probe responses from second gateway 112B. When a probe response is received, the “Y” flag can be removed from the data path uplink session table, and second gateway 112B can be re-established as the default gateway.

[0186] In some examples, a cluster is established using a first gateway 112A and a second gateway 112B. In this example, the uplink manager 1204 of the first gateway 112A can expect to identify probe responses during any health check process described herein, including cluster health checks, WAN health checks, or LAN heartbeats. Each of these health checks can operate in different sessions stored in a session table by the uplink manager 1204, and a flag can be assigned to each of these sessions. If no probe response is received within a threshold time period, the uplink manager 1204 can detect a transmission delay of the probe response for that session. In another example, the uplink manager 1204 can expect to identify probe responses within a session. If no response packet is identified for the same session, the uplink manager 1204 can detect a transmission delay of the response packet. In either example, the absence of a probe response can indicate a potential uplink loss, or a disconnected connection or a disconnected device. For confirmation, the FPAPPS process can mark probes with flags or identifiers.

[0187] In some examples, uplink manager 1204 can initiate a default gateway change within an established cluster, as described herein. Uplink manager 1204 can also notify cluster manager 1202 to initiate a default gateway change, as described herein. When no cluster is established, uplink manager 1204 can initiate a default gateway change by removing the gateway from the affected gateway's DHCP table or by otherwise updating the routing path used to send or receive data (e.g., using the FPAPPS procedure to identify the route used to send data).

[0188] In some examples, once a session is marked as unresponsive by uplink manager 1204, uplink manager 1214 can check the virtual uplink between first gateway 112A and second gateway 112B in one or more ways. When a cluster is established between first gateway 112A and second gateway 112B, FPAPPS can check the status of the data path from cluster manager 1202. Cluster manager 1202 can be aware of potential loss, as indicated by cluster splitting as described herein. In another example, the FPAPPS procedure can check the status of the virtual uplink by actively checking heartbeat probe responses by communicating directly with uplink manager 1204 of first gateway 112A or second gateway 112B (e.g., via virtual uplink engine 220 or another uplink manager). The result of receiving or not receiving each heartbeat probe response can be sent to the uplink manager to adjust the default connection from the virtual uplink to the physical uplink.

[0189] In some examples, after pausing LAN connectivity and WAN health checks, cluster manager 1202 can continue to attempt to re-establish connections with the second cluster device (e.g., using probes or a cluster heartbeat process). When a response to the cluster heartbeat is received from the second gateway, cluster manager 1202 can leverage the second gateway device to modify the gateway device cluster (e.g., because it is able to generate and send responses to attempts to re-establish connections).

[0190] In some examples, in response to a change in the default gateway by uplink manager 1204 or cluster manager 1202, the data path for data transmission can be updated to communicate with different ISPs. For example, the data path can be changed from virtual uplink 1124 to physical uplink 1120 to the first ISP 130.

[0191] In some examples, the source of uplink connectivity status can be associated with priority. For example, the FPAPPS procedure can identify a virtual uplink problem and notify the uplink manager 1204 at the first gateway 112A. If the uplink manager 1204 does not detect the same uplink problem, it can by default accept the uplink problem identified by the FPAPPS procedure as having a higher priority than the procedure executed at the first gateway 112A (e.g., HCM status check). In some examples, the FPAPPS procedure identifying the uplink connectivity status of gateway 112 can be more reliable than performing an HCM status check at an individual gateway to identify an uplink problem.

[0192] In some examples, cluster manager 1202 or uplink manager 1204 can notify SD-WAN 140 (e.g., SD-WAN overlay tunnel orchestrator 142) that the cluster or uplink connection is lost.

[0193] As described throughout the disclosure, additional heartbeats and health probes can be stopped or restarted in response to an indication of a failed connection. Pausing these processes can help reduce the amount of traffic sent in the network and reduce the amount of traffic lost due to failed connections.

[0194] The threshold number of retries or the session timeout value during a session can be configured at the user interface (e.g., by an administrative user). After a threshold number of attempts to receive responses to the probe during a session, HCM status checks can be stopped or paused. As an illustrative example, HCM status checks could correspond to a 1-second frequency and a 3-second retry value. Session monitoring can be implemented using an HA VLAN connection or a virtual uplink 1124 using a faster frequency or retry value (e.g., a 900-millisecond frequency and a 3-second retry value). When using a virtual uplink 1124 and stopping HCM status checks, the timeout value can be set to a value faster than the HCM status detection timeout value using the virtual uplink 1124. The session can remain open until the session timeout value is exceeded.

[0195] In some examples, for verification purposes, the identifier of an interrupted uplink identified in one session can be compared with data identified in a second session. For instance, a cluster health check probe with a lost response probe can be compared with data received from an HCM health check running between the first gateway 112A and the second gateway 112B.

[0196] The confirmation of this uplink disconnection process utilizing the FPAPPS procedure is much faster than that of a typical WAN connection relying on HCM. For example, an HCM heartbeat probe can be sent with three retries at a 10-second probe interval, and a connection failure can only be confirmed after three consecutive probes. This means that the first gateway can lose WAN traffic on the virtual uplink during this period (e.g., waiting for the retry attempts to end). Conversely, by utilizing the FPAPPS procedure, which marks potential problems in the data path, FPAPPS can send instructions to the first gateway 112A to adjust the default connection from the virtual uplink to the physical uplink when no probe response is received within a threshold time period.

[0197] Example computing components and implementations

[0198] If a virtual uplink becomes unreachable due to events such as unplanned reloading, cluster splitting, or other events, implementing one or more of the proposed embodiments can significantly reduce packet loss on the virtual uplink. For example, using a virtual uplink tunnel heartbeat can limit traffic loss to a period of just over three seconds and reduce the amount of data packets sent over the network when the system detects a problem with the uplink or device preventing other devices from receiving these packets. Implementing any of the processes or systems described herein can reduce traffic loss to millisecond intervals and also reduce the amount of traffic sent. For example, once a virtual uplink is confirmed to be unaccessible, there is no need to wait for the health check manager process, which could take nearly 40 seconds to confirm the loss of the virtual uplink. The virtual uplink can be immediately and proactively paused, probes sent through the virtual uplink in the legacy system can be canceled, stopped, or paused, and other necessary services can continue solely through the still operational physical uplink. Because services can be sent through the physical uplink, there is no need to impair the remaining services on the network.

[0199] Figure 15 Example computing components that can be used to implement the various methods and systems described herein are illustrated. In this illustration, computing platform 1500 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 15 In the example implementation, computing platform 1500 includes hardware processor 1502 and machine-readable storage medium 1504.

[0200] Hardware processor 1502 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in machine-readable storage medium 1504. Hardware processor 1502 may acquire, decode, and execute instructions throughout the description herein. As an alternative to or in addition to retrieving and executing instructions, hardware processor 1502 may include one or more electronic circuits comprising electronic components for performing the functions of one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0201] Machine-readable storage media (such as machine-readable storage media 1504) can be any electronic, magnetic, optical, or other physical storage device that includes or stores executable instructions. Therefore, machine-readable storage media 1504 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some embodiments, machine-readable storage media 1504 can be a non-transitory storage medium, wherein the term "non-transitory" excludes transient propagation signals. As described in detail below, machine-readable storage media 1504 can be encoded with executable instructions, such as instructions 1506-1514 described above.

[0202] Hardware processor 1502 can execute instructions 1506 to monitor network accessibility from the gateway. For example, the gateway device can use a Health Check Manager (HCM) procedure to monitor network accessibility via both physical and virtual uplinks. In some examples, the gateway device monitors accessibility by sending a first Internet Control Message Protocol (ICMP) probe from the gateway device to a first ISP on the physical uplink and a second ICMP probe from the gateway device to a switch and subsequently to a second gateway on the virtual uplink. Accessibility to the device via each uplink can be determined by receiving one or more responses from the first ISP or the second gateway via each uplink, such as... Figures 3-4 As shown.

[0203] Hardware processor 1502 can execute instructions 1508 to mark the status of uplinks. For example, the gateway device can identify whether each uplink in the uplink is reachable. This can be based, for example, on receiving a response to the HCM procedure via a virtual uplink rather than via a physical uplink within a threshold time period. The gateway device can mark the status of a virtual uplink as reachable and the status of a physical uplink as unreachable.

[0204] Hardware processor 1502 can execute instruction 1510 to update the default gateway. For example, the gateway device can update the default gateway to a virtual uplink and remove the physical uplink. The default gateway can utilize the virtual uplink to maintain data transmission from the gateway device to the network.

[0205] Hardware processor 1502 can execute instruction 1512 to suspend or stop a connection to an uplink while maintaining data transmission via different uplinks. For example, a gateway device can suspend or stop a connection via a physical uplink to remove data transmission from the gateway device to the network via the physical uplink, while maintaining data transmission from the gateway device to the network via a virtual uplink.

[0206] Hardware processor 1502 can execute instruction 1514 to continue monitoring uplink accessibility. For example, a gateway device can continue monitoring network accessibility via the physical uplink by listening to the physical uplink tunnel heartbeat from the physical uplink.

[0207] Figure 16 Example computing components that can be used to implement the various methods and systems described herein are illustrated. In this illustration, computing platform 1600 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. Figure 16 In the example implementation, computing platform 1600 includes a hardware processor 1602 and a machine-readable storage medium 1604. In some examples, computing platform 1600 is a first gateway device.

[0208] The hardware processor 1602 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 1604. The hardware processor 1602 may acquire, decode, and execute instructions throughout the description herein. Alternatively or otherwise, the hardware processor 1602 may include one or more electronic circuits comprising electronic components for performing the functions of one or more instructions, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits.

[0209] Machine-readable storage media (such as machine-readable storage media 1604) can be any electronic, magnetic, optical, or other physical storage device that includes or stores executable instructions. Therefore, machine-readable storage media 1604 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some embodiments, machine-readable storage media 1604 can be a non-transitory storage medium, wherein the term "non-transitory" excludes transient propagation signals. As described in detail below, machine-readable storage media 1604 can be encoded with executable instructions, such as instructions 1606-1612 described above.

[0210] Hardware processor 1602 can execute instructions 1606 to determine a second gateway. The second gateway can be communicatively connected to the first gateway device via a virtual uplink through a local area network (LAN) connection, wherein the first gateway device enables a physical uplink to a first network accessible by an Internet Service Provider (ISP) via the first gateway device, and the second gateway device enables the virtual uplink of the first gateway device via the LAN connection to connect to the second network accessible by the ISP via the second gateway device.

[0211] Hardware processor 1602 can execute instructions 1608 to form a gateway cluster with the first gateway device and the second gateway device. In some examples, the gateway device cluster can be formed between the first gateway device (e.g., computing platform 1600) and the second gateway device. The gateway device cluster can be formed via a cluster manager located at a cloud-based software-defined WAN (SD-WAN) or within the first gateway device. In addition to virtual uplinks to each ISP, the first and second gateway devices can also communicate via a LAN connection.

[0212] Hardware processor 1602 can execute instruction 1610 to initiate a cluster heartbeat process. For example, using a cluster manager, the first gateway can initiate a cluster heartbeat process to the second gateway via a LAN connection.

[0213] Hardware processor 1602 can execute instruction 1612 to suspend the virtual uplink and WAN health check, update the default gateway, and simultaneously maintain data transmission outside the gateway cluster when no response is received from the cluster heartbeat process from the second gateway device. In some examples, the virtual uplink is suspended due to unreachability, and the WAN health check sent to the second gateway device via the virtual uplink is also suspended. In some examples, the default gateway is updated to remove the virtual uplink, and the first gateway device can utilize a physical uplink outside the gateway device cluster as the default connection to a first network accessible by an ISP via the first gateway device. In some examples, the first gateway device can simultaneously maintain data transmission from a physical uplink outside the gateway device cluster to a first network accessible by an ISP via the first gateway device.

[0214] Figure 17 Example computing components that can be used to implement the various methods and systems described herein are illustrated. In this illustration, computing platform 1700 can be, for example, a server computer, a controller, or any other similar computing component capable of processing data. In some examples, computing platform 1700 may be implemented wholly or partially in at least one of a first gateway device and a second gateway device (described below). In some examples, computing platform 1700 may be implemented wholly or partially in a cloud-based application / platform. In some examples, computing platform 1700 may be implemented partially in at least one of a first gateway device and a second gateway device, and partially in a cloud-based application / platform.

[0215] exist Figure 17 In the example implementation, computing platform 1700 includes hardware processor 1702 and machine-readable storage medium 1704.

[0216] The hardware processor 1702 may be one or more central processing units (CPUs), semiconductor-based microprocessors, and / or other hardware devices suitable for retrieving and executing instructions stored in the machine-readable storage medium 1704. The hardware processor 1702 may acquire, decode, and execute instructions throughout the description herein. Alternatively or otherwise, the hardware processor 1702 may include one or more electronic circuits comprising electronic components, such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), or other electronic circuits, for performing the functions of one or more instructions.

[0217] Machine-readable storage media (such as machine-readable storage media 1704) can be any electronic, magnetic, optical, or other physical storage device that includes or stores executable instructions. Therefore, machine-readable storage media 1704 can be, for example, random access memory (RAM), non-volatile RAM (NVRAM), electrically erasable programmable read-only memory (EEPROM), storage devices, optical discs, etc. In some embodiments, machine-readable storage media 1704 can be a non-transitory storage medium, wherein the term "non-transitory" excludes transient propagation signals. As described in detail below, machine-readable storage media 1704 can be encoded with executable instructions, such as instructions 1706-1716 described above.

[0218] Hardware processor 1702 can execute instructions 1706 to determine a subset of shortest interval heartbeats for at least two types of heartbeat traffic detected between a first gateway device and a second gateway device. The first gateway device can communicatively connect to the second gateway device via a virtual uplink through a local area network (LAN) connection. Among the types of heartbeat traffic detected between the first and second gateway devices, at least two types of heartbeat traffic in the subset of shortest interval heartbeats can have the shortest interval. In some examples, the first and second gateway devices can include a gateway device cluster, and one of the at least two types of heartbeat traffic in the subset of shortest interval heartbeats can include a cluster heartbeat (as described above, the time interval between cluster heartbeats can be on a millisecond scale). In some examples, the first gateway device can enable a physical uplink to a first network accessible via an Internet Service Provider (ISP) through the first gateway device, and the second gateway device can enable a virtual uplink via a LAN connection for the first gateway device to connect to a second network accessible via an ISP through the second gateway device.

[0219] Since this technical example is designed for comprehension, different types of heartbeat services (e.g., VRRP advertising heartbeat probes and responses, GRE heartbeat probes and responses, cluster heartbeat probes and responses, branch gateway peer tunnel heartbeat probes and responses, etc.) can have different time intervals between heartbeats. As mentioned above, this technical example can utilize heartbeats with shorter time intervals (e.g., millisecond intervals versus 10-second intervals) for LAN health checks to achieve faster / more responsive LAN health checks. Therefore, by determining a subset of heartbeats with the shortest time intervals for at least two types of detected heartbeat services, the hardware processor 1702 can improve the speed / efficiency of performing LAN health checks.

[0220] Hardware processor 1702 can execute instruction 1708 to determine the loss of heartbeat response from the second gateway device to the first gateway device for at least two of the two types of heartbeat services in the shortest time interval heartbeat subset.

[0221] In some examples, this could include monitoring forward and reverse session entries in a data session table to detect reverse session entries associated with a first gateway device and a second gateway device, and marking at least two of the two types of heartbeat traffic in the shortest time interval heartbeat subset as out of sync with the forward session entries. As described above, the data session table could include a table / representation recording heartbeat probes and responses between the first gateway device and the second gateway device (e.g., combined with...). Figure 14 The data session table is illustrated in Figures 1400 and 1410. In various examples, an FPAPP running on at least one of the first and second gateway devices can maintain the data session table, including marking reverse session entries as synchronous and asynchronous. A forward session entry in the data session table can be associated with a heartbeat probe sent from the first gateway device to the second gateway device, and a reverse session entry in the data session table can be associated with a heartbeat response sent from the second gateway device to the first gateway device (indicating the reception of the heartbeat probe). When a heartbeat response is detected from the second gateway device, the heartbeat response can be recorded as a “synchronous” reverse session entry associated with the forward session entry / heartbeat probe. However, when no heartbeat response is detected for a given heartbeat probe / forward session entry, an “unsynchronized” reverse session entry can be recorded in the data session table, indicating a “missing” heartbeat response (i.e., indicating that no heartbeat response was detected for the given heartbeat probe / reverse session entry). As mentioned above, the “lost” heartbeat response may be caused by LAN health-related issues affecting the virtual uplink between the first gateway device and the second gateway device, which may in turn affect the transmission of the heartbeat probe to the second gateway device (causing the second gateway device not to send a heartbeat response) and / or the “lost” heartbeat response from the second gateway device to the first gateway device.

[0222] It should also be noted that the hardware processor 1702 determines that heartbeat responses from the second gateway device to the first gateway device are lost for at least two types of heartbeat services within the shortest time interval heartbeat subset. This feature better ensures that the determined lost heartbeat responses are caused by LAN health-related issues affecting the virtual uplink between the first and second gateway devices, rather than by a problem caused by one type of heartbeat service. In other words, if the shortest time interval heartbeat subset includes only a single type of heartbeat service, the hardware processor 1702 will have a harder time determining whether the determined lost heartbeat responses are caused by LAN health-related issues affecting the virtual uplink between the first and second gateway devices, rather than a problem with the heartbeat service type (e.g., a problem with cluster heartbeat probe responses). In contrast, the likelihood of at least two types of heartbeat services simultaneously encountering non-LAN health-related issues is lower. Therefore, if the hardware processor 1702 determines that heartbeat responses for at least two types of heartbeat services within the shortest time interval heartbeat subset are lost, the determined lost heartbeats are more likely caused by LAN health-related issues affecting the virtual uplink between the first and second gateway devices. For the reasons described above, hardware processor 1702 determines that heartbeat responses from the second gateway device to the first gateway device have been lost for two of at least two types of heartbeat traffic within the shortest time interval heartbeat subset. In some examples, taking this into account, hardware processor 1702 may reduce the number of heartbeat traffic types within the shortest time interval heartbeat subset (e.g., to two or three types) to improve computational efficiency / speed. That is, hardware processor 1702 can save processing resources by reducing the size of the shortest time interval heartbeat subset (e.g., to two or three types).

[0223] As described above, when hardware processor 1702 determines that only one or more heartbeat responses are missing for at least two types of heartbeat services in the shortest time interval heartbeat subset, hardware processor 1702 can determine that the lost heartbeat responses are caused by heartbeat-related problems affecting that type of heartbeat service (e.g., there may be problems affecting only cluster heartbeats). Here, hardware processor 1702 can again determine that the lost heartbeat responses are not caused by LAN health-related problems affecting the virtual uplink between the first gateway device and the second gateway device, because other types of heartbeat services have not lost heartbeat responses. Therefore, when hardware processor 1702 determines that only one or more heartbeat responses are missing for at least two types of heartbeat services in the shortest time interval heartbeat subset, hardware processor 1701 can issue an alert indicating that there is a heartbeat-related problem for that type of heartbeat service.

[0224] When a heartbeat response from the second gateway device to the first gateway device is lost in at least two types of heartbeat traffic within the shortest time interval heartbeat subset, the hardware processor 1702 may execute instruction 1710 to suspend (e.g., make unreachable) the virtual uplink between the first and second gateway devices. In some examples, when suspending the virtual uplink, the hardware processor 1702 may: (1) update the default gateway to remove the virtual uplink and utilize the physical uplink as the default connection to a first network accessible by an ISP via the first gateway device (as described above); and (2) simultaneously maintain data transmission from the physical uplink to the first network accessible by an ISP via the first gateway device (as described above). In this way, a (determined) failed virtual uplink between the first and second gateway devices can be avoided until the LAN health is restored.

[0225] In some examples, after suspending the virtual uplink between the first and second gateway devices, the hardware processor 1702 can resume the virtual uplink between the first and second gateway devices upon detecting a heartbeat response from the second gateway device to the first gateway device. As described above, in some examples, this may include monitoring forward and reverse session entries in the data session table to detect that a reverse session entry associated with the first and second gateway devices is marked as synchronized with a forward session entry.

[0226] Figure 18 A block diagram of an example computer system 1800 in which various embodiments described herein may be implemented is shown. The computer system 1800 includes a bus 1802 or other communication mechanism for transmitting information, and one or more hardware processors 1804 coupled to the bus 1802 to process information. The hardware processors 1804 may be, for example, one or more general-purpose microprocessors.

[0227] Computer system 1800 also includes main memory 1806 coupled to bus 1802, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 1806 is used to store information and instructions to be executed by processor 1804. Main memory 1806 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1804. When stored in storage media accessible to processor 1804, such instructions render computer system 1800 as a special-purpose machine customized to perform the operations specified in the instructions.

[0228] Computer system 1800 also includes read-only memory (ROM) 1808 or other static storage devices coupled to bus 1802 for storing static information and instructions of processor 1804. Storage devices 1810, such as disks, optical discs, or USB thumb drives (flash drives), are provided and coupled to bus 1802 to store information and instructions.

[0229] Computer system 1800 may be coupled to display 1812, such as a liquid crystal display (LCD) (or touchscreen), via bus 1802 to display information to the computer user. Input device 1814, including alphanumeric and other keys, is coupled to bus 1802 to transmit information and command selections to processor 1804. Another type of user input device is cursor control 1816, such as a mouse, trackball, or arrow keys, which transmits directional information and command selections to processor 1804 and controls cursor movement on display 1812. In some embodiments, the same directional information and command selections as those of the cursor control can be received via touch on the touchscreen without a cursor.

[0230] The computing system 1800 may include a user interface module for implementing a GUI, which may be stored in a mass storage device as executable software code to be executed by the computing device(s). For example, this module and other modules may include components such as software components, object-oriented software components, class components and task components, procedures, functions, properties, processes, subroutines, program code segments, drivers, firmware, microcode, circuit systems, data, databases, data structures, tables, arrays, and variables.

[0231] Generally, the terms "component," "engine," "system," "database," and "data storage" used in this document can refer to logic embodied in hardware or firmware, or to a collection of software instructions that may have entry and exit points, written in a programming language such as Java, C, or C++. Software components can be compiled and linked into an executable program, installed in a dynamic link library, or written in an interpreted programming language such as BASIC, Perl, or Python. It should be understood that software components can be invoked from other components or from themselves, and / or can be invoked in response to detected events or interrupts. Software components configured to execute on a computing device may be provided on computer-readable media, such as optical discs, digital video discs, flash drives, disks, or any other tangible media, or as digital downloads (and may be initially stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution). Such software code may be stored, in part or in whole, on a storage device executing the computing device for execution by the computing device. Software instructions may be embedded in firmware such as EPROM. It will be further understood that hardware components may include connected logic units, such as gates and flip-flops, and / or may include programmable units, such as programmable gate arrays or processors.

[0232] Computer system 1800 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in conjunction with the computer system, cause or program the computer system 1800 into a special-purpose machine. According to one embodiment, the techniques herein are executed by computer system 1800 in response to processor(s) 1804 executing one or more sequences of one or more instructions contained in main memory 1806. Such instructions may be read into main memory 1806 from another storage medium, such as storage device 1810. Execution of the instruction sequence contained in main memory 1806 causes processor(s) 1804 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used in place of or in combination with software instructions.

[0233] As used herein, the term "non-transitory media" and similar terms refer to any medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such non-transitory media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as storage device 1810. Volatile media include dynamic memory such as main memory 1806. Common forms of non-transitory media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips or cartridges, and their network versions.

[0234] Non-transient media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transmission of information between non-transient media. Examples of transmission media include coaxial cables, copper wires, and optical fibers, including conductors and Bus 1802. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0235] Computer system 1800 also includes an interface 1818 coupled to bus 1802. Interface 1818 provides bidirectional data communication coupled to one or more network links connected to one or more local networks. For example, interface 1818 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem for providing data communication connectivity to a corresponding type of telephone line. As another example, interface 1818 may be a Local Area Network (LAN) card for providing data communication connectivity to a LAN-compatible network (or a WAN component communicating with a WAN). Wireless links may also be implemented. In any such implementation, interface 1818 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0236] A network link typically provides data communication to other data devices via one or more networks. For example, a network link can provide a connection to a host computer or to a data device operated by an Internet Service Provider (ISP) via a local network. The ISP, in turn, provides data communication services through a global packet data communication network now commonly referred to as the "Internet." Both local networks and the Internet use electrical, electromagnetic, or optical signals that carry streams of digital data. Signals through various networks, as well as signals on network links and through interface 1818, are example forms of transmission media that transmit digital data to and from computer system 1800.

[0237] Computer system 1800 can send and receive messages, including program code, through (multiple) networks, network links, and interface 1818. In the Internet example, the server can send application request codes through the Internet, ISP, local network, and interface 1818.

[0238] The received code may be executed by processor 1804 upon receipt and / or stored in storage device 1810 or other non-volatile storage device for later execution.

[0239] Each process, method, and algorithm described in the preceding sections can be embodied in a code component executed by one or more computer systems or computer processors, including computer hardware, and can be fully or partially automated by them. One or more computer systems or computer processors can also operate to support the performance of related operations in a “cloud computing” environment or as “Software as a Service” (SaaS). Processes and algorithms can be implemented partially or wholly in a dedicated circuit system. The various features and processes described above can be used independently of each other or can be combined in various ways. Different combinations and sub-combinations are intended to fall within the scope of this disclosure, and certain method or process blocks may be omitted in some implementations. The methods and processes described herein are not limited to any particular sequence, and associated blocks or states can be executed in other suitable sequences, or can be executed in parallel, or in some other way. Blocks or states can be added to or removed from the disclosed example embodiments. The performance of certain operations or processes can be distributed among computer systems or computer processors, residing not only within a single machine but also deployed across multiple machines.

[0240] As used herein, the circuit can be implemented using any form of hardware, software, or a combination thereof. For example, one or more processors, controllers, ASICs, PLAs, PALs, CPLDs, FPGAs, logic components, software routines, or other mechanisms can be used to construct the circuit. In implementation, the various circuits described herein can be implemented as discrete circuits, or the described functions and features can be shared partially or wholly among one or more circuits. Even if various features or functional elements can be described separately or claimed as separate circuits, these features and functions can be shared among one or more common circuits, and such description does not require or imply the need for a separate circuit to implement such features or functions. Where the circuit is implemented wholly or partially using software, such software can be implemented to operate in conjunction with a computing or processing system (e.g., computer system 1800) capable of performing the functions described with respect to the circuit.

[0241] As used herein, the term “or” can be interpreted as inclusive or exclusive. Furthermore, singular descriptions of resources, operations, or structures should not be construed as excluding plural forms. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “can,” “could,” “might,” or “may” is generally intended to convey that certain embodiments include certain features, elements, and / or steps, while other embodiments do not.

[0242] Unless otherwise expressly stated, the terms and phrases used in this document, and their variations thereof, should be interpreted as open-ended rather than restrictive. Adjectives such as “routine,” “traditional,” “normal,” “standard,” “known,” and similar terms should not be interpreted as limiting the described items to those available at a given time or period, but should be understood to include routine, traditional, normal, or standard techniques that may be available or known now or in the future. In some cases, the appearance of broadening words and phrases such as “one or more,” “at least,” “but not limited to,” or other similar phrases should not be interpreted as an intentional or necessary use of a narrower case where such a broadening phrase might not exist.

Claims

1. A first gateway device comprising: a memory; and one or more processors configured to execute machine-readable instructions stored in the memory to: determine a second gateway device communicatively connected to the first gateway device via a virtual uplink over a local area network (LAN) connection, wherein the first gateway device enables a physical uplink to a first network accessible via an internet service provider (ISP) of the first gateway device and the second gateway device enables the virtual uplink for the first gateway device to connect to a second network over the LAN connection, the second network accessible via an ISP of the second gateway device; form a gateway device cluster with the second gateway device, wherein the gateway device cluster is formed via a cluster manager located at a cloud-based software-defined wide area network (SD-WAN) or internal to the first gateway device, and the first gateway device and the second gateway device communicate via the LAN connection in addition to the virtual uplink to each ISP; initiate, using the cluster manager, a cluster heartbeat process from the first gateway device to the second gateway device over the LAN connection; and when a response to the cluster heartbeat process is received from the first gateway device and not the second gateway device within a threshold time period, notify an uplink manager of the first gateway device of a network issue affecting communication with the second network device, wherein the uplink manager is configured to: suspend the virtual uplink as unreachable, suspend WAN health checks sent to the second gateway device via the virtual uplink, update a default gateway device to remove the virtual uplink and utilize the physical uplink outside of the gateway device cluster as a default connection to the first network accessible via the ISP of the first gateway device, and while maintaining data transmission from the physical uplink outside of the gateway device cluster to the first network accessible via the ISP of the first gateway device.

2. The first gateway device of claim 1, wherein the threshold time period is less than a time to perform a health check manager (HCM) process, the HCM process comprising: sending a first probe from the first gateway device to a first internet service provider (ISP) over the physical uplink; sending a second probe from the first gateway device to a switch and in turn to a second ISP of the second gateway device over the virtual uplink; and receiving one or more responses from the first ISP or from the second ISP of the second gateway device.

3. The first gateway device of claim 2, wherein the first probe and the second probe are internet control message protocol (ICMP) probes.

4. The first gateway device of claim 2, wherein the first probe and the second probe are user datagram protocol (UDP) probes. ​ 5. The first gateway device of claim 1, wherein the one or more processors execute machine readable instructions to further: receive a profile and configuration settings from the SD-WAN prior to forming the gateway device cluster with the second gateway device.

6. The first gateway device of claim 1, wherein the LAN connection is implemented as an overlay tunnel through a generic routing encapsulation (GRE) tunnel.

7. The first gateway device of claim 1, wherein the cluster heartbeat process exchanges information about the gateway device cluster and a media access control (MAC) address to the second gateway device.

8. The first gateway device of claim 1, wherein the one or more processors execute machine readable instructions to further: send configuration settings to one or more access points (APs) associated with the gateway device cluster, wherein the configuration settings include a bucket map that indicates how the one or more APs forward client traffic, and wherein the bucket map contains an internet protocol (IP) address of each of the gateway devices in the gateway device cluster.

9. The first gateway device of claim 8, wherein the APs use the bucket map to support client load balancing when the LAN connection is implemented as an overlay tunnel through a generic routing encapsulation (GRE) tunnel.

10. The first gateway device of claim 1, wherein suspending the virtual uplink immediately to be unreachable is performed by the uplink manager, and the default gateway device for the virtual uplink is removed by a fast path application (FP APPS) process.

11. The first gateway device of claim 1, wherein the one or more processors execute machine readable instructions to further: after suspending the virtual uplink and the WAN health check, continue attempting to reestablish a connection with the second gateway device by the cluster manager; and when a response to the cluster heartbeat process is received from the second gateway device, reform the gateway device cluster with the second gateway device.

12. A method for network management, comprising: determining, by a first gateway device, a second gateway device that is communicatively connected to the first gateway device through a local area network (LAN) connection via a virtual uplink, wherein the first gateway device enables a physical uplink to a first network that is accessible via an internet service provider (ISP) of the first gateway device, and the second gateway device enables the virtual uplink for the first gateway device to connect to a second network through the LAN connection, the second network being accessible via an ISP of the second gateway device; forming a gateway device cluster with a second gateway device, wherein the gateway device cluster is formed via a cluster manager located at a cloud-based software-defined wide area network (SD-WAN) or internal to the first gateway device, and the first and second gateway devices communicate via the LAN connection in addition to the virtual uplink to each ISP; initiating, using the cluster manager, a cluster heartbeat process from the first gateway device to the second gateway device via the LAN connection; and when a response to the cluster heartbeat process is received from the first gateway device and not the second gateway device within a threshold time period, notifying an uplink manager of the first gateway device of a network issue affecting communication with the second network device, wherein the uplink manager is configured to: suspend the virtual uplink as unreachable, suspend WAN health checks sent to the second gateway device via the virtual uplink, update a default gateway device to remove the virtual uplink and utilize the physical uplink outside of the gateway device cluster as a default connection to the first network accessible via the ISP of the first gateway device, and while maintaining data transmission from the physical uplink outside of the gateway device cluster to the first network accessible via the ISP of the first gateway device.

13. The method for network management of claim 12, wherein the threshold time period is less than a time to perform a health check manager (HCM) process, the HCM process comprising: sending a first probe from the first gateway device to a first internet service provider (ISP) over the physical uplink; sending a second probe from the first gateway device to a switch over the virtual uplink and in turn to a second ISP of the second gateway device; and receiving one or more responses from the first ISP or from the second ISP of the second gateway device.

14. The method for network management of claim 12, further comprising: receiving a profile and configuration settings from the SD-WAN prior to forming the gateway device cluster with the second gateway device.

15. The method for network management of claim 12, wherein the LAN connection is implemented as an overlay tunnel over a generic routing encapsulation (GRE) tunnel.

16. The method for network management of claim 12, wherein the cluster heartbeat process exchanges information about the gateway device cluster and a media access control (MAC) address to the second gateway device.

17. The method for network management of claim 12, further comprising: sending configuration settings to one or more access points (APs) associated with the gateway device cluster, wherein the configuration settings include a bucket map indicating how the one or more APs forward client traffic, and wherein the bucket map contains an internet protocol (IP) address of each of the gateway devices in the gateway device cluster.

18. The method for network management of claim 17, wherein the AP uses the bucket graph to support client load balancing when the LAN connection is implemented as an overlay tunnel through a generic routing encapsulation (GRE) tunnel.

19. The method for network management of claim 12, wherein suspending the virtual uplink as unreachable is performed by the uplink manager, and the default gateway device for the virtual uplink is removed by a fast path application (FPA) PS process.

20. A non-transitory computer-readable storage medium storing a plurality of instructions executable by one or more processors, the plurality of instructions, when executed by the one or more processors, cause the one or more processors to: determine a second gateway device communicatively connected to a first gateway device by a local area network (LAN) connection via a virtual uplink, wherein the first gateway device enables a physical uplink to a first network accessible via an internet service provider (ISP) of the first gateway device, and the second gateway device enables the virtual uplink for the first gateway device to connect to a second network by the LAN connection, the second network accessible via an ISP of the second gateway device; form a gateway device cluster with the second gateway device, wherein the gateway device cluster is formed via a cluster manager located at a cloud-based software-defined wide area network (SD-WAN) or inside the first gateway device, and the first gateway device and the second gateway device communicate via the LAN connection in addition to the virtual uplink to each ISP; initiate, using the cluster manager, a cluster heartbeat process from the first gateway device to the second gateway device via the LAN connection; and when a response to the cluster heartbeat process is received from the first gateway device but not the second gateway device within a threshold period of time, notify an uplink manager of the first gateway device about a network issue affecting communication with the second network device, wherein the uplink manager is configured to: suspend the virtual uplink as unreachable, suspend WAN health checks sent to the second gateway device via the virtual uplink, update a default gateway device to remove the virtual uplink, and utilize the physical uplink outside of the gateway device cluster as a default connection to the first network accessible via the ISP of the first gateway device, and while maintaining data transmission from the physical uplink outside of the gateway device cluster to the first network accessible via the ISP of the first gateway device.

Images