Method and apparatus for application-aware central clustering technology for ultra-large-scale SD-WAN

Abstract

Today, a single cluster of forwarding hub nodes in a software-defined wide area network (SD-WAN) is bound to a fixed expansion ratio. For example, a cluster of N nodes has an expansion factor of 1:N as a fixed ratio. If the first allocated cluster node is overloaded, the next node in the cluster (i.e., the second node) takes over, and so on until the span reaches all N available nodes. Today's cluster services ignore application requirements and are bound to a rigid scheme for providing clustering services to multiple peer edge nodes (e.g., in a hub-and-spoke topology). In this way, high-priority real-time application traffic flows are handled in the same way as low-priority (e.g., bulk) traffic flows relative to the expansion ratio within the cluster. This may subsequently result in suboptimal performance in providing and load-balancing traffic within the cluster and, in some cases, may also lead to underutilization of cluster resources.

Description

BACKGROUND OF THE INVENTION

[0001] Today, a single cluster of forwarding hub nodes in a software-defined wide area network (SD-WAN) is bound to a fixed expansion ratio. For example, a cluster of N nodes has an expansion factor of 1:N as a fixed ratio. If the first allocated cluster node is overloaded, the next node in the cluster (i.e., the second node) takes over, and so on until the span reaches all N available nodes. Today's cluster services ignore application requirements and are bound to a rigid scheme for providing clustering services to multiple peer edge nodes (e.g., in a hub-and-spoke topology). In this way, high-priority real-time application traffic flows are handled in the same way as low-priority (e.g., bulk) traffic flows relative to the expansion ratio within the cluster. This may subsequently result in sub-optimal performance in providing and load-balancing traffic within the cluster and, in some cases, may also result in under-utilization of cluster resources. SUMMARY OF THE INVENTION

[0002] Some embodiments provide a software-defined wide area network (SD-WAN) that includes a first branch location (first branch) and a data center location (data center). The data center includes a plurality of forwarding hub nodes, and the branch site includes at least one edge forwarding node. The edge node at the branch site receives packets of a specific flow that have flow attributes. The edge node uses the flow attributes of the packets to identify a hub selection rule from a plurality of hub selection rules, each of the plurality of hub selection rules identifying a set of one or more forwarding hub nodes of a data center for receiving one or more flows from the branch site. At least one hub selection rule identifies at least one forwarding hub node that is unique to that hub selection rule (i.e., not identified by other hub selection rules). The edge node uses the identified hub selection rule to identify the forwarding hub node for the specific flow and sends the packet from the branch site to the identified forwarding hub node of the data center.

[0003] In some embodiments, the forwarding hub node acts as a gateway for the SD-WAN that provides access from the first branch site to other branch sites or third-party data centers. In some embodiments, the third-party data center includes a software-as-a-service (SaaS) data center (e.g., a data center for a video conferencing SaaS provider, a data center for a middleware (e.g., firewall) service provider, a data center for a storage service provider, etc.). In some embodiments, the branch sites and the third-party data centers are topologically arranged in a hub-and-spoke topology around the data center such that traffic between the two sites passes through the forwarding hub nodes in the data center (i.e., regardless of the geographical location of the sites).

[0004] In combination or alternatively, in some embodiments, a forwarding hub node provides a branch site with access to computing, storage, and service resources of a data center. Examples of such resources include computing machines (e.g., virtual machines and / or containers providing server operations), storage machines (e.g., database servers), and middleware service operations (e.g., firewall services, load balancing services, encryption services, etc.). In some embodiments, the connection between a first branch site and a data center hub node is a secure encrypted connection that encrypts packets exchanged between an edge node at the first branch site and the data center hub node. Examples of secure encrypted connections used in some embodiments include VPN (Virtual Private Network) connections or secure IPsec (Internet Protocol Security) connections.

[0005] In some embodiments, a branch edge node identifies a hub selection rule for a received packet by matching flow attributes of the packet with match criteria of the hub selection rule, where the hub selection rule associates the match criteria with one or more identifiers of one or more forwarding hub nodes of a data center. According to some embodiments, the match criteria of the hub selection rule are defined based on flow attributes. In some embodiments, flow attributes used for the matching operation include a flow identifier of the received packet (e.g., a five-tuple identifier of the received packet, i.e., source and destination Internet Protocol (IP) addresses / port numbers and protocol).

[0006] In combination or alternatively, in some embodiments, the flow identifier used for the matching operation includes flow attributes other than layer 2-4 (L2-L4) header values, such as layer 7 (L7) attributes. Examples of L7 attributes include AppID (e.g., traffic type identifier), user identifier, group identifier (e.g., Active Directory (AD) identifier), threat level, and application name / version. To obtain L7 attributes, some embodiments perform deep packet inspection (DPI) on the packet.

[0007] By using L7 attributes to define the match criteria of the hub selection rule, some embodiments allow forwarding flows to different forwarding hub nodes based on different context attributes associated with the flow (i.e., assigning different forwarding hub nodes to different classes of flows). For example, in some embodiments, the hub selection rule associates different sets of flows containing different types of traffic (identified by different AppIDs) with different sets of forwarding hub nodes. In some embodiments, assigning forwarding hub nodes based on L7 attributes allows certain classes of traffic to be prioritized over other classes of traffic. For example, a first class of flows containing a first type of traffic (e.g., VoIP) determined to be of a high-priority type may be assigned more forwarding hub nodes compared to a second class of flows containing a second type of traffic determined to be of a low-priority type.

[0008] As described above, one or more matching criteria for the hub selection rules can be defined based on other L7 context attributes, such as user identifiers, group identifiers, threat levels, and application names / versions. For example, in some embodiments, the hub selection rules associate a set of flows having user identifiers corresponding to executives or finance personnel with a first set of forwarding hub nodes, while associating a set of flows having user identifiers other than those corresponding to executive or finance status with a second set of forwarding hub nodes.

[0009] In some embodiments, each hub selection rule identifies a different set of forwarding hub nodes that can be used for selection (e.g., for processing flows belonging to the same class as the matching packet). Accordingly, in some embodiments, when a matching hub selection rule is found, the edge node selects a forwarding hub node from the set of forwarding hub nodes identified by the hub selection rule. In some embodiments, the edge node selects a forwarding hub node from the set based on load balancing criteria (e.g., weight values) and load balancing policies (e.g., round robin, etc.).

[0010] In some embodiments, the SD-WAN controller provides the hub selection rules to the branch edge nodes. The controller receives network traffic statistics from the forwarding hub nodes, aggregates the received statistics by flow class, and analyzes the statistics to identify flow classes that require additional or fewer forwarding hub nodes in their respective sets of forwarding hub nodes. In some embodiments, when the total amount of traffic associated with a particular class of flows is found to exceed a maximum threshold of traffic or fall below a minimum threshold of traffic, the controller determines that additional or fewer forwarding hub nodes are needed to process the particular class of flows. According to some embodiments, when the controller determines that additional forwarding hub nodes are needed for a particular flow class, the controller instructs the manager (e.g., server) of the data center to generate additional forwarding hub nodes. Conversely, in some embodiments, when the controller determines that fewer forwarding hub nodes are needed for a particular flow class, the controller can reassign the redundant forwarding hub nodes to other flow classes.

[0011] When the controller instructs the manager of the data center to generate additional forwarding hub nodes, in some embodiments, the controller sends an updated list of the forwarding hub node group to the branch edge nodes. In some embodiments, the updated list is provided via updated hub selection rules (e.g., having updates to the forwarding hub node groups specified for each hub selection rule). In some embodiments, the forwarding hub node groups specified for each hub selection rule are identified by group identifiers. Thus, in some embodiments, the controller simply provides the updated group identifiers to the edge nodes. Conversely, or alternatively, in some embodiments, the controller provides the updated group identifiers in the form of updated hub selection rules that reference the updated group identifiers.

[0012] The foregoing summary is intended as a brief introduction to some embodiments of the present invention. It is not meant to be an introduction or overview of all inventive subject matter disclosed herein. The following detailed description and the drawings referred to in the detailed description will further illustrate the embodiments described in the summary as well as other embodiments. Accordingly, a full review of the summary, detailed description, drawings, and claims is needed to understand all embodiments described herein. Additionally, the claimed subject matter is not limited by the illustrative details set forth in the summary, detailed description, and drawings. Brief Description of the Drawings

[0013] The novel features of the invention are set forth in the appended claims. However, for purposes of explanation, several embodiments of the invention are set forth in the following drawings.

[0014] Figure 1 Conceptually illustrates an example of an SD-WAN according to some embodiments, the SD-WAN including a plurality of branch sites connected to a hub of a data center.

[0015] Figure 2 Conceptually illustrates another example of an SD-WAN according to some embodiments, the SD-WAN including a cluster of controllers for configuring components of the SD-WAN.

[0016] Figure 3 Conceptually illustrates an example component of an edge node of a branch site according to some embodiments.

[0017] Figure 4 Illustrates the process of an edge node for selecting a hub to which to forward packets according to some embodiments.

[0018] Figure 5 Illustrates the process of a controller for managing the configuration of edge nodes and hubs of an SD-WAN according to some embodiments.

[0019] Figure 6Conceptual map illustrating a computer system implementing some embodiments of the present invention. Detailed Description

[0020] In the following detailed description of the present invention, numerous details, examples, and embodiments of the present invention are set forth and illustrated. However, it will be clear and apparent to those skilled in the art that the present invention is not limited to the described embodiments, and that the present invention may be practiced without some of the specific details and examples discussed.

[0021] Some embodiments provide a software-defined wide area network (SD-WAN) that includes one or more branch sites (branch locations) and data centers (data center locations). The data center includes a plurality of forwarding hub nodes (hereinafter referred to as "hubs"), and each branch site includes at least one edge node. In some embodiments, the edge nodes are deployed in the form of high-availability pairs at each branch site, such that each branch site includes an active edge node and a standby edge node in case of failure. The edge nodes at the branch site receive packets of a flow, the packets having flow attributes. The edge nodes use the flow attributes of the packets to identify a hub selection rule from a plurality of hub selection rules, each of the plurality of hub selection rules identifying a set of one or more hubs in a data center for receiving one or more flows from the branch site and including matching criteria defined according to the flow attributes. In some embodiments, at least one hub selection rule identifies at least one hub that is unique to that hub selection rule (i.e., not identified by other hub selection rules). The edge nodes use the identified hub selection rule to identify the hub of the flow and send the packets from the branch site to the identified hub in the data center (i.e., in accordance with the identified hub selection rule).

[0022] Figure 1 Conceptual map illustrating an SD-WAN network (hereinafter also referred to as a virtual network) for interconnecting multiple branch sites with each other and connecting to resources of a centralized data center. In this example, SD-WAN 100 is created for interconnecting branch sites 130-136 with each other and connecting to resources 160 of data center 105 (data center) and SaaS data center 140 via a set of hubs 112-116 (also referred to herein as forwarding hub nodes) of hub cluster 110. SD-WAN 100 is established by a controller cluster (not shown), a set of hubs 112-116, and four edge nodes 120-126, with one edge node in each of the branch sites 130-136.

[0023] In some embodiments, an edge node is an edge machine (e.g., a virtual machine (VM), a container, a program executing on a computer, etc.) and / or a stand-alone device operating at a multi-computer location of a particular entity (e.g., at the entity's office or data center) to connect the computers at its respective location to the hub and other edge nodes (if so configured). In some embodiments, the edge node is a cluster of edge nodes at each branch site. In other embodiments, the edge nodes are deployed to each branch site in the form of a high-availability pair such that one edge node in the pair is the active edge node and the other edge node in the pair is a standby edge node that can take over as the active edge node in case of a failure. Further, in this example, the set of hubs 112 - 116 are deployed as machines (e.g., VMs or containers) in the same common data center 105. In other embodiments, the hubs can be deployed in different common data centers.

[0024] Examples of entities for which such a virtual network can be established include commercial entities (e.g., companies), non-profit entities (e.g., hospitals, research institutions, etc.), and educational entities (e.g., universities, colleges, etc.) or any other type of entity. Examples of public cloud providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, etc., while examples of entities include enterprises (e.g., companies, partnerships, etc.), organizations (e.g., schools, non-profit organizations, government entities, etc.), and so on. In other embodiments, the hubs can also be deployed in the private cloud data center of a virtual WAN provider that hosts the hubs to establish an SD-WAN for different entities.

[0025] In Figure 1 the example of, the hub is a multi-tenant forwarding element that can be used to establish a secure connection link (e.g., a tunnel) with edge nodes at a multi-computer site of a particular entity (such as a branch site (branch office), a data center (e.g., a third-party data center), etc.). For example, the set of hubs 112 - 116 in cluster 110 provides access from each branch site 130 - 136 to each other branch site 130 - 136, as well as access to the SaaS data center 140 via the connection links 150 that terminate at cluster 110 as shown. According to some embodiments, these multi-computer sites are typically located at different physical locations (e.g., different buildings, different cities, different states, etc.). In some embodiments, the forwarding hub nodes can be deployed as physical nodes or virtual nodes. Additionally, in some embodiments, the forwarding hub nodes can be deployed at the premises where the data center is located, while in other embodiments, the forwarding hub nodes can be deployed in the cloud (e.g., as a set of virtual edges configured as a cluster).

[0026] Additionally, Figure 1For example, the set of hubs 112-116 also provides access to resources 160 (e.g., machines) in the data center 105. More specifically, the set of hubs 116 provides access to resources 160. In some embodiments, the resources include a set of one or more servers (e.g., web servers, database servers) within a microservices container (e.g., a pod). Optionally or alternatively, some embodiments include multiple such microservices containers, each accessible via a different set of one or more hubs in the data center. In accordance with some embodiments, the resources and the hubs are within the data center location.

[0027] In accordance with some embodiments, the edge nodes 120-126 are forwarding elements that exchange packets with one or more hubs and / or other edge nodes via one or more secure connection links. In this example, all the secure connection links of the edge nodes use the set of hubs 112-116. Figure 1 It is also illustrated that via the set of hubs 112, the SD-WAN 100 allows the edge nodes to connect to the SaaS data center 140. Although not shown, some embodiments include SaaS data centers, and in accordance with some embodiments, each of the multiple different SaaS data centers is accessible via a different set of hubs. In some embodiments, the SaaS data centers include data centers for video conferencing SaaS providers, data centers for middleware (such as firewalls) service providers, data centers for storage service providers, etc. As shown, the branch sites 130-136 and the SaaS data center 140 are topologically arranged in a hub-and-spoke topology around the data center 105. Thus, regardless of the geographical location of the sites, the traffic between any two sites must pass through the set of hubs 112-116 in the data center 105.

[0028] In some embodiments, the set of hubs 112-116 provides the branch sites 130-136 with access to the computing, storage, and service resources (such as resources 160) of the data center. Examples of such resources include computing machines (e.g., virtual machines and / or containers providing server operations), storage machines (e.g., database servers), and middleware service operations (e.g., firewall services, load balancing services, encryption services, etc.). In some embodiments, the connection between the branch site and the data center hub is a secure encrypted connection that encrypts the packets exchanged between the edge node at the branch site and the data center hub. Examples of secure encrypted connections used in some embodiments include VPN (Virtual Private Network) connections or secure IPsec (Internet Protocol Security) connections.

[0029] In some embodiments, multiple secure connection links (e.g., multiple secure tunnels) may be established between an edge node and a hub. In some embodiments, when multiple such links are defined between an edge node and a hub, each secure connection link is associated with a different physical network link between the edge node and the external network. For example, in some embodiments, to access the external network, the edge node has one or more commercial broadband Internet links (e.g., cable mode and fiber optic links), wireless cellular links (e.g., 5G LTE network), etc. for accessing the Internet.

[0030] In some embodiments, each secure connection link between the hub and the edge node is formed as a VPN tunnel between the hub and the edge node. As Figure 1 illustrated in the figure, the collection of hubs 112 also connects the edge node to the SaaS data center 140. In some embodiments, these connections are through secure VPN tunnels. The edge node, the hub, and the collection of secure connections between the edge node, the hub, and the SaaS data center form the SD-WAN 100 for a particular entity.

[0031] Since the collection of hubs 112-116 is a multi-tenant hub, in accordance with some embodiments, they are used to define other virtual networks for other entities (e.g., other companies, organizations, etc.). In some such embodiments, a tenant identifier is stored in the tunnel header that encapsulates the packets to traverse the tunnels defined between the hub and the branch site or other data centers to distinguish the packet flow received from the edge node of one entity from the packet flow received along other tunnels of other entities. In other embodiments, the hub is single-tenant and is specifically deployed for use by only one entity.

[0032] As described above, the edge nodes of some embodiments forward packets to the hub based on hub selection rules, each hub selection rule identifying a collection of one or more hubs (e.g., the collection of hubs 112-116) of a data center for receiving one or more flows from a branch site. In some embodiments, the edge node uses the flow attributes of the received packets to identify the hub selection rule. In accordance with some embodiments, the edge node identifies the hub selection rule for the received packets by matching the flow attributes of the received packets with the matching criteria of the hub selection, where the hub selection associates the matching criteria with one or more identifiers of one or more forwarding hub nodes of the data center. For example, Figure 1Describes two flows 170 and 175 of edge nodes 120 that all originate from branch site 130. The first flow 170 is forwarded to a set of hubs 112 that provide access to SaaS data center 140, while the second flow 175 is forwarded to a set of hubs 116 that provide access to a set of resource machines 160 of data center 105.

[0033] In some embodiments, the matching criteria of the hub selection rule are defined based on flow attributes. In some embodiments, the flow attributes used for the matching operation include the flow identifier of the received packet (e.g., the five-tuple identifier of the received packet, i.e., the source and destination Internet Protocol (IP) addresses / port numbers and protocol). Optionally or alternatively, in some embodiments, the flow identifier used for the matching operation includes flow attributes other than layer 2-4b (L2-L4) header values, such as layer 7 (L7) attributes. Examples of L7 attributes include AppID (e.g., traffic type identifier), user identifier, group identifier (e.g., Active Directory (AD) identifier), threat level, and application name / version. To obtain L7 attributes, some embodiments perform deep packet inspection (DPI) on the packet. Alternatively, some embodiments can utilize a context engine to collect L7 attributes, as further described below.

[0034] By using L7 attributes to define the matching criteria of the hub selection rule, some embodiments allow flows to be forwarded to different hubs based on different context attributes associated with the flow (i.e., assign different hubs to different classes of flows). For example, in some embodiments, the hub selection rule associates different sets of flows containing different types of traffic (i.e., identified by different AppIDs) with different sets of hubs. In some embodiments, allocating hubs based on L7 attributes allows certain classes of traffic to be prioritized over other classes of traffic. For example, a first class of flows containing a first type of traffic (e.g., VoIP) determined to be of a high-priority type may be assigned more hubs compared to a second class of flows containing a second type of traffic determined to be of a low-priority type. Some embodiments also add an attribute to the traffic flow to indicate that the traffic has a higher priority in order to affect the hub selection rule. For example, some embodiments include the location of the edge node (e.g., latitude / longitude, geographical location) as an additional attribute that affects the hub selection rule.

[0035] As described above, one or more matching criteria for the hub selection rules can be defined based on other L7 context attributes, such as user identifiers, group identifiers, threat levels, and application name / version. For example, in some embodiments, the hub selection rules associate a set of flows having user identifiers corresponding to executives or finance personnel with a first set of forwarding hub nodes, while associating a set of flows having user identifiers other than those corresponding to executives or finance status with a second set of hubs. Thus, in some embodiments, congestion is reduced, and it is allowed to more easily prioritize network traffic by allocating hubs based on flow attributes, such that certain flow categories that require a larger number of hubs or resources can be provided with a larger number of hubs or resources.

[0036] In some embodiments, different hub selection rules identify different groups of hubs available for selection for flows that match the rules. Thus, in some embodiments, when a matching hub selection rule is identified for the flow of a received packet, the edge node selects a hub from the group of hubs identified by the matching hub selection rule. In some embodiments, the edge node performs a load balancing operation that distributes the flows matching the hub selection rule among the hubs specified by the hub selection rule based on a set of load balancing criteria (e.g., weight values).

[0037] For example, in some embodiments, the load balancing operation uses weight values to distribute the flows matching the rule among the hubs specified by the hub selection rule in a round-robin manner (e.g., for three hubs with weight values of 2, 3, and 3, the load balancing operation will allocate the first two matching flows to the first hub, the next three matching flows to the second hub, the next three matching flows to the third hub, and then repeat by returning to the first hub for the next two flows).

[0038] In some embodiments, the load balancing weight values are dynamically adjusted based on packet processing statistics collected from edge nodes and / or hubs in some embodiments. In some embodiments, these statistics are collected and distributed by a controller cluster (not shown) of the SD-WAN. In some embodiments, the controller cluster also distributes hub selection. The controller cluster and its operations will be described in more detail below.

[0039] Figure 2 FIG. illustrates an SD-WAN network 200 for interconnecting multiple branch sites 230-236 with each other and with resources of a centralized data center 205. In this example, the SD-WAN 200 is established by a controller cluster 260, a hub cluster 212-216, and four edge nodes 220-226 in a private data center 265, with one edge node in each of the branch sites 230-236.

[0040] The controller cluster 260 acts as a central point for managing (e.g., defining and modifying) configuration data that is provided to edge nodes and / or hubs to configure some or all of the operations. In some embodiments, the controller cluster has a set of manager servers that define and modify the configuration data, and a set of controller servers that distribute the configuration data to the edge nodes and / or hubs. In other embodiments, the controller cluster has only one set of servers that define, modify, and distribute the configuration data. In some embodiments, the control cluster instructs the edge nodes to use certain hubs for different classes of traffic, as described in more detail below.

[0041] Although Figure 2 FIG. illustrates the controller cluster 260 residing in a private data center 265, but in some embodiments, the controller cluster resides in one or more public cloud data centers and / or private cloud data centers. Additionally, some embodiments deploy one or more hubs in one or more private data centers (e.g., the data centers of the entity that deploys the hubs and provides the controller cluster for configuring the hubs to implement a virtual network).

[0042] Figure 2 FIG. further illustrates a set of hub groups 212-216 in the data center 205. In some embodiments, each hub group 212-216 is designated to handle a different class of traffic based on the configuration of the controller cluster 260. The hub group 212 is designated as the hub group for receiving traffic associated with the SaaS data center 240, as shown. In some embodiments, a higher-priority class of traffic is assigned more hubs compared to a lower-priority class of traffic. For example, each hub group 212-216 includes a different number of hubs, where the hub group 216 has the highest number of hubs. In some embodiments, the number of hubs assigned to each traffic class is based on input from a user (e.g., a network administrator).

[0043] As described above, in some embodiments, the controller cluster 260 (the controller) of the SD-WAN provides hub selection rules to the edge nodes 220-226 at the branch sites 230-236 for selecting the hub and / or hub group to which to send packets of traffic. In some embodiments, the hubs of the hub group are configured to provide network traffic statistics collected from the traffic received at the hub to the controller cluster. In some embodiments, the configuration of the hub specifies that the statistics are provided periodically.

[0044] The controller cluster 260 receives network traffic statistics from the hubs of the hub groups 212 - 216, aggregates the received statistics by flow category (e.g., by AppID, user identifier, etc.), and analyzes the statistics to identify flow categories that require additional or fewer hubs in their respective hub groups. For example, in some embodiments, when it is found that the total amount of traffic associated with a particular category of flows exceeds a maximum threshold of traffic or is below a minimum threshold of traffic, the controller cluster 260 determines that additional hubs are required to handle the particular category of flows. In some embodiments, the maximum and minimum thresholds are defined by a user (e.g., a network administrator).

[0045] In accordance with some embodiments, when the controller cluster 260 determines that additional hubs are required for a particular flow category, the controller instructs a manager of the data center (not shown) to generate additional hubs. Conversely, in some embodiments, when the controller determines that fewer hubs are required for a particular flow category, the controller may remove the redundant hubs from the hub group designated for the particular flow category. In some embodiments, the controller may reassign the redundant hubs to other flow categories.

[0046] When the controller instructs the manager of the data center to generate additional hubs, in some embodiments, the controller cluster 260 sends an updated list of the hub groups to the edge nodes 220 - 226. In some embodiments, the updated list is provided via updated hub selection rules (e.g., having updates to the hub groups specified for each hub selection rule). In some embodiments, the hub groups specified for each hub selection rule are identified by using group identifiers. Thus, in some embodiments, the controller cluster simply provides the updated group identifiers to the edge nodes. Conversely, or alternatively, in some embodiments, the controller cluster provides the updated group identifiers in the form of updated hub selection rules that reference the updated group identifiers. The addition and removal of hubs will be discussed further below with reference to Figure 5 Further discussion.

[0047] Figure 3 The conceptual diagram illustrates an example of an edge node 300 of some embodiments of the present invention. As shown, the edge node 300 includes a packet processor 302, a load - balancing hub selector 310, a flow classifier 320, and a connection tracker 350. In some embodiments, the components of the edge node operate on a single machine, while in other embodiments (e.g., when the edge node is a cluster of edge nodes), they operate on separate machines.

[0048] The packet processor 302 is the forwarding engine of the edge forwarding node in some embodiments. For packets of a received flow, in some embodiments, the packet processor 302 first determines whether the connection tracker 350 includes any records related to the flow. The connection tracker stores records 360 about flows that have previously been processed by the edge node. In Figure 3 In the example illustrated in the figure, the stored records 360 of the connection tracker 350 include a flow identifier (e.g., a quintuple identifier), the matching central selection rule of the flow, and the IP address of the selected central of the flow. Although each flow ID is illustrated as having one selected central for each flow, other embodiments may include a list of two or more centrals that have been selected for different packets of the same flow. In other words, in some embodiments, the central is selected based on each flow, while in other embodiments, the central is selected based on each packet. For example, in some embodiments, when the edge node processes and forwards additional packets of the same flow, the records of the connection tracker 350 are updated. In some embodiments, the updated records include statistics about the number of packets in the flow that are forwarded to each central.

[0049] When the packet processor 302 determines that the connection tracker has a record that matches the flow of the received packet (e.g., determines that the quintuple identifier of the packet matches the quintuple identifier of the record in the connection tracker), the packet processor selects a central for the packet by selecting the central specified in the matching connection tracking record. On the other hand, when the packet processor 302 determines that the connection tracker does not store any records related to the flow of the received packet, in some embodiments, the packet processor 302 uses the flow classifier 320 to identify a central selection rule that specifies one or more centrals to be used for the received packet's flow.

[0050] In some embodiments, the flow classifier 320 matches the attributes of the flow with the matching criteria of the central selection rules 340 stored in the storage device 330. As shown, the central selection rules 340 include matching criteria and a corresponding list of available centrals. In some embodiments, the matching attributes are defined based on (1) the quintuple header values of the packet flow (i.e., source IP address, source port address, destination IP address, destination port address, and protocol), and / or (2) context attributes associated with the packet flow. In this example, the matching criteria are defined based on both the quintuple identifier and the traffic type. In some embodiments, some or all of the quintuple header values may be specified as wildcard values.

[0051] Additionally, in this example, each rule specifies a list of its pivots by designating a pivot group identifier (GID), where the GID of each pivot group is an index to another data store of the identifiers (e.g., IP addresses) of the pivots designated in that pivot group. For example, Rule 1 of pivot selection rule 340: (1) matches a flow with a header value that matches 5-tuple ID1 and carries audio streaming content, and (2) designates the corresponding pivot group GID 5. Thus, a flow with a matching 5-tuple identifier and an AppID that identifies audio streaming as the traffic type of that flow will be forwarded to the pivots of pivot group 5. Combinatorially or alternatively, instead of providing a group ID, some embodiments list the available pivots in each pivot group by listing the respective network addresses (i.e., IP addresses) of the available pivots in each pivot group in the pivot selection rule. Similarly, in addition to the traffic type, for the matching criteria, the matching criteria of some embodiments may use different context attributes, or a combination of two or more context attributes.

[0052] In some embodiments, to select a pivot from the available pivots indicated by a matching pivot selection rule, packet processor 302 uses load balancing pivot selector 310. In some embodiments, load balancing pivot selector 310 performs a load balancing operation to identify and select a pivot to which to forward a packet. In some embodiments, load balancing pivot selector 310 uses load balancing criteria stored in storage device 315 to perform its load balancing and pivot selection operations.

[0053] The edge node performs its load balancing operation to distribute the flows that match the rule among the pivots specified by the pivot selection rule. For example, in some embodiments, the load balancing operation uses weight values to distribute the flows that match the rule among the specified pivots of the pivot selection rule in a round-robin manner (e.g., for three pivots with weight values of 2, 3, 3, the load balancing operation will distribute the first two matching flows to the first pivot, the next three matching flows to the second pivot, the next three matching flows to the third pivot, and then repeat by returning to the first pivot for the next two flows). In some embodiments, the weight values are periodically adjusted based on statistical data about the packets processed by the pivot.

[0054] Figure 4 Illustrated is processing 400 of an edge node that receives packets of a specific flow. As shown, processing 400 begins at 410 by receiving a packet having a flow identifier associated with a specific packet flow. In some embodiments, the received packet may be the first packet of the flow, while in other embodiments, the packet may be a subsequent packet of the flow.

[0055] After receiving a packet at 410, at 420, process 400 determines whether a record associated with a particular flow is stored in the connection tracker. As described above with respect to Figure 3 as described, in some embodiments, the connection tracker (e.g., connection tracker 350) stores records about flows that have been processed by the edge node. As described above, in accordance with some embodiments, these stored records include an identifier of the flow, the identified central selection rule for the flow, and one or more central nodes to which the packets of the flow have been forwarded. When a record associated with a particular flow is identified in the connection tracker, the process transfers to 430, where the process identifies the central node previously selected for the flow from the connection tracker record. The process then transfers to 480 to forward the packet to the selected central node.

[0056] Otherwise, when no record associated with a particular flow is stored in the connection tracker, the process transfers to 440 to identify the context attributes of the packet flow. In some embodiments, the context attributes include AppID (e.g., traffic type identifier), user identifier, group identifier (e.g., Active Directory (AD) identifier), threat level, and application name / version. To identify the context attributes of the received packet, some embodiments perform deep packet inspection (DPI) on the received packet. Alternatively, some embodiments utilize a context engine that collects context attributes on the edge node through one or more guest introspection (GI) agents executed on the edge node. In some such embodiments, the context engine provides the collected context attributes to, for example, a flow classifier, such as Figure 3 flow classifier 320.

[0057] After identifying the context attributes of the received packet, process 400 matches the identified context attributes of the flow with the matching criteria of the central selection rule. As described above, in some embodiments, the matching criteria are defined based on flow attributes (e.g., context attributes). For example, in the case of edge node 300, flow classifier 320 accesses the central selection rule from storage device 330 to match the identified context attributes with the matching criteria listed for central selection rule 340. In some embodiments, the central selection rule is received from a controller of the SD-WAN (e.g., controller cluster 260), and as described above, each central selection rule associates the matching criteria with one or more identifiers of one or more central nodes or central node groups in the data center. In some embodiments, the matching criteria of the central selection rule are defined based on flow attributes.

[0058] Next, at 460, the process selects a central node from the central node groups identified as available by the matching central selection rule. Some embodiments utilize the group identifier associated with the central node group to identify the available central node groups for each central selection rule, such as inFigure 3 In an example embodiment. In some embodiments, a controller (e.g., controller cluster 260) may provide a mapping of group identifiers to their respective central groups to edge nodes (e.g., edge nodes 220-226) for the edge nodes to use to identify a particular central in the central group to which to send packets.

[0059] In some embodiments, such as Figure 3 , a load-balancing central selector of an edge node (e.g., load-balancing central selector 310) is responsible for selecting a central. For example, as described above, in some embodiments, load-balancing operations use periodically adjusted weight values to distribute flows matching the rule among the specified centrals in a central selection rule in a round-robin manner.

[0060] In some embodiments, for a packet belonging to a flow having a corresponding record stored by a connection tracker, the same central may be selected for the current packet of the flow. However, as described in more detail below, the available centrals in each central group are dynamically allocated and thus may change between the processing of different packets of a flow. Thus, in some embodiments, the central selected for one packet of a flow may no longer be available for selection for subsequent packets of that flow. In some such embodiments, the load-balancing central selector may select the next available central from the available centrals identified by the matching central selection rule for the flow.

[0061] After selecting a central, the process proceeds to 470 to create a record in the connection tracking storage 360 to identify the central selected for the flow. For example, in some embodiments, the created connection tracking record includes an identifier of the flow, the matching central selection rule, and the central(s) selected for the flow. Whenever process 400 matches a packet with a connection tracking record, in some embodiments, the process updates the connection tracker with other information about the particular flow. For example, in some embodiments, the process updates an existing record to reflect the central selected for the received packet (i.e., if the selected central is a central other than those already reflected in the record).

[0062] After creating the connection tracking record, the process (at 480) forwards the packet to the selected hub. As described above, in some embodiments, the edge node forwards the packet to the selected hub using a direct tunnel established between the edge node and the hub and / or hub group. In some embodiments, multiple secure connection links (e.g., multiple secure tunnels) may be established between the edge node and the hub. When multiple such links are defined between the edge node and the hub, in some embodiments, each secure connection link is associated with a different physical network link between the edge node and the external network. For example, in some embodiments, to access the external network, the edge node has one or more commercial broadband Internet links (e.g., cable mode and fiber optic links), wireless cellular links (e.g., 5G LTE network), etc. to access the Internet. In some embodiments, each secure connection link between the hub and the edge node is formed as a VPN tunnel between the hub and the edge node. Process 400 then ends.

[0063] Figure 5 Illustrates process 500 of a controller of an SD-WAN (e.g., controller cluster 260 of SD-WAN 200). Process 500 begins at 505 by receiving network traffic statistics from a hub / hub group of a data center (e.g., hub group 212-216 of data center 205). As described above, in accordance with some embodiments, the hub / hub group is configured to provide network traffic statistics to the controller / controller cluster.

[0064] At 510, the process aggregates the received network traffic statistics by flow category. In some embodiments, the flows are classified by traffic type (e.g., identified by the AppID of the packet). In some such embodiments, each traffic type has a specified priority level (e.g., high priority, low priority, etc.), which corresponds to the number of hubs that can be allocated to receive the flows of that traffic type. For example, in some embodiments, a first type of traffic designated as high priority may be allocated 70% of the hubs of the data center, while a second type of traffic designated as low priority may be allocated the other 30% of the hubs of the data center. In accordance with some embodiments, the number of hubs allocated for a particular traffic type is defined by a user (e.g., network administrator).

[0065] Once the received network traffic statistics have been aggregated, processing 500 selects a flow category for analysis at 515. Examples of flow categories can include categories based on AppID (e.g., traffic type), user identifier (e.g., administrator, junior employee, etc.), threat level (e.g., high, low, neutral, etc.), and so on. In some embodiments, as described above, each flow category is assigned a priority level. For example, some embodiments that classify flows by traffic type can assign a high priority level to, for example, VoIP traffic and a low priority level to, for example, peer-to-peer email traffic.

[0066] Next, at 520, processing 500 determines whether the amount of traffic associated with the selected flow category has exceeded a maximum threshold specified for that flow category within a minimum duration (e.g., hours, days, weeks, etc.). In some embodiments, the maximum threshold and the minimum duration are each specified by a user (e.g., a network administrator). In some embodiments, the specified maximum threshold and minimum duration may vary between each flow category, while in other embodiments, they are consistent for each flow category.

[0067] When the processing determines that the amount of traffic has not exceeded the maximum threshold within the specified minimum duration, the processing transfers to 525 to determine whether the amount of traffic has fallen below a minimum threshold within the minimum duration. In some embodiments, the minimum duration specified for the maximum threshold and the minimum duration specified for the minimum threshold are equal, while in other embodiments, the specified minimum durations are different.

[0068] When the processing determines at 525 that the amount of traffic associated with the flow is below the minimum threshold within the minimum duration, the processing transfers to 530 to remove redundant hubs from the hub group specified for the selected flow category. In some embodiments, removing redundant hubs includes reassigning the redundant hubs to other flow categories (e.g., other flow categories that may require additional hubs). Otherwise, the processing transfers to 540 to determine whether there are any additional flow categories to analyze.

[0069] Alternatively, when the processing determines at 520 that the amount of traffic associated with the selected flow category has exceeded the maximum threshold within the minimum duration, the processing transfers to 535 to instruct the manager of the data center (e.g., the VeloCloud coordinator) to generate additional hubs, which will be added to the hub group assigned to serve the selected flow category. In some embodiments, when redundant hubs are found for a particular category of flow as described above, these redundant hubs can be assigned to the flow categories that are determined to also require additional hubs along with the newly generated hubs, or as an alternative to generating new hubs.

[0070] Next, at 540, the process determines whether there are any additional flow classes to analyze. When the process determines that there are additional flow classes to analyze, the process transfers back to 515 to select a flow class for analysis. Otherwise, the process transfers to 545 to send the updated central selection rules to the edge nodes of the branch sites, where the updated central selection rules identify any changes (e.g., additions, removals) to the central group. Process 500 then ends.

[0071] Many of the features and applications described above are implemented as software processes, which are specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the one or more processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard disk drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or by wire.

[0072] In this specification, the term "software" means including firmware residing in read-only memory or applications stored in magnetic memory, which can be read into memory for processing by a processor. Additionally, in some embodiments, multiple software inventions may be implemented as sub-parts of a larger program while retaining the different software inventions. In some embodiments, multiple software inventions may also be implemented as separate programs. Finally, any combination of independent programs that together implement the software inventions described herein are within the scope of the present invention. In some embodiments, a software program, when installed to operate on one or more electronic systems, defines one or more specific machine implementations that execute and perform the operations of the software program.

[0073] Figure 6 Conceptual diagram illustrating a computer system 600 that implements some embodiments of the present invention. The computer system 600 can be used to implement any one of the above-described host, controller, central, and edge forwarding elements. Thus, it can be used to execute any one of the above processes. The computer system includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 600 includes a bus 605, a processing unit 610, a system memory 625, a read-only memory 630, a permanent storage device 635, an input device 640, and an output device 645.

[0074] Bus 605 collectively represents all system buses, peripheral buses, and chipset buses that communicatively couple a number of internal devices of computer system 600. For example, bus 605 communicatively couples processing unit 610 with read-only memory 630, system memory 625, and persistent storage device 635.

[0075] Processing unit 610 retrieves instructions to be executed and data to be processed from these various storage units in order to perform the processing of the present invention. In different embodiments, the processing unit may be a single processor or a multi-core processor. Read-only memory (ROM) 630 stores static data and instructions required by processing unit 610 and other modules of the computer system. On the other hand, persistent storage device 635 is a read-write memory device. This device is a non-volatile storage unit that stores instructions and data even when computer system 600 is turned off. Some embodiments of the present invention use a mass storage device (such as a magnetic disk or an optical disk and its corresponding disk drive) as persistent storage device 635.

[0076] Other embodiments use a removable storage device (such as a floppy disk, a flash drive, etc.) as persistent storage device. Similar to persistent storage device 635, system memory 625 is a read-write memory device. However, different from storage device 635, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data required by the processor during operation. In some embodiments, the processing of the present invention is stored in system memory 625, persistent storage device 635, and / or read-only memory 630. Processing unit 610 retrieves instructions to be executed and data to be processed from these various storage units in order to perform the processing of some embodiments.

[0077] Bus 605 is also connected to input devices and output devices 640 and 645. Input devices enable users to convey information to and select commands for the computer system. Input device 640 includes an alphanumeric keyboard and an indicating device (also referred to as a "cursor control device"). Output device 645 displays images generated by the computer system. Output devices include printers and display devices, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include a device that functions as both an input device and an output device, such as a touch screen.

[0078] Finally, as Figure 6 shown, bus 605 also couples computer system 600 to network 665 via a network adapter (not shown). In this way, the computer can be part of a computer network (such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet), or part of a network of networks (such as the Internet). Any or all components of computer system 600 can be used in conjunction with the present invention.

[0079] Some embodiments include electronic components, such as microprocessors, storage devices, and memories, that store computer program instructions in a machine-readable or computer-readable medium (or referred to as computer-readable storage medium, machine-readable medium, or machine-readable storage medium). Some examples of such computer-readable media include RAM, ROM, compact disc read-only memory (CD-ROM), recordable compact disc (CD-R), rewritable compact disc (CD-RW), digital versatile disc read-only (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid state disk drives, read-only and recordable compact discs, hyperdensity discs, any other optical or magnetic medium, and floppy disks. The computer-readable medium can store a computer program executable by at least one processing unit and including an instruction set for performing various operations. Examples of computer programs or computer code include machine code such as that produced by a compiler, and files including high-level code executed by a computer, an electronic component, or a microprocessor using an interpreter.

[0080] Although the discussion above mainly relates to microprocessors or multi-core processors that execute software, some embodiments are carried out by one or more integrated circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0081] The terms "computer", "server", "processor", and "memory" used in this specification all refer to electronic or other technical devices. These terms exclude humans or groups of humans. For this specification, the term "display" means display on an electronic device. The terms "computer-readable medium" and "machine-readable medium" used in this specification are entirely limited to tangible, physical objects that store information in a computer-readable form. These terms exclude any wireless signal, wired download signal, and any other transient or temporary signal.

[0082] Although the present invention has been described with reference to numerous specific details, those of ordinary skill in the art will recognize that the present invention can be embodied in other specific forms without departing from the spirit of the present invention. For example, several of the above embodiments deploy the central hub in a public cloud data center. However, in other embodiments, the central hub is deployed in a third-party private cloud data center (e.g., a data center used by a third party to deploy cloud hubs for different entities in order to deploy virtual networks for these entities). Thus, those of ordinary skill in the art will understand that the present invention is not limited by the foregoing illustrative details, but rather is defined by the appended claims.

Claims

1. For a software-defined wide area network (SD-WAN) connecting a first site, a second site, and a third site, where the first site and the second site are branch sites and respectively include a first forwarding edge and a second forwarding edge, and the third site is a data center and includes a plurality of forwarding hubs, a method for forwarding data comprises: at a forwarding edge of the first branch site, receiving packets of a specific flow, the packets originating from the first branch site and addressed to a destination at the second branch site, the flow being associated with flow attributes; identifying a hub selection rule from a plurality of hub selection rules using the flow attributes, each hub selection rule identifying at least one specific forwarding hub at the data center third site for receiving one or more flows from the first branch site and indicating the received flow to the specific forwarding hub for forwarding to the second branch site, the data center third site including a plurality of server machines accessible by the first site and the second site through the hubs, the plurality of hub selection rules including at least one hub selection rule identifying at least one forwarding hub not identified by other hub selection rules; using the identified hub selection rule to identify a forwarding hub for the specific flow; and sending the packets from the forwarding edge at the first branch site to the identified forwarding hub at the data center third site.

2. The method according to claim 1, wherein the flow attributes include a five-tuple identifier of the packets.

3. The method according to claim 1, wherein the flow attributes include attributes other than layer 2-4 header values.

4. The method according to claim 3, further comprising performing deep packet inspection (DPI) on the packets to identify the flow attributes, wherein the flow attributes include layer 7 attributes of the packets.

5. The method according to claim 4, wherein the layer 7 attributes identify a traffic type of the specific flow.

6. The method according to claim 1, wherein each of the plurality of hub selection rules designates a different set of forwarding hubs for receiving flows of a specific flow category.

7. The method according to claim 1, wherein each of the plurality of hub selection rules includes a matching criterion defined based on flow attributes.

8. The method according to claim 7, wherein identifying a hub selection rule using the flow attributes includes comparing the flow attributes with the matching criteria of the plurality of hub selection rules to identify a matching hub selection rule.

9. The method according to claim 1, wherein at least two hub selection rules identify the same set of forwarding hubs for receiving flows of a first category and a second category from the first site.

10. The method according to claim 1, wherein each of the plurality of forwarding hubs is associated with a different network address, and wherein sending the packets from the forwarding edge at the first site to the identified hub at the data center third site includes sending the packets from the forwarding edge at the first site to the network address associated with the identified hub at the data center third site.

11. The method according to claim 1, wherein the plurality of forwarding hubs act as gateways for accessing resources at other branch sites including a second site or at a third-party data center.

12. The method according to claim 11, wherein the plurality of forwarding hubs also provide access to the resources of the data center.

13. The method according to claim 11, wherein the plurality of branch sites are arranged topologically in a hub-and-spoke topology around the data center.

14. The method according to claim 1, wherein the plurality of hub selection rules are received from a controller of the SD-WAN.

15. The method according to claim 14, further comprising receiving an updated set of hub selection rules from the controller, wherein the updated set of hub selection rules includes at least one hub selection rule that identifies at least one additional forwarding hub for receiving a particular class of flows from the first site.

16. The method according to claim 15, wherein the controller generates the updated set of hub selection rules based on an analysis of network traffic statistics from the plurality of forwarding hubs.

17. The method according to claim 16, wherein the controller analyzes the network traffic statistics to identify a group of forwarding hubs that require additional or fewer forwarding hubs to receive flows from the first site.

18. The method according to claim 1, wherein sending the packet to the identified forwarding hub includes sending the packet via any one of an optical fiber, a cable, 5G, and a digital subscriber line (DSL).

19. The method according to claim 11, wherein the plurality of branch sites are connected to the plurality of forwarding hubs at the data center via one of a virtual private network (VPN) domain and an IPSec domain, wherein the packet is encrypted at a forwarding edge of the plurality of branch sites and decrypted by the plurality of forwarding hubs.

20. The method according to claim 1, wherein at least one hub selection rule identifies a first group of forwarding hubs for first and second different sets of flows, the method further comprising receiving a new hub selection rule that identifies a second group of forwarding hubs to which the second set of flows is to be sent.

21. The method according to claim 1, wherein the forwarding edge of the first site is the first forwarding edge of a high-availability forwarding edge pair of the first site, wherein the first forwarding edge is the active edge of the first site, and the second forwarding edge of the high-availability forwarding edge pair is the standby forwarding edge of the first site.

22. A machine-readable medium storing a program that, when implemented by at least one processing unit, implements the method according to any one of claims 1-21.

23. An electronic device, comprising: a set of processing units; and a machine-readable medium storing a program that, when implemented by at least one of the processing units, implements the method according to any one of claims 1-21.

24. A system comprising means for implementing the method according to any one of claims 1-21.

25. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1-21.

Images