Method for modifying SD-WAN using metric-based heat maps

Abstract

Some embodiments provide a method for modifying an SD-WAN (Software Defined Wide Area Network) deployed for a set of geographic locations using a heat map. From a managed forwarding element (MFE) set that forwards a plurality of data message streams over the SD-WAN to a set of destination clusters, the method collects a plurality of metrics associated with the plurality of data message streams. Based on the collected plurality of metrics, the method generates a heat map that accounts for (1) the plurality of data message streams, (2) a location of the MFE set, and (3) a location of the one or more destination clusters. The method uses the generated heat map to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message streams.

Description

Background Art

[0001] Today, cloud-based applications are deployed across hybrid and multi-clouds for both availability and resiliency. However, for reliable and secure connectivity toward cloud-based applications, today's SD-WAN networks have essentially static standard policies, which can lead to suboptimal routing and degraded user experience. Currently, cloud-based transit is provided in close proximity to cloud-based applications across different regions around the world, which is primarily driven by business value. Therefore, consideration of the evolving needs for next-generation cloud-native applications is lost. In addition, secure and resilient services are statically provided to serve as way stations for dynamic application clusters, resulting in suboptimal performance and / or application degradation due to the placement of services that are often located closer to the source rather than the destination. Summary of the invention

[0002] Some embodiments of the present invention provide a method for generating a heat map and using the generated heat map to modify an SD-WAN (Software Defined Wide Area Network) deployed for a set of geographic locations. Application traffic is handled as flows that are forwarded by a set of managed forwarding elements (MFEs) that generate multiple metrics associated with each of the flows. Based on the collected metrics, the method generates a heat map that illustrates multiple data message flows, the locations of the MFE sets, and the locations of the destinations of the data message flows (e.g., SD-WAN applications hosted by public or private data centers, SaaS (Software as a Service) applications hosted by third-party data centers, etc.). The method uses the generated heat map to identify at least one modification to be made to the SD-WAN to improve the forwarding of the data message flows.

[0003] In some embodiments, the method is performed by a management and control server (e.g., Velocloud Orchestrator (VCO)) for the SD-WAN or a cluster of management and control servers. The management and control server of some embodiments collects metrics from a set of MFEs by collecting metrics from a computing machine that is designated to collect metrics and location context associated with data message flows in the SD-WAN from the set of MFEs. In some embodiments, the metrics collected by the designated computing machine include quality of experience (QoE) metrics, such as loss rate, packet delay rate, packet jitter rate, and throughput. In some embodiments, the designated computing machine uses the collected QoE metrics to calculate multiple QoE scores associated with the data message flow, and the metrics collected by the management and control server include QoE scores. In some embodiments, the QoE scores specify a traffic density associated with the data message flow, which can be used in conjunction with a heat map to identify modifications to be made to the SD-WAN.

[0004] In some embodiments, the MFE set includes an edge router, a cloud gateway router, and a hub router for connecting a data center to the SD-WAN. The edge routers of some embodiments are deployed at the edge of a data center (e.g., a branch site, a cloud data center, etc.) of an enterprise network that implements the SD-WAN, and connect these data centers to other forwarding elements (e.g., a hub router and a gateway router) of the SD-WAN. In some embodiments, the gateway router connects the edge router to a third-party data center via the SD-WAN, and in some embodiments, other operations of the SD-WAN are also performed, such as routing advertisements. According to some embodiments, the edge router is connected to the cloud gateway router via two channels, one of which is a secure channel and the other is an unsecure channel. The hub routers of some embodiments connect different edge routers to each other. For example, in some embodiments, the hub router connects the edge router at the branch site to other edge routers at other branch sites and at the data center hosting the SD-WAN application.

[0005] In some embodiments, the heat map groups destinations into various destination clusters based on their geographic proximity to one another. For example, multiple SaaS applications may be distributed across a large geographic region (e.g., the United States), with some locations in the geographic region having a higher concentration of SaaS applications than other locations (e.g., a higher concentration near a metropolitan area). In some embodiments, any modifications to the SD-WAN are identified by first identifying a specific destination cluster at a specific location that does not include a geographically close MFE for forwarding data message flows to and from the specific destination cluster, and then providing and deploying a new MFE to the specific location to improve forwarding to and from the specific destination cluster.

[0006] In another example, some embodiments use a heat map to identify any destination clusters that are experiencing congestion due to high traffic volumes to those destination clusters. In some embodiments, the locations of the identified destination clusters may already have one or more local MFEs for forwarding data message flows to and from the destination clusters, and new MFEs may be provisioned and deployed to these locations to increase the amount of resources available for forwarding data message flows to and from those destination clusters. Alternatively or in combination, some embodiments may implement other modifications, such as modifying physical links at branch sites and data centers.

[0007] In some embodiments, heat maps are used to identify SD-WAN applications that need improvement. For example, metrics associated with a particular SD-WAN application (e.g., throughput, latency, packet loss, and jitter) may indicate anomalies detected by the MFE when processing data message flows to and from the particular SD-WAN application. Based on the detected anomaly or anomalies, some embodiments modify the number of edge forwarding elements (e.g., edge routers) that connect data centers to each other via the SD-WAN, the number of hubs (e.g., hub routers) that connect edge forwarding elements to each other via the SD-WAN, and / or the link capacity of the link set used to connect to the particular SD-WAN application. According to some embodiments, instances of SD-WAN applications include VOIP applications, database applications, and applications for running virtual machines (VMs).

[0008] In some embodiments, a visualization of the heat map is presented through a user interface (UI) for viewing and analysis by a user (e.g., a network administrator). In some embodiments, the UI is provided by a management and control server, which in some embodiments also generates the heat map. In some embodiments, according to some embodiments, the visualization includes a representation of the data message flow, a representation of the MFE set at its respective location, and a representation of the destination and one or more destination clusters at their respective locations.

[0009] In some embodiments, the visualization is a map across the geographic area in which the SD-WAN is deployed, with representations of the MFEs, destinations, destination clusters, and data message flows overlaid on the map. In addition to providing visualizations, the UI of some embodiments enables a user to identify and select modifications to the SD-WAN for implementation by components of the SD-WAN (i.e., management and control server, MFEs, etc.). For example, a user may cause the management and control server to provision and deploy additional cloud gateway routers, and also define forwarding rules associated with the additional MFEs for use by, for example, edge routers of the SD-WAN.

[0010] The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. It is not meant to be an introduction or overview of all the subject matter of the invention disclosed in this document. The following detailed description and the drawings referenced in the detailed description will further describe the embodiments described in the summary as well as other embodiments. Therefore, in order to understand all the embodiments described by this document, a comprehensive reading of the summary, the detailed description, the drawings and the claims is required. In addition, the claimed subject matter is not limited by the illustrative details in the summary, the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following figures.

[0012] Figure 1 An example of a visualization illustrating a simplified heat map of the SD-WAN of a particular entity that is generated and presented through a UI in some embodiments.

[0013] Figure 2 A process of some embodiments for generating a heat map and using the generated heat map to identify and implement modifications is conceptually illustrated.

[0014] Figure 3 Visualization of a heat map illustrating some embodiments, where thermal applications are identified.

[0015] Figure 4 A visualization illustrating some embodiments in which destination clusters (ie, groups of geographically proximate applications) have been identified.

[0016] Figure 5 A visualization illustrating some embodiments in which a hot application is relocated from its initial location within a first destination cluster to a location within a second destination cluster.

[0017] Figure 6 Visualizations illustrating some embodiments after a hot application has been relocated to a location within a second destination cluster.

[0018] Figure 7 A process performed in some embodiments for provisioning a new cloud gateway router for reaching an application or application cluster is conceptually illustrated.

[0019] Figure 8 Visualizations illustrating some embodiments of a heat map after potential locations for cloud gateway routers have been identified.

[0020] Fig. 9 A visualization illustrating some embodiments where a specific location for a new cloud gateway has been selected for an edge router for reaching hot applications at its new location.

[0021] Fig.10 Conceptually illustrates a process performed in some embodiments to modify the SD-WAN to improve forwarding of one or more flows determined to be hot flows (e.g., flows to hot applications).

[0022] Fig.11 Conceptually illustrated is a diagram showing a cloud gateway that provides connectivity to a set of applications for a set of edge routers.

[0023] Fig.12 A computer system is conceptually described with which some embodiments of the present invention are implemented. DETAILED DESCRIPTION

[0024] In the following detailed description of the invention, many details, examples and embodiments of the invention are set forth and described. However, it will be clear and apparent to those skilled in the art that the invention is not limited to the embodiments set forth, and the invention can be practiced without some of the specific details and examples discussed.

[0025] Some embodiments of the present invention provide a method for generating a heat map and using the generated heat map to modify an SD-WAN (Software Defined Wide Area Network) deployed for a set of geographic locations. From a set of managed forwarding elements (MFEs) that forward data message flows through the SD-WAN, the method collects multiple metrics associated with the data message flows (e.g., metrics generated by the MFEs that process the data message flows). Based on the collected metrics, the method generates a heat map that illustrates multiple data message flows, the locations of the MFE sets, and the locations of the destinations of the data message flows (e.g., SD-WAN applications hosted by public or private data centers, SaaS (Software as a Service) applications hosted by third-party data centers, etc.). The method uses the generated heat map to identify at least one modification to be made to the SD-WAN to improve the forwarding of the data message flows.

[0026] In some embodiments, the method is performed by a management and control server (e.g., Velocloud Orchestrator (VCO)) for the SD-WAN or a cluster of management and control servers. The management and control server of some embodiments collects metrics from a set of MFEs by collecting metrics from a computing machine that is designated to collect metrics and location context associated with data message flows in the SD-WAN from the set of MFEs. In some embodiments, the metrics collected by the designated computing machine include quality of experience (QoE) metrics, such as loss rate, packet delay rate, packet jitter rate, and throughput. In some embodiments, the designated computing machine uses the collected QoE metrics to calculate multiple QoE scores associated with the data message flow, and the metrics collected by the management and control server include QoE scores. In some embodiments, the QoE scores specify a traffic density associated with the data message flow, which can be used in conjunction with a heat map to identify modifications to be made to the SD-WAN.

[0027] In some embodiments, the MFE set includes an edge router, a cloud gateway router, and a hub router for connecting a data center to the SD-WAN. The edge routers of some embodiments are deployed at the edge of a data center (e.g., a branch site, a cloud data center, etc.) of an enterprise network that implements the SD-WAN, and connect these data centers to other forwarding elements (e.g., a hub router and a gateway router) of the SD-WAN. In some embodiments, the gateway router connects the edge router to a third-party data center via the SD-WAN, and in some embodiments, other operations of the SD-WAN, such as router advertisements, are also performed. According to some embodiments, the edge router is connected to the cloud gateway router via two channels, one of which is a secure channel and the other is an unsecure channel. The hub routers of some embodiments connect different edge routers to each other. For example, in some embodiments, the hub router connects the edge router at the branch site to other edge routers at other branch sites and at the data center hosting the SD-WAN application.

[0028] Figure 1 An example of a visualization illustrating a simplified heat map of an SD-WAN for a particular entity that is generated in some embodiments and presented through a UI. In some embodiments, the heat map is generated by a management and control server for the SD-WAN (e.g., Velocloud Orchestrator (VCO)), and the visualization 100 is presented through a UI also provided by the management and control server. In some embodiments, the management and control server is a centralized controller, while in other embodiments it is a distributed controller with controller agents executing on devices in the SD-WAN (e.g., on forwarding elements described below). In still other embodiments, the controller is a cloud gateway that performs the functionality of a controller, or the controller and cloud gateway share controller functionality.

[0029] As shown, visualization 100 includes a map 110 of a geographic area covered by the SD-WAN and analyzed for potential modification. Across map 110, a plurality of applications 115 are distributed. In some embodiments, each application 115 represents a currently running application (i.e., an application known to the SD-WAN), while in other embodiments, each application 115 represents a potential application, and in still other embodiments, applications 115 represent a combination of currently running applications and potential applications. It should be noted that while visualization 100 is illustrated in black and white, other embodiments of the present invention use various colors to present the heat map to distinguish different components of the heat map, as will be further described below. For example, according to some embodiments, currently running applications and potential applications may be presented differently (e.g., different colors, different color intensities, different opacities, etc.).

[0030] In addition to the application 115, the map 110 also includes a cloud gateway 120 (i.e., a cloud gateway router) and a plurality of edge routers 130 connected to the cloud gateway 120 via links 140. The edge routers of some embodiments are edge machines (e.g., virtual machines (VMs), containers, programs executed on computers, etc.) and / or standalone devices that operate at a multi-computer location of a particular entity (e.g., at an office or data center of the entity) to connect computers at their respective locations to other elements (e.g., gateways, hubs, etc.) in a virtual network. In some embodiments, the elements are clusters of elements at each branch site. In other embodiments, edge elements are deployed to each branch site as a high availability pair, such that one edge element in the pair is an active element and the other edge element in the pair is a standby element that can take over as an active edge element in the event of a failover.

[0031] Examples of entities for which such a virtual network may be established include commercial entities (e.g., companies), non-profit entities (e.g., hospitals, research institutions, etc.), and educational entities (e.g., universities, colleges, etc.), or any other type of entity. Examples of public cloud providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, etc., while examples of entities include companies (e.g., corporations, partnerships, etc.), organizations (e.g., schools, non-profit organizations, government entities, etc.), etc. In other embodiments, the hub may also be deployed in a dedicated cloud data center of a virtual WAN provider hosting the hub to establish SD-WAN for different entities.

[0032] In some embodiments, a branch site (e.g., a multi-user computing site) is a location that has multiple user computing and / or other user-operated devices and serves as a source computer and device for communicating with other computers and devices at other sites (e.g., other branch sites, data center sites, etc.). In some embodiments, a branch site may also include servers that are not operated by users. In some embodiments, the multi-machine site is a multi-tenant data center, such as a data center of a software as a service (SaaS) provider. In some embodiments, when the multi-tenant data center is a data center of a SaaS provider, the forwarding element that provides access to the multi-tenant data center is a multi-tenant gateway router.

[0033] In some embodiments, cloud gateway router 120 (also referred to herein as cloud gateway) is a forwarding element residing in a private or public data center. In some embodiments, link 140 between cloud gateway 120 and edge router 130 is a secure connection link (e.g., tunnel). In some embodiments, multiple secure connection links (e.g., multiple secure tunnels established on multiple physical links) may be established between one edge router and the cloud gateway.

[0034] When multiple such links are defined between the edge router and the cloud gateway, in some embodiments, each secure connection link is associated with a different physical network link between the edge router and the external network. For example, in order to access the external network, in some embodiments, the edge router has one or more commercial broadband Internet links (e.g., cable modems, fiber optic links) for accessing the Internet, MPLS (Multi-Protocol Label Switching) links for accessing the external network through the MPLS provider's network, wireless cellular links (e.g., 5G LTE networks), etc. In some embodiments, the different physical links between the edge router 130 and the cloud gateway 120 are links of the same type (e.g., different MPLS links).

[0035] In some embodiments, one edge router 130 may also have multiple direct links (e.g., secure connection links established through multiple physical links) to another edge router and / or to a data center hub router (not shown). Likewise, in some embodiments, different links may use different types of physical links or the same type of physical links. Furthermore, in some embodiments, different edge routers at different branch sites are connected: (1) directly through one or more links, (2) through a cloud gateway or data center hub router to which one of the edge routers is connected through two or more links, or (3) through another edge router at another branch site that may enhance its role to that of a hub forwarding element.

[0036] In some embodiments, cloud gateway 120 is used to connect two SD-WAN forwarding elements (e.g., edge router 130 and forwarding element located in the same data center as one of applications 115) through at least two secure connection links between gateway 120 and two forwarding elements at two SD-WAN sites (e.g., a branch site and a data center site (not shown)). In some embodiments, cloud gateway 120 also provides network data from one multi-machine site to another multi-machine site (e.g., providing an accessible subnet of one site to another site).

[0037] In some embodiments, each secure connection link between two SD-WAN forwarding elements (i.e., cloud gateway 120 and edge router 130) is formed as a VPN tunnel (e.g., an overlay tunnel) between the two forwarding elements. In addition, in some embodiments, secure connection links are defined between gateways in different public cloud data centers to allow paths through virtual networks to traverse from one public cloud data center to another public cloud data center, while in other embodiments, such links are not defined. In addition, in some embodiments, cloud gateway 120 is a multi-tenant gateway for defining other virtual networks for other entities (e.g., other companies, organizations, etc.). Some such embodiments use a tenant identifier to create a tunnel between a gateway and an edge router of a particular entity, and then use the tunnel identifier of the created tunnel to allow the cloud gateway to distinguish between the packet flows it receives from the edge forwarding element of one entity and the packet flows it receives along other tunnels of other entities. In other embodiments, the cloud gateway is single-tenant and is deployed specifically for use by only one entity.

[0038] In some embodiments, the heat map 100 is generated by the management and control server based on metrics collected from various MFEs of the SD-WAN (e.g., cloud gateways 120 and edge routers 130). In some embodiments, the management and control server receives (or collects) metrics from a computing machine designated for collecting metrics (also referred to herein as a discovery node (DN)) and location context associated with data message flows in the SD-WAN from the MFEs of the SD-WAN. In some embodiments, the metrics collected by the DN include quality of experience (QoE) metrics, such as loss rate, packet delay rate, packet jitter rate, and throughput. In some embodiments, the DN uses the collected QoE metrics to calculate multiple QoE scores associated with the data message flow and provides these QoE scores to the management and control server. In some embodiments, the QoE score specifies the traffic density associated with the data message flow, which can be used in conjunction with the heat map to identify modifications to be made to the SD-WAN. In other embodiments, the DN provides additional metrics to the management and control server in conjunction with the QoE score.

[0039] In some embodiments, the heat map is used to identify problems within the SD-WAN and modifications to the SD-WAN to mitigate the identified problems and improve forwarding through the SD-WAN. For example, in some embodiments, the heat map can be used to identify modifications to improve forwarding for: congestion points, locations with large destination clusters without any local MFE to forward data message flows to the clusters (e.g., locations where a large number of applications are running), specific applications that experience an above-average amount of traffic (e.g., an amount of traffic that exceeds a specified traffic threshold), and specific applications that have specified specific service requirements (e.g., latency requirements). In some embodiments, the management and control server identifies problems and modifications to mitigate the problems (e.g., based on policies and service rules defined by a network administrator for the SD-WAN) and implements these modifications. In other embodiments, a user (e.g., a network administrator) uses visualization 100 to identify problems in the SD-WAN and defines modifications to the SD-WAN through the UI.

[0040] For example, Figure 2 A process 200 of some embodiments for generating a heat map and using the generated heat map to identify and implement modifications is conceptually illustrated. In some embodiments, the process 200 is performed by a management and control server, while in other embodiments, the process 200 is performed by a combination of a management and control server and a user through a UI. The process 200 begins by collecting (at 210) metrics associated with data message flows in the SD-WAN from an MFE that forwards the data message flows through the SD-WAN.

[0041] In some embodiments, as mentioned above, the management and control server collects metrics from the DN, which is designated to collect metrics from the MFE of the SD-WAN, such as QoE metrics (e.g., packet loss rate, packet delay rate, packet jitter rate, throughput, etc.). In some embodiments, the metrics collected from the DN include a QoE score calculated by the DN. Alternatively or in combination, the collected metrics of some embodiments also include other scores calculated by the DN, such as a flow density data score and a bandwidth score.

[0042] In some embodiments, the DN collects metrics only from the cloud gateway, while in other embodiments, the DN collects metrics from all MFEs in the SD-WAN (i.e., cloud gateway routers, edge routers, and hub routers). For example, in some embodiments, each cloud gateway is configured to profile a specific set of destinations to discover the QoE of the applications corresponding to the destinations, and derive QoE metrics. In some such embodiments, the cloud gateway exports a set of application QoE metrics to the DN, including a list of destinations (e.g., destination network addresses). The management and control server of some embodiments maintains a registry of cloud providers with appropriate cloud service availability and associated policies. For example, VMware, Inc's VCO maintains a registry of cloud providers with VMware Cloud (VMC) service availability and associated policies.

[0043] The DN of some embodiments then begins probing the location context for a given list of destinations and reaches the closest cloud provider with an appropriate cloud service available. In some embodiments, the DN uses established services to gather the location context for the list of destinations. Examples of established services used in some embodiments include databases that provide context data for comprehensive IP address profiles (e.g., Maxmind and IPinfo).

[0044] Process 200 uses (at 220) the collected metrics to generate a heat map that illustrates the flow of data messages, the locations of the MFEs, and the locations of the destinations. For example, visualization 100 includes representations of applications 115 distributed across map 110 (i.e., the destinations and the locations of the destinations), as well as representations of cloud gateway routers 120 and edge routers 130. In addition to or in lieu of distinguishing known applications from potential applications, in some embodiments, representations of applications 115 may also be presented with varying degrees of intensity (e.g., color intensity) to distinguish high-traffic applications from low-traffic applications.

[0045] Process 200 uses (at 230) the generated heat map to identify one or more modifications to be made to the SD-WAN to improve forwarding of data message flows. In some embodiments, examples of modifications include adding one or more cloud gateway routers or other forwarding elements (e.g., hub routers, edge routers used as hub routers, etc.) to the SD-WAN, changing which cloud gateways and / or other forwarding elements are used to forward all or groups of a particular flow, adding or changing which links are used for all or a particular flow, etc.

[0046] For example, in some embodiments, a network administrator views a heat map that includes a visualization of a group of flows (e.g., file transfer flows) sent from edge router 130 to a particular application located in Illinois via cloud gateway router 120. Based on the heat map and a QoE metric for the flow group associated with cloud gateway router 120, in some embodiments, the network administrator determines that one or more modifications are needed to improve forwarding of the flow group (e.g., in order to meet a service level agreement (SLA) associated with the flow group).

[0047] In some embodiments, the first potential modification identified to improve forwarding of the flow group is to route the flow group through a different next-hop MFE. In some embodiments, the different next-hop MFE is a hub router (not shown) that is geographically closer to a specific application in Illinois than cloud gateway router 120. In other embodiments, a new MFE (e.g., cloud gateway router) is provided near the specific application for forwarding the flow group to and from the specific application, such as when there is no geographically close hub router or other MFE for the specific application.

[0048] In some embodiments, a second potential modification for flow group identification is to add hops to the route from edge router 130 to a specific application in Illinois to reduce the distance traversed between each hop. For example, in some embodiments, an existing second MFE (e.g., a hub router (not shown) or a second cloud gateway router (not shown)) is identified as a potential next hop between cloud gateway router 120 and a specific application to reduce the distance of the last mile connection. Alternatively or in combination, in some embodiments, one or more cloud gateway routers (or other MFEs) are provided as additional next hops between cloud gateway router 120 and a specific application in Illinois to reduce the distance between each hop.

[0049] In some embodiments, the identification is made by the management and control server (e.g., based on policy and service rules defined for the SD-WAN) and / or by a user (e.g., a network administrator) through a UI provided by the management and control server. For example, in some embodiments, a user determines that specific applications or application clusters require their own corresponding cloud gateways to forward data message flows to and from the specific applications or clusters, and then provisions the cloud gateways to be deployed to locations near the specific applications or clusters. In other embodiments, the management and control server determines that the number of hops between the source set of machines and the destination application should be reduced to improve the QoE metric, and generates new forwarding rules for flows between the source set of machines and the destination application to bypass the intermediate MFE and reduce the number of hops.

[0050] Process 200 then implements (at 240) the identified one or more modifications to the SD-WAN. For example, when a new cloud gateway is provided for the SD-WAN, the management and control server of some embodiments provides a forwarding rule set defined for the new cloud gateway to edge routers of the SD-WAN to direct the edge routers to use the new cloud gateway to forward data messages according to the forwarding rule set. In some embodiments, the forwarding rule set may include a list of cloud gateways, and in some embodiments, specify the use of the new cloud gateway for flows to a particular application, or, for example, for flows to network addresses at a particular location or within a particular region. After 240, process 200 ends.

[0051] In some embodiments, the management and control server also sends out a dynamic flow map to the edge router of the SD-WAN, the dynamic flow map including a destination list (e.g., IP address, port, protocol, etc.) and a unique flow group identifier (e.g., a unique universal identifier (UUID)) and dynamic transit point information of dynamic transit points assigned to different flow groups. The dynamic transit point of some embodiments is a cloud gateway that forwards data message flows through the SD-WAN. In some embodiments, each cloud gateway that is a dynamic transit point registers with the management and control server as a dynamic transit gateway.

[0052] In some embodiments, each flow group is defined based on location discovery performed by the DN and is identified by a flow group identifier, which in some embodiments is assigned to a corresponding dynamic transit identifier. In addition, in some embodiments, the flow group is defined based on one or more attributes associated with each flow in the flow group. In some embodiments, examples of such attributes include one or more of a destination address or set of destination addresses for the flow, a source address or set of source addresses for the flow, a specific category associated with the flow (e.g., VOIP (Voice over IP), video conferencing, file transfer, etc.), etc.

[0053] In some embodiments, different flow groups are defined based on different attributes. For example, in some embodiments, a first flow group is defined based on layer 7 (L7) information, such as an application identifier (appID) that identifies the type of data contained in the payload of the packets of the flows in the flow group (e.g., video, VOIP, etc.), while a second flow group is defined based on L7 or context attributes (i.e., attributes other than L2 to L4 header values) that identify the set of source applications from which the flows are emitted (e.g., a specific video conferencing application or a video streaming service application). To obtain such L7 attributes, some embodiments perform deep packet inspection at the edge device, as further described below. In conjunction with or in lieu of using L7 attributes, some embodiments also define flow groups based on other L2 to L4 header values ​​and / or other non-L2 to L4 context attributes associated with the flows in the flow group.

[0054] In some embodiments, the dynamic flow graphs are each defined and formatted according to a five-tuple identifier corresponding to a particular destination, a flow group identifier assigned to a flow group going to the particular destination, and an identifier associated with a dynamic transit point through which the particular destination can be reached. The edge devices of some embodiments use software defined routing to utilize the dynamic flow graphs and forward application traffic toward the best available cloud gateway.

[0055] In some embodiments, each edge router processes a received dynamic flow graph and installs a special aggregated route based on a flow group UUID uniquely associated with a dynamic transit gateway identifier as a next-hop logical identifier. In some embodiments, when a flow is received at an edge device, each edge device performs a flow graph check and identifies a flow group associated with the received flow. For example, in some embodiments, each edge device collects attributes from received packets and uses the collected attributes to perform a flow graph check to identify the associated flow group. As described above, in some embodiments, each flow group is defined based on one or more attributes, and thus, each edge device of some embodiments collects attributes from each received packet to identify the flow group associated with the packet. In some embodiments, each edge device includes a deep packet inspector for performing deep packet inspection (DPI) on received packets to extract and collect context attributes (e.g., L7 attributes) for performing flow graph inspection.

[0056] In some embodiments, once the edge device has identified the flow group corresponding to the received packet and the UUID associated with the identified flow group, the edge device performs a special aggregated route lookup action to identify a route based on the UUID associated with the identified flow group. According to some embodiments, the edge device of some embodiments then routes the received flow using the logical identifier of the dynamic transit gateway corresponding to the flow group UUID (e.g., by sending the traffic on an overlay tunnel associated with the dynamic transit gateway).

[0057] Figures 3 to 9 Additional example visualizations of heat maps illustrating SD-WAN in some embodiments. Figure 3 300 illustrates a heat map in which hot applications 350 are identified. In some embodiments, hot applications are applications that receive above a threshold amount of traffic. Further, in some embodiments, hot applications may include a particular class of applications (e.g., video conferencing applications). Further, the heat map may display hot areas, which are areas with a large number of other applications (e.g., dense application clusters). As with visualization 100, visualization 300 includes a map 310 of the geographic area spanned by the SD-WAN, representations of applications 315 distributed across map 310, a cloud gateway 320, and a plurality of edge routers 330 connected to the cloud gateway 320 via links 340.

[0058] As shown, cloud gateway 320 connects edge router 330 to at least hot application 350 via one or more links 360. Although illustrated as direct links between cloud gateway 320 and hot application 350, according to some embodiments, one or more links 360 connect cloud gateway 320 to an edge router of, for example, a data center hosting a server running hot application 350. In this example, hot application 350 is a SaaS application hosted by a third-party data center (not shown), and link 360 is an unmanaged link. In other embodiments, hot application 350 is an SD-WAN application running on a server belonging to an entity for which the SD-WAN is implemented, and link 360 is a managed link.

[0059] Figure 4 A visualization 400 is illustrated in which destination clusters (i.e., groups of geographically close applications) have been identified. In some embodiments, location context (e.g., location context collected by the DN described above) is used to identify larger destination groups (e.g., using a clustering-like approach). As shown, two destination clusters 470 and 475 have been identified, with the hot application 350 being located within the destination cluster 470. In some embodiments, the center points (centroids) of the destination clusters 470 and 475 are identified and labeled using an algorithm such as a Gaussian mixture method (GMM) or using a local density estimator such as DBSCAN. Although two clusters are identified in the visualization 400, other embodiments may include additional or fewer destination clusters than illustrated.

[0060] In some embodiments, an application can be relocated while maintaining the same destination network address. In some embodiments, an application is relocated when a server machine (e.g., a virtual machine (VM) or Pod) on which the application executes is migrated to a new location. In other embodiments, a new server machine is deployed at the new location and is configured like the previous server machine on which the application executes. In some embodiments, to relocate an application to a server machine at a new location, a new instance of the application is deployed to the new server machine and configured in the same manner as the previous instance of the application.

[0061] For example, Figure 5 Visualization 500 is illustrated in which hot application 350 is relocated from its initial location within destination cluster 470 to a location within another destination cluster 475. As shown, cluster 470 is geographically much closer to cloud gateway 320 than cluster 475. However, edge router 330 is still configured to forward data messages to hot application 350 using cloud gateway 320. As a result, data message flows sent between cloud gateway 320 and hot application 350 have a longer distance to travel before reaching their destination.

[0062] For example, Figure 6 600 after the hot application 350 has been relocated to a location within the destination cluster 475. The last mile connectivity 680 from the cloud gateway 320 to the hot application 350 at its new location covers a much longer distance than the last mile connectivity (i.e., link 360) before the hot application 350 was relocated. Due to the increased distance, some embodiments experience network issues that affect the user's QoE, such as increased latency. Thus, some embodiments use heat maps to identify potential locations for dynamic staging points (e.g., additional cloud gateways) for reaching the hot application 350, and in some embodiments, for reaching some or all other applications located in and around the destination cluster 475.

[0063] In some embodiments, additional dynamic transit points are required to reach hot applications 350 based on the service requirements associated with hot applications 350. In some embodiments, additional dynamic transit points are also required for other applications (e.g., applications that receive less than a threshold amount of business), as well as hot applications that have not yet been relocated. For example, in some embodiments, applications are associated with low latency requirements, and therefore a cloud gateway that is geographically close to the application is required to ensure that the low latency requirements are met. In some embodiments, using a cloud gateway that is closer to the application can result in a longer RTT to reach the application compared to the round trip time (RTT) associated with using a cloud gateway that is farther away from the application but closer to the source edge router. In some such embodiments, longer RTT is preferred due to lower latency of the last mile connection to the application.

[0064] Figure 7 Conceptually illustrating a process 700 performed in some embodiments for provisioning a new cloud gateway router for reaching an application or application cluster. In some embodiments, the application or application cluster may include an application in a remote location (e.g., the two applications shown within the boundaries of Minnesota in the visualization of map 110), as well as an application in destination cluster 475 before or after application 350 is relocated. In some embodiments, process 700 is performed by a management and control server of the SD-WAN.

[0065] Process 700 begins when the process determines (at 710) that additional MFEs are needed for a particular geographic region based on flow patterns of multiple data message flows to various SaaS applications distributed across multiple geographic regions. As discussed above, after hot applications 350 are relocated, additional dynamic transit points (e.g., cloud gateways) are needed to reach the hot applications at their new locations. In other embodiments, an application or application cluster may require a new cloud gateway based on a determination that there are no cloud gateways near the application or application cluster. In some such embodiments, as also mentioned above, one or more applications may be associated with service requirements that can be satisfied only by providing a local cloud gateway for the application or application cluster.

[0066] Process 700 identifies (at 720) locations within a particular geographic area where additional MFEs are to be deployed. Figure 8 A visualization 800 illustrating a heat map after potential locations for a cloud gateway router have been identified. As shown, eight (8) potential locations (ie, dynamic transit points) for a cloud gateway router 890 have been identified and presented on the heat map.

[0067] In some embodiments, proximity scoping is used by the cloud gateway 320 or the management and control server of the SD-WAN to identify potential locations for dynamic transit points. In some embodiments, tools such as MyTraceroute (MTR) are used to trace lossy network segments and find potential locations. In some embodiments, a capacity-constrained P-center algorithm is also used to identify the optimal location for such dynamic transit points. In some embodiments, the centroid with lossy network segments is added to the destination clusters 470 and 475. In addition, in some embodiments, available edge computing stacks (i.e., existing MFE instances) close to the identified potential locations are added to the edge computing stack set. The management and control server of some embodiments selects a location from the identified potential locations based on the QoE score associated with the location.

[0068] The process 700 then provisions and deploys (at 730) the additional MFEs to the identified locations. The management and control server of some embodiments registers with a controller service in a cloud provider that is proximate to the SaaS application in the identified location. In some embodiments, the steps for allocating a transit point are as follows. First, the identified centroids of the application clusters (e.g., destination clusters 470 and 475) are added to the set C. k , where C represents the set of clusters and k represents the number of identified clusters. k , select the transfer point set T closest to the cluster m , where T is the set of transit points, and m is the number of selected transit points (i.e., the number of transit points in the set of M transit points). Then, the optimal transit point is located by applying the capacity-constrained P-center method as mentioned above, and then the optimal transit point is assigned to the E-givenn edge routers, where E is the set of edges, and n is the number of edges in the set of N edges, x n,m,k It is through T m From E n Access C k The estimated utilization (i.e., load), L m It is T m The maximum load, I m is the current load of m, and Q n is the maximum number of transit points that can be assigned to edge router n.

[0069] Using the above, the goal is to determine the maximum utilization and allocation of transfer points, W = ∑n∑m∑kx n，m，k *X n,m +∑ml m Y m , where: If m is selected for n, then X n,m is 1, or 0 otherwise; and if m is deployed, then Y m is 1, or 0 otherwise. This goal is subject to a set of constraints. For example, the total number of assigned transit points cannot exceed the maximum number of transit points M, and the sum of the estimated total load of a transit point and the current load cannot exceed the transit point L. m The maximum load of the edge n, and the number of transit points assigned to the edge n cannot exceed the maximum allowed number of transit points per edge Q n In addition, the specified integrity constraints include X m,n For any / all m, n equal to 0, 1; and Y m For any / all m equal to 0, 1.

[0070] In some embodiments, the management and control server provides a gateway by triggering automatic provisioning and activation of a gateway instance based on a gateway template. In some embodiments, the template-based automatic provisioning is API-based and provides an automated solution for hosting a gateway on a target cloud provider (e.g., AWS, GCP, Microsoft Azure, etc.). In some embodiments, once a gateway has been provided, the gateway is registered with the management and control server as a dynamic transit gateway. The management and control server of some embodiments receives metrics (e.g., QoE scores) associated with a provisioned dynamic transit gateway and compares these metrics with metrics received before the dynamic transit gateway is provided to identify improvements. In some embodiments, the management and control server performs automatic expansion and deactivation of a dynamic transit gateway (e.g., when the flow density drops below an established threshold).

[0071] Once the additional MFE has been provided and deployed, process 700 provides (at 740) forwarding rules to the edge router to direct the edge router to forward data message flows to a specific geographic region using the additional MFE. In some embodiments, the forwarding rules may specify a specific application or set of applications that will utilize the additional MFE, while all other data message flows to other applications not specified by the rules will be forwarded using, for example, a default MFE, even if the other applications are in the same region as the specified specific application or set of applications.

[0072] Fig. 9 900 is illustrated in which a specific location of a new cloud gateway 995 has been selected for an edge router for reaching hot applications at their new location. As shown, edge router 330 has a connection 940 to the new cloud gateway 995, which has a significantly shorter last mile connection 945 to hot applications 350. In some embodiments, after deploying the new cloud gateway 995, the management and control server provides forwarding rules to the edge routers to direct the edge routers to use the new cloud gateway 995 for all flows to applications in and around cluster 475. In other embodiments, the management and control server provides forwarding rules to direct the edge routers to use the new cloud gateway 995 only for flows to hot applications 350, and to use the existing cloud gateway 320 for every other flow to the region.

[0073] In some of these embodiments, the management and control server also provides new forwarding rules, records and / or configuration data to the new cloud gateway 995 to guide the new cloud gateway 995 to appropriately forward flows received from the edge router 330 (e.g., a list of service IP addresses of service applications, data for establishing tunnels to computers and / or machines on which applications are executed or to their associated forwarding elements, etc.) to applications running in data centers in Georgia and one or more neighboring states (e.g., applications in and around the destination cluster 475). Returning to process 700, after 740, the process ends.

[0074] In some embodiments, after modifications to the SD-WAN have been made, the DN continues to calculate QoE scores from the QoE metrics collected from the MFE of the SD-WAN. The management and control server collects the calculated QoE scores from the DN, and in some embodiments, compares these scores to previously collected QoE scores to identify and highlight improvements caused by the SD-WAN modifications. In some embodiments, the dynamic transit gateways are monitored, and in some embodiments, the dynamic transit gateways are automatically scaled out and deactivated when the flow density drops below an established threshold.

[0075] In several embodiments described above, new cloud gateways are deployed in a region to reach hot applications that are relocated to the region (e.g., cloud gateway 995 deployed to reach hot applications 350 that are relocated from destination cluster 470 to destination cluster 475). In some embodiments, a network administrator can use a heat map to deploy new cloud gateways even if no applications have been relocated, for example, to deploy new cloud gateways in a region for applications that are currently operating in the region or in a nearby region. The following two examples illustrate such use of heat maps.

[0076] As a first example, a network administrator of some embodiments views a heat map to identify a destination cluster (e.g., destination cluster 475 in the south) in one region that is being accessed by a computing device in an SD-WAN connection site in another region (e.g., by a machine connected to edge router 330 in California) through a cloud gateway in another region (e.g., cloud gateway 320 in California) deployed near the computing device. After noticing this, the network administrator of some embodiments may then decide to deploy a cloud gateway (e.g., cloud gateway 995 in Georgia) closer to the destination cluster to reduce the distance of the last mile connection to the destination cluster.

[0077] Another example involves a network administrator reviewing a heat map to identify hot applications (e.g., two applications located within the border of Minnesota on map 110) that are frequently accessed by computing devices located in SD-WAN connection sites located in other regions (e.g., machines connected to edge router 330 in California) located in regions with sparse or relatively sparse server density through cloud gateways located in the other regions (e.g., cloud gateway 320 in California). After reviewing the heat map and identifying such hot applications, the network administrator may decide to deploy a cloud gateway (e.g., a cloud gateway in a public or private cloud data center in Minnesota) closer to the identified hot applications so that there is at least one geographically close cloud gateway for reaching the hot applications in order to shorten the last mile connectivity to the hot applications.

[0078] In addition, while the embodiments described above provide examples in which a single cloud gateway router is provided to improve forwarding, in other embodiments, two or more cloud gateway routers are provided to improve forwarding. For example, in some such other embodiments, a first cloud gateway router is provided for forwarding flows identified as hot flows, while a second cloud gateway router is provided for forwarding all other flows for a particular area. In still other embodiments, one or more additional cloud gateway routers are provided, for example, to reduce the load on existing cloud gateway routers that forward flows to and from, for example, a dense destination cluster.

[0079] Fig.10A process 1000 is conceptually illustrated that is performed in some embodiments to modify an SD-WAN to improve forwarding of one or more flows determined to be hot flows (e.g., flows to hot applications). In some embodiments, process 1000 is performed by a management and control server of the SD-WAN. Process 1000 begins when the management and control server collects (at 1010) metrics associated with data message flows sent between the MFEs from the MFEs. As discussed above, the management and control server of some embodiments collects QoE metrics from DNs designated for collecting metrics from the MFEs of the SD-WAN. In some embodiments, each cloud gateway is configured to profile a specific set of destinations to discover the QoE of applications corresponding to the destinations, and derive QoE metrics that are then collected by (or exported to) the DNs.

[0080] Process 1000 analyzes (at 1020) the collected metrics to group data message flows according to type, and identifies a ranking of data message flow groups according to traffic throughput. In some embodiments, data message flows with high packet rates are defined as hot flows. In some embodiments, hot flows also include flows to hot applications, and / or include flows from a specific hot source or source set that sends a large number of packets. In yet other embodiments, hot flows include flows belonging to a specific category (e.g., video conferencing flows, VOIP flows, etc.). In yet other embodiments, hot flows are defined as all or any combination of the above hot flows.

[0081] The process 1000 uses (at 1030) the ranking to identify a set of one or more groups of data message flows. For example, the ranking may group flows based on maximum and minimum thresholds for throughput (i.e., packet rate) and identify the top N groups to be included in the set of one or more groups of data message flows. In some embodiments, the identified set of one or more groups will be designated as hot flows for which modifications to the SD-WAN will be made.

[0082] Now refer to Fig.11 Description: Rank the streams based on their "hotness". Fig.11Conceptually illustrated is a diagram 1100 showing a cloud gateway that provides a set of edge routers with connections to a set of applications. Each edge router 1110 includes two secure connection links 1112 and 1114, respectively, to connect to a cloud gateway router 1120. Link 1112 is a commercial broadband Internet link (e.g., cable modem, fiber link), while link 1114 is an optimized MPLS (Multi-Protocol Label Switching) link. In some embodiments, each link 1112 and 1114 includes multiple links. Cloud gateway router 1120 is connected to applications 1130 and 1135 via link 1125. In some embodiments, link 1125 is a managed link, in other embodiments an unmanaged link, and in still other embodiments a combination of managed and unmanaged links.

[0083] In this example, application 1130 is ranked and defined as a hot application (e.g., an application that receives more than a threshold amount of traffic), while application 1135 is an application that experiences an average amount of traffic (e.g., an application that receives less than a threshold amount of traffic). Thus, flows to hot application 1130 will be hot flows (at 1030) that will have a higher ranking, while flows to non-hot application 1135 will be non-hot flows (at 1030) that will have a lower ranking.

[0084] Returning to process 1000, the process modifies (at 1040) the SD-WAN to improve forwarding of the identified set of one or more groups of data message flows through the SD-WAN. For example, in diagram 1100, in some embodiments, based on metrics collected from cloud gateway 1120, controller cluster 1105 provides new or updated forwarding rules to cloud gateway for distribution to edge router 1110 to direct edge router 1110 to use link 1112 when forwarding data message flows associated with non-hot applications 1135, and to use link 1114 when forwarding data message flows associated with hot applications 1130 (i.e., hot flow sets). In other embodiments, other modifications to the SD-WAN are implemented to improve forwarding of hot flows, such as adding links (e.g., adding fiber links) between edge routers and cloud gateways, deploying one or more additional cloud gateways designated for forwarding hot flows, etc. After 1040, process 1000 ends.

[0085] Many of the above features and applications are implemented as software processes specified as a set of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electronic signals transmitted wirelessly or through wired connections.

[0086] In this specification, the term "software" is intended to include firmware residing in a read-only memory or an application stored in a magnetic storage device, which can be read into a memory to be processed by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while retaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that implement the software inventions described herein together is within the scope of the present invention. In some embodiments, when installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.

[0087] Fig.12 A computer system 1200 is conceptually illustrated with which some embodiments of the present invention are implemented. The computer system 1200 can be used to implement any of the hosts, controllers, gateways, and edge forwarding elements described above. Thus, it can be used to perform any of the processes described above. This computer system 1200 includes various types of non-transitory machine-readable media and interfaces for various other types of machine-readable media. The computer system 1200 includes a bus 1205, a processing unit 1210, a system memory 1225, a read-only memory 1230, a permanent storage device 1235, an input device 1240, and an output device 1245.

[0088] Bus 1205 collectively represents all system, peripheral, and chipset buses that communicatively connect the many internal devices of computer system 1200. For example, bus 1205 communicatively connects processing unit 1210 with read-only memory 1230, system memory 1225, and permanent storage 1235.

[0089] Processing unit 1210 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the processes of the present invention. In different embodiments, processing unit 1210 may be a single processor or a multi-core processor. Read-only memory (ROM) 1230 stores static data and instructions required by processing unit 1210 and other modules of computer system 1200. On the other hand, permanent storage device 1235 is a read and write memory device. Such device 1235 is a non-volatile memory unit that stores instructions and data even when computer system 1200 is turned off. Some embodiments of the present invention use a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) as permanent storage device 1235.

[0090] Other embodiments use removable storage devices (e.g., floppy disks, flash drives, etc.) as permanent storage devices. Similar to permanent storage device 1235, system memory 1225 is a read and write memory device. However, unlike storage device 1235, system memory 1225 is a volatile read and write memory, such as random access memory. System memory 1225 stores some instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in system memory 1225, permanent storage device 1235, and / or read-only memory 1230. Processing unit 1210 retrieves instructions to be executed and data to be processed from these various memory units in order to perform the processes of some embodiments.

[0091] The bus 1205 is also connected to input and output devices 1240 and 1245. Input device 1240 enables a user to communicate information and select commands to the computer system 1200. Input device 1240 includes an alphanumeric keyboard and a pointer device (also called a "cursor control device"). Output device 1245 displays images generated by the computer system 1200. Output device 1245 includes a printer and a display device, such as a cathode ray tube (CRT) or a liquid crystal display (LCD). Some embodiments include devices that function as both input and output devices 1240 and 1245, such as a touch screen.

[0092] Finally, if Fig.12 12, bus 1205 also couples computer system 1200 to a network 1265 through a network adapter (not shown). In this manner, computer 1200 may be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of computer system 1200 may be used in conjunction with the present invention.

[0093] Some embodiments include electronic components, such as microprocessors, storage devices, and memories, that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable CD-ROM, ultra-density CD-ROM, any other optical or magnetic media, and floppy disk. Computer-readable media can store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer code include machine code, such as produced by a compiler, and files containing higher-level code that is executed by a computer, electronic component, or microprocessor using an interpreter.

[0094] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits, such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.

[0095] As used in this specification, the terms "computer", "server", "processor" and "memory" all refer to electronic or other technical devices. These terms do not include people or groups of people. For the purposes of this specification, the terms display or display means display on an electronic device. As used in this specification, the terms "computer-readable medium", "computer-readable medium" and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other short-lived or transient signals.

[0096] Although the present invention has been described with reference to many specific details, it will be appreciated by those skilled in the art that the present invention may be embodied in other specific forms without departing from the spirit of the present invention. Therefore, it will be appreciated by those skilled in the art that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.

Claims

1. A method for modifying an SD-WAN (Software Defined Wide Area Network) deployed for a set of geographic locations using a heat map, the method comprising: collecting a plurality of metrics associated with the plurality of data message flows from a set of managed forwarding elements (MFEs) that forward the plurality of data message flows through the SD-WAN to a set of destination clusters; generating a heat map based on the collected plurality of metrics, the heat map illustrating (i) the plurality of data message flows, (ii) locations of the MFE sets, and (iii) locations of the one or more destination clusters; and The generated heat map is used to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message flows. 2 . The method of claim 1 , wherein the one or more destination clusters include a plurality of SaaS (Software as a Service) applications.

3. The method of claim 1 , wherein using the generated heat map to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message flows comprises: identifying, from the heat map, a particular destination cluster at a particular location, the particular location not including a geographically proximate MFE for forwarding data message flows to and from the particular destination cluster; and A new MFE is deployed to the specific location to improve forwarding to and from the specific destination cluster.

4. The method of claim 1 , wherein using the generated heat map to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message flows comprises: using the heat map to identify one or more locations of one or more destination clusters in the set of destination clusters that experience congestion; and For each of the identified one or more destination clusters, a new MFE is deployed to the location of the destination cluster to improve forwarding to and from the destination cluster.

5. The method of claim 1 , wherein using the generated heat map to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message flows comprises: identifying a set of SD-WAN applications using the generated heat map; identifying, from the set of identified SD-WAN applications, a particular SD-WAN application associated with a particular metric from the collected plurality of metrics to be improved; and Modifying the SD-WAN to improve the specific metric for the specific SD-WAN application.

6. The method of claim 5, wherein the specific SD-WAN application comprises one of a VOIP application, a database application, and an application for running a virtual machine (VM).

7. The method of claim 6, wherein the particular metric comprises one of throughput, latency, packet loss, and jitter.

8. A method according to claim 5, wherein modifying the SD-WAN to improve the specific metric of the specific SD-WAN application includes modifying one of: (i) the number of edge forwarding elements connecting data centers to each other through the SD-WAN, (ii) the number of hubs connecting edge forwarding elements to each other through the SD-WAN, and (iii) the link capacity of the link set used to connect to the specific SD-WAN application.

9. The method of claim 1 further comprises presenting a visualization of the heat map through a user interface (UI), wherein using the generated heat map to identify at least one modification to be made to the SD-WAN to improve forwarding of the plurality of data message flows comprises implementing a user-specified modification received through the UI.

10. The method of claim 9, wherein the visualization comprises (i) a representation of the plurality of data message flows, (ii) a representation of the MFE sets at their respective locations, and (iii) a representation of the one or more destination clusters at their respective locations.

11. The method of claim 1 , wherein the MFE set comprises (i) an edge router set for connecting a data center to the SD-WAN, (ii) a cloud gateway router set for connecting edge routers in the edge router set to a third-party data center via the SD-WAN, and (iii) a hub router set for connecting each edge router in the edge router set to each other edge router in the edge router set.

12. A method for dynamically deploying a managed forwarding element (MFE) in a software defined wide area network (SD-WAN) for a specific geographic area across which a plurality of SaaS applications are distributed, the method comprising: determining that the specific geographic region requires an additional MFE based on a flow pattern of a plurality of flows to the plurality of SaaS applications distributed across the specific geographic region; configuring the additional MFE to be deployed at a specific location in the specific geographic area for forwarding the plurality of flows to the plurality of SaaS applications; and A set of forwarding rules is provided to a specific set of MFEs connecting a set of branch sites to the SD-WAN to direct the specific set of MFEs to forward subsequent data messages belonging to the plurality of flows to the plurality of SaaS applications using the additional MFEs.

13. The method of claim 12, wherein determining that the specific geographic area requires the additional MFE based on flow patterns of the multiple flows to the multiple SaaS applications comprises receiving multiple quality of experience (QoE) scores associated with the multiple SaaS applications from a specific computing machine, the specific computing machine collecting QoE metrics from existing MFEs connected by the SD-WAN.

14. The method of claim 13, wherein the method is performed by a management and control server, wherein the specific computing machine consolidates the collected QoE metrics to calculate the plurality of QoE scores to provide to the management and control server for analysis.

15. The method of claim 14, wherein in addition to the QoE metrics, the particular computing machine also collects location context associated with destination network addresses of the plurality of SaaS applications to provide to the management and control server along with the QoE scores.

16. The method of claim 15, further comprising using the location context to (i) identify one or more groups of geographically proximate destination network addresses, and (ii) identify a centroid of each of the one or more identified groups of destination network addresses.

17. The method of claim 12, wherein configuring the additional MFE to be deployed at the specific location in the specific geographic area for forwarding the plurality of flows to the plurality of SaaS applications further comprises: performing a proximity scoping operation to identify a set of potential locations for deploying the additional MFEs; and The particular location is selected from the set of potential locations.

18. The method of claim 17, wherein the particular location is selected from the set of potential locations based on an operation for identifying a capacity-constrained P-center, wherein the capacity-constrained P-center includes the particular location.

19. The method according to claim 12, wherein configuring the additional MFE to be deployed at the specific location in the specific geographic area for forwarding the multiple flows to the multiple SaaS applications further includes (i) registering the additional MFE with a management and control server of the SD-WAN, and (ii) deploying the additional MFE at the specific location.

20. The method of claim 12, wherein: The plurality of streams is a first plurality of streams; The additional MFE is a first MFE; The forwarding rule set is a first forwarding rule set; and The set of edge routers uses a second MFE to forward data messages belonging to a second plurality of flows according to a second set of forwarding rules.

21. The method of claim 20, wherein the second set of forwarding rules comprises a default forwarding rule and the second MFE comprises a default MFE.

22. The method of claim 12, wherein: The additional MFE comprises a cloud gateway router; and The specific MFE set includes an edge router set.

23. A method for modifying an SD-WAN (Software Defined Wide Area Network), the method comprising: collecting, from a set of managed forwarding elements (MFEs), a plurality of metrics associated with a plurality of data message flows sent between the set of MFEs; analyzing the collected plurality of metrics to group the data message flows according to a plurality of types and identifying a ranking of the plurality of groups of data message flows according to traffic throughput; identifying a set of one or more groups of data message flows using the ranking; and The SD-WAN is modified to improve forwarding of the identified set of one or more groups of data message flows through the SD-WAN.

24. The method of claim 23, wherein collecting from the set of MFEs the plurality of metrics associated with the plurality of data message flows sent between the set of MFEs comprises collecting from a specific computing machine a plurality of quality of experience (QoE) scores associated with the plurality of SaaS applications, the specific computing machine (i) collecting QoE metrics from the set of MFEs, and (ii) using the collected QoE metrics to calculate the plurality of QoE scores.

25. The method of claim 24, wherein the QoE metrics include at least packet loss rate, packet delay rate, packet jitter rate, and throughput.

26. The method of claim 24, wherein the plurality of QoE scores specify traffic throughput associated with the plurality of flows, wherein analyzing the collected plurality of metrics to rank the plurality of data message flows according to traffic throughput comprises identifying the specified traffic throughput from the plurality of QoE scores.

27. A method according to claim 23, wherein at least one of the data message flows in the set of one or more groups of data message flows is associated with a destination network address assigned to a specific SaaS (Software as a Service) application hosted in a specific data center in a specific geographic area, and wherein modifying the SD-WAN to improve forwarding of the identified set of one or more groups of data message flows includes deploying additional MFEs to the specific geographic area to improve forwarding to and from the specific SaaS application.

28. The method of claim 27, wherein: the at least one data message flow destined for the particular SaaS application being one of a plurality of data message flows associated with a destination network address located in the particular geographic area; The additional MFE forwards data messages belonging only to the at least one data message flow destined for the specific SaaS application; and Each other of the plurality of data message flows associated with a destination network address located in the particular geographic area is forwarded to and from the destination network address by one or more existing MFEs in the set of MFEs.

29. The method of claim 28, wherein the additional MFE forwards data messages belonging only to the at least one data message flow destined for the specific SaaS application based on a first forwarding rule set, wherein the first forwarding rule set specifies the additional MFE as a next hop for reaching the destination network address assigned to the specific SaaS application.

30. The method of claim 29, wherein a second set of forwarding rules designates the one or more existing MFEs in the set of MFEs as next hops for reaching other destination network addresses located in the particular geographic area.

31. A method according to claim 23, wherein modifying the SD-WAN to improve forwarding of the identified set of one or more groups of data message flows through the SD-WAN includes increasing the capacity of a managed set of links across which the identified set of one or more groups of data message flows are forwarded.

32. The method of claim 23, wherein the identified set of one or more groups of data message flows are designated as hot flows.

33. A machine-readable medium storing a program which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 32.

34. An electronic device comprising: Processing unit set; and A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1 to 32.

35. A system comprising means for implementing the method according to any one of claims 1 to 32.

36. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 32.

Images