Collect and analyze data about flows associated with DPI parameters
Through local and remote deep packet inspectors working together, DPI parameter sets are generated and compared, performance differences are identified and paths are adjusted, and performance problems caused by differences between edge nodes and cloud gateways in SD-WAN are solved, and more efficient packet streaming is achieved and service level optimization.
Patent Information
- Application Number
- CN202080060181.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2020-02-18
- Filing Date
- 2020-11-04
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-11-04
AI Technical Summary
The existing software-defined wide area network (SD-WAN) solutions perform deep packet inspection (DPI), differences between edge nodes and cloud gateways lead to poor performance and difficult to effectively optimize the path and service level of packet flows.
Local and remote deep packet inspectors work together to generate and compare DPI parameter sets, identify performance differences by recording and analyzing, and adjust forwarding paths of edge nodes and gateways to improve packet flow transmission.
The transmission efficiency and service level of packet flows in SD-WAN are improved, and the overall network performance is improved by dynamically adjusting paths and optimizing the configuration of edge nodes and gateways.
Smart Images

Figure CN114342330B_ABST
Abstract
Description
Background Art
[0001] In recent years, several companies have introduced to the market solutions for deploying software-defined (SD) wide area networks (WANs) for enterprises. Some SD-WAN solutions use external third-party private or public cloud data centers (clouds) to define different virtual WANs for different enterprises. These solutions typically have edge forwarding elements (called edge devices) at enterprise edge nodes, which are connected to one or more gateway forwarding elements (called gateway devices or gateways) deployed in third-party clouds.
[0002] In such a deployment, the edge device is connected to the gateway via one or more secure connections that traverse one or more network links that connect the edge device to the external network. Examples of such network links include MPLS links, 5G LTE links, commercial broadband Internet links (e.g., cable modem links or fiber optic links), etc.
[0003] Edge nodes include branch offices of an enterprise (called branches), and these offices are often geographically distributed with network links to gateways of various different network connection types. These SD-WAN solutions employ deep packet inspection to inform certain actions they perform. Summary of the invention
[0004] Some embodiments provide a method for performing deep packet inspection (DPI) for an SD-WAN (software defined wide area network) established as an entity by multiple edge nodes and one or more cloud gateway sets. At a specific edge node, the method uses local and remote deep packet inspectors to perform DPI of a packet flow. Specifically, the method initially uses a local deep packet inspector to perform a first DPI operation on a set of packets of a first packet flow to generate a DPI parameter set for the first packet flow.
[0005] The method then forwards a copy of the set of packets to a remote deep packet inspector to perform a second DPI operation to generate a second set of DPI parameters. In some embodiments, a controller cluster that configures edge nodes and gateways can access the remote deep packet inspector. In some such embodiments, the method forwards a copy of the set of packets to the controller cluster, which then uses the remote deep packet inspector to perform the remote DPI operation. The method receives the result of the second DPI operation, and when the generated first and second DPI parameters are different, generates a record of the difference.
[0006] In some embodiments, the method uses the generated records to improve the operation of the local deep packet inspector. For example, in some embodiments, the local deep packet inspector is a third-party inspector used by a particular edge node, and the generated records are used to identify different flows for which the third-party inspector has poor DPI performance. When the generated records specify a difference between the first and second sets of generated DPI parameters, in some embodiments, the method sends data about the difference to the remote machine to be aggregated with other data about other differences in DPI operations performed on other packet flows over the WAN.
[0007] In some embodiments, after completing the first DPI operation, the method designates the generated first DPI parameter set as the DPI parameter set associated with the first packet flow. When the first and second DPI parameter sets are different, in some embodiments, the method modifies the DPI parameter set associated with the first packet flow based on the generated second DPI parameter set. For example, in some embodiments, the method modifies the DPI parameter set by storing the second DPI parameter set as the DPI parameter set associated with the first packet flow.
[0008] In some embodiments, the method forwards each packet to its destination after the local deep packet inspector has processed it. However, in other embodiments, the method delays forwarding packets of the first stream to the destination of the stream while performing the first DPI operation. During this period, the method stores the delayed packets in a storage queue of a particular edge node. Once the first DPI operation has been completed, the method forwards the set of packets stored in the storage queue and subsequent packets of the first stream to the destination. It then also forwards a copy of the set of packets to the remote deep packet inspector.
[0009] In some embodiments, the method forwards packets of the first packet flow based on the generated first DPI parameter set. For example, in some embodiments, the method uses at least one parameter in the generated first DPI parameter set to select a path for forwarding packets of the first packet flow through the WAN. When the generated first and second DPI parameter sets are different, in some embodiments, the method modifies the forwarding of packets of the first packet flow by forwarding (e.g., selecting a path for) packets of the first packet flow using the second DPI parameter set.
[0010] In some embodiments, the method forwards at least a subset of the generated DPI parameters from a particular edge node to other edge nodes directly or indirectly through a controller set. In addition, in some embodiments, the method again forwards at least a subset of the generated DPI parameters from a particular edge node to at least one gateway directly or indirectly through a controller set. In some embodiments, the generated DPI parameter set includes an identifier that identifies a type of service carried in a payload of a packet.
[0011] In these or other embodiments, the generated DPI parameter set includes an identifier that identifies an application that is the source of the first packet flow and / or an identifier that identifies a class of application to which the source belongs. In some embodiments, the remote or local deep packet inspector does not generate an identifier of the source application or class. In these embodiments, the edge node or controller cluster generates one or both of these identifiers by mapping a traffic type identifier generated by the DPI operation to an application or class identifier.
[0012] In some embodiments, a particular edge node is an edge machine (e.g., a virtual machine (VM), a container, a standalone appliance, a program executed on a computer, etc.) that operates at an entity (e.g., a branch office) or a data center with several computers, and the edge node connects the computers to a WAN. In some of these embodiments, the local deep packet inspector operates on a first computing device with the edge node machine (e.g., as a VM or container), while the remote deep packet inspector operates on a separate second computing device at a remote location (e.g., in a building, neighborhood, city, state, etc. that is different from the location where the particular edge node operates). In some embodiments, the first and second computing devices are computers, while in other embodiments, they are standalone DPI devices. In other embodiments, the first computing device is an appliance, and the second computing device is a computer on which the remote deep packet inspector executes.
[0013] Some embodiments provide a method of using DPI generated parameters to evaluate and in some cases modify how flows associated with a particular application traverse an SD-WAN defined by a number of edge nodes and one or more cloud gateways. At a set of one or more servers, the method receives a set of DPI parameters collected for packet flows processed by a first set of two or more edge nodes on which DPI operations are performed. From these collected sets, the method identifies a subset of DPI parameters associated with a plurality of flows involving a particular application identifier specified by the DPI operation.
[0014] In some embodiments, the received set of DPI parameters includes operational statistics and metrics related to packet flows processed by the first set of edge nodes (e.g., packet transmission time, payload size, current number of packets processed by the node, etc.). In some embodiments, the statistics are accompanied by other data, such as flow identifiers, application classification details, and forwarding decisions (e.g., identifying the selected path), etc. In some embodiments, the operational statistics, metrics, and other data are collected and provided by edge nodes and / or gateways. The method then analyzes the identified subset of parameters to determine whether any packet flows associated with one or more specific DPI parameters have undesirable metrics related to their flow through the WAN.
[0015] When the analysis produces a determination that the edge node should use a different path for flows associated with the particular application identifier, the method then distributes the adjusted next-hop forwarding record to a second set of one or more edge nodes to modify the path that the edge node uses to forward flows associated with the particular application identifier. In some embodiments, the first and second sets of edge nodes are the same, while in other embodiments, the first set of edge nodes is a subset of the second set of edge nodes (e.g., the second set includes at least one node that is not in the first set).
[0016] In some embodiments, the DPI operation of a flow is performed at a source edge node (also referred to as an ingress edge node) where the flow enters the WAN and is passed from the source edge node to another edge node or cloud gateway. In conjunction with the DPI operation, the source edge node collects operational metrics and statistical information (e.g., packet transmission time, payload size, current number of packets processed by the node, etc.) of the packets of the flow that it passes to another edge node or cloud gateway, and provides the DPI-generated parameters together with the collected statistical information to the server set for analysis.
[0017] In some embodiments, the source edge node collects statistics for the flow based on the initial number of packets it uses to perform its DPI operation. In some of these embodiments, the source edge node provides the server set with its initial set of packets for its DPI operation for the flow, as well as the operational metrics and statistics for the new flow it provides to the server set. In some embodiments, the number of packets in the initial set of packets analyzed by the DPI operation of the source edge node depends on the application identified as the source of the flow by the DPI operation. Therefore, the DPI operation analyzes different numbers of packets for different flows from different applications or different types of applications.
[0018] In some embodiments, the destination edge nodes (also referred to as egress edge nodes) also perform DPI operations and collect operational metrics / statistics of flows at the beginning of flows they receive over the WAN (i.e., from a cloud gateway or other edge node). In other embodiments, the destination edge nodes do not perform DPI operations, but collect operational metrics / statistics of flows at the beginning of flows. In some embodiments, the destination edge nodes receive one or more DPI parameters (e.g., application identifiers) generated by the DPI operations of the source edge nodes (e.g., in-band of packets via tunnel headers, or out-of-band via other packets).
[0019] In conjunction with or alternatively to performing DPI operations at edge nodes, some embodiments perform DPI operations outside of edge nodes (e.g., at a physical location remote from the physical location where the edge node operates). In some embodiments, the method also collects statistics / metrics about the processing of flows from the gateway. In some embodiments, source edge nodes set flags in the tunnel encapsulation headers that they use to forward packets to the gateway to direct the gateway to collect statistics for certain flows.
[0020] In some embodiments, the server set uses a flow identifier (e.g., a five-tuple identifier of the flow) to correlate the metrics / statistics it collects from different forwarding elements of the SD-WAN (e.g., from a source edge node, a destination edge node, and / or a gateway). Once the collected metrics / statistics are associated with a particular flow, the server set analyzes the collected metrics / statistics to derive additional operational data that quantifies whether the particular flow is obtaining the expected service level. In some embodiments, the correlated metrics / statistics data is associated with parameters of a particular DPI generation (e.g., an application identifier, etc.) so that the analysis can be completed based on the DPI parameters. For example, in some embodiments, the derived data is used to determine whether a particular flow associated with a particular application identifier reaches its destination within the expected duration, whether the particular flow is delayed too much at a particular gateway, etc.
[0021] When the derived data indicates that a particular flow is not obtaining the desired level of service (e.g., a flow associated with a particular application identifier is not reaching its destination quickly enough), the server set then distributes an adjusted next-hop forwarding record to the edge nodes and / or gateways, which directs the edge nodes and / or gateways to modify the forwarding of the particular flow or similar future flows (e.g., flows from an application with the same DPI identification and / or to the same destination). For example, in some embodiments, based on the distributed next-hop forwarding record, the source edge node selects a different gateway to forward packets for the particular flow and other similar subsequent flows. In other embodiments, the source edge node uses the adjusted next-hop forwarding record to select (one or more) gateways for forwarding subsequent flows similar to the particular flow (e.g., flows with an application with the same DPI identification and to the same destination).
[0022] The foregoing summary is intended to serve as a brief introduction to some embodiments of the present invention. It is not an introduction or overview of all the subject matter disclosed in this document. The following detailed description and the drawings cited in the detailed description will further describe the embodiments described in the summary and other embodiments. Therefore, in order to understand all the embodiments described in this document, a comprehensive review of the summary, the detailed description, the drawings and the claims is required. In addition, the subject matter claimed is not limited by the illustrative details in the summary, the detailed description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The novel features of the invention are set forth in the appended claims. However, for purposes of illustration, several embodiments of the invention are set forth in the following drawings.
[0024] Figure 1 An example is shown of an edge node of an SD-WAN network using local and remote deep packet inspectors to perform a set of robust DPI operations.
[0025] Figure 2 The process performed by an edge node upon receiving a packet for forwarding in some embodiments is conceptually illustrated.
[0026] Figure 3 The process performed when an edge node receives the results of a DPI operation by a remote deep packet inspector for a particular flow is conceptually illustrated.
[0027] Figure 4 An example of modifying the path selected for a particular flow is shown.
[0028] Figure 5 Components of a controller cluster that perform the above operations are shown.
[0029] Figure 6 A process that a controller cluster periodically performs in some embodiments is conceptually illustrated.
[0030] Figure 7 The process that the evaluator performs to identify flows with poor performance and congested gateways is conceptually illustrated.
[0031] Figure 8 Shown is a new cloud gateway being deployed to handle VOIP calls after a controller set detects that the VOIP call load on two previously deployed cloud gateways has exceeded a certain level that prevents the VOIP calls from receiving their expected service level.
[0032] Fig. 9It is shown that the controller set detects that the VOIP call load on a previously deployed cloud gateway has exceeded a certain level that prevents the VOIP calls from receiving their expected service level.
[0033] Fig.10 A computer system is conceptually illustrated with which some embodiments of the present invention are implemented. DETAILED DESCRIPTION
[0034] In the following detailed description of the present invention, many details, examples and embodiments of the present invention are described and illustrated. However, it is clear and obvious to those skilled in the art that the present invention is not limited to the embodiments described, and the present invention can be implemented without some of the specific details and examples discussed.
[0035] Some embodiments provide a method for performing deep packet inspection (DPI) on an SD-WAN (software defined wide area network) established as an entity by a plurality of edge nodes and one or more cloud gateway sets. At a specific edge node, the method uses local and remote deep packet inspectors to perform DPI on a packet flow. Specifically, the method first uses a local deep packet inspector to perform a first DPI operation on a set of packets of a first packet flow to generate a DPI parameter set for the first packet flow.
[0036] The method then forwards a copy of the set of packets to a remote deep packet inspector to perform a second DPI operation to generate a second set of DPI parameters. In some embodiments, a controller cluster that configures edge nodes and gateways can access the remote deep packet inspector. In some such embodiments, the method forwards a copy of the set of packets to the controller cluster, which then uses the remote deep packet inspector to perform the remote DPI operation. The method receives the result of the second DPI operation, and when the generated first and second DPI parameters are different, generates a record of the difference.
[0037] In some embodiments, the method uses the generated record to improve the operation of the local deep packet inspector. For example, in some embodiments, the local deep packet inspector is a third-party inspector used by a particular edge node, and the generated record is used to identify different flows for which the third-party inspector has poor DPI performance. When the generated record specifies a difference between the first and second sets of generated DPI parameters, in some embodiments, the method sends data about the difference to the remote machine to be aggregated with other data about other differences in DPI operations performed on other packet flows over the WAN.
[0038] In some embodiments, after completing the first DPI operation, the method designates the generated first DPI parameter set as the DPI parameter set associated with the first packet flow. When the first and second DPI parameter sets are different, in some embodiments, the method modifies the DPI parameter set associated with the first packet flow based on the generated second DPI parameter set. For example, in some embodiments, the method modifies the DPI parameter set by storing the second DPI parameter set as the DPI parameter set associated with the first packet flow.
[0039] Figure 1 An example of an edge node of an SD-WAN network (hereinafter also referred to as a virtual network) using local and remote deep packet inspectors to perform a robust set of DPI operations is shown. In this example, edge node 120 is an edge node using local and remote deep packet inspectors 190 and 192, and the SD-WAN is an SD-WAN 100 created for a specific entity to connect two branches 150 and 152 of the entity to two of its data centers 154 and 156, and a data center 158 of a SaaS (software as a service) provider used by the entity. SD-WAN 100 is established by a controller cluster 140, two cloud gateways 105 and 107, and four edge nodes 120-126, one edge node in each branch and data center 154 and 156.
[0040] The edge nodes in some embodiments are edge machines (e.g., virtual machines (VMs), containers, programs executed on computers, etc.) and / or standalone devices that operate at a multi-computer location of a particular entity (e.g., at an office or data center of the entity) to connect computers at their respective locations to cloud gateways and other edge nodes (if so configured). In addition, in this example, the two gateways 105 and 107 are deployed as machines (e.g., VMs or containers) in two different public cloud data centers 110 and 112 of two different public cloud providers.
[0041] Examples of entities for which such a virtual network may be established include commercial entities (e.g., companies), non-profit entities (e.g., hospitals, research institutions, etc.), and educational entities (e.g., universities, colleges, etc.), or any other type of entity. Examples of public cloud providers include Amazon Web Services (AWS), Google Cloud Platform (GCP), Microsoft Azure, etc., while examples of entities include companies (e.g., enterprises, partnerships, etc.), organizations (e.g., schools, non-profit organizations, government entities, etc.), etc. In other embodiments, the gateway may also be deployed in a private cloud data center of a virtual WAN provider that hosts the gateway to establish SD-WANs for different entities.
[0042] exist Figure 1In the present invention, a gateway is a multi-tenant forwarding element that can be used to establish a secure connection link (e.g., a tunnel) with an edge node at a multi-computer site of a specific entity such as a branch office, a data center, etc. These multi-computer sites are usually located in different physical locations (e.g., different buildings, different cities, different states, etc.), and are also referred to as multi-machine computing nodes hereinafter. Figure 1 In the example, edge nodes 120-126 include forwarding elements that exchange data messages with one or more gateways or other edge node forwarding elements through one or more secure connection links. In this example, only edge nodes 120 and 122 have secure connection links between them. All other secure connection links of edge nodes have gateways.
[0043] In some embodiments, multiple secure connection links (e.g., multiple secure tunnels) may be established between the edge node and the gateway. When multiple such links are defined between the edge node and the gateway, in some embodiments, each secure connection link is associated with a different physical network link between the edge node and the external network. For example, in order to access the external network, in some embodiments, the edge node has one or more commercial broadband Internet links (e.g., cable modems, fiber links), wireless cellular links (e.g., 5G LTE networks), etc. for accessing the Internet.
[0044] Furthermore, multiple paths can be defined between a pair of edge nodes. Figure 1 Two examples of this are shown. It shows two paths through gateways 105 and 107 between edge nodes 120 and 124. It also shows two paths between edge nodes 120 and 126, one of which passes through cloud gateway 107 and the other of which passes through the MPLS (Multi-Protocol Label Switching) network 185 of the MPLS provider to which both edge nodes 120 and 126 are connected. Figure 1 It is also shown that through cloud gateways 105 and 107, the SD-WAN 100 allows edge nodes to connect to the SaaS provider's data center 158.
[0045] In some embodiments, each secure connection link between the gateway and the edge node is formed as a VPN (Virtual Private Network) tunnel between the gateway and the edge node. In some embodiments, the gateway is also connected to the SaaS data center 158 via a secure VPN tunnel. The collection of edge nodes, gateways, and secure connections between edge nodes, gateways, and SaaS data centers forms the SD-WAN 100 of a specific entity. In this example, the SD-WAN spans two public cloud data centers 110 and 112 and an MPLS network to connect branch offices 150 and 152 and data centers 154, 156, and 158.
[0046] In some embodiments, secure connection links are defined between gateways to allow paths through virtual networks to traverse from one public cloud data center to another, while in other embodiments no such links are defined. In addition, because gateways 105 and 107 are multi-tenant gateways, in some embodiments they are used to define other virtual networks for other entities (e.g., other companies, organizations, etc.). Some such embodiments store a tenant identifier in a tunnel header that encapsulates packets to traverse a tunnel defined between a particular entity's gateway and edge forwarding element. The tunnel identifier allows the gateway to distinguish packet flows it receives from an edge forwarding element of an entity from packet flows it receives along other tunnels of other entities. In other embodiments, the gateway is single-tenant and is deployed specifically for use by only one entity.
[0047] Figure 1 A controller cluster 140 in a private data center 117 is shown. The controller cluster 140 serves as a central point for managing (e.g., defining and modifying) configuration data provided to edge nodes and / or gateways to configure some or all operations. In some embodiments, the controller cluster has a set of manager servers that define and modify configuration data, and in some embodiments, a set of controller servers that distribute configuration data to edge forwarding elements and / or gateways. In other embodiments, the controller cluster has only one set of servers that define, modify, and distribute configuration data. In some embodiments, the controller cluster directs edge nodes to use certain gateways (i.e., assigns gateways to edge nodes) and establishes direct connections with other edge nodes.
[0048] although Figure 1 The controller cluster 140 is shown as residing in one private data center 117, but in some embodiments, the controller resides in one or more public cloud data centers and / or private cloud data centers. In addition, some embodiments deploy one or more gateways in one or more private data centers (e.g., a data center where the gateway is deployed and an entity that provides a controller for configuring the gateway to implement a virtual network).
[0049] exist Figure 1In the example shown, the deep packet inspectors 190 and 192 used by the edge node 120 are located in two different physical locations. The local deep packet inspector 190 is in the same physical location as the edge node 120 (i.e., in the branch 150), while the remote deep packet inspector 192 is co-located with the controller set 140 in the data center 117. In some embodiments, the local deep packet inspector operates on a first computing device with the edge node machine (e.g., as a VM or container). In other embodiments, the local deep packet inspector operates on another device other than the edge node machine or device. For example, in these embodiments, the local deep packet inspector 190 is a standalone device or a machine (e.g., a VM or container) executed on another computer.
[0050] The remote deep packet checker 192 operates at a remote location away from the physical location of the edge node 120 (e.g., in another building, neighborhood, city, state, etc. outside the location where the particular edge node operates). In some embodiments, the remote deep packet checker 192 is part of the controller cluster (e.g., is another process or thread executed by the controller cluster). In other embodiments, the remote packet checker 192 operates near the controller cluster (e.g., is a VM executed in the same computer cluster as the controller cluster and has a stable communication link with the controller cluster). In addition, in some embodiments, the local and remote deep packet checkers are executed by dedicated hardware accelerators, which are part of the CPU, exist as one or more coprocessors, exist as one or more add-in cards, and / or utilize dedicated processing units (such as one or more GPUs).
[0051] For a particular packet flow, the edge node 120 initially uses the local deep packet inspector 190 to perform a first DPI operation on an initial set of packets for the particular packet flow. For the particular packet flow, the DPI operation generates a DPI parameter set, which in different embodiments includes different DPI parameters or combinations of these parameters, such as an identifier that specifies a traffic type carried in the payload of the packet, an identifier that specifies an application that is the source of the flow, an identifier that specifies a class type associated with the flow, etc. In some embodiments, the local deep packet inspector does not generate an identifier of the source application or class. In these embodiments, the edge node generates one or both of these identifiers by mapping the traffic type identifier generated by the DPI operation to an application or class identifier.
[0052] In some embodiments, the edge node 120 forwards packets of a particular packet flow based on the generated first DPI parameter set. For example, in some embodiments, the edge node 120 uses at least one parameter in the generated first DPI parameter set to select a path for forwarding packets of the first packet flow through the WAN. In some embodiments, the edge node 120 forwards a copy of the set of packets it used for the first DPI operation of the local deep packet inspector 190 to the remote deep packet inspector 192 to perform a second DPI operation to generate a second DPI parameter set. The edge node 120 receives the result of the second DPI operation. When the generated first and second DPI parameters are different, the edge node 120 generates a record of the difference.
[0053] In some embodiments, the edge node 120 uses the generated record to improve the operation of the local deep packet inspector. For example, in some embodiments, the local deep packet inspector is a third-party inspector used by a particular edge node, and the generated record is used to identify different flows for which the third-party inspector has poor DPI performance. When the generated record specifies a difference between the first and second sets of generated DPI parameters, in some embodiments, the edge node 120 sends data about the difference to the remote machine to be aggregated with other data about other differences in DPI operations performed for other packet flows over the WAN.
[0054] In some embodiments, after completing the first DPI operation, the edge node 120 specifies the generated first DPI parameter set as the DPI parameter set associated with the first packet flow. When the first and second DPI parameter sets are different, the edge node 120 modifies the DPI parameter set associated with the first packet flow based on the generated second DPI parameter set. For example, in some embodiments, the edge node 120 modifies the DPI parameter set by storing the second DPI parameter set as the DPI parameter set associated with the first packet flow.
[0055] In addition, in embodiments where the edge node 120 forwards packets of a particular packet flow based on the generated DPI parameters, when the generated first and second DPI parameter sets are different, the edge node 120 modifies the forwarding of packets of the first packet flow by using the second DPI parameter set. In some embodiments, the edge node 120 directly or indirectly forwards at least a subset of the generated first and / or second DPI parameters to other edge nodes through the controller cluster 140 (e.g., through in-band or out-of-band communication with other edge nodes). In addition, in some embodiments, the edge node 120 directly or indirectly forwards at least a subset of the generated first and / or second DPI parameters to at least one gateway through the controller cluster 140 (e.g., through in-band or out-of-band communication with the gateway).
[0056] Figure 2 Conceptually illustrated is a process 200 that edge node 120 performs in some embodiments when it receives a packet for forwarding. In some embodiments, edge node 120 performs the process for each outgoing packet it receives from inside branch 150 for forwarding outside branch 150, or for each incoming packet it receives from outside branch 150 for forwarding to a machine within branch 150. In other embodiments, edge node 120 performs the process only for each outgoing packet.
[0057] As shown, process 200 begins when an edge node receives (at 205) a packet for forwarding. Next, at 210, the process determines whether the packet is part of an existing flow that the edge node is currently processing. In some embodiments, the existing flows are two opposing flows (i.e., bidirectional flows) in the same connection session between a machine in branch 150 and a machine outside branch 150 (e.g., in branch 152 or in data center 154, 156, or 158) because DPI operations analyze packets exchanged in both directions in a connection session. In other embodiments, the existing flow is a unidirectional flow between the two machines (e.g., from an internal machine to an external machine).
[0058] Additionally, at 210, when the first and second flows are part of a set of flows that have certain header values in common (e.g., source IP address, destination IP address, and destination port), in some embodiments, process 200 treats the DPI operation performed on the first flow as a DPI operation on a subsequent second flow. In other words, in some embodiments, the flow determination at 210 determines whether the DPI operation is performed on a set of flows that may be grouped together based on certain criteria.
[0059] To determine whether the packet received at 205 is part of an existing flow, in some embodiments, the process checks a connection tracking storage device that stores a record for each flow it is currently processing. In some embodiments, the connection tracking storage device stores a record for each flow, where the record for a flow stores an identifier for the flow (e.g., a five-tuple identifier for the flow, which includes source and destination IP addresses, source and destination port addresses, and a protocol). Therefore, in these embodiments, process 200 determines (at 210) whether the flow identifier of the received packet is stored in the connection tracking storage device. In embodiments where process 200 performs local DPI operations on a set of flows, process 200 determines (at 210) whether the flow attributes of the received packet match the flow attributes of a set of flows stored in the connection tracker.
[0060] If not, the process creates (at 215) a flow container to store a copy of the initial packet of the flow in the flow container. At 215, the process also creates a record in its connection tracker for the flow of the received packet (e.g., stores the packet's five-tuple flow identifier in the connection tracker). The process transitions from 215 to 220. The process also transitions to 220 when the process determines (at 210) that the received packet is part of the flow it is currently processing.
[0061] At 220, the process determines whether it has completed a DPI operation on the flow of the received packet. To make this determination at 220, in some embodiments, the process checks another connection tracking storage device that stores a record for each flow or set of flows for which it has previously completed a DPI operation. In some embodiments, each record in the connection tracking storage device stores a flow identifier (e.g., a five-tuple identifier) for the flow or set of flows for which the process has previously completed a DPI operation, and the DPI parameter set that the process previously identified for the flow. In conjunction with or alternatively storing the DPI parameter sets, each record stores a forwarding decision or other forwarding action (such as egress queue selection) that the edge node previously made based on the DPI parameter set that it previously identified for the flow.
[0062] When the process determines (at 220) that it has previously completed DPI operations for the flow or set of flows for the received packet, it transitions to 250 where it will forward the packet based on the forwarding operation(s) it previously decided based on the DPI parameters previously identified for the flow or set of flows. In some embodiments, these forwarding operations include any combination of selecting a path along which the packet should be sent, selecting an egress queue in which the packet should be stored before forwarding, specifying QoS parameters for the packet for other gateways or edge nodes to use, etc.
[0063] When the process determines (at 220) that it has not previously completed a DPI operation for the flow or set of flows for the received packet, the process stores (at 225) a copy of the received packet in a flow container that was defined at 215 or that was previously defined for an earlier packet in the same flow. Next, at 230, the process provides the received packet to the local deep packet inspector 190 to perform its DPI operation.
[0064] At 235, the process determines whether local deep packet inspector 190 is able to complete its operation based on the received packet. In some embodiments, the process makes this determination based on a response it receives from local deep packet inspector 190. In some embodiments, local inspector 190 returns a set of one or more DPI parameters for the flow of received packets when it has completed its operation, and returns a reply indicating that it has not yet completed its operation when it needs to analyze more packets for the flow.
[0065] When the process determines (at 235) that the local inspector 190 needs more packets to analyze, the process performs (at 245) a forwarding classification operation without reference to any DPI parameter values, forwards (at 250) the received packet based on the forwarding classification operation, and then ends. In some embodiments, the forwarding classification operation involves matching attributes of the received packet (e.g., its flow identifier or a set of attributes of its flow set) with one or more match action rules that specify a next hop interface for the packet and tunnel attributes that should be used to encapsulate the packet and forward it to the next hop.
[0066] In the above method, neither the edge node nor the local deep packet inspector performs a soft termination of the connection session associated with the received packet, while the local DPI inspector can perform its DPI operation. Under this method, the packet is forwarded (at 250) after its classification (at 245). In other embodiments, the edge node or the local deep packet inspector performs a soft termination on the connection session associated with the received packet so that the local DPI operation can perform its DPI operation. In some of these embodiments, the edge node 120 does not forward any initial packets in the flow out of the branch 150, but instead stores the packets in a container until the local DPI operation has completed so that it can perform an action (e.g., a forwarding decision) based on the DPI operation.
[0067] When the process 230 determines (at 235) that the local deep packet inspector 190 is able to complete its operations based on the received packet, it determines (at 240) whether a DPI-based action must be performed on the packet. As described above, in some different embodiments, the returned DPI parameter set includes different DPI parameters. Examples of these parameters include business type identifiers, source application type identifiers, class identifiers, etc. In some embodiments, the local deep packet inspector does not generate an identifier for the source application or class. In these embodiments, the controller cluster generates one or both of these identifiers by mapping the generated business type identifier to a local DPI operation of the application or class identifier.
[0068] Based on the returned DPI parameter set, in some embodiments, the edge node 120 performs its forwarding operation on the packet flow associated with some DPI parameters. For example, in some embodiments, the edge node 120 selects a faster path (i.e., a path with low latency) or a more resilient path (i.e., a path with a very low failure rate) for packets associated with VOIP calls (which must use the best available path). In combination or alternatively, in some embodiments, the edge node 120 associates these packets with a higher priority queue so that these packets can enjoy a higher quality of service (QoS).
[0069] In some of these embodiments, the edge node does not perform any special actions on the packet flow unless the DPI inspector 190 or 192 associates the packet flow with one or more specific DPI parameters. Thus, when the process determines (at 240) that the DPI parameters set for the received packet flow are not associated with any special type of action, the process performs (at 245) its forwarding operations without reference to any DPI parameter values and forwards (at 250) the received packet and any packets it previously stored for the flow based on these forwarding operations.
[0070] In some embodiments, the process performs these forwarding operations by matching the identifier of the flow to one or more match-action forwarding rules that identify tunnel parameters (e.g., tunnel identifier, etc.) and forwarding parameters (e.g., next-hop forwarding interface, destination network address (IP, port, MAC, etc.), etc.), and then encapsulating and forwarding the packet(s) of the flow based on the tunnel and forwarding parameters, as described above. At 245, in some embodiments, the process also stores an indication that the local DPI inspector 190 did not provide any DPI parameters that require any special processing for the flow, while in other embodiments, it does not store any such indication at 245.
[0071] In addition, Figure 2 In the embodiment, the process 200 does not request the remote deep packet inspector 192 to analyze the received flow of packets when the process 200 determines (at 240) that it does not have to perform any special operation on the flow based on the DPI parameter(s) returned by the local inspector 190. However, in other embodiments, the process instructs the remote inspector 192 to analyze the received flow of packets even when the process determines (at 240) that it should not perform any special operation on the flow based on the parameters returned by the local inspector.
[0072] When the process determines that it should perform special actions on the received packet flow based on the parameters returned by the local inspector 190, the process transitions from 240 to 255. For the received packet flow, the process stores (at 255) a locally generated DPI parameter set (i.e., generated by the local DPI inspector 190) in a storage device (e.g., a database) that associates the flow with the DPI parameter set. The process stores the DPI parameter set for subsequent reporting operations and / or for comparison with remotely generated DPI parameters.
[0073] It also sends (at 255) a copy of the initial set of packets inspected by the local deep packet inspector 190 to the remote deep packet inspector 192. The initial set of packets includes any packets stored in the flow container that was created at 215 for the packets of the initial flow and in which one or more packets are stored at 225 in each iteration of the process 200 for that flow. In some embodiments, the process sends (at 255) to the remote deep packet inspector 192 a locally generated set of DPI parameters (i.e., a set of DPI parameters generated by the local inspector 190) along with the initial set of packets.
[0074] At 260, the process performs its forwarding classification operations using the one or more DPI parameters identified by local deep packet inspector 190. In some embodiments, the process performs these forwarding operations by matching the identifier of the flow and the one or more DPI parameters to one or more match-action forwarding rules that identify tunnel parameters (e.g., tunnel identifier, etc.) and forwarding parameters (e.g., next-hop forwarding interface, etc.). Based on the tunnel and forwarding parameters identified at 260, the process then encapsulates and forwards (at 250) the received packet and then ends.
[0075] In some embodiments, the DPI-based forwarding classification operation at 260 may change the path through the WAN that was selected for an earlier packet of the flow by the DPI-independent forwarding classification operation at 245. For example, after selecting a slow path through the WAN for a particular flow before the local DPI operation is completed, in some embodiments, the edge node may select a faster path once the local DPI operation has completed and the DPI operation specifies that the flow is associated with an application that requires the use of the best available path.
[0076] In other embodiments, rather than modifying the path of the flow on which the local DPI operation is performed, process 200 stores the identified DPI parameters of the associated flow set and then uses the stored DPI parameters to select a fast path for subsequent flows in the same flow set as the current flow. In addition, for the current flow or subsequent flows in the same flow set, process 200 performs (at 250 or 260) other forwarding operations based on the DPI parameter set identified by the local DPI operation of the current flow. For example, in some embodiments, process 200 specifies a higher priority egress queue based on the identified DPI parameter set to provide a higher QoS for the current flow or subsequent flow in the flow set. In some embodiments, process 200 also includes in the tunnel header of the current flow or subsequent flow a notification of the higher priority QoS parameters of the current flow or subsequent flow to the gateway (or one or more) or the destination edge node.
[0077] In some embodiments, the edge node 120 forwards each packet to its destination after the local deep packet inspector has processed it. However, in other embodiments, the edge node 120 delays forwarding the packet to the destination of the flow while performing local DPI operations. During this period, the edge node stores the delayed packets in a designated flow container of the packet flow (i.e., a storage queue defined by the edge node for the packet flow). Once the first DPI operation has been completed, the edge node then forwards the set of packets stored in the storage queue and subsequent packets of the first flow to the destination. For certain locally identified DPI parameters, the forwarding is based on the DPI parameters (e.g., for certain DPI parameters, the next hop / path selection is based on the DPI parameters). The edge node 120 in these embodiments also forwards a copy of the set of packets stored in the storage queue to the remote deep packet inspector.
[0078] In some embodiments, the number of packets stored in a flow container for a particular packet flow depends on the number of packets required by the local deep packet inspector 190 to complete its DPI operations. Specifically, in some embodiments, the local deep packet inspector needs to inspect different numbers of packets for flows from different types of source applications in order to assess the traffic type, source application type, category type, etc. However, typically, for many applications, the number of packets is in the range of 10-20 packets.
[0079] Figure 3 A process 300 is conceptually illustrated that is performed when an edge node 120 receives the results of a DPI operation by a remote deep packet inspector 192 for a particular flow. For certain packet flows (e.g., packet flows for which the local packet inspector 190 generates particular DPI parameters), in some embodiments, the edge node 120 forwards to the remote deep packet inspector 192 a copy of the initial set of packets that the local deep packet inspector 190 uses to perform its DPI operation. In other embodiments, the edge node 120 forwards to the remote deep packet inspector 192 a greater number of packets of the flow to be analyzed than the number of packets it provides to the local deep packet inspector 190.
[0080] like Figure 3 As shown, the process begins (at 305) when the edge node 120 receives the result of the second DPI operation from the remote deep packet inspector 192. The process then determines (at 310) whether the second DPI operation produces a second DPI parameter set that is different from the first DPI parameter set produced by the local deep packet inspector 190. When the two DPI parameter sets match, in some embodiments, the remote deep packet inspector 192 simply returns an indication of the match. Alternatively, when the second DPI parameter set does not match the first DPI parameter set, the remote deep packet inspector 192 returns the second DPI parameter set in an encoded or unencoded format.
[0081] When the process determines (at 310) that the second set of DPI parameters generated by the remote DPI operation matches the first set of DPI parameters generated by the local DPI operation, the process creates (at 315) a record for the particular flow to indicate that there is no difference between the two sets of DPI parameters, and then ends. In some embodiments, the record is simply another field in the record that process 200 creates (at 255) in the DPI parameter storage device of the edge node to store the set of DPI parameters for the particular flow.
[0082] Alternatively, when the process determines (at 310) that the first and second DPI parameter sets do not match, the process creates (at 320) a record of the difference. For example, in some embodiments, the process identifies (at 320) the second DPI parameter set as the DPI parameter set associated with the particular flow. In some embodiments, the process does this by storing the second DPI parameter set in a record created for the particular flow in a DPI parameter storage device of the edge node. In some embodiments, the process also sets (at 320) a value of a field in the record to specify the difference between the local and remote DPI operations.
[0083] The process 300 also stores (at 320) in the record or another record a first set of DPI parameters generated by the local deep packet inspector 190 and replaced by a second set of DPI parameters. In some embodiments, the process 300 maintains the first set of DPI parameters because the record is used to improve the operation of the local deep packet inspector. For example, in some embodiments, the local deep packet inspector is a third-party inspector used by a particular edge node, and the generated record is used to identify different flows for which the third-party inspector has poor DPI performance. When the generated record specifies a difference between the first and second sets of generated DPI parameters, in some embodiments, the edge node 120 sends data about the difference to a remote machine to be aggregated with other data about other differences in DPI operations performed on other packet flows over the WAN. Then, in some embodiments, the data is analyzed to modify the operation of the local deep packet inspector.
[0084] In embodiments where edge node 120 forwards packets of a particular packet flow based on the generated DPI parameters, process 300 determines (at 325) whether it needs to modify its forwarding of packets of the particular flow based on the second set of DPI parameters received from remote deep packet inspector 192. If so, edge node 120 modifies the forwarding.
[0085] Figure 4 An example of modifying the path selected for a particular flow is shown. Figure 1 The example shown in Figure 4Edge node 120 is shown initially forwarding a particular flow to gateway 105, which follows a first path to edge node 124. The selection of gateway 105 as the next hop is based on a first DPI parameter set generated by local deep packet inspector 190. Edge node 190 uses one or more parameters in the set to select gateway 105 instead of gateway 107 on a second path to edge node 124. The second path in this example has lower latency and is used for higher priority packets. However, edge node 120 does not initially select gateway 107 and its associated second path because the first DPI parameter set does not include any parameters associated with a high priority flow.
[0086] Figure 4 Also shown is an edge node that receives a second set of DPI parameters from a remote deep packet inspector 192. Based on the second set of DPI parameters, the edge node 120 begins forwarding the particular flow through the gateway 107 and the second path. In this example, the second set of DPI parameters has one DPI parameter associated with a high priority flow (e.g., with a traffic type identifier that specifies that the payload of the flow contains VOIP data). The edge node 120 matches the second set of DPI parameters and the flow identifier of the particular flow with a match action rule that specifies the gateway 107 as the next hop of the path to be selected.
[0087] Some embodiments provide a method of using DPI generated parameters to evaluate and in some cases modify how flows associated with a particular application traverse an SD-WAN. At a set of one or more servers, the method receives a set of DPI parameters collected for packet flows processed by a first set of edge nodes on which DPI operations are performed. From these collected sets, the method involves a subset of DPI parameters associated with a plurality of flows involving a particular application identifier specified by the DPI operation.
[0088] In some embodiments, the received DPI parameter set includes operational statistics and metrics related to packet flows processed by the first set of edge nodes (e.g., packet transmission time, payload size, current number of packets processed by the node, etc.). In some embodiments, the statistics are accompanied by other data, such as flow identifiers, application classification details, and forwarding decisions (e.g., identifying a selected path), etc. In some embodiments, the operational statistics, metrics, and other data are collected and provided by edge nodes and / or gateways.
[0089] The method then analyzes the identified subset of parameters to determine whether any packet flows associated with the one or more specific DPI parameters have an undesirable metric related to their flow through the WAN. When the analysis results in a determination that the edge node should use a different path for flows associated with the specific application identifier, the method then distributes the adjusted next-hop forwarding record to a second set of one or more edge nodes to modify the path that the edge node uses to forward flows associated with the specific application identifier. In some embodiments, the first and second sets of edge nodes are the same, while in other embodiments, the first set of edge nodes is a subset of the second set of edge nodes (e.g., the second set includes at least one node that is not in the first set).
[0090] In some embodiments, the above method comprises Figure 1 The controller cluster 140 is implemented. Figure 5 Components of a controller cluster that performs the above operations are shown. As shown, in some embodiments, the controller cluster 140 includes a data collector 505, a data correlator 510, a data aggregator 515, a data evaluator 517, a gateway deployment manager 520, and a path generator 525. In some embodiments, these components operate on one computer, while in other embodiments, they operate on multiple computers. For scalability, in some embodiments, each component can be implemented by a cluster of similar processes.
[0091] Reference Figure 6 describe Figure 5 operation of the components of the controller cluster 140, Figure 6 A process 600 is conceptually illustrated that is periodically performed by the controller cluster 140 in some embodiments. The process collects data for flows associated with certain DPI parameters from edge nodes and / or gateways. It correlates the collected data with associated data about the same flows, and then analyzes the collected data to derive additional statistics / metrics about each flow. The process then compares the collected and derived data for the flows with the expected service level metrics / statistics associated with the DPI parameters for the flows to identify when the flows are not obtaining the expected service level (e.g., flows associated with a particular application identifier are not reaching their destination quickly enough).
[0092] When it identifies one or more flows that are not obtaining the desired service level, the process 600 distributes an adjusted next-hop forwarding record to the edge nodes and / or gateways, which directs the edge nodes and / or gateways to modify the forwarding of the specific flow or similar future flows (e.g., flows from applications with the same DPI identification and / or to the same destination). For example, in some embodiments, based on the distributed path adjustment value, the source edge node selects a different gateway to forward packets for the specific flow and other similar subsequent flows. In other embodiments, the source edge node uses the distributed adjusted next-hop forwarding record to select a gateway for forwarding subsequent flows similar to the specific flow (e.g., flows with applications with the same DPI identification and to the same destination).
[0093] like Figure 6 As shown, in some embodiments, the data collector 505 collects (at 605) data from the first set of SD-WAN forwarding nodes regarding the processing of flows associated with a set of one or more DPI parameters by the nodes. For example, in some embodiments, the data collector collects data regarding the processing of flows associated with one or more traffic type identifiers (e.g., VOIP calls, video conferencing, etc.). In other embodiments, the data collector collects data regarding the processing of all flows passing through the first set of forwarding nodes.
[0094] In some embodiments, the first set of forwarding nodes is only a subset of the SD-WAN forwarding nodes, and the collected data set is related to flows for a subset of tenants of the SD-WAN (e.g., one tenant). As further described below, process 600 collects data only from a subset of forwarding nodes, but shares the analysis results of the data with a larger set of SD-WAN forwarding nodes. Similarly, when only data for a subset of tenants is collected, the process uses the analysis results of the data with a larger set of tenants (e.g., all tenants or all tenants using a particular SaaS application). In some embodiments, the first set of forwarding nodes is simply an SD-WAN edge node that acts as a source and destination for flows through the network. In other embodiments, the first set of forwarding nodes also includes SD-WAN cloud gateways (e.g., gateways 105 and 107).
[0095] In some embodiments, the collected data includes operational statistics and metrics of the flow (e.g., average packet transmission time, average packet delay, average payload size, etc.). These operational statistics and metrics are collected by the first set of forwarding nodes for the packet flows processed by these nodes. In some embodiments, the collected data also includes operational statistics and metrics of the forwarding nodes. Examples of such statistics and metrics include queue depth, queue delay, number of packets processed by a node in a certain duration, etc.
[0096] As described above, the collected records identify flow identifiers, application classification details, and forwarding decisions (e.g., identifying the selected path), etc. The collected records include flow and / or forwarding node statistics / metrics associated with one or more DPI parameters, which in some embodiments are specified by DPI operations performed while processing these flows at the forwarding nodes. In some embodiments, the DPI operations for the flows are performed at the source edge node (also known as the ingress edge node), where the flow enters the WAN and is passed from the source edge node to another edge node or cloud gateway. In conjunction with the DPI operations, the source edge node collects operational metrics and statistics (e.g., packet transmission time, payload size, current number of packets processed by the node, etc.) for the packets of the flows it passes to another edge node or cloud gateway, and provides the DPI-generated parameters together with the collected statistics to the server set for analysis.
[0097] In some embodiments, the SD-WAN forwarding node continuously collects statistics / metrics of the flow while processing the flow. In other embodiments, these forwarding nodes collect statistics / metrics of the flow just at the beginning of the flow in order to reduce the amount of resources consumed by data collection (e.g., CPU resources, memory resources). For example, in some embodiments, the source edge node collects statistics of the flow based on the number of initial packets used to perform its DPI operation. In some of these embodiments, the source edge node provides the controller set with its initial set of packets for its DPI operation for the flow, as well as the operational metrics and statistics provided to the server set for the new flow. In some embodiments, the number of packets in the initial set of packets analyzed by the DPI operation of the source edge node depends on the application identified as the source of the flow by the DPI operation. Therefore, the DPI operation analyzes different numbers of packets for different flows from different applications or different types of applications.
[0098] In some embodiments, the destination edge nodes (also referred to as egress edge nodes) also perform DPI operations and collect operational metrics / statistics of flows at the beginning of flows they receive over the WAN (i.e., from a cloud gateway or other edge node). In other embodiments, the destination edge nodes do not perform DPI operations, but collect operational metrics / statistics of flows (e.g., continuously or only at the beginning of flows). In some embodiments, the destination edge node receives (e.g., in-band of a packet via a tunnel header, or out-of-band via other packets) one or more DPI parameters (e.g., application identifiers) generated by the DPI operations of the source edge node.
[0099] In some embodiments, the destination edge node receives instructions from the source edge node directing the destination edge node to collect statistics / metrics about certain flows. For example, in some embodiments, the source edge node sets flags in the tunnel encapsulation headers that these edge nodes use to forward packets to the gateway to direct the destination edge node to collect statistics for certain flows. The gateway in these embodiments forwards these flags when forwarding the encapsulated packets to the destination edge node.
[0100] In some embodiments, the data collector 505 also collects statistics / metrics about the processing of flows from the gateways. In some embodiments, the source edge nodes set flags in the tunnel encapsulation headers that these edge nodes use to forward packets to the gateways to direct the gateways to collect statistics for certain flows. In addition, in conjunction with or alternatively to performing DPI operations at edge nodes, some embodiments perform DPI operations external to edge nodes (e.g., at a physical location that is remote from the physical location where the edge nodes operate).
[0101] The data collector 505 stores the data received at 605 in the raw data storage device 530 of the controller cluster 140. In some embodiments, the correlator 510 then correlates (at 610) different records for the same flow collected from different edge nodes and / or gateways stored in the raw data storage device 530. To correlate the records, the correlator 510 uses a flow identifier (e.g., a five-tuple identifier of the flow) to identify records related to the same flow collected from different forwarding elements of the SD-WAN (e.g., from a source edge node, a destination edge node, and / or a gateway).
[0102] In different embodiments, the correlator 510 associates related collected flow records in different ways. In some embodiments, it creates associations between related records of a flow (e.g., a reference in each record to a data structure storing related records). In other embodiments, it merges sets of related records of a flow into one record. Still other embodiments associate related flow records in different ways. Furthermore, in some embodiments, each related set of related records is associated with a set of parameters generated by the DPI (e.g., with a specific application identifier or a traffic type identifier).
[0103] The correlator 510 stores the relevant records for each flow in a relevant data storage device 535. The aggregator 515 retrieves the relevant records from the storage device 535, derives additional statistics / metrics from these records, stores the provided and derived statistics / metrics for flows that it has not previously identified, and mixes the provided and derived statistics / metrics with the statistics / metrics that it has previously stored for flows that it has previously identified.
[0104] Specifically, once the collected metrics / statistics are correlated for a particular flow, the aggregator 515 analyzes the collected metrics / statistics to derive additional operational data that quantifies whether the particular flow is obtaining the desired service level. In some embodiments, the correlated metrics / statistics data is associated with parameters of a particular DPI generation (e.g., application identifier, etc.) such that, in some embodiments, the analysis is done based on the DPI parameters. For example, in some embodiments, the derived data is used to determine whether a particular flow associated with a particular application identifier reaches its destination within the expected duration, whether the particular flow is delayed too much at a particular gateway, etc.
[0105] The following is an example of how the aggregator derives statistics / metrics for a flow from the collected correlation records of the flow. In some embodiments, the collected flow records specify, on a per-packet basis, the time at which the packet left the source edge node, arrived at the gateway node, left the gateway node, and arrived at the destination edge node. After these records are correlated, the aggregator 515 calculates the average transmission time taken for the packets of the flow to traverse from the source edge node to the destination edge node.
[0106] If the aggregator has not previously processed the statistics / metrics for the stream, the aggregator creates a record for the stream in the aggregate data storage device 519 and stores in the record the collected and associated statistics / metrics for the stream and any statistics / metrics derived by the aggregator for the stream (e.g., the calculated average transmission time for the stream). For some streams, the storage device has previously stored records as statistics / metrics for these streams previously processed by the aggregator. Therefore, for each such stream, in some embodiments, the aggregator 515 aggregates the newly collected and derived statistics / metrics with the statistics / metrics for the previously collected and derived streams. In some embodiments, the aggregation operation uses a weighted sum to mix the new statistics / metrics with the previously stored statistics / metrics. In some embodiments, the weighted sum ensures that the associated statistics / metrics for the stream do not fluctuate dramatically each time a new set of statistics / metrics is received.
[0107] In some embodiments, the aggregator also processes the statistics / metrics stored in the associated data storage device 535 for the gateway to blend new statistics / metrics for the gateway stored in the storage device with the statistics / metrics for the gateway that it previously stored in the aggregated data storage device 519. To blend these statistics / metrics, in some embodiments, the aggregator 515 uses a weighted sum to ensure that the associated statistics / metrics for the gateway do not fluctuate wildly each time a new set of statistics / metrics is received.
[0108] The data evaluator 517 analyzes the statistics / metrics stored in the aggregate data storage device 519 to identify any flows associated with a particular set of DPI parameters that are not obtaining the desired service level from the SD-WAN. The data evaluator 517 also analyzes the stored statistics / metrics to identify any congested gateways. Figure 7 A process 700 is conceptually illustrated that the evaluator 517 performs to identify such flows and gateways. In some embodiments, the data evaluator performs the process 700 periodically.
[0109] As shown, the process selects (at 705) a record for a flow in the aggregate data storage device 519 and identifies (at 710) a subset of DPI parameters associated with the flow (e.g., application identifier, traffic type identifier, etc.). In some embodiments, the identified subset of DPI parameters is stored with the record for the selected flow, while in other embodiments, it is referenced by the record. Then, from the service level storage device 522, the process 700 retrieves (at 715) a desired set of service performance statistics / metrics for the identified subset of DPI parameters from the service level storage device 531.
[0110] The process next determines (at 720) whether the statistics / metrics stored in the records for the retrieved flows do not satisfy any expected service performance statistics / metrics for the identified subset of DPI parameters (e.g., are above an expected service performance threshold). Some embodiments have different service level guarantees for flows associated with different DPI parameters. For example, in some embodiments, flows associated with a first service type identifier cannot have a delay at the cloud gateway that exceeds a first time duration, while flows associated with a second service type identifier cannot have a delay at the cloud gateway that exceeds a second time duration. In conjunction or alternatively, in some embodiments, flows associated with a first service type identifier must reach their destination edge node within a time period, while flows associated with a second service type identifier must reach their destination edge node within another time period.
[0111] When the process determines that the statistics / metrics stored in the retrieved flow record do not satisfy any desired service performance statistics / metrics for the identified DPI parameter subset, the process stores (at 725) the flow record in the path analysis storage device 523 so that the flow's path through the SD-WANN can be further analyzed, and then transitions to 730. When it is determined (at 720) that the stored statistics / metrics of the flow satisfy the desired service performance statistics / metrics, the process also transitions to 730. At 730, the process determines whether it has examined all flow records. If not, it returns to 705 to select another flow record and repeat its operations on that record. Otherwise, transition to 735.
[0112] At 735, the process traverses the records of cloud gateways in the aggregate data storage device 519 to identify any cloud gateways that are too congested. In some embodiments, the process generally determines whether a cloud gateway is too congested for all flows in general. In other embodiments, the process makes this determination for flows associated with a particular set of one or more DPI parameters. For example, in some such embodiments, the process determines whether a cloud gateway is too congested to process flows associated with a particular traffic type identifier. The process 700 stores (at 740) a record of each cloud gateway that it identifies as being too congested in the gateway analysis storage device 529 and then ends.
[0113] After the data evaluator 517 identifies congested gateways and underperforming flows, the gateway deployment manager 520 evaluates (at 625) the gateway data, determines when and where additional cloud gateways should be deployed, and deploys these cloud gateways. In some embodiments, a cloud gateway is a machine (e.g., VM) that executes on a host computer in a cloud data center and performs forwarding operations.
[0114] In some of these embodiments, the gateway deployment manager 520 instantiates and configures a new machine to function as a new gateway in the same cloud data center as one or more other gateways, or to function as a new gateway in a new cloud data center without any other gateways. In other embodiments, the gateway is pre-instantiated, and the deployment manager 520 simply assigns the previously instantiated gateway to perform the desired cloud gateway service for the SD-WAN of the entity in question.
[0115] In some embodiments, when an existing gateway has too much load for a particular duration, the gateway deployment manager 520 deploys a new gateway to relieve the load on the existing congested gateway. For example, in some embodiments, the gateway deployment manager maintains a count of the number of time periods that an existing gateway has too much load, and deploys a new gateway to relieve the load on the existing gateway only when the count it maintains for that gateway reaches a particular value before being reset. In some of these embodiments, the deployment manager 520 reduces or resets when the newly aggregated data does not identify a gateway previously identified as congested as congested.
[0116] In some embodiments, the gateway deployment manager 520 deploys a new gateway for use by all flows. In other embodiments, the gateway deployment manager 520 deploys a new gateway for use by flows associated with certain DPI parameters. For example, when the process 600 determines that the gateway used for VOIP calls is too congested, in some embodiments, the deployment manager 520 deploys another cloud gateway to handle flows associated with the VOIP service identifier.
[0117] Figure 8An example of this is shown. Specifically, the figure shows that a new cloud gateway 815 is deployed to handle VOIP calls after the controller set detects that the VOIP call load on two previously deployed cloud gateways 805 and 810 has exceeded a certain level that prevents the VOIP calls from receiving their expected service level. In this example, the new gateway is added to a new data center 830 that is different from the data centers 820 and 825 that host cloud gateways 805 and 810. In some embodiments, the controller cluster reduces the load on one or more cloud gateways by deploying one or more gateways in the same data center as the overloaded previously deployed gateways.
[0118] Once the deployment manager 520 deploys the new gateway, it instructs (at 625) the path generator 525 to identify new paths for flows using the newly deployed gateway and to generate next-hop forwarding records for one or more edge nodes and gateways to use these newly identified paths. The path generator 525 stores the generated next-hop forwarding records in the record storage device 538, and the record distributor 540 retrieves the forwarding records from the record storage device 538 and distributes them to the specified edge nodes and / or gateways.
[0119] The path generator also specifies (at 630) adjusted next-hop forwarding records for a second set of edge nodes for one or more flows identified in the path analysis storage device 523 as requiring a better path through the SD-WAN, or for future flows having similar attributes to these identified flows. Specifically, as described above, the data evaluator 517 (1) analyzes the statistics / metrics stored in the records of the retrieved flows to identify any flows that do not meet the expected service performance metrics for the subset of DPI parameters associated with the flow, and (2) stores (at 720) the records of the flows in the path analysis storage device 523 so that the paths of the flows through the SD-WAN can be further analyzed. At 630, the path generator 525 explores alternative paths for each flow identified in the path analysis storage device to attempt to identify better paths for these flows or future similar flows so that these flows can meet the service level guarantees of the DPI parameters associated with the flow.
[0120] This exploration may result in the path generator identifying new gateways to be deployed. When the path generator identifies such a gateway, it instructs the gateway deployment manager 520 to deploy the new gateway. The path generator 525 (1) generates next-hop forwarding records for one or more edge nodes and gateways to use these newly deployed gateways in order to implement the new paths it has identified, and (2) stores these next-hop forwarding records in a record storage device 538, from which the record distributor retrieves the forwarding records and distributes them to the designated edge nodes and / or gateways. Figure 8is an example of adding a gateway to improve the performance of the SD-WAN path used by flows associated with certain DPI parameters, in this figure, it is the flows associated with the VOIP service type.
[0121] In some embodiments, the path generator's exploration of alternative paths may also move a subset of flows away from the gateway while maintaining another subset of flows with the gateway. Fig. 9 An example of this is shown. Specifically, in the figure, the controller set detects that the VOIP call load on a previously deployed cloud gateway 805 has exceeded a certain level that prevents the VOIP calls from receiving their expected service level. Therefore, the controller set reconfigures the branch edge node 924 to use the previously deployed cloud gateway 910 in the data center 920 for its VOIP calls in order to reduce the load on the cloud gateway 805.
[0122] At 630, in some embodiments, the path generator 525 provides its new next-hop forwarding record only to forwarding nodes that are members of the first set of forwarding nodes from which statistics / metrics are collected. However, in other embodiments, the path generator 525 provides its new next-hop forwarding record even to SD-WAN forwarding nodes from which the controller cluster did not collect statistics / metrics at 605. In other words, in some embodiments, the first set of forwarding nodes and the second set of forwarding nodes are the same, while in other embodiments, the first set of edge nodes is a subset of the second set of edge nodes (e.g., the second set includes at least one node that is not in the first set).
[0123] For example, in some embodiments, the controller cluster analyzes metrics associated with flows for one entity involving a SaaS provider's application, such as Office 365. After assessing that certain gateways do not meet the expected service level performance for one entity's monitored flows, the controller cluster not only configures the entity's edge nodes to reduce their usage or avoid the problem gateways altogether, but also configures other entities' edge nodes for the same SaaS provider application in the same manner.
[0124] In some embodiments, the controller cluster collects statistics / metrics from only a subset of the entity's branches and data centers in order to conserve resources. However, in these embodiments, the controller cluster uses the knowledge it gains from analyzing the collected data to configure edge nodes and gateways for all of the entity's branches and data centers that are part of the SD-WAN.
[0125] A person of ordinary skill will recognize that the above process is performed in different ways in other embodiments. Figure 6A set of operations is shown that are periodically performed by the controller cluster, but in some embodiments, the cluster performs these operations at a different frequency. In addition, rather than simply adjusting the next-hop forwarding record to adjust the path of certain flows, the controller cluster distributes path adjustment values to adjust how edge nodes select among multiple feasible paths to the same destination in order to reduce the load on a particular gateway and / or direct more flows through a better performing gateway.
[0126] In different embodiments, the server set distributes different types of path adjustment values. In some embodiments, the distributed path adjustment values include path selection weight values that edge nodes use to select among different paths to the same destination (e.g., for flows associated with a particular application to the same destination edge node). In other embodiments, the distributed path adjustment values include packet processing statistics and / or other congestion metrics associated with different gateways and / or different edge nodes. In some embodiments, the source edge node uses such statistics to select among different candidate gateways, different candidate gateways are used by different candidate paths to the same destination, and / or when there are multiple different such nodes or destinations (e.g., when there are multiple candidate data centers), to select among different candidate destination edge nodes. In other embodiments, the server set also uses other types of path adjustment values.
[0127] Many of the above features and applications are implemented as software processes designated as sets of instructions recorded on a computer-readable storage medium (also referred to as a computer-readable medium). When these instructions are executed by one or more processing units (e.g., one or more processors, cores of a processor, or other processing units), they cause the (one or more) processing units to perform the actions indicated in the instructions. Examples of computer-readable media include, but are not limited to, CD-ROMs, flash drives, RAM chips, hard drives, EPROMs, etc. Computer-readable media do not include carrier waves and electrical signals transmitted over wireless or wired connections.
[0128] In this specification, the term "software" is intended to include firmware residing in a read-only memory or an application stored in a magnetic storage device, which can be read into a memory for processing by a processor. In addition, in some embodiments, multiple software inventions can be implemented as sub-parts of a larger program while maintaining different software inventions. In some embodiments, multiple software inventions can also be implemented as separate programs. Finally, any combination of separate programs that jointly implement the software inventions described herein are within the scope of the present invention. In some embodiments, when the software program is installed to operate on one or more electronic systems, the software program defines one or more specific machine implementations that execute and implement the operations of the software program.
[0129] Fig.10A computer system 1000 is conceptually illustrated with which some embodiments of the present invention are implemented. The computer system 1000 can be used to implement any of the above-mentioned hosts, controllers, gateways, and edge forwarding elements. Therefore, it can be used to perform any of the above-mentioned processes. The computer system includes various types of non-transient machine-readable media and interfaces for various other types of machine-readable media. The computer system 1000 includes a bus 1005, (one or more) processing units 1010, a system memory 1025, a read-only memory 1030, a permanent storage device 1035, an input device 1040, and an output device 1045.
[0130] Bus 1005 collectively represents all system, peripheral, and chipset buses that communicatively connect the numerous internal devices of computer system 1000. For example, bus 1005 communicatively connects processing unit(s) 1010 with read-only memory 1030, system memory 1025, and permanent storage device 1035.
[0131] The processing unit(s) 1010 retrieves instructions to be executed and data to be processed from these different storage units in order to perform the processing of the present invention. In different embodiments, the processing unit(s) may be a single processor or a multi-core processor. The read-only memory (ROM) 1030 stores static data and instructions required by the processing unit(s) 1010 and other modules of the computer system. On the other hand, the permanent storage device 1035 is a read-write storage device. This device is a non-volatile storage unit that stores instructions and data even when the computer system 1000 is turned off. Some embodiments of the present invention use a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) as the permanent storage device 1035.
[0132] Other embodiments use removable storage devices (such as floppy disks, flash drives, etc.) as permanent storage devices. Like permanent storage device 1035, system memory 1025 is a read-write storage device. However, unlike storage device 1035, system memory is a volatile read-write memory, such as random access memory. System memory stores some instructions and data that the processor needs at run time. In some embodiments, the processes of the present invention are stored in system memory 1025, permanent storage device 1035, and / or read-only memory 1030. (One or more) processing units 1010 retrieve instructions to be executed and data to be processed from these different storage units in order to perform the processes of some embodiments.
[0133] The bus 1005 is also connected to input and output devices 1040 and 1045. Input devices enable a user to transmit information and select commands to the computer system. Input devices 1040 include an alphanumeric keyboard and a pointing device (also called a "cursor control device"). Output devices 1045 display images generated by the computer system. Output devices include printers and display devices, such as cathode ray tubes (CRTs) or liquid crystal displays (LCDs). Some embodiments include devices such as touch screens that serve as input and output devices.
[0134] Finally, if Fig.10 As shown, bus 1005 also couples computer system 1000 to a network 1065 via a network adapter (not shown). In this manner, the computer may be part of a computer network, such as a local area network ("LAN"), a wide area network ("WAN"), or an intranet, or a network of networks, such as the Internet. Any or all components of computer system 1000 may be used in conjunction with the present invention.
[0135] Some embodiments include electronic components such as microprocessors, storage devices, and memories that store computer program instructions in machine-readable or computer-readable media (alternatively referred to as computer-readable storage media, machine-readable media, or machine-readable storage media). Some examples of such computer-readable media include RAM, ROM, compact disk-read only (CD-ROM), compact disk-recordable (CD-R), compact disk-rewritable (CD-RW), read-only digital versatile disks (e.g., DVD-ROM, dual-layer DVD-ROM), various recordable / rewritable DVDs (e.g., DVD-RAM, DVD-RW, DVD+RW, etc.), flash memory (e.g., SD card, mini SD card, micro SD card, etc.), magnetic and / or solid-state hard drives, read-only and recordable Blu-ray discs. A computer readable medium may store a computer program that is executable by at least one processing unit and includes a set of instructions for performing various operations. Examples of computer programs or computer codes include machine code, such as produced by a compiler, and files including high-level code that is executed by a computer, an electronic component, or a microprocessor using an interpreter.
[0136] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, some embodiments are performed by one or more integrated circuits such as application specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs). In some embodiments, such integrated circuits execute instructions stored on the circuit itself.
[0137] As used in this specification, the terms "computer," "server," "processor," and "memory" all refer to electronic or other technical devices. These terms do not include individuals or groups. For the purposes of this specification, the terms "display" or "exhibit" refer to displaying on an electronic device. As used in this specification, the terms "computer-readable medium," "computer-readable media," and "machine-readable medium" are entirely limited to tangible physical objects that store information in a computer-readable form. These terms do not include any wireless signals, wired download signals, and any other transient or temporary signals.
[0138] Although the present invention has been described with reference to many specific details, those of ordinary skill in the art will recognize that the present invention may be implemented in other specific forms without departing from the spirit of the present invention. For example, several of the above embodiments deploy gateways in public cloud data centers. However, in other embodiments, the gateway is deployed in a third party's private cloud data center (e.g., a data center used by a third party to deploy cloud gateways for different entities in order to deploy virtual networks for these entities). Therefore, those of ordinary skill in the art will understand that the present invention is not limited by the foregoing illustrative details, but is defined by the appended claims.
Claims
1. For a software defined wide area network (SD-WAN) established by a set of multiple edge nodes and one or more cloud gateways, a method for controlling packet flow through the SD-WAN using deep packet inspection (DPI) operations, the method comprising: At the server, collecting parameters from a first set of two or more edge nodes and a set of gateways processing a plurality of packet flows, at least one edge node performing a DPI operation on the set of packet flows; correlating the parameters collected from the first set of two or more edge nodes and the set of gateways to identify a correlated set of parameters, the correlated set of parameters related to processing of packet flows processed by the first set of two or more edge nodes and the set of gateways; identifying, from the associated set of parameters, a subset of parameters associated with a plurality of packet flows associated with a specific application identifier specified by the DPI operation; analyzing the identified subset of parameters to identify at least one specific packet flow having an undesirable path through the SD-WAN; as well as Distributing the adjusted forwarding record to a second set of one or more edge nodes to modify a path used by the second set of edge nodes for the identified specific packet flow associated with the specific application identifier and the undesired path through the SD-WAN.
2. The method of claim 1, wherein at least a subset of the DPI operations are performed at a physical location remote from a physical location of a first set of operations of the two or more edge nodes. The method of claim 1 , wherein the collected parameters include operational metrics. The method of claim 3 , wherein at least a set of operational metrics is collected at a first set of the two or more edge nodes. The method of claim 3 , wherein at least a set of operational metrics is collected by at least one gateway.
6. The method of claim 1, wherein analyzing the identified subset of parameters comprises determining a duration for packets in each of a plurality of packet flows to travel from a source edge node of the packet flow through the SD-WAN to a destination edge node of the packet flow.
7. The method of claim 1, wherein analyzing the identified subset of parameters comprises determining a duration for packets in each of a plurality of packet flows to pass through a gateway to reach a destination edge node for the packet flow.
8. The method according to claim 1, wherein The first node initially receives the first forwarding record and then receives the second forwarding record; Based on the first forwarding record, the first node forwards the first packet flow through a first gateway before receiving the second forwarding record; as well as Based on the second forwarding record, the first node stops forwarding the first packet flow through the first gateway, and instead forwards the first packet flow through a second gateway.
9. The method according to claim 1, wherein The first node initially receives the first forwarding record and then receives the second forwarding record; Based on the first forwarding record, the first node forwards a first packet flow associated with a first DPI parameter to a first destination of the first packet flow using a first gateway; as well as Based on the second forwarding record, the first node forwards a second packet flow associated with the first DPI parameter to a second destination of the second packet flow using a second gateway.
10. The method of claim 9, wherein the first and second destinations are the same destination. The method of claim 1 , wherein the first set of edge nodes is a subset of the second set of edge nodes.
12. The method of claim 11, wherein the second set of edge nodes includes at least one edge node that is not in the first set.
13. The method of claim 1, wherein the collected parameters include statistical information collected by the first set of the two or more edge nodes.
14. The method according to claim 1, further comprising: For each of a set of packet flows through the SD-WAN, packets collected by source and destination edge nodes and parameters associated with the collected packets are received.
15. The method of claim 14, wherein the collected packets include packets associated with the specific application identifier specified by the DPI operation performed on the collected packets.
16. The method of claim 15, wherein the DPI operations are performed at both the source and destination edge nodes.
17. The method of claim 15, wherein the DPI operation is performed at the source edge node and an application identifier is provided by the source edge node to the destination edge node.
18. The method of claim 15, wherein the number of packets collected for a corresponding flow relates to a number of packets required for a DPI operation on the corresponding flow.
19. A machine-readable medium storing a program, which, when executed by at least one processing unit, implements the method according to any one of claims 1 to 18.
20. An electronic device comprising: a collection of processing units; and A machine-readable medium storing a program which, when executed by at least one of the processing units, implements the method according to any one of claims 1-18.
21. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 18.
Citation Information
Patent Citations
Optimized control system for aggregation of multiple broadband connections over radio interfaces
CN104769895A
Inline service switch
CN107005584A
SD-WAN system, method for using SD-WAN system, and related device
CN108713309A
Determining routing decisions in a software-defined wide area network
EP3509256A1
Data routing across multiple autonomous network systems
US20160261493A1
