Context-aware route computation and selection

Abstract

To provide apparatuses and methods for selecting a transmission path for network traffic.SOLUTION: In a mesh network including a path management hub and proxy server nodes that form a mesh network, the hub receives link performance metrics from each of the nodes, determines optimal paths between first and second nodes for each of a plurality of metric types or a combination of metric types, maps a service identifier to the determined optimal paths for a context representing link performance metric types, and transmits a next hop for each of the optimal paths along with the mapping to the first node. The first node receives the service identifier mapped to the next hops and traffic for the first connection. In response to determining that the received traffic is for the service, the first node selects a first next hop and forwards the traffic.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] FIELD OF THEINVENTION

[0001] Various embodiments described herein relate to route computation and selection for mesh networks. In particular, embodiments deliver application or data content over a mesh network of proxy servers by dynamically computing and selecting optimal routes based on traffic context. Summary of the Invention [Problem to be solved by the invention]

[0002] Quick Overview

[0002] There is a need for an apparatus and method for selecting a transmission path for network traffic, where such a system and method optionally complements or replaces conventional systems for selecting a transmission path. [Means for solving the problem]

[0003] According to some embodiments, a method includes receiving, by a route management hub, link performance metrics from each of a plurality of proxy server nodes forming a mesh network. Each of the plurality of nodes forms part of the mesh network via a link to each of two or more other nodes in the plurality of nodes. The link performance metrics represent a plurality of performance metric types. The method also includes, in response to receiving the link performance metrics, evaluating, by the route management hub, a plurality of potential routes between each first node and a second node in the plurality of nodes using the received link performance metrics to determine an optimal route for each of the plurality of metric types and / or combinations of metric types. The method also includes determining, by the route management hub, a context representing one or more link performance metric types that are preferred for the service. The method also includes mapping, by the route management hub, a service identifier for the service to the plurality of determined optimal routes between the first node and the second node using the one or more link performance metric types of the determined context. The method also includes transmitting, from the path management hub to the first node, a respective next hop for each of the plurality of mapped optimal routes and a mapping between each of the respective next hops and a service identifier. The method also includes receiving, by the first node, from the path management hub, the service identifier mapped to the plurality of next hops for the plurality of mapped optimal routes. The method also includes receiving, by the first node, traffic for a first connection with the service. The method also includes selecting, by the first node, a first next hop mapped to the service identifier of the plurality of next hops for the plurality of mapped optimal routes in response to determining that the received traffic is for the service. The method also includes forwarding, by the first node, the received traffic to the selected first next hop.

[0004]

[0004] In some embodiments, the method also includes tracking, by the first node, link performance metrics for traffic of the service through each of the multiple next hops. The method also includes determining, by the first node, using the link performance metrics tracked by the first node, that a second next hop mapped to a service identifier of the multiple next hops received from the path management hub is better than the first next hop in at least one of the tracked performance metrics. The method also includes receiving, by the first node, traffic of a second connection with the service. The method also includes, in response to determining, by the first node, that the second next hop is better than the first next hop, forwarding the received traffic of the second connection with the service to the second next hop.

[0005]

[0005] In some embodiments, the method also includes sending by the first node to a route management hub an indication that the first node should use the second next hop to forward traffic of the service and the tracked link performance metrics.

[0006]

[0006] In some embodiments, the method also includes tracking, by the first node, link performance metrics for traffic of the service through each of a plurality of next hops, where the link performance metrics tracked by the first node include, for each connection, a number of bytes directed to the service over a period of time and a number of bytes received from the service over a period of time. The method also includes determining, by the first node, using the link performance metrics tracked by the first node, that a ratio of the number of bytes directed to the service to the number of bytes received from the service exceeds a first threshold. The method also includes receiving, by the first node, traffic of a second connection with the service. The method also includes, in response to determining, by the first node that the ratio exceeds the first threshold, forwarding the received traffic of the second connection with the service to the second next hop.

[0007]

[0007] In some embodiments, the method also includes tracking, by the first node, link performance metrics for traffic of the service through each of a plurality of next hops, where the link performance metrics tracked by the first node include, for each connection, a number of bytes directed to the service over a period of time and a number of bytes received from the service over a period of time. The method also includes determining, by the first node, whether a ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies a first sorting criterion using the link performance metrics tracked by the first node. The method also includes receiving, by the first node, traffic of a second connection with the service. The method also includes forwarding, by the first node, the received traffic of the second connection with the service to the second next hop in response to determining that the ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies the first sorting criterion.

[0008] In some embodiments, the method also includes tracking, by the first node, link performance metrics for traffic of a first connection for the service via the first next hop. The method also includes determining, by the first node, using the link performance metrics tracked by the first node that at least one of the link performance metrics for the first connection exceeds a second threshold. The method also includes receiving, by the first node, traffic of a second connection with the service. The method also includes forwarding, in response to the first node determining that the at least one link performance metric for the first connection exceeds the second threshold, the received traffic of the second connection with the service to the second next hop.

[0009]

[0009] In some embodiments, the method also includes tracking, by the first node, link performance metrics for traffic of a first connection for the service via the first next hop. The method also includes determining, by the first node, using the link performance metrics tracked by the first node that at least one of the link performance metrics for the first connection meets a second sorting criterion. The method also includes receiving, by the first node, traffic of a second connection with the service. The method also includes forwarding, in response to the first node determining that the at least one link performance metric for the first connection meets the second sorting criterion, the received traffic of the second connection with the service to the second next hop.

[0010]

[0010] In some embodiments, the link performance metrics include, for each connection, a number of bytes directed to each service over a period of time and a number of bytes received from each service over a period of time. The method also includes determining, by the path management hub, whether a ratio of the number of bytes directed to the service to the number of bytes received from the service exceeds a third threshold using the link performance metrics. The method also includes selecting, by the path management hub, a context representing one or more link performance metric types in response to determining that the ratio of the number of bytes directed to the service to the number of bytes received from the service exceeds the third threshold.

[0011]

[0011] In some embodiments, the link performance metrics include, for each connection, a number of bytes directed to each service over a period of time and a number of bytes received from each service over a period of time. The method also includes determining, by the path management hub, whether a ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies a context adjustment criterion using the link performance metrics. The method also includes selecting, by the path management hub, a context representing one or more link performance metric types in response to determining that the ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies the context adjustment criterion.

[0012]

[0012] In some embodiments, a service identifier mapped to a next hop for multiple optimal routes sent from a route management hub to a first node includes a mapping to a context representing one or more link performance metric types.

[0013] In some embodiments, the multiple link performance metric types include one or more of: bandwidth, requests per second, packet loss, jitter, and latency.

[0014] In some embodiments, determining a context indicative of one or more link performance metric types prioritized for a service includes receiving, by a route management hub, a selection of the context.

[0015]

[0015] According to some embodiments, a non-transitory computer-readable medium stores instructions that, when executed by a processing device, cause the processing device to perform a method. The method includes receiving, by a path management hub, link performance metrics from each of a plurality of proxy server nodes forming a mesh network. Each of the plurality of nodes forms part of the mesh network via links to each of two or more other nodes in the plurality of nodes. The link performance metrics represent a plurality of performance metric types. The method also includes, in response to receiving the link performance metrics, evaluating, by the path management hub, a plurality of potential paths between each first node and a second node in the plurality of nodes using the received link performance metrics to determine an optimal path for each of the plurality of metric types and / or combinations of metric types. The method also includes determining, by the path management hub, a context representing one or more link performance metric types that are preferred for the service. The method also includes mapping, by the path management hub, a service identifier for the service to the plurality of determined optimal paths between the first node and the second node using the one or more link performance metric types of the determined context. The method also includes transmitting, from the route management hub to the first node, a respective next hop for each of the plurality of mapped optimal routes and a mapping between each of the respective next hops and a service identifier. The method also includes receiving, by the first node, the service identifier mapped to the plurality of next hops for the plurality of mapped optimal routes from the route management hub. The method also includes receiving, by the first node, traffic for a first connection with the service. The method also includes selecting, by the first node, a first next hop mapped to the service identifier of the plurality of next hops for the plurality of mapped optimal routes in response to determining that the received traffic is for the service.The method also includes forwarding, by the first node, the received traffic to the selected first next hop.

[0016] In some embodiments, the instructions cause the processing unit to track, by the first node, link performance metrics for traffic of the service through each of the multiple next hops. The instructions also cause the processing unit to determine, by the first node, using the link performance metrics tracked by the first node, that a second next hop mapped to a service identifier of the multiple next hops received from the path management hub is better than the first next hop in at least one of the tracked performance metrics. The instructions also cause the processing unit to receive, by the first node, traffic of a second connection with the service. The instructions also cause the processing unit to forward, by the first node, the received traffic of the second connection with the service to the second next hop in response to determining that the second next hop is better than the first next hop.

[0017]

[0017] In some embodiments, the instructions also cause the processing device to send, by the first node to the path management hub, an instruction for the first node to use the second next hop for forwarding traffic of the service and the tracked link performance metrics.

[0018]

[0018] In some embodiments, the instructions also cause the processing unit to track, by the first node, link performance metrics for traffic of the service through each of the multiple next hops. The link performance metrics tracked by the first node include, for each connection, a number of bytes directed to the service over a period of time and a number of bytes received from the service over a period of time. The instructions also cause the processing unit to determine, by the first node, using the link performance metrics tracked by the first node, that a ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies a first sorting criterion. The instructions also cause the processing unit to receive, by the first node, traffic of a second connection with the service. The instructions also cause the processing unit to forward the received traffic of the second connection with the service to the second next hop in response to determining, by the first node, that a ratio of the number of bytes directed to the service to the number of bytes received from the service satisfies the sorting criterion.

[0019] In some embodiments, the instructions also cause the processing unit to track, by the first node, link performance metrics for traffic of a first connection for the service via the first hop. The instructions also cause the processing unit to determine, by the first node, using the link performance metrics tracked by the first node, that at least one of the link performance metrics for the first connection meets a second sorting criterion. The instructions also cause the processing unit to receive, by the first node, traffic of a second connection with the service. The instructions also cause the processing unit to forward, in response to the first node determining that at least one link performance metric for the first connection meets the second sorting criterion, the received traffic of the second connection with the service to a second next hop.

[0020]

[0020] In some embodiments, the link performance metrics include, for each connection, a number of bytes directed to each service over a period of time and a number of bytes received from each service over a period of time. The instructions also cause the processing unit to determine, by the path management hub, whether a ratio of the number of bytes directed to a service to the number of bytes received from the service satisfies a context adjustment criterion using the link performance metrics. The instructions also cause the processing unit to select a context representing one or more link performance metric types in response to determining, by the path management hub, that a ratio of the number of bytes directed to a service to the number of bytes received from the service satisfies a context adjustment criterion.

[0021]

[0021] In some embodiments, the service identifier mapped to the next hop for multiple optimal routes sent from the route management hub to the first node includes a mapping to a context representing one or more link performance metric types.

[0022] In some embodiments, the multiple link performance metric types include one or more of: bandwidth, requests per second, packet loss, jitter, and latency.

[0023] In some embodiments, determining a context indicative of one or more link performance metric types that are prioritized for a service includes receiving, by a route management hub, a selection of the context.

[0024]

[0024] According to some embodiments, a system includes a path management hub including a first processing device and a first memory coupled to the first processing device. The first memory stores instructions that, when executed by the first processing device, cause the path management hub to receive link performance metrics from each of a plurality of proxy server nodes forming a mesh network. Each of the plurality of nodes forms part of the mesh network via links to each of two or more other nodes in the plurality of nodes. The link performance metrics represent a plurality of performance metric types. The instructions in the first memory also cause the path management hub, in response to receiving the link performance metrics, to evaluate a plurality of potential paths between each first node and a second node in the plurality of nodes using the received link performance metrics to determine an optimal path for each of the plurality of metric types and / or combinations of metric types. The instructions in the first memory also cause the path management hub to determine a context representing one or more link performance metric types to be prioritized for service. The instructions in the first memory also cause the path management hub to map a service identifier for the service to a plurality of determined optimal paths between the first node and the second node using one or more link performance metric types of the determined context. The instructions in the first memory also cause the path management hub to transmit to the first node a respective next hop for each of the plurality of mapped optimal paths and a mapping between each of the respective next hops and the service identifier. The first node comprises a second processing unit and a second memory coupled to the second processing unit. The second memory stores instructions that, when executed by the second processing unit, cause the first node to receive from the path management hub a service identifier mapped to a plurality of next hops for the plurality of mapped optimal paths. The instructions in the second memory also cause the first node to receive traffic for a first connection with the service.The instructions in the second memory also cause the first node to select a first next hop mapped to the service identifier of the multiple next hops for the multiple mapped optimal paths in response to determining that the received traffic is for the service. The instructions in the second memory also cause the first node to forward the received traffic to the selected first next hop.

[0025]

[0025] In some embodiments, execution of the instructions stored in the second memory further causes the first node to track link performance metrics for traffic of the service through each of the multiple next hops. The instructions stored in the second memory also cause the first node to determine, using the link performance metrics tracked by the first node, that a second next hop mapped to a service identifier of the multiple next hops received from the path management hub is better than the first next hop in at least one of the tracked performance metrics. The instructions stored in the second memory also cause the first node to receive traffic of a second connection with the service. The instructions stored in the second memory also cause the first node to forward the received traffic of the second connection with the service to the second next hop in response to determining that the second next hop is better than the first next hop.

[0026]

[0026] According to some embodiments, a computer-implemented method includes receiving, by a route management hub, link performance metrics from each of a plurality of proxy server nodes forming a mesh network. Each of the plurality of nodes forms part of the mesh network via a link to each of two or more other nodes in the plurality of nodes. The link performance metrics represent a plurality of performance metric types. The method also includes, in response to receiving the link performance metrics, evaluating, by the route management hub, a plurality of potential routes between each first node and a second node in the plurality of nodes using the received link performance metrics to determine an optimal route for each of the plurality of metric types and / or combinations of metric types. The method also includes determining, by the route management hub, a context representing one or more link performance metric types that are preferred for the service. The method also includes mapping, by the route management hub, a service identifier for the service to the plurality of determined optimal routes between the first node and the second node using the one or more link performance metric types of the determined context. The method also includes transmitting, from the route management hub to the first node, respective next hops for each of the plurality of mapped optimal routes and a mapping between each of the respective next hops and a service identifier. [Background technology]

[0027] 2. Background of the Invention

[0027] Content delivery networks (CDNs) aim to provide content to end users with high availability and high performance. In particular, CDNs deploy proxy servers in multiple geographically diverse data centers to cache and serve content. When content is requested, the request is directed to, for example, the node that requires the fewest number of hops, the fewest network seconds from the client requesting the content, etc. Typically, this results in the selection of the edge node closest to the client. For example, proxy servers in a telecommunications service provider (TSP) network cache content and serve it to the TSP's customers, reducing demand on the network backbone and delivering content quickly and reliably. However, caching content in many places to maximize the chances of having a proxy server close to any client can incur significant costs.

[0028] BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The present embodiments are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numbers indicate similar elements and in which: [Brief description of the drawings]

[0029] [Figure 1]

[0029] FIG. 1 is a block diagram illustrating an example mesh network of proxy servers that perform context-aware path computation. [Diagram 2]

[0030] 1 is a flowchart illustrating an exemplary method for context-aware path computation and selection. [Diagram 3]

[0031] FIG. 1 illustrates an example traffic flow used to determine a context indicative of one or more link performance metric types prioritized for a service. [Figure 4]FIG. 1 illustrates an example traffic flow used to determine a context that represents one or more link performance metric types that are prioritized for a service. [Diagram 5] FIG. 1 illustrates an example traffic flow used to determine a context that represents one or more link performance metric types that are prioritized for a service. [Figure 6] FIG. 1 illustrates an example traffic flow used to determine a context that represents one or more link performance metric types that are prioritized for a service. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] Detailed Description

[0032] This specification describes an embodiment for delivering application or data content over a mesh network of proxy servers by dynamically computing and selecting optimal paths based on traffic context. The optimal path is selected by a path management hub using link performance metrics received from proxy server nodes that make up the mesh base and based on service-based context. The path management hub may be implemented as a single server or a distributed system at one location or across multiple locations. Each service-based context represents one or more link performance metric types that are preferred for that service. In some embodiments, an administrator of a service provides the selection of a particular context. In some embodiments, the hub sets the context to a default setting or sets the context based on an evaluation of the traffic flow patterns of the service. As an example of selecting an optimal path, the path management hub may compute a directed graph using various performance metrics of links along potential paths through the mesh network and select multiple paths as optimal paths based on each service-based context. The path management hub maps the context-based optimal paths to corresponding services.

[0031]

[0033] Once an optimal route is selected and mapped, the route management hub distributes the next hop information to the nodes. In some embodiments, a node receiving traffic for a given service uses the mapping to select a next hop provided by the route management hub and forwards the traffic to that next hop. In some embodiments, one or more nodes track link performance metrics and report back to the route management hub. In some embodiments, a node uses the tracked link performance metrics to select from multiple next hops mapped to a single service.

[0032]

[0034] As a result, embodiments reduce the number of locations required to cache content or host applications to provide high performance. Instead, embodiments achieve high performance in delivering application or data content using intelligent selection of paths through the mesh of proxy servers. Additionally, embodiments use out-of-band path selection based on a global view of the hub's mesh network, and in-band selection based on evaluation of individual node performance metrics.

[0033]

[0035] 1 illustrates in block diagram form an exemplary mesh network 100 of proxy server nodes 105 (e.g., nodes 105a, 105b, 105c, 105d, 105e, 105f, 105g, 105h, 105i, 105j, 105k, and 105l) that perform context-aware path computation. In some embodiments, the mesh network 100 is a content delivery network (CDN) and / or an application delivery network (ADN). The mesh network 100 includes a set of geographically distributed proxy server nodes 105 deployed in multiple data centers. The mesh network 100 may include more or fewer nodes 105 than are illustrated.

[0034]

[0036] In some embodiments, each node 105 (e.g., as shown by node 105a) comprises hardware 110. Hardware 110 comprises one or more processors ("CPUs"), data storage and memory (e.g., "RAM"), and a network interface controller ("NIC"). The data storage and memory may be used to store data, metadata, and programs for execution on the processors (e.g., link assessment module 115, DNS module 120, forwarding module 125, and / or path computation module 130). In some embodiments, the data storage and memory may include one or more of volatile and non-volatile memory, such as random access memory ("RAM"), read only memory ("ROM"), solid state disk ("SSD"), flash, phase change memory ("PCM"), and / or other types of data storage, such as magnetic disk drives, optical disk drives, etc. In some embodiments, the memory comprises internal memory and / or distributed memory. In some embodiments, one or more buses (not shown) interconnect various components of hardware 110. In some embodiments, the NIC connects one of the nodes 105 (e.g., nodes 105a-105l) to other nodes 105, client devices outside the mesh network 100, and / or server devices outside the mesh network 100 via wired and / or wireless networks.

[0035]

[0037] In some embodiments, each node (e.g., as shown by node 105a) comprises multiple modules. Each of these modules may be implemented in hardware, software, or a combination of hardware and software. The link assessment module 115 tracks metrics between the nodes 105 (e.g., nodes 105a-105l). For example, the link assessment module 115 of a first node may track the number of bytes per second per connection sent to a service through the first node and the number of bytes per second per connection received from the service by the first node. In some embodiments, these metrics provide a representation of data flow characteristics over a connection between a client and a service. Exemplary data flow characteristics are described in further detail with reference to FIGS. 3-6. In some embodiments, the link assessment module 115 tracks other metric types including one or more of requests per second per connection, receive packet loss per network address, transmit packet loss per network address, latency per network address, bandwidth per network address, jitter per network address, network utilization, traffic content (e.g., by inspection of headers, packets, or other data / metadata in transit), various request and / or response latencies (e.g., requests and / or responses for HTML pages, JavaScript, CSS, images, APIs, and / or media), and the like.

[0036]

[0038] The link assessment module 115 stores the tracked metrics in a data storage or memory in the hardware 110. Additionally, the link assessment module 115 transmits the tracked metrics to a path management hub using a NIC in the hardware 110. In some embodiments, the link assessment module 115 maps one or more metrics to a service of the corresponding traffic. For example, the link assessment module 115 may maintain a data structure in the data storage or memory that maps an identifier for a service to a metric for traffic sent to and received from the service. Exemplary services include providing content for a website, streaming video, providing data for an application, etc.

[0037]

[0039] In some embodiments, the link assessment module 115 injects test traffic to track metrics between a first respective node 105 (e.g., node 105a) and each of a plurality of second nodes 105 (e.g., nodes 105b-105l). For example, each respective node 105 may store a list of network addresses of the other nodes 105 and data for test transmissions. By sending test traffic to the other nodes 105 and receiving responses from the other nodes 105, the link assessment module 115 can track metrics for links to the other nodes 105 in addition to, or instead of, tracking actual traffic directed to and received from one or more services.

[0038]

[0040] In some embodiments, the link assessment module 115 uses the tracked link performance metrics to select a next hop for the traffic, eg, as described in more detail with reference to FIGS.

[0039]

[0041] In some embodiments, each node 105 includes a Domain Name System (DNS) module 120 (e.g., as shown by node 105a). The DNS module 120 stores next hop information. For example, each node 105 may determine its neighboring nodes 105 and track next hops for traffic using an address discovery protocol. In some embodiments, the next hop information is provided by a route management hub. For example, as described in more detail herein, the route management hub may determine optimal routes for each service and distribute next hop information to nodes 105 for those routes. The next hop information for a service may include a list of optimal next hops mapped to the service in a ranked order. In some embodiments, the link assessment module 115 adjusts the ranked order of the next hops based on tracked link performance metrics, for example, as described in more detail with reference to FIGS. 2-6.

[0040]

[0042] In some embodiments, each node 105 (e.g., as shown by node 105a) comprises a forwarding module 125. The forwarding module 125 receives traffic from a client, service, or another node 105, determines a service and / or destination from the received traffic, and / or selects a next hop using a mapping stored by the DNS module 120.

[0041]

[0043] In some embodiments, a node 105 (e.g., node 105a) comprises a path computation module 130. In some embodiments, a node comprising the path computation module 130 serves as a path management hub for the mesh network 100. For example, the mesh network 100 may have one active path management hub that computes and distributes optimal paths to the nodes 105.

[0042]

[0044] In some embodiments, the path computation module 130 maintains a registry of nodes 105 in the mesh network 100. In some embodiments, the path computation module 130 receives link performance metrics from each of the nodes 105 and uses the received link performance metrics to determine an optimal path between the nodes 105 for each of a plurality of metric types or combinations of metric types. In some embodiments, the path computation module 130 maps a service identifier to a plurality of determined optimal paths for a context representing one or more link performance metric types and transmits a next hop for each of the plurality of mapped optimal paths along with a mapping of the service to the next hop to the one or more nodes. In some embodiments, the path computation module 130 maps the context or the link performance metric type to one or more of the next hops and transmits the mapping to the one or more nodes. The path computation module 130 implemented in a path management hub is described in more detail with reference to Figures 2-6.

[0043]

[0045] FIG. 2 is a flow chart illustrating an example method 200 of context-aware path computation and selection. At block 205, a path management hub receives link performance metric data from nodes 105 in a mesh network 100. For example, the mesh network 100 is shown as including twelve nodes 105a-105l, one of which (e.g., node 105a) functions as a path management hub by actively using a path computation module 130. Each link assessment module 115 of each of the respective nodes 105 tracks and transmits link performance metrics for actual and / or test traffic transmitted and / or received by that respective node 105 and transmits the link performance metrics to the path management hub. In some embodiments, the path management hub receives link performance metrics at periodic intervals (e.g., defined by a time period, number of connections evaluated, etc.) for each node 105. The path management hub receives the link performance metrics, for example, via client / server communication with each of the respective nodes 105 and / or via a message queue. The received link performance metrics may include one or more of: bytes per second per connection sent to the service through the node, bytes per second per connection received from the service by the node, an indication of the type of data flow during the connection between the client and the service expressed in bytes sent / received, requests per second per connection, receive packet loss per network address, transmit packet loss per network address, latency per network address, bandwidth per network address, jitter per network address, network utilization, traffic content type (e.g., by inspection of headers, packets, or other data / metadata in transit), context / link performance metric type for the service as determined by the respective node 105, and the like.In some embodiments, the link performance metrics are mapped to one or more of a service, a duration, a source address or other identifier, and a destination address or other identifier.

[0044]

[0046] At block 210, the path management hub determines a context representing one or more link performance metric types for each service using the mesh network 100. In some embodiments, the context indicates that traffic to and / or from the service is bandwidth sensitive. In such embodiments, the path management hub prioritizes paths through the mesh network 100 that provide the best performance in terms of bandwidth. In some embodiments, the context may indicate that traffic to and / or from the service is latency sensitive. In such embodiments, the path management hub prioritizes paths through the mesh network 100 that provide the best performance in terms of latency. The context may represent multiple link performance metric types. For example, the context may indicate that traffic to and / or from the service is latency sensitive and cost sensitive. In such an example, the path management hub prioritizes paths through the mesh network 100 that provide the best performance in terms of latency while still being cost-effective and / or not exceeding a cost threshold. The context is not limited to these examples and may represent other link performance metric types or combinations of link performance metric types.

[0045]

[0047] The path management hub determines the context for the service by looking up a stored mapping between the service and the context or by making a determination based on data received from the node 105. For example, the path management hub may receive and store a preferred context from an administrator or other representative of the service attempting to deliver the content or application over the mesh network 100. Alternatively, the path management hub may receive and store a preferred context determined by the node 105. As another alternative, the path management hub's determination includes determining that there is no pre-configured context and setting the context to a default setting.

[0046]

[0048] In some embodiments, the path management hub determines the context based on the node 105, identifies the traffic type by analyzing the traffic header / metadata and / or traffic payload data, and sends the traffic type to the path management hub. For example, the path management hub may receive from the node 105 a mapping between an identifier for a service and a traffic type for that service determined by the node 105. The traffic type may classify the content, for example, as images, audio, video, application programming interface (API) requests and / or responses, Voice over Internet Protocol (VoIP) calls, etc. Alternatively, the traffic type received from the node 105 may classify the traffic. For example, the node 105 may classify images and API requests and / or responses as latency sensitive traffic, streaming video as bandwidth sensitive traffic, VoIP calls as jitter sensitive traffic, etc.

[0047]

[0049] In some embodiments, the path management hub determines the context based on data flow patterns detected by the node 105. For example, the node 105 may track data flows and send a mapping between identifiers and traffic types for the service to the path management hub based on the tracked data flows, e.g., as described with reference to Figures 3-6. For example, the node 105 may classify traffic flows as latency sensitive traffic, bandwidth sensitive traffic, jitter sensitive traffic, etc., based on the number of bytes per second per connection sent to the service through the node, the number of bytes per second per connection received from the service by the node, the amount of received packets, the amount of sent packets, the symmetry between received and sent traffic, etc.

[0048]

[0050] At block 215, the path management hub calculates or otherwise determines one or more optimal paths for each link performance metric type. For example, in response to receiving the link performance metrics, the path management hub uses the received link performance metrics to assign a value to each link between two nodes 105 in the mesh network 100. For each link performance metric type, the path management hub calculates a directed graph between the nodes 105 using the values ​​of that performance metric type as input to Dijkstra's algorithm or another algorithm that determines the shortest paths between the nodes 105.

[0049]

[0051] In some embodiments, the path management hub determines one or more optimal paths for combinations of link performance metric types. For example, the path management hub may use multiple link performance metric types to assign a value to each link between two nodes 105. In some embodiments, using multiple performance metric types includes creating a multi-dimensional value where each dimension represents a performance metric type. In some embodiments, using multiple performance metric types includes calculating a sum, product, or other combination of performance metrics of each type for each link. In some embodiments, using multiple performance metric types includes calculating a shortest path of a first performance metric type that does not exceed or fall below a threshold of a second performance metric type.

[0050]

[0052] In some embodiments, assigning a value to each link includes normalizing the received link performance metrics. Normalizing the received link performance metrics may include adjusting the values ​​based on whether high or low values ​​are optimal for a given link performance metric type. For example, a shortest path in terms of latency is based on low latency links, and a shortest path in terms of bandwidth is based on high bandwidth links. Additionally, normalizing the received link performance metrics may include weighting the link performance metrics based on the link performance metric type.

[0051]

[0053] In some embodiments, in determining an optimal path, the path management hub ranks multiple paths between nodes 105 for each link performance metric type. For example, the path management hub may determine the top two or three optimal paths between two nodes 105. If a service may be provided by multiple nodes 105 (e.g., may be provided using cached content at each of the multiple nodes 105, or each of the multiple nodes 105 may serve as an origin entry / exit point for a given service), the path management hub may determine the top two or three optimal paths between a first node 105 and one of the multiple other nodes. Alternatively, the path management hub determines a single optimal path between each set of two nodes 105.

[0052]

[0054] At block 220, the route management hub maps a service identifier for each service to one or more of the optimal routes based on the determined context for that service. For example, if the determined context for a service indicates that high bandwidth routes are preferred, the route management hub maps the identifier for that service to the route determined to be optimal in terms of bandwidth. In embodiments where multiple optimal routes are determined for a context, the route management hub maps the identifier to a ranked list of optimal routes. The service identifier is a port number, a network address, a combination of a port number and a network address, or another unique identifier for the service.

[0053]

[0055] In some embodiments, the route management hub uses the context to map the service identifier to multiple optimal routes, with each route selected as optimal for a different link performance metric type. For example, a default context may be used to map the service identifier to a list that includes the best route for bandwidth, the best route for latency, and the best route for jitter. Using the determined traffic type (e.g., based on traffic flow or inspection as described herein), the node 105 may select the optimal route corresponding to the traffic type. For example, for a traffic type that is bandwidth sensitive, the node 105 selects the optimal route for bandwidth.

[0054]

[0056] In addition, the path management hub distributes next hop information of the mapped optimal path to the nodes 105. For example, the optimal path may include an intermediate node 105 between a first node 105 forwarding traffic and a second node 105 receiving traffic. The path management hub determines the next hop along each optimal path for each node 105 for each service and transmits a mapping of service identifiers to one or more next hops for the context determined to apply to that service. In some embodiments, the path management hub includes a mapping of context or link performance metric type to next hop when distributed to the nodes 105.

[0055]

[0057] At block 225, each node 105 receives next hop information from the route management hub. For example, for each service, a first node 105 receives an identifier for the service that is mapped to a ranked list of next hops that represent other nodes 105 to which the first node will forward traffic for the service. In some embodiments, the ranked list is mapped to both services and contexts. In some embodiments, the ranked list is mapped to services, and each next hop in the ranked list is mapped to a context. The DNS module 120 stores the mapped ranked list.

[0056]

[0058] At block 230, node 105 receives the network traffic and forwards the traffic using the received next hop information. For example, forwarding module 125 receives network traffic that includes a domain, a source address, a destination address, a port number, a combination of an address and a port number, and / or another unique identifier associated with a service. Using the service identifier, DNS module 120 selects a next hop. In an embodiment in which node 105 receives a ranked list of next hops mapped to a service identifier, DNS module 120 selects the next hop that is ranked highest in the list, and forwarding module 125 forwards the traffic to node 105 at that next hop.

[0057]

[0059] At block 235, the node 105 tracks link performance metrics. In some embodiments, the link assessment module 115 injects test traffic to track metrics between the first node 105 and each of the plurality of second nodes 105. For example, each node 105 may store a list of network addresses of the other nodes 105 and data for test transmission. In some embodiments, the stored list of network addresses is a collection of next hops received from a path management hub. By sending test traffic to the other nodes 105 and receiving responses from the other nodes 105, the link assessment module 115 can track metrics of links to the other nodes 105 (e.g., one or more of the link performance metric types described herein) in addition to or instead of tracking actual traffic directed to and received from one or more services. In some embodiments, the link assessment module 115 sends and receives test traffic via the forwarding module 125 in response to expiration of a periodic time interval, completion of a threshold number of connections, or another interval / threshold.

[0058]

[0060] The node 105 transmits the tracked link performance metrics to the route management hub. In some embodiments, the link assessment module 115 transmits the tracked link performance metrics to the route management hub once the assessment of a batch of links is complete. In some embodiments, the link assessment module 115 transmits the tracked link performance metrics as determined.

[0059]

[0061] The link assessment module 115 optionally reorders the next hop information stored by the DNS module 120 based on the tracked link performance metrics. As a result of such reordering, the node 105 makes internal / in-band adjustments to the optimal path during external / out-of-band adjustments to the optimal path made by the path management hub. For example, the link assessment module 115 may use the tracked link performance metrics to determine that the top ranked next hop determined by the path management hub is no longer the optimal path for a given context. As a specific example, the next hop information received by the node 105 indicates that the service is configured to prioritize bandwidth and is mapped to three next hops based on the context. In response to the link assessment module 115 determining that the tracked link performance metrics indicate that the second next hop in the ranked list is better in bandwidth than the first next hop in the ranked list, the link assessment module reorders the ranked list to make the second next hop first.

[0060]

[0062] In some embodiments, the link assessment module 115 makes the decision to sort the ranked list of next hops for a service by determining a link performance metric to use to evaluate the ranked list of next hops for a given service. For example, a context identifier received from a route management hub along with a service and mapped to a next hop for that service may be resolved by the link assessment module 115 as representing one or more link performance metrics.

[0061]

[0063] In some embodiments, the link assessment module 115 makes the determination to reorder the ranked list of next hops for a service by examining the traffic for that service. For example, by examining headers, packets, or other data / metadata in transit, the link assessment module 115 determines a traffic type for the service. The traffic type may classify content, for example, image, audio, video, application programming interface (API) requests and / or responses, Voice over Internet Protocol (VoIP) calls, etc., and / or map the traffic type to a context / link performance metric type. The link assessment module 115 reorders the ranked list of next hops if a lower ranked next hop outperforms a higher ranked next hop in tracked link performance metrics for the determined link performance metric type for the traffic.

[0062]

[0064] In some embodiments, the ranked list may include next hops that prioritize different contexts or link performance metric types, in which case the link assessment module 115 reorders the ranked list of next hops such that lower ranked next hops are prioritized and higher ranked next hops are not prioritized for the link performance metric type determined for the traffic.

[0063]

[0065] In some embodiments, the link assessment module 115 makes a decision to reorder the ranked list of next hops for a service based on estimating the load of the link, node 105, or service. For example, a service may be provided by multiple origin servers connected to different nodes 105, or by different nodes 105 as proxies. The tracked link performance metrics may provide an indication of low-loaded paths, nodes 105, or origins, and / or paths, nodes 105, or origins that are above a load threshold. In response to a determination that there is a low-load option, or that the top ranked next hop is an option that is above a load threshold, the link assessment module 115 reorders the ranked list to provide load balancing, throttle traffic, etc.

[0064]

[0066] In some embodiments, the link assessment module 115 makes the decision to reorder the ranked list of next hops for a service by characterizing traffic flow data for the service. For example, the traffic flows may be characterized as corresponding to a context or one or more link performance metric types. As described above, the context or link performance metric types may be used to evaluate the link performance metrics and reorder the next hops.

[0065]

[0067] Method 200 continues in parallel at blocks 205 and 230. The route management hub receives updated link performance metrics from node 105 and updates optimal routes and corresponding next hop information as described in connection with blocks 205-220. Additionally, meanwhile, node 105 continues to receive and forward traffic, track link performance metrics, and optionally reorder next hops as described in connection with blocks 230 and 235.

[0066]

[0068] 3-6 illustrate example traffic flows that may be used to determine a context that represents one or more link performance metric types that are preferred for a service. For example, the link assessment module 115 may track, for each connection, the number of bytes sent per second through a node to a service and / or the number of bytes received per second from a service by a node as part of the link performance metrics. The link assessment module 115 may perform determinations as described with respect to FIGS. 3-6 to determine whether next hop reordering criteria and / or context adjustment criteria are met.

[0067]

[0069] FIG. 3 illustrates an exemplary traffic flow for file requests and responses. For example, a service may provide one or more Cascading Style Sheets (CSS), JavaScript, HyperText Markup Language (HTML) files, images, or other files. As illustrated, the traffic flow for such file requests and responses may have a relatively low number of bytes sent and received per second. In addition, the illustrated traffic flow for such files shows a relatively equal ratio of bytes received compared to bytes sent. The link assessment module 115 uses the amount and / or ratio of bytes received and bytes sent to determine that the next hop for the traffic should be latency prioritized. For example, the link assessment module 115 determines that the amount of bytes received and / or the amount of bytes sent (and / or the ratio of bytes received to bytes sent) meets a sorting criterion (e.g., the amount of bytes sent is below a threshold and / or the ratio of bytes received to bytes sent is within a predefined ratio value range), thereby mapping to a latency prioritized context.

[0068]

[0070] 4 illustrates an example traffic flow of a streaming service request and response. The illustrated traffic flow shows a relatively low number of received bytes per second and a high number of transmitted bytes per second. The link assessment module 115 uses the amount of transmitted bytes and / or the ratio of received bytes per second to transmitted bytes per second to determine that the next hop for the traffic should be bandwidth prioritized. For example, the link assessment module 115 determines that the sorting criteria is met, e.g., if the amount of transmitted bytes exceeds a threshold and / or the ratio of received bytes to transmitted bytes is less than a threshold ratio (e.g., over a predetermined period of time), thereby mapping to a bandwidth prioritized context.

[0069]

[0071] FIG. 5 illustrates an example traffic flow of a request and response for a streaming service that transmits data in chunks. Similar to FIG. 4, the illustrated traffic flow shows a relatively low number of received bytes per second and a high number of transmitted bytes per second. The link assessment module 115 uses the amount and / or ratio of received and transmitted bytes to determine that the next hop for the traffic should be bandwidth prioritized. For example, the link assessment module 115 determines that the reordering criteria is met, e.g., if the amount of transmitted bytes exceeds a threshold and / or the ratio of the amount of received bytes to transmitted bytes is less than a threshold ratio (e.g., over a predetermined period of time), thereby mapping to a bandwidth-prioritized context.

[0070]

[0072] Figure 6 shows an example traffic flow for requests and responses for a frame-based protocol. For example, in the case of the HTTP / 2 protocol, the server may start sending data as soon as the client connects and before the client has sent any request or other bytes.

[0071]

[0073] It is apparent from this specification that aspects of the present invention may be implemented at least partially in software. That is, the link assessment module 115, the DNS module 120, the forwarding module 125, the path computation module 130, and / or the computer-implemented method 200 may be implemented or performed in a computer system or other data processing system, such as one or more respective nodes 105, as a processor therein executes sequences of instructions contained in a memory or other non-transitory machine-readable storage medium. The software may also be transmitted or received over a network (not shown) via a network interface. In various embodiments, hardwired circuitry may be used in combination with software instructions to implement the present embodiments. Thus, the present technique is not limited to any particular combination of hardware circuitry and software, or to any particular source of instructions executed by the node 105. It should also be understood that additional components not shown may be part of the node 105, and that in certain embodiments, the node 105 may use fewer components than those shown in FIG. 1.

[0072]

[0074] An article of manufacture may be used to store program code that provides at least some of the functionality of the above embodiments. In addition, an article of manufacture may be used to store program code created using at least some of the functionality of the above embodiments. The article of manufacture that stores the program code may be embodied as, but is not limited to, one or more memories (e.g., one or more flash memories, random access memories - static, dynamic, or other), optical disks, CD-ROMs, DVD-ROMs, EPROMs, EEPROMs, magnetic or optical cards, solid-state drives (SSDs), or other types of non-transitory computer-readable media suitable for storing electronic instructions. In addition, embodiments of the present invention may be implemented in hardware or firmware using computer systems including, but not limited to, FPGAs, ASICs, processors, computers, or networks. Modules and components of hardware or software implementations may be divided or combined without significantly modifying the embodiments of the present invention.

[0073]

[0075] In the preceding portions of this specification, the invention has been described in connection with certain exemplary embodiments thereof. Various embodiments and aspects of the invention are described in connection with the details set forth herein, and the accompanying drawings illustrate various embodiments. The above description and drawings are intended to illustrate the invention and should not be construed as limiting the invention. References herein to "one embodiment," "one embodiment," "exemplary embodiment," and the like indicate that the described embodiment may include a particular feature, structure, or characteristic, but not necessarily all embodiments include that particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with one embodiment, such feature, structure, or characteristic may be implemented in connection with other embodiments, whether or not explicitly described. Additionally, as used herein, the term "exemplary" refers to an embodiment that serves merely as an example or illustration. Use of "exemplary" should not be construed as indicating a preferred example. Blocks with dashed borders (e.g., large dashes, small dashes, dot-dashes, dots) are used to indicate virtualized resources or optional operations in a flowchart that add further features to embodiments of the present invention. However, such notations should not be interpreted as implying that these are the only optional or optional operations and / or that blocks with solid borders in a particular embodiment of the present invention are not optional. Numerous specific details are described to provide a thorough understanding of various embodiments of the present invention. However, in certain instances, well-known or conventional details are not described in order to simplify the discussion of the embodiments of the present invention.

[0074]

[0076] It should be understood that various modifications may be made without departing from the broader spirit and scope of the present invention as set forth in the appended claims. For example, the methods described herein may be performed with fewer or more features / blocks or the features / blocks may be performed in a different order. In addition, the methods described herein may be repeated or performed in parallel with each other or with different instances of the same or similar methods.

Claims

1. Receiving, by a route management hub, link performance metrics from a plurality of proxy server nodes forming a mesh network, the link performance metrics representing a plurality of link performance metric types; evaluating, by the route management hub, a plurality of potential routes between a first proxy server node and a second proxy server node using the link performance metrics to determine a next hop for each of a plurality of optimal routes corresponding to each link performance metric type and / or combination of link performance metric types; determining, by the route management hub, a context representative of one or more link performance metric types for each of a plurality of services between the first proxy server node and the second proxy server node; mapping, by the route management hub, each service of the plurality of services to a corresponding optimal route of the plurality of optimal routes; sending, by the route management hub, to the first proxy server node, the mapping of the plurality of services to the plurality of optimal routes, thereby causing the first proxy server node to forward received traffic of a service to a first next hop; and the link performance metrics include, for each connection, a number of bytes directed to each service over a period of time and a number of bytes received from each service over a period of time, the method comprising: determining, by the first proxy server node, whether a ratio of bytes directed to the service to bytes received from the service using the link performance metric exceeds a third threshold; selecting, by the route management hub, a context representative of the one or more link performance metric types in response to determining that the ratio of bytes directed to the service to bytes received from the service exceeds the third threshold; Including, Computer-implemented method.

2. receiving, by a route management hub, link performance metrics from a plurality of proxy server nodes forming a mesh network, the link performance metrics representing a plurality of link performance metric types; evaluating, by the route management hub, a plurality of potential routes between a first proxy server node and a second proxy server node using the link performance metrics to determine a next hop for each of a plurality of optimal routes corresponding to each link performance metric type and / or combination of link performance metric types; determining, by the route management hub, a context representative of one or more link performance metric types for each of a plurality of services between the first proxy server node and the second proxy server node; mapping, by the route management hub, each service of the plurality of services to a corresponding optimal route of the plurality of optimal routes; sending, by the route management hub, to the first proxy server node, the mapping of the plurality of services to the plurality of optimal routes, thereby causing the first proxy server node to forward received traffic of a service to a first next hop; and the link performance metrics include, for each connection, a number of bytes directed to each service over a period of time and a number of bytes received from each service over a period of time, the method comprising: determining, by the first proxy server node, whether a ratio of bytes directed to the service to bytes received from the service using the link performance metric satisfies a context adjustment criterion; selecting, by the route management hub, a context representative of the one or more link performance metric types in response to determining that the ratio of bytes directed to the service to bytes received from the service satisfies the context adjustment criterion; , including, Computer-implemented method.

3. receiving, by the first proxy server node, an indication that the first proxy server node should use a second next hop for forwarding traffic of the service; updating, by the route management hub, a mapping of the plurality of services to the plurality of best routes based on the first proxy server node receiving the indication to use the second next hop; The computer-implemented method of claim 1 or 2, further comprising: tracking, by the first proxy server node, link performance metrics for traffic of the service through each of the plurality of next hops; in response to determining that a second next hop of the plurality of next hops is better than the first next hop in at least one of the link performance metrics for traffic of the service, (i) updating, by the route management hub, a mapping of the plurality of services to the plurality of optimal paths; and (ii) transmitting, to the first proxy server node, the mapping of the plurality of services to the plurality of optimal paths, thereby causing the first proxy server node to forward received traffic of the service to the second next hop. The computer-implemented method of claim 1 or 2, further comprising:

5. Tracking, by the first proxy server node, link performance metrics for traffic of the service through each of the plurality of next hops, the tracked link performance metrics including, for each connection, a number of bytes directed to the service in a given period of time and a number of bytes received from the service in a given period of time; in response to determining that a ratio of bytes directed to the service to bytes received from the service exceeds a first threshold, (i) updating, by the route management hub, a mapping of the plurality of services to the plurality of best paths based on the link performance metric for traffic of the service; and (ii) sending the mapping of the plurality of services to the plurality of best paths to the first proxy server node, thereby causing the first proxy server node to forward received traffic of the service to a second next hop. The computer-implemented method of claim 1 or 2, further comprising:

6. tracking, by the first proxy server node, link performance metrics for traffic of the service through each of the plurality of next hops, the tracked link performance metrics including, for each connection, a number of bytes directed to the service in a period of time and a number of bytes received from the service in a period of time; and in response to determining that a ratio of bytes directed to the service to bytes received from the service satisfies a first sorting criterion for next hop information, by the route management hub, (i) updating a mapping of the plurality of services to the plurality of best paths based on the link performance metric for traffic of the service; and (ii) transmitting the mapping of the plurality of services to the plurality of best paths to the first proxy server node, thereby causing the first proxy server node to forward received traffic of the service to a second next hop. The computer-implemented method of claim 1 or 2, further comprising: tracking, by the first proxy server node, link performance metrics related to traffic of a first connection for the service via the first next hop; in response to determining that at least one link performance metric for the first connection exceeds a second threshold, causing the route management hub to cause the first proxy server node to forward traffic for a second connection with the service to a second next hop; The computer-implemented method of claim 1 or 2, further comprising: tracking, by the first proxy server node, link performance metrics related to traffic of a first connection for the service via the first next hop; in response to determining that at least one link performance metric for the first connection satisfies a second sorting criterion for next hop information, causing the route management hub to cause the first proxy server node to forward traffic for a second connection with the service to a second next hop; The computer-implemented method of claim 1 or 2, further comprising:

9. 3. The computer-implemented method of claim 1 or 2, wherein the multiple link performance metric types include one or more of: bandwidth, requests per second, packet loss, jitter, and latency.

10. A non-transitory computer readable medium storing instructions which, when executed by a processor, cause the processor to perform the method of claim 1 or 2.

11. A system comprising a path management hub comprising a first processing unit and a first memory connected to the first processing unit, the first memory storing instructions that, when executed by the first processing unit, cause the path management hub to perform the method of claim 1 or 2.

Images