Optimizing network delivery of 5g edge application services

Abstract

A method and apparatus for optimizing network delivery of fifth generation (5G) edge application services by weighting network latency and application server load, capacity, and preference to determine where to transmit new packet flows. By using a weighting approach, network traffic can be balanced among multiple instances (e.g., applications) with the same address.

Description

[0001] Cross-citation of related applications

[0002] This patent application claims the benefit of U.S. Provisional Patent Application No. 63 / 094,673, filed October 21, 2020, entitled “Methods and Algorithms to Optimize Network Delivery of Encrypted 5G Edge Application Services,” and U.S. Provisional Patent Application No. 63 / 104,285, filed October 22, 2020, entitled “Network Protocol Extension For 5G Edge Computing Services,” each of which is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the telecommunications sector, and more specifically to the delivery of application services to users via telecommunications networks. Background Technology

[0004] In a fifth-generation (5G) edge computing environment, applications can be served by various application servers in different local data networks (LDNs) (also known as edge computing data centers). LDNs are typically located close to, or even co-located with, 5G sites to minimize latency and optimize user experience.

[0005] When a user equipment (UE) initiates an application packet (or simply packet) using a destination from a domain name service (DNS) reply or from a cache of the UE, the packet from the UE is carried in a protocol data unit (PDU) session over a 5G core to a 5G user plane function (UPF) PDU session anchor (PSA). The PSA decapsulates a 5G general packet radio service (GPRS) tunneling protocol (GTP) outer information header and forwards the packet to an ingress router of a local data network (LDN). The LDN is responsible for forwarding the packet to the intended destination.

[0006] 5G sites are coupled to different LDNs through an internet protocol (IP) network. Routers in the IP network attempt to select the best application server to service a request by the UE. However, selecting the best application server can be difficult. SUMMARY

[0007] The disclosed aspects / embodiments provide techniques for optimizing network delivery of 5G edge application services by weighting network latency and load, capacity, and preference of application servers to determine where to transmit new packet flows. By using a weighting approach, network traffic can be balanced among multiple instances of an application (e.g., application) having the same address.

[0008] A first aspect relates to a first ingress router comprising: a memory to store instructions; one or more processors to execute the instructions to cause the first ingress router to: receive a flow associated with an anycast address; determine a first internet protocol (IP) layer metric corresponding to a first application server having the anycast address and a second IP layer metric corresponding to a second application server having the anycast address, wherein the first IP layer metric comprises a first network latency, a first load measurement, a first capacity index, and a first network preference, and the second IP layer metric comprises a second network latency, a second load measurement, a second capacity index, and a second network preference; compute a cost to reach the first application server and the second application server according to the first IP layer metric and the second IP layer metric; and select a path from a first path connecting the first application server and a second path connecting the second application server based on the computed cost.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the one or more processors are further to execute the instructions to cause the first ingress router to receive a first border gateway protocol (BGP) message from a first egress router connected to the first application server by the first path that includes at least a portion of the first IP layer metric.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the one or more processors are further to execute the instructions to cause the first ingress router to receive a second border gateway protocol (BGP) message from a second egress router connected to the second application server by the second path that includes at least a portion of the second IP layer metric.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the one or more processors are further to execute the instructions to cause the first ingress router to receive a second border gateway protocol (BGP) message from an ingress router of the second path that includes at least a portion of the second IP layer metric.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the cost is determined according to:

[0013]

[0014] where a represents a weighting factor, with a value between 0 and 1, L i represents the first load measurement, L j represents the second load measurement, b i represents the first capacity index, b j represents the second capacity index, D i represents the first network latency, D j represents the second network latency, g i represents the first network preference, g j represents the second network preference.

[0015] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first network latency and the second network latency each include a round trip time (RTT).

[0016] Optionally, in any of the foregoing aspects, another implementation of the aspect proposes that the first network latency and the second network latency are each measured by the ingress routers of the corresponding paths in the first path and the second path.

[0017] Optionally, in any of the foregoing aspects, another implementation of the aspect proposes that the first network latency and the second network latency are each measured by the egress routers of the corresponding paths in the first path and the second path.

[0018] Optionally, in any of the foregoing aspects, another implementation of the aspect proposes that the first load measurement and the second load measurement each include a weighted combination of the total number of data packets sent by the egress node to the application server, the total number of data packets received by the egress node from the application server, the total number of bytes sent by the egress node to the application server, and the total number of bytes received by the egress node from the application server.

[0019] The second aspect relates to a routing method implemented by a first ingress router, comprising: receiving a flow associated with an anycast address; determining a first Internet Protocol (IP) layer metric corresponding to a first application server having the anycast address and a second IP layer metric corresponding to a second application server having the anycast address, wherein the first IP layer metric includes a first network latency, a first load measurement, a first capacity index, and a first network preference, and the second IP layer metric includes a second network latency, a second load measurement, a second capacity index, and a second network preference; calculating the cost of reaching the first application server and the second application server based on the first IP layer metric and the second IP layer metric; and selecting a path from a first path connecting the first application server and a second path connecting the second application server based on the calculated cost.

[0020] Alternatively, in any of the foregoing aspects, another implementation of the aspect proposes receiving a first border gateway protocol (BGP) message that includes at least a portion of the first IP layer metric from a first egress router connected to the first application server via the first path.

[0021] Optionally, in any of the foregoing aspects, another implementation of the aspect proposes receiving a second border gateway protocol (BGP) message, including at least a portion of the second IP layer metric, from a second egress router connected to the second application server via the second path.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving, from an ingress router of the second path, a second border gateway protocol (BGP) message including at least a portion of the second IP layer metric.

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining the cost according to:

[0024]

[0025] where a represents a weighting factor, with a value between 0 and 1, L i represents the first load measurement, L j represents the second load measurement, b i represents the first capacity index, b j represents the second capacity index, D i represents the first network latency, D j represents the second network latency, g i represents the first network preference, g j represents the second network preference.

[0026] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first network latency and the second network latency each comprise a round trip time (RTT).

[0027] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first network latency and the second network latency are each measured by an ingress router of a corresponding one of the first path and the second path.

[0028] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first network latency and the second network latency are each measured by an egress router of a corresponding one of the first path and the second path.

[0029] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first load measurement and the second load measurement each comprise a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0030] A third aspect relates to a first egress router comprising: a memory to store instructions; one or more processors to execute the instructions to cause the first egress router to: determine a first load measurement of a first application server attached to the first egress router; determine a first capacity index of the first application server attached to the first egress router; determine a first network preference corresponding to the first egress router; transmit the first load measurement, the first capacity index, and the first network preference to a first ingress router.

[0031] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first load measurement, the first capacity index, and the first network preference are transmitted in a type length value (TLV) structure.

[0032] Optionally, in any of the preceding aspects, another implementation of the aspect provides that a load measurement sub-TLV structure in the TLV structure includes a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0033] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first load measurement includes a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0034] A fourth aspect relates to a routing method implemented by a first egress router comprising: determining a first load measurement of a first application server attached to the first egress router; determining a first capacity index of the first application server attached to the first egress router; determining a first network preference corresponding to the first egress router; transmitting the first load measurement, the first capacity index, and the first network preference to a first ingress router.

[0035] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first load measurement, the first capacity index, and the first network preference are transmitted in a type length value (TLV) structure.

[0036] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the load measurement sub-TLV structure in the TLV structure includes a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0037] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the first load measurement includes a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0038] A fifth aspect relates to a first ingress router comprising: a memory device for storing instructions; a processing device for executing the instructions to cause the first ingress router to perform any of the disclosed embodiments.

[0039] A sixth aspect relates to a first egress router comprising: a memory device for storing instructions; a processing device for executing the instructions to cause the first egress router to perform any of the disclosed embodiments.

[0040] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present application.

[0041] These and other features will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0042] For a more complete understanding of the present application, reference is now made to the following brief description of the drawings taken in conjunction with the detailed description below, wherein like reference numerals represent like parts.

[0043] Figure 1 is a schematic diagram of a telecommunications network.

[0044] Figure 2 is a schematic diagram of a telecommunications network according to an embodiment of the application.

[0045] Figure 3 is a type length value (TLV) structure according to an embodiment of the application, which can be used to signal load measurements, capacity indices, and network preferences.

[0046] Figure 4Load Measurement sub-TLV structure included in the TLV structure of Figure 3

[0047] Figure 5 Capacity Index sub-TLV structure included in the TLV structure of Figure 3

[0048] Figure 6 Site Preference Index sub-TLV structure included in the TLV structure of Figure 3

[0049] Figure 7 A routing method implemented by an ingress router according to an embodiment of the present application.

[0050] Figure 8 A routing method implemented by an egress router according to an embodiment of the present application.

[0051] Figure 9 A schematic diagram of a network device according to an embodiment of the present application. DETAILED DESCRIPTION

[0052] It should be understood at the outset that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including exemplary design and implementation set forth in this description, but can be modified in various ways.

[0053] As mentioned above, the task of a router in an IP network is to select the best application server to serve the request of a UE. In some cases, the router simply selects the application server closest to the UE. For example, each time a packet moves from one router (or device) to another, it is considered a "hop". Thus, the application server with the least number of hops from the UE is selected. However, when there are other application servers nearby, it can be undesirable to use distance alone as a measure to select the best application server. In fact, there is no guarantee that the closest application server can provide the best user experience (e.g., lowest latency, etc.).

[0054] ​​​Routers can also rely on DNS to select the optimal application server to service a request by a UE. However, there can be drawbacks to using DNS. For example, DNS can be used to provide the address of the application server closest to the requesting UE. However, the requesting UE can cache that address indefinitely. Thus, when the UE moves to another 5G site that can be farther away, the cached address can still be used. As a result, the UE can experience undesirable latency (a.k.a. network delay).

[0055] Furthermore, the DNS-identified application server can be overloaded. Thus, the application server's response to the UE's request can be slow or can not provide a response at all. This outcome is worse when there are other available application servers in close proximity and capable of providing the requested application to the UE.

[0056] Anycast is a method of network addressing and routing in which a single destination IP address is shared by different devices (e.g., application servers) in multiple locations. Routers direct packets destined for the single destination IP address to the application server closest to the sender. As used herein, closest refers to the application server that can be reached with the lowest number of hops (e.g., border gateway protocol (BGP) network hops). Anycast routing is widely used by content delivery networks such as web and DNS hosts to bring their content closer to end users.

[0057] Anycast is increasingly being used by application providers and content delivery networks (CDNs) because anycast makes dynamic load balancing between different application servers possible based on network conditions. However, using anycast to select an application server can result in selecting an overloaded application server when there are other available application servers in close proximity and capable of providing the requested application to the UE. This can occur in a 5G environment where the UE frequently moves from one 5G site to another 5G site.

[0058] Disclosed herein are techniques to optimize network delivery of 5G edge application services by weighting network latency along with load, capacity, and preference of application servers to determine where to transmit new packet flows. By using a weighting approach, network traffic can be balanced among multiple instances (e.g., applications) with the same address.

[0059] The present invention assumes that most application traffic is encrypted and that application servers do not expose their internal information for security reasons. In addition, the present invention is about the network using its own network probes and measurements to implement optimized traffic distribution to specific ANYCAST (also known as Anycast) application instances.

[0060] The present invention includes techniques to estimate traffic load of different instances of an Anycast address using the network's routing and innovative measurements and probes without application cooperation. The present invention includes techniques to discover the edge router closest to a specific Anycast instance. The present invention includes techniques to combine innovative measurements with dynamically routing application flows to lower load instances. The present invention includes techniques to optimize the distribution of encrypted application traffic to its Anycast instances. The present invention includes techniques to use the underlay network's internal information to better balance the load among multiple instances of an application server. Some application servers have their own load balancer to distribute the load to multiple instances. However, there are many problems with application's own load balancer, such as the App load balancer itself becomes a bottleneck, and network congestion of the selected App load balancer can degrade performance. The present invention includes techniques to use intelligent steering of user traffic through application server instances by network segment based on innovative network measurements to achieve optimal end-to-end performance.

[0061] Figure 1 is a schematic diagram of a telecommunications network 100. As shown, the telecommunications network 100 includes a first 5G site 102 (5G Site A), a second 5G site 104 (5G Site B), a first PSA 106 (PSA1), a second PSA 108 (PSA2), a first ingress router 110, a second ingress router 112, an IP network 114, a first egress router 116, a second egress router 118, a third egress router 120, a first LDN 122 (LDN 1), a second LDN 124 (LDN 2), and a third LDN 126 (LDN 3). It should be appreciated that the telecommunications network 100 can include additional components in practical applications. Notably, the 5G PSA (PDU Session Anchor) can be a server blade that is not routable. There should then be a separate router directly connected to the PSA as an ingress router to the local data network (LDN). For ease of description, the PSA is used herein to represent the combination of the 5G PSA and the ingress router to the LDN.

[0062] The first 5G site 102 can be referred to as a 5G core network, including a first PSA 106 and one or more base stations (not shown). The first PSA 106 represents a user plane function (UPF) that terminates the N6 interface of a PDU session within the first 5G site 102. For example, the base stations can be nodeBs or evolved nodeBs (eNBs) for communicating with the UE 128. The UE 128 can be a cell phone (a.k.a. cellular telephone, smart phone), a smart tablet, a smart watch, or other electronic device configured with a cellular communication device. The first PSA 106 communicates with the first ingress router 110. In an embodiment, the first PSA 106 is co-located or co-sited with the first ingress router 110.

[0063] The second 5G site 104 includes a second PSA 108 and one or more base stations (not shown). The second PSA 108 represents a UPF that terminates the N6 interface of a PDU session within the second 5G site 104. For example, the base stations can be nodeBs or evolved nodeBs (eNBs) for communicating with the UE 128. The second PSA 108 communicates with the second ingress router 110. In an embodiment, the second PSA 108 is co-located or co-sited with the second ingress router 112.

[0064] The first ingress router 110, the second ingress router 112, the first egress router 116, the second egress router 118, and the third egress router 120 are capable of communicating with each other over the IP network 114. The IP network 114 is a communication network that uses the Internet Protocol (IP) to send and receive messages between connected devices (e.g., routers, computers, servers, etc.). IP networks are one of the most commonly used global networks, implemented in the Internet, local area networks (LANs), and enterprise networks. Hosts or network nodes in an IP network are typically configured with the transmission control protocol (TCP) / IP protocol suite.

[0065] The first ingress router 110 and the second ingress router 120 can be collectively referred to as a neighbor group. While Figure 1 While two ingress routers (e.g., the first ingress router 110 and the second ingress router 112) are depicted, it is recognized that more or fewer ingress routers can be included in a practical application.

[0066] The first egress router 116 is disposed between the IP network 114 and the first LDN 122. Thus, the first egress router 116 is capable of passing service requests received from the first ingress router 110 or the second ingress router 112 to the first LDN 122 over the IP network 114. The second egress router 118 is disposed between the IP network 114 and the second LDN 124. Thus, the second egress router 118 is capable of passing service requests received from the first ingress router 110 or the second ingress router 112 to the second LDN 124 over the IP network 114. The third egress router 120 is disposed between the IP network 114 and the third LDN 126. Thus, the third egress router 120 is capable of passing service requests received from the first ingress router 110 or the second ingress router 112 to the third LDN 126 over the IP network 114.

[0067] The first egress router 116, the second egress router 118, and the third egress router 120 can be collectively referred to as a neighbor group. While three egress routers (e.g., the first egress router 116, the second egress router 118, and the third egress router 120) are depicted in Figure 1 While three egress routers (e.g., the first egress router 116, the second egress router 118, and the third egress router 120) are depicted in

[0068] Notably, from the perspective of the first LDN 122, the second LDN 124, and the third LDN 126, the first egress router 116, the second egress router 118, and the third egress router 120 will be viewed as ingress routers. Similarly, from the perspective of the first LDN 122, the second LDN 124, and the third LDN 126, the first ingress router 110 and the second ingress router 112 will be viewed as egress routers.

[0069] The first LDN 122 hosts a plurality of application servers (a.k.a. anycast servers) denoted by the names S1, S2, and S3. As shown, the application server S1 is configured to provide an application associated with the address 10.1.2.x / 24, the application server S2 is configured to provide an application associated with the address 11.1.2.x / 24, and the application server S3 is configured to provide an application associated with the address 12.1.2.x / 24.

[0070] The second LDN 124 hosts multiple application servers, which are denoted by the names S4, S5, and S6. As shown, application server S4 is used to provide applications associated with the address 10.1.2.x / 24, application server S5 is used to provide applications associated with the address 11.1.2.x / 24, and application server S6 is used to provide applications associated with the address 12.1.2.x / 24.

[0071] The third LDN 126 hosts multiple application servers, which are denoted by the names S7, S8, and S9. As shown, application server S7 is used to provide applications associated with the address 10.1.2.x / 24, application server S8 is used to provide applications associated with the address 11.1.2.x / 24, and application server S8 is used to provide applications associated with the address 12.1.2.x / 24. The first LDN 122, the second LDN 124, and the third LDN 126 can be collectively referred to as a neighbor group.

[0072] As can be seen, application servers S1, S4, and S7 all provide the same application, although they are disposed in different LDNs. In addition, application servers S2, S5, and S8 all provide the same application, although they are disposed in different LDNs. Likewise, application servers S3, S6, and S9 all provide the same application, although they are disposed in different LDNs. While three application servers are described in each LDN, in actual applications, more or fewer application servers can be included in each LDN. In addition, in actual applications, the telecommunication network 100 can include more or fewer LDNs.

[0073] Today, most applications have multiple server instances, located in different regions or different edge data centers that are accessible through a 5G local data network. Application server instances can not respond to messages sent by network devices, and application communications can be encrypted. Therefore, it cannot be assumed that application server instances can respond to network ping or internet control message protocol (ICMP) query messages sent by routers or network devices.

[0074] The network latency measurement that is closest to reaching the application server is then the measurement of the latency between the ingress router (or 5G PSA) and the router to which the application server instance is directly attached. In the present invention, the term “application instance egress router” (AIER) is used to describe the last router to which an application instance is attached.

[0075] As Figure 1 shown, the UE 128 can migrate or move from the first 5G site 102 to a second 5G site. When this occurs, the task of the second ingress router 112 is to select an appropriate application server to service the request. As noted above, simply selecting the nearest application server can not be ideal. For example, when the second ingress router 112 simply selects the nearest application server (e.g., the application server reachable with the fewest hops), the UE can experience network latency when the nearest application server is overloaded or the path to the nearest application server is congested.

[0076] Figure 2 is a schematic diagram of a telecommunications network 200 in accordance with an embodiment of the application. As shown, the telecommunications network 200 includes a first 5G site 202 (5G Site A), a first PSA 206 (PSAl), a second PSA 208 (PSA2), a third PSA 209 (PSA3), a first ingress router 210, a second ingress router 212, a third ingress router 213, an IP network 214, a first egress router 216, a second egress router 218, a third egress router 220, a first LDN 222 (LDN 1), a second LDN 224 (LDN 2), and a third LDN 226 (LDN 3). It should be appreciated that the telecommunications network 200 can include additional components in practical applications.

[0077] Figure 2 The first 5G site 202 in Figure 1 is similar to the first 5G site 102 in Figure 2 The first PSA 206, the second PSA 208, and the third PSA 209 in Figure 1 are similar to the first PSA 106 and the second PSA 108 in Figure 2 The first ingress router 210, the second ingress router 212, and the third ingress router 213 in Figure 1 are similar to the first ingress router 110 and the second ingress router 112 in Figure 2 The IP network 214 in Figure 1 is similar to the IP network 114 in Figure 2 The first egress router 216, the second egress router 218, and the third egress router 220 in Figure 1 are similar to the first egress router 116, the second egress router 118, and the third egress router 120 in Figure 2 ​​​​​​​the first LDN 222, the second LDN 224, and the third LDN 226 in the first network 200 are similar to the first LDN 122, the second LDN 124, and the third LDN 126 in the first network 100. Thus, for the sake of brevity, the detailed description of these elements in the first network 200 will not be repeated. Figure 1 the first LDN 122, the second LDN 124, and the third LDN 126 in the first network 100. Thus, for the sake of brevity, the detailed description of these elements in the first network 200 will not be repeated. Figure 2 the first LDN 122, the second LDN 124, and the third LDN 126 in the first network 100. Thus, for the sake of brevity, the detailed description of these elements in the first network 200 will not be repeated.

[0078] When a new data packet stream associated with an anycast address (an IP address) arrives at the first ingress router 210 from the UE 128, the task of the first ingress router 210 is to determine where to forward the new data packet stream. As used herein, a new data packet stream begins when the UE 128 requests a new application (e.g., when the UE 128, which has been using an application associated with 10.1.2.x / 24, now requests an application associated with 12.1.2.x / 24). Rather than simply selecting the nearest application server with the requested application, the ingress router 210 selects the “best” or “optimal” egress router coupled to the application server with the requested application. The first ingress router 210 selects the best or optimal egress router (e.g., the first egress router 216, the second egress router 218, or the third egress router 220) by considering the network latency between the ingress router and the egress router, and by considering the load, capacity, and / or preference of the application server with the requested application, as will be explained more fully below.

[0079] The first ingress router 210 periodically measures the network latency (a.k.a. delay) between the first ingress router 210 and the first egress router 216 attached to the application server (e.g., S1) with the requested application. Network latency refers to the amount of time required for a signal to be transmitted from the first ingress router 210 to the first egress router 216 plus the amount of time required for the first ingress router 210 to receive an acknowledgement of the receipt of the signal by the first egress router 216. Network latency can be referred to as round trip time (RTT). In embodiments, the network latency represents an average network latency over the period.

[0080] The first ingress router 210 also receives from the first egress router 216 a load measurement, a capacity index, and a network preference each corresponding to the application server (e.g., S1) with the requested application. In embodiments, the load measurement comprises a weighted combination of the total number of data packets sent by the egress node (e.g., the first egress node 216) to the application server (e.g., S1), the total number of data packets received by the egress node from the application server, the total number of bytes sent by the egress node to the application server, and the total number of bytes received by the egress node from the application server. In embodiments, the load measurement represents an average load measurement over the measurement period.

[0081] In embodiments, the load measurement is based on the following equation:

[0082] load = w1*ToPackets + w2*FromPackets + w3*ToBytes + w4*FromBytes

[0083] where load represents the load measurement, w1, w2, w3, and w4 are each a weighting factor, ToPackets is the total number of packets sent by the egress node to the application server, FromPackets is the total number of packets received by the egress node from the application server, ToBytes is the total number of bytes sent by the egress node to the application server, and FromBytes is the total number of bytes received by the egress node from the application server. Notably, each weight can have a different value. In embodiments, each weight has a value between 0 and 1 (e.g., 0 < wi < 1, where i is 1, 2, 3, or 4). In embodiments, w1 + w2 + w3 + w4 = 1. In embodiments, one or more of the weights are provided by a network engineer. In embodiments, one or more of the weights are based on user feedback or based on self-learning.

[0084] The capacity index represents the capacity of the application server (e.g., S1). As used herein, capacity refers to the ability of the application server to handle service requests. The capacity can be determined based on the processing power of the application server, the speed of the application server, and the like. The higher the value of the capacity index, the higher the capacity of the application server. The lower the value of the capacity index, the lower the capacity of the application server.

[0085] The network preference represents the preference of the application server (e.g., S1). The network preference of one server (e.g., S1) can be relative to the network preference of another application server (e.g., S4). The higher the value of the network preference, the higher the preference of the application server. The lower the value of the network preference, the lower the preference of the application server.

[0086] The second ingress router 212 and the third ingress router 213 also periodically measure network latency and receive from the egress routers (e.g., the second egress router 218 and the third egress router 220) the load measurement, the capacity index, and the network preference each corresponding to an application server (e.g., S4, S7) having the requested application.

[0087] Assuming the second ingress router 212 and the third ingress router 213 are in the same neighbor group as the first ingress router 210, and each router can reach different anycast servers (e.g., S1, S4, S7, etc.) providing the same application, the first ingress router 210, the second ingress router 212, and the third ingress router 213 share their network latency, load measurements, capacity indices, and network preferences with each other. In embodiments, the first ingress router 210, the second ingress router 212, and the third ingress router 213 periodically transmit their network latency, load measurements, capacity indices, and network preferences with each other. In embodiments, each of the first ingress router 210, the second ingress router 212, and the third ingress router 213 stores the network latency, load measurements, capacity indices, and network preferences in memory for later use.

[0088] Once the first ingress router 210 has the various network latencies, load measurements, capacity indices, and network preferences, the first ingress router 210 is able to determine the best or optimal egress router for a new flow. In embodiments, the ingress router 210 selects the best or optimal router for a new flow by comparing the cost to reach each application server (e.g., S1, S4, S7) providing the same service according to the following formula:

[0089]

[0090] where a represents a weighting factor, with a value between 0 and 1, L i represents the first load measurement, L j represents the second load measurement, b i represents the first capacity index, b j represents the second capacity index, D i represents the first network latency, D j represents the second network latency, g i represents the first network preference, g j represents the second network preference, min j represents the egress router j that achieves the minimum cost. In embodiments, the weighting factor is used to balance server load and network preference.

[0091] In the above formula, represents the cost to reach an App server attached to Site-j relative to the cost to reach an App server attached to another site Site-i (identified by cost-j).

[0092]

[0093] In an embodiment, the first ingress router 210 selects the best or optimal router for a new flow by comparing the cost to reach each application server (e.g., S1, S4, S7) that provides the same service according to the following simplified formula:

[0094]

[0095] where a represents a weighting factor, with a value between 0 and 1, L i represents the first load measurement, L j represents the second load measurement, D i represents the first network latency, D j represents the second network latency, min j represents the egress router j that achieves the minimum cost.

[0096] With respect to Figure 2 , an example is provided of how the first ingress router 210 uses the above simplified formula. This example assumes that the latency between the first ingress router 210 and the second ingress router 212 is the same as the latency between the first ingress router 210 and the third ingress router 213. Thus, this latency is not included in the calculation.

[0097] In Figure 2 , the network latency D i between the first ingress router 210 and the first egress router 216 is measured to be 100 microseconds (ps). The network latency D j between the second ingress router 212 and the second egress router 218 is measured to be 200 microseconds (ps). The network latency D j between the third ingress router 213 and the third egress router 220 is measured to be 150 microseconds (ps). The load measurement L i of the first application server S1 that provides the requested application is 3,000. The load measurement L j of the second application server S4 that provides the requested application is 2,000. The load measurement L j of the third application server S7 that provides the requested application is 1,000. Assume that the weighting factor a is 0.6. Thus, the first ingress router 210 gets the following results:

[0098] Cost to reach S1:

[0099] Cost to reach S4:

[0100] Cost to reach S7:

[0101] Since the cost of reaching the application server S7 in the third LDN 226 is the lowest relative to the cost of reaching the application server S4 in the second LDN 224 and the cost of reaching the application server S1 in the first LDN 222, the application server S7 is selected and the new data packet stream is routed to the third egress router 220.

[0102] There are two cases. In scenario a), the network controller is informed of the routers (represented by router ID or IP address) to which a particular application server instance is attached. These routers can be gateway routers to the micro data center hosting the application instance. In scenario b), the network controller is not configured and no information is provided to the network controller about the location and manner in which the application server instance is attached. This scenario requires the network to intelligently discover the application instance egress routers. There can be multiple “application instance egress routers” for an application server, located in different regions or 5G LDNs.

[0103] For the above scenario a), the network latency or RTT to reach the application server instance can be measured by the ingress router (a logical function within the PSA or a router directly connected to the 5G PSA) sending a ping request to the application instance egress router, and the router replying to the ping request. The ingress router can measure the time interval between the ping request and the ping reply and record it as the RTT at time T.

[0104] Using the time interval between the ping and the ping reply can not be accurate in measuring the RTT of application data packets between two routers, as there can be multiple paths between the two routers and the ping message can go through a different path than the actual data packets.

[0105] A more accurate RTT measurement of the data packets by the network is by attaching metadata to the actual data packets. The metadata includes a timestamp (for a sequence number in the case where the ingress router can map the sequence number to an actual time). The application instance router receiving the data packet containing the metadata needs to extract the metadata and send the extracted metadata back to the sender.

[0106] Since the application data packets are only sent to one application instance egress router, when there are no application data packets available, the ping and ping reply RTT measurement is used periodically to measure the RTT of the best route instance that is not based on the routing protocol forwarding table to all other application egress routers.

[0107] For scenario b) above, the network needs to discover which router is attached to the application server instance. Below is the method to discover all application instance egress routers for a specific application.

[0108] One or more mini data center gateway routers periodically send reverse address resolution protocol (ARP) IP version 4 (IPv4) or neighbor discovery scans with the address in the application server address to discover if the application server instance is hosted in the edge data center. If yes, one or more routers will be identified as the application instance egress router (or routers) for the application.

[0109] Each DC gateway router needs to advertise its attached application instance to other relevant LDNs and all routers in the network through the Interior Gateway Protocol (IGP and BGP). The built-in distribution of the routing protocol can also distribute additional information to the routers in other areas.

[0110] It is worth noting that when multiple instances are instantiated in close edge data centers, the network routing protocol can not produce a large enough cost difference to reach the instances in different edge data centers for 5G edge services. In this case, the network controller injects an additional cost metric to the route to reach each instance to prevent the packets of one flow from being forwarded to a different instance, which can degrade the application communication due to the oscillation of packet delivery for one flow. The detailed mechanism of how the network controller injects the cost to different routes is not provided herein.

[0111] In embodiments, each application server can be used by a small group of UEs. Therefore, it is not necessary for the egress router to advertise all IP layer metrics to all routers. Also, each edge computing (EC) data center gateway router can have only a small number of EC instances.

[0112] “Application bound group router” is used to refer to a group of routers that are interested in a specific anycast address for an application. The network (IP) layer metrics of the EC server instances should be advertised among the routers in the “application bound group router”.

[0113] The BGP RT Constrained distribution described in IETF document RFC 4684, published in November 2006 by P. Marques et al, entitled "Constrained Route Distribution for Border Gateway Protocol / MultiProtocol Label Switching (BGP / MPLS) Internet Protocol (IP) Virtual Private Networks (VPNs)", can be used to form the "application binding group".

[0114] Since the EC / application servers are typically much more numerous than the routers in the 5G LDN, a more practical approach to forming the "application binding group" is for each ingress router, upon receiving the first packet, to query the network controller for the specific anycast address included in the "application binding group router". There should be a timer associated with the ingress router, since the UE using the EC service can move. When the timer expires, the ingress router will be removed from the "application binding group router".

[0115] The 5G network connects to many small local data networks (LDNs) at which application servers are instantiated. Each LDN can have multiple other LDNs in close proximity. When the application server has multiple instances in different LDNs, sometimes the performance is better if the application traffic from the UE is not sent to its closest LDN, but to an application server instance in a neighboring LDN.

[0116] In particular, when a UE moves from one 5G local access wireless network to another, traffic performance can be better if connected to the application server instance in the old LDN rather than switching to the application server instance in the new LDN for some reasons. The new LDN can be overloaded, making the network latency to the application server instance longer. The application server instance in the new LDN is overloaded due to high demand, for example, a gathering or other event of aggregation in the area. For a particular application, its instances can only be instantiated in a subset of LDNs. For ease of description, this set of LDNs is called the application binding neighborhood set for this particular application. The neighborhood set for application A can be different from that for application B. The application binding neighborhood set is discovered through an extension of a routing protocol, for example, BGP.

[0117] Assuming an external border gateway protocol (eBGP) is running between the LDNs, if the load router can add a very large cost value when propagating a route update received from a neighboring LDN through its own LDN to another neighboring LDN, all routers can easily compute whether a particular application address is within its own LDN, within a directly neighboring LDN, or within a number of LDNs far away. Through this method, the application binding neighborhood set can be formed.

[0118] An application can have its instances instantiated in different geographical areas, for example, different countries or continents. Below is an algorithm to fine-tune the application binding neighborhood set to exclude instances in too far away areas. Since the geographical distance is significant, some instances can be manually removed.

[0119] Assume LDN-Neighbor[App-i] is a set of neighboring LDNs in which application App-i is instantiated.

[0120] - Each PSA, say PSA-a, measures the network performance of the list of PSAs in all LDN-Neighbor[App-i].

[0121] - Current LDN-Neighbor[App-i] is the current application binding neighborhood set for application “App-i”.

[0122] When (True)

[0123]

[0124] Anycast used in 5G edge computing environment is slightly different from typical anycast and the typical anycast address deployed. The typical anycast address is used to represent instances in very different geographical locations (e.g. different continents). The anycast address of the “app.net” in Asia directs the data packet to the “app.net” server instance hosted in Asia.

[0125] 5G edge computing environment can host multiple instances of an edge computing server in multiple mini edge computing DCs close by. The router, the ingress router of the local data network, can forward the data packet of the anycast address of “app.net” to different egress routers attached with “app.net” instances.

[0126] If “app.net” has 4 instances in 4 different 5G edge computing data centers. All these 4 instances have the same anycast address. The RTT of “app.net” anycast address should be a set of values (not a singlecast address one RTT value). The value refers to the egress router specific address (i.e. loopback address) that the instance is attached to.

[0127] The RTT of “app.net” ANYCAST address is represented as:

[0128] {egress router loopback address, RTT value} list

[0129] This list is called “RTT-ANYCAST”.

[0130] To better optimize anycast traffic, each router adjacent to a 5G PSA needs to periodically measure the RTT of the list of egress routers that advertise the anycast address. The RTT of the egress router at Site-i is considered as the RTT of the anycast instance at Site-i.

[0131] To better optimize anycast traffic, each PSA uses the method disclosed in this paper to discover the list of routers that directly attach the application server instances in the adjacent group of application bindings. Each PSA needs to periodically perform this RTT measurement on these routers and refresh its measurement results.

[0132] Since the network cannot access the internal logic of the application server instance, this section introduces a method to passively measure the workload of the application server instance.

[0133] Although this method is not as accurate as the application server itself reporting its own workload, this method provides a smart estimate without the cooperation of the application or the corresponding application controller.

[0134] The specific process is as follows.

[0135] - For a specific application that matches the address of an application server instance, the application instance egress router (AIER), such as the egress router 116 in Figure 1 measures the traffic to the address and the traffic from the specific address.

[0136] o Total packets to the instance

[0137] o Total packets from the instance

[0138] o Total bytes to the instance

[0139] o Total bytes from the instance

[0140] - All AIERs in the adjacent group of an application binding report the above measurements on different time spans.

[0141] o By sending independent messages to the controller, or

[0142] o Marking to the client route BGP UPDATE.

[0143] - If the measurements or configured time spans (e.g., total 1 minute, total 15 minutes, total every hour, and total 24 hours) are propagated to their corresponding network controller or to the ingress node using BGP Update, the time span can be the BGP UPDATE frequency.

[0144] It should be noted that the load index can be the actual packet amount (To and From traffic, respectively) for a specific time span (e.g., 15 minutes or 24 hours) or a ratio of the packet amount divided by a fixed value.

[0145] The disclosed process is based on balancing the traffic to multiple instances with the same edge server based on the following attributes.

[0146] - The traffic load or load index of the instances in the LDN, as described above

[0147] - The capacity index,

[0148] - The site preference index,

[0149] - The network latency, as described above.

[0150] For any particular application, its deployment in different regions or data centers of 5G edge services can have different capacities. Some data centers can have hundreds or thousands of instances connected to a frontend application load balancer that is reachable from external clients. Other data centers can have very few application instances. The "capacity index" is a number 1-10 that indicates the capacity of the application server in a particular location. The capacity index of any particular application server in a particular location is provided by the application controller. If no capacity index is given, the default value for all locations is 5.

[0151] The following is the algorithm to determine the cost of traffic to a particular anycast instance.

[0152]

[0153] where a represents a weighting factor, with a value between 0 and 1 (if less than 0.5, network latency has a greater impact; otherwise, server load and capacity have a greater impact), L i represents a first load measurement (e.g., total bytes / packets sent to / from the server), L j represents a second load measurement, b i represents a first capacity index (the larger the value, the higher the capacity), b j represents a second capacity index, D i represents a first network latency (e.g., RTT between two edge routers between the client and the server, for 5G, RTT between the PSA and the application instance egress router), D j represents a second network latency, g i represents a first network preference (e.g., the larger the value, the higher the preference, certain LDNs can have higher capacity in terms of router capacity and / or bandwidth capacity than others; the value can be between 1 and 10; can be configured by the network controller to indicate the preference of the LDN, if not configured, the default value is 5), g j represents a second network preference.

[0154] The moving average counts needed for load and latency are as follows.

[0155] • The moving average (MA) of the server load measurement l_i for node i is L_i, and the moving average (MA) of the network RTT measurement d_i for node i is D_i.

[0156] • l_i is the count of bytes to / from the server from node i in some measurement period.

[0157] • d_i is the edge-to-edge RTT of flows on the path from node i

[0158] • Assume node i receives L_j and D_j periodically from all neighbors

[0159] When a new flow f arrives at node i, node i forwards the flow to the application instance with the minimum cost A. The above is discussed with respect to Figure 3 An example of how the algorithm works is provided (without considering capacity indices and network preference values).

[0160] Figure 3 is a type length value (TLV) structure 300 of the type discussed above that can be used to signal load measurements, capacity indices, and network preferences. In embodiments, the TLV structure 300 is a switching capability specific information (SCSI) TLV as defined in D. Ceccarelli et al., “Generalized SCSI: A Generic Structure for Interface Switching Capability Descriptor (ISCD) Switching Capability Specific Information (SCSI),” Internet Engineering Task Force (IETF) Request for Comment (RFC) 8258, October 2017. In embodiments, the TLV structure 300 can be encoded as an update message, a BGP update message, or other type of message. In embodiments, the TLV structure 300 is transmitted by an egress router (e.g., the first egress router 216, the second egress router 218, or the third egress router 220) to an ingress router (e.g., the first ingress router 210, the second ingress router 212, or the third ingress router 213).

[0161] As shown in the figure, the TLV structure 300 includes a type field 302, a length field 304, and a value field 306. The type field 302 is a 16-bit field used to identify the type and structure of the information in the value field 306. The length field 304 is a 16-bit field representing the length of the value field 306 in octets (e.g., bytes). The length field 304 is either zero or a multiple of four octets, depending on the number or amount of information in the value field 306.

[0162] Value field 306 includes a load measurement sub-TLV structure 308, a capacity index sub-TLV structure 310, and a site preference index sub-TLV structure 312. The load measurement sub-TLV structure 308 carries the load measurements discussed above. The capacity index sub-TLV structure 310 carries the capacity index discussed above. The site preference index sub-TLV structure 312 carries the network preferences discussed above. Figure 3 As shown, when using, for example, the simplified formula discussed above, the capacity index sub-TLV structure 310 and the site preference index sub-TLV structure 312 may be optional.

[0163] Figure 4 Is included Figure 3 Additional details of the load measurement sub-TLV structure 308 in the TLV structure are provided below. As shown, the load measurement sub-TLV structure 308 includes a type field 402, a length field 404, and a value field 406. The type field 402 is a 16-bit field used to identify the type and structure of the information in the value field 406. In an embodiment, the value of the type field 402 is 1, indicating the cumulative number of packets since the last message (e.g., a BGP update message). In an embodiment, the value of the type field 402 is 1, indicating a user-specified time period, which may be coarser than when the type value is 1. The length field 404 is a 16-bit field representing the length of the value field 406 in octets (e.g., bytes).

[0164] Value field 406 includes measurement period field 408, total number of packets to instance field 410, total number of packets from instance field 412, total number of bytes to instance field 414, and total number of bytes from instance field 416. Measurement period field 408 includes a default value based on the BGP update period.

[0165] The total number of packets to instance field 410 indicates the total number of packets sent to the application server, as described above. The total number of packets from instance field 412 indicates the total number of packets received from the application server, as described above. The total number of bytes to instance field 414 indicates the total number of bytes sent to the application server, as described above. The total number of bytes from instance field 416 indicates the total number of bytes received from the application server, as described above.

[0166] Figure 5 is additional detail of the capacity index sub-TLV structure 310 included in the TLV structure of Figure 3 As shown, the capacity index sub-TLV structure 310 includes a type field 502, a length field 504, and a value field 506. The type field 502 is a 16-bit field used to identify the type and structure of the information in the value field 506. In an embodiment, the value of the type field 502 can be 20. The length field 504 is a 16-bit field that indicates the length of the value field 506 in octets (e.g., bytes). The value field 506 carries the capacity index, as described above. Since the capacity index can be a more stable value than, for example, the load of an application server, the capacity index sub-TLV structure 310 can not be included in every message from the egress router to the ingress router.

[0167] Figure 6 is additional detail of the site preference index sub-TLV structure 312 included in the TLV structure of Figure 3 As shown, the site preference index sub-TLV structure 312 includes a type field 602, a length field 604, and a value field 606. The type field 602 is a 16-bit field used to identify the type and structure of the information in the value field 606. In an embodiment, the value of the type field 602 can be thirty. The length field 604 is a 16-bit field that indicates the length of the value field 606 in octets (e.g., bytes). The value field 606 carries the network preference, as described above. Since the network preference can be a more stable value than, for example, the load of an application server, the site preference index sub-TLV structure 312 can not be included in every message from the egress router to the ingress router.

[0168] Figure 7 is a routing method 700 implemented by an ingress router (e.g., the first ingress router 210) according to an embodiment of the application. The method 700 can be implemented to route a new data packet flow to a best or optimal application server.

[0169] In block 702, the first ingress router (e.g., the first ingress router 210) receives a flow associated with an anycast address.

[0170] In block 704, the first ingress router determines a first internet protocol (IP) layer metric corresponding to a first application server having the anycast address and a second IP layer metric corresponding to a second application server having the anycast address. In an embodiment, the first IP layer metric includes a first network latency, a first load measurement, a first capacity index, and a first network preference. In an embodiment, the second IP layer metric includes a second network latency, a second load measurement, a second capacity index, and a second network preference.

[0171] In block 706, the first ingress router computes a cost to reach the first application server and the second application server from the first IP layer metric and the second IP layer metric.

[0172] In block 708, the first ingress router selects a path from the first path connecting to the first application server and the second path connecting to the second application server based on the computed cost.

[0173] In an embodiment, the first ingress router receives a first border gateway protocol (BGP) message including at least a portion of the first IP layer metric from a first egress router connected to the first application server through the first path. In an embodiment, the first ingress router receives a second border gateway protocol (BGP) message including at least a portion of the second IP layer metric from a second egress router connected to the second application server through the second path.

[0174] In an embodiment, the first ingress router receives a second border gateway protocol (BGP) message including at least a portion of the second IP layer metric from an ingress router of the second path. In an embodiment, the first ingress router selects the path from the first path and the second path according to a determination that the cost:

[0175]

[0176] where a represents a weighting factor, with a value between 0 and 1, L i represents the first load measurement, L j represents the second load measurement, b i represents the first capacity index, b j represents the second capacity index, D i represents the first network latency, D j represents the second network latency, g i represents the first network preference, g j represents the second network preference.

[0177] In an embodiment, the first network latency and the second network latency each include a round trip time (RTT). In an embodiment, the first network latency and the second network latency are each measured by an ingress router of a corresponding path of the first path and the second path. In an embodiment, the first network latency and the second network latency are each measured by an egress router of a corresponding path of the first path and the second path.

[0178] In embodiments, the first load measure and the second load measure each include a weighted combination of a total number of packets sent by the egress node to the application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0179] Figure 8 is a routing method 800 implemented by an egress router (e.g., the first egress router 216) in accordance with embodiments of the application. The method 800 can be implemented to provide information to an ingress router (e.g., the first ingress router 210) so that the ingress router can determine the best or optimal application server to receive a new packet flow.

[0180] In block 802, a first load measure of a first application server (e.g., SI) attached to the first egress router is determined. In block 804, a first capacity index of the first application server attached to the first egress router is determined. In block 806, a first network preference corresponding to the first egress router is determined. In block 808, the first load measure, the first capacity index, and the first network preference are transmitted to the first ingress router.

[0181] In embodiments, the first load measure, the first capacity index, and the first network preference are transmitted in a TLV structure (e.g., the TLV structure 300). In embodiments, the TLV structure includes a load measure sub-TLV structure (e.g., the load measure sub-TLV structure 308) containing the first load measure. In embodiments, the load measure sub-TLV structure includes a total number of packets sent by the egress node to the application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0182] In embodiments, the first load measure includes a weighted (e.g., wl) combination of a total number of packets sent by the egress node to the application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

[0183] In embodiments, the TLV structure includes a capacity index sub-TLV structure (e.g., the capacity sub-TLV structure 310) containing the first capacity index. In embodiments, the TLV structure includes a network preference sub-TLV structure (e.g., the site preference index sub-TLV structure 312) containing the first network preference.

[0184] Figure 9is a schematic diagram of a network device 900 (e.g., ingress router, egress router, network appliance, etc.). The network device 900 is suitable for implementing the disclosed embodiments as described herein. The network device 900 includes an ingress port / ingress device 910 and a receiver unit (Rx) / receiving device 920 for receiving data; a processor, logic unit, or central processing unit (CPU) / processing device 930 for processing data; a transmitter unit (Tx) / transmitting device 940 and an egress port / egress device 950 for transmitting data; and a memory / storage device 960 for storing data. The network device 900 can also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress port / ingress device 910, the receiver unit / receiving device 920, the transmitter unit / transmitting device 940, and the egress port / egress device 950 for egress or ingress of optical or electrical signals.

[0185] The processor / processing device 930 is implemented by hardware and software. The processor / processing device 930 can be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing device 930 is in communication with the ingress port / ingress device 910, the receiver unit / receiving device 920, the transmitter unit / transmitting device 940, the egress port / egress device 950, and the memory / storage device 960. The processor / processing device 930 includes a routing module 970. The routing module 970 is capable of implementing the methods disclosed herein. Thus, the inclusion of the routing module 970 provides a substantial improvement to the functionality of the network device 900 and effects a transformation of the network device 900 to a different state. Alternatively, the routing module 970 is implemented as instructions stored in the memory / storage device 960 and executed by the processor / processing device 930.

[0186] The network device 900 can also include input and / or output (I / O) devices / I / O devices 980 for data input and output with a user. The I / O devices I / O devices 980 can include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices I / O devices 980 can also include input devices, such as a keyboard, mouse, navigation ball, etc., and / or corresponding interfaces for interacting with such output devices.

[0187] The memory / storage 960 includes one or more disks, tape drives, and solid-state drives, and can be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data read during program execution. The memory / storage 960 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0188] While several embodiments have been provided in the present disclosure, it is to be understood that the disclosed systems and methods can be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to limit the disclosure to the details given herein. For example, the various elements or components can be combined or integrated in another system or certain features can be omitted, or not implemented.

[0189] Also, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate can be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and can be made without departing from the spirit and scope of the present disclosure.

Claims

1. A first ingress router, characterized by, comprising: a memory to store instructions; one or more processors to execute the instructions to cause the first ingress router to: receive a flow associated with an anycast address; determine a first internet protocol (IP) layer metric corresponding to a first application server having the anycast address and a second IP layer metric corresponding to a second application server having the anycast address, wherein the first IP layer metric comprises a first network latency, a first load measurement, a first capacity index, and a first network preference, and the second IP layer metric comprises a second network latency, a second load measurement, a second capacity index, and a second network preference; compute a cost to reach the first application server and the second application server according to the first IP layer metric and the second IP layer metric; select a path from a first path connecting the first application server and a second path connecting the second application server based on the computed cost; wherein the cost is determined according to the following equation: wherein a represents a weighting factor, with a value between 0 and 1, represents the first load measure, represents the second load measure, represents the first capacity index, represents the second capacity index, represents the first network latency, represents the second network latency, represents the first network preference, represents the second network preference.

2. The first ingress router of claim 1, wherein, the one or more processors are further to execute the instructions to cause the first ingress router to: receive a first border gateway protocol (BGP) message comprising at least a portion of the first IP layer metric from a first egress router connected to the first application server through the first path.

3. The ingress router of any of claims 1 to 2, wherein, the one or more processors are further to execute the instructions to cause the first ingress router to: receive a second border gateway protocol (BGP) message comprising at least a portion of the second IP layer metric from a second egress router connected to the second application server through the second path.

4. The ingress router of any of claims 1 to 2, wherein, the one or more processors are to execute the instructions to cause the first ingress router to: receive a second border gateway protocol (BGP) message comprising at least a portion of the second IP layer metric from an ingress router of the second path.

5. The first ingress router of any of claims 1 to 2, wherein, the first network latency and the second network latency each comprise a round trip time (RTT).

6. The ingress router of any of claims 1 to 2, wherein, the first network latency and the second network latency are each measured by an ingress router of a corresponding path of the first path and the second path.

7. The ingress router of any of claims 1 to 2, wherein, the first network latency and the second network latency are each measured by an egress router of a corresponding path of the first path and the second path.

8. The first ingress router of any of claims 1 to 2, wherein, the first load measurement and the second load measurement each comprise a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

9. A routing method implemented by a first ingress router, characterized in that, comprising: receiving a flow associated with an anycast address; determining a first internet protocol (IP) layer metric corresponding to a first application server having the anycast address and a second IP layer metric corresponding to a second application server having the anycast address, wherein the first IP layer metric comprises a first network latency, a first load measurement, a first capacity index, and a first network preference, and the second IP layer metric comprises a second network latency, a second load measurement, a second capacity index, and a second network preference; calculating a cost to reach the first application server and the second application server based on the first IP layer metric and the second IP layer metric; selecting a path from a first path connecting the first application server and a second path connecting the second application server based on the calculated cost; wherein the cost is determined according to the following equation: wherein a represents a weighting factor, with a value between 0 and 1, denotes the first load measure, denotes the second load measure, denotes the first capacity index, denotes the second capacity index, denotes the first network latency, denotes the second network latency, denotes the first network preference, denotes the second network preference.

10. The method of claim 9, wherein, further comprising receiving a first border gateway protocol (BGP) message including at least a portion of the first IP layer metric from a first egress router connected to the first application server through the first path.

11. The method of any one of claims 9-10, wherein, further comprising receiving a second border gateway protocol (BGP) message including at least a portion of the second IP layer metric from a second egress router connected to the second application server through the second path.

12. The method of any one of claims 9-10, wherein, further comprising receiving a second border gateway protocol (BGP) message including at least a portion of the second IP layer metric from an ingress router of the second path.

13. The method of any one of claims 9-10, wherein, the first network latency and the second network latency each comprise a round trip time (RTT).

14. The method of any one of claims 9-10, wherein, the first network latency and the second network latency are each measured by an ingress router of a corresponding path of the first path and the second path.

15. The method of any one of claims 9-10, wherein, the first network latency and the second network latency are each measured by an egress router of a corresponding path of the first path and the second path.

16. The method of any one of claims 9-10, wherein, the first load measurement and the second load measurement each comprise a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

17. A first egress router, characterized by comprising: a memory for storing instructions; one or more processors for executing the instructions to cause the first egress router to: determine a first load measurement of a first application server attached to the first egress router; determine a first capacity index of the first application server attached to the first egress router; determine a first network preference corresponding to the first egress router; transmit the first load measurement, the first capacity index, and the first network preference to a first ingress router; The first ingress router is the ingress router of claim 1.

18. The first export router of claim 17, wherein, The first load measurement, the first capacity index, and the first network preference are transmitted in a type-length-value (TLV) structure.

19. The first export router of claim 18, wherein, The load measurement sub-TLV in the TLV structure includes a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

20. The first export router of claim 17, wherein, The first load measurement includes a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

21. A routing method implemented by a first egress router, characterized by, Comprising: determining a first load measurement of a first application server attached to the first egress router; determining a first capacity index of the first application server attached to the first egress router; determining a first network preference corresponding to the first egress router; transmitting the first load measurement, the first capacity index, and the first network preference to a first ingress router; The first ingress router is the ingress router of claim 1.

22. The method of claim 21, wherein, The first load measurement, the first capacity index, and the first network preference are transmitted in a type-length-value (TLV) structure.

23. The method of claim 22, wherein, The load measurement sub-TLV in the TLV structure includes a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

24. The method of claim 21, wherein, The first load measurement includes a weighted combination of a total number of packets sent by an egress node to an application server, a total number of packets received by the egress node from the application server, a total number of bytes sent by the egress node to the application server, and a total number of bytes received by the egress node from the application server.

25. A first ingress router, comprising: Comprising: a memory device for storing instructions; a processing device for executing the instructions to cause the first ingress router to perform any one of claims 9 to 16.

26. A first egress router, comprising: Comprising: a memory device for storing instructions; a processing device for executing the instructions to cause the first egress router to perform any one of claims 21 to 24.

Images