Mechanism for optimizing large scale handover triggered by cloud DC site failure or degradation

Abstract

A method performed by an ingress router of an edge cloud datacenter for optimizing large scale handover triggered by site failure or degradation at the datacenter. The method comprises: receiving a border gateway protocol (BGP) UPDATE message, the BGP UPDATE message comprising a site reference identifier (ID) and operation capacity information, the site reference identifier (ID) corresponding to a set of routes within a site of a cloud data center; attaching the site reference ID to the group of routes in the routing table; selecting a path for forwarding traffic corresponding to the group of routes based on the operation capacity information; and forwarding the flow of the selected service along the path.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 63 / 391,370, filed by Futurewei Technologies, Inc. on July 22, 2022, and entitled “Mechanism to Optimize Mass Switching Triggered by Cloud DC Site Failures or Degradation,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to network communications, and more particularly to techniques for optimizing large-scale switchovers triggered by cloud data center (DC) site failures or degradations. Background Art

[0004] Fifth generation (5G) edge computing enables cloud servers to run and provide services closer to endpoints, thereby reducing latency and speeding up local processing. Cloud data center (DC) gateway (GW) routers connect external clients to multiple sites or container sets (pods) owned or managed by one or more cloud DC center operators. These cloud internal sites or container sets are invisible to clients using cloud services. Enterprise clients typically have their own customer premises equipment (CPE), which connects to the cloud GW or virtual GW using dedicated paths over the public Internet.

[0005] There are many operations, administration and management (OAM) and diagnostic tools available for enterprise CPE to detect the connection of cloud GW and the performance of cloud GW. However, network layer OAM cannot detect the failure or degradation of cloud sites / containers attached to cloud GW.

[0006] When a failure event occurs, the cloud DC GW visible to the client is usually operating normally. Therefore, the client GW cannot use bidirectional forwarding detection (BFD) to detect the failure. When the site capacity degrades or goes dark, there will be a large number of routes that need to be changed. Summary of the invention

[0007] A first aspect relates to a method implemented by a gateway of a cloud data center. The method includes: sending a first border gateway protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; determining that an operating capacity change affecting the set of routes within the site has occurred; and sending a second BGP UPDATE message, the second BGP UPDATE message including operating capacity information of the site reflecting the operating capacity change.

[0008] Optionally, in a first implementation according to the first aspect, the operating capacity information indicates an operating capacity percentage of the site.

[0009] Optionally, in a second implementation according to the first aspect or its implementation, the group of routes is all routes within the site.

[0010] Optionally, in a third implementation according to the first aspect or its implementation, the site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

[0011] Optionally, in a fourth implementation according to the first aspect or its implementation, the site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

[0012] Optionally, in a fifth implementation according to the first aspect or its implementation, the operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

[0013] Optionally, in a sixth implementation according to the first aspect or its implementation, the operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

[0014] Optionally, in a seventh implementation according to the first aspect or its implementation, the method further includes: monitoring subsequent operating capacity changes of the site; determining that the subsequent operating capacity changes occur; and sending a subsequent BGP UPDATE message, wherein the subsequent BGP UPDATE message includes subsequent operating capacity information of the site corresponding to the subsequent operating capacity changes.

[0015] Optionally, in an eighth implementation according to the first aspect or its implementation, the subsequent operation capacity change further reduces the capacity of the site.

[0016] Optionally, in a ninth implementation according to the first aspect or its implementation, the subsequent operation capacity change increases the capacity of the site.

[0017] A second aspect relates to a method implemented by an ingress router of a cloud data center. The method includes: receiving a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; appending the site reference ID to the set of routes in a routing table; receiving a second BGP UPDATE message, the second BGP UPDATE message including operating capacity information; based on the operating capacity information, selecting a path for forwarding traffic corresponding to the set of routes; and forwarding traffic of the selected service along the path.

[0018] Optionally, in a first implementation according to the second aspect, selecting a path for forwarding traffic corresponding to the set of routes includes: calculating a first cost of the path based on multiple factors including the operating capacity information; and comparing the first cost of the path with a second cost of a second path.

[0019] Optionally, in a second implementation according to the second aspect or its implementation, the multiple factors include load index, capacity index, network delay measurement and preference index.

[0020] Optionally, in a third implementation according to the second aspect or its implementation, forwarding traffic of the selected service along the path includes: performing a lookup on the set of routes in a forwarding information base (FIB) to obtain a destination prefix.

[0021] Optionally, in a fourth implementation according to the second aspect or its implementation, forwarding traffic of the selected service along the path includes: forwarding data packets from the same flow to the same egress router.

[0022] Optionally, in a fifth implementation according to the second aspect or its implementation, the operating capacity information indicates an operating capacity percentage of the site.

[0023] Optionally, in a sixth implementation according to the second aspect or its implementation, the group of routes is all routes within the site.

[0024] Optionally, in a seventh implementation according to the second aspect or its implementation, the site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

[0025] Optionally, in an eighth implementation according to the second aspect or its implementation, the site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

[0026] Optionally, in a ninth implementation according to the second aspect or its implementation, the operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

[0027] Optionally, in a tenth implementation according to the second aspect or its implementation, the operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

[0028] A third aspect relates to a gateway router of a cloud data center, the gateway comprising: a memory storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the gateway to perform the following operations: send a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; determine that an operating capacity change affecting the set of routes within the site has occurred; and send a second BGP UPDATE message, the second BGP UPDATE message including operating capacity information of the site reflecting the operating capacity change.

[0029] Optionally, in a first implementation according to the third aspect, the operating capacity information indicates an operating capacity percentage of the site.

[0030] Optionally, in a second implementation according to the third aspect or its implementation, the group of routes is all routes within the site.

[0031] Optionally, in a third implementation according to the third aspect or its implementation, the site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

[0032] Optionally, in a fourth implementation according to the third aspect or its implementation, the site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

[0033] Optionally, in a fifth implementation according to the third aspect or its implementation, the operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

[0034] Optionally, in a sixth implementation according to the third aspect or its implementation, the operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

[0035] Optionally, in a seventh implementation according to the third aspect or its implementation, the method further includes: monitoring subsequent operating capacity changes of the site; determining that the subsequent operating capacity changes occur; and sending a subsequent BGP UPDATE message, wherein the subsequent BGP UPDATE message includes subsequent operating capacity information of the site corresponding to the subsequent capacity changes.

[0036] Optionally, in an eighth implementation according to the third aspect or its implementation, the subsequent operation capacity change further reduces the capacity of the site.

[0037] Optionally, in a ninth implementation according to the third aspect or its implementation, the subsequent operation capacity change increases the capacity of the site.

[0038] A fourth aspect relates to a router, comprising: a memory storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the router to perform the following operations: receive a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; append the site reference ID to the set of routes in a routing table; receive a second BGP UPDATE message, the second BGP UPDATE message including operating capacity information; based on the operating capacity information, select a path for forwarding traffic corresponding to the set of routes; and forward traffic of the selected service along the path.

[0039] Optionally, in a first implementation according to the fourth aspect, selecting a path for forwarding traffic corresponding to the set of routes includes: calculating a first cost of the path based on multiple factors including the operating capacity information; and comparing the first cost of the path with a second cost of a second path.

[0040] Optionally, in a second implementation according to the fourth aspect or its implementation, the multiple factors include load index, capacity index, network delay measurement and preference index.

[0041] Optionally, in a third implementation according to the fourth aspect or its implementation, forwarding traffic of the selected service along the path includes: performing a lookup on the set of routes in a forwarding information base (FIB) to obtain a destination prefix.

[0042] Optionally, in a fourth implementation according to the fourth aspect or its implementation, forwarding traffic of the selected service along the path includes: forwarding data packets from the same flow to the same egress router.

[0043] Optionally, in a fifth implementation according to the fourth aspect or its implementation, the operating capacity information indicates an operating capacity percentage of the site.

[0044] Optionally, in a sixth implementation according to the fourth aspect or its implementation, the group of routes is all routes within the site.

[0045] Optionally, in a seventh implementation according to the fourth aspect or its implementation, the site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

[0046] Optionally, in an eighth implementation according to the fourth aspect or its implementation, the site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

[0047] Optionally, in a ninth implementation according to the fourth aspect or its implementation, the operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

[0048] Optionally, in a tenth implementation according to the fourth aspect or its implementation, the operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

[0049] A fifth aspect relates to a method implemented by a gateway of a cloud data center, the method comprising: determining that an operational capacity of a site has occurred; and sending a BGP UPDATE message, the BGP UPDATE message comprising a site reference identifier (ID) of the site and operational capacity information, wherein the site reference ID corresponds to a set of routes within the site of the cloud data center.

[0050] A sixth aspect relates to a method implemented by an ingress router of a cloud data center. The method includes: receiving a Border Gateway Protocol (BGP) UPDATE message, the BGP UPDATE message including a site reference identifier (ID) and operating capacity information, wherein the site reference ID corresponds to a set of routes within a site of the cloud data center; appending the site reference ID to the set of routes in a routing table; based on the operating capacity information, selecting a path for forwarding traffic corresponding to the set of routes; and forwarding traffic of a selected service along the path.

[0051] A seventh aspect relates to a network device, comprising means for executing any one of the above aspects or their implementations.

[0052] The eighth aspect relates to a computer program product, which includes computer executable instructions stored on a non-transitory computer-readable storage medium, and the computer executable instructions, when executed by one or more processors of the device, cause the device to perform any one of the above aspects or their implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] For a more complete understanding of the present disclosure, reference is now made to the following brief description taken in conjunction with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0054] Figure 1 is a schematic diagram illustrating a network according to an embodiment of the present disclosure.

[0055] Figure 2 is a flow chart illustrating a process for announcing site capacity degradation according to an embodiment of the present disclosure.

[0056] Figure 3 is a flow chart illustrating a process performed by an ingress router according to an embodiment of the present disclosure.

[0057] Figure 4 It is a site capacity opaque extended community attribute according to an implementation of the present disclosure.

[0058] Figure 5 is a metadata path attribute according to an implementation of the present disclosure.

[0059] Figure 6 is a site preference index sub-TLV according to an embodiment of the present disclosure.

[0060] Figure 7 is a degraded index sub-TLV according to an embodiment of the present disclosure.

[0061] Figure 8 is a schematic diagram of an apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0062] First, it should be understood that although illustrative implementations of one or more embodiments are provided below, the disclosed systems and / or methods can be implemented using any number of techniques, whether currently known or existing. The present disclosure is in no way limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims and the full scope of their equivalents.

[0063] The present disclosure provides a mechanism for optimizing the processing when a large number of service instances are affected by a cloud site / container set that encounters a failure or degradation. The disclosed mechanism not only significantly reduces the number of large number of service instances that the cloud GW announces to the affected CPE or ingress router, but also speeds up the switching of the CPE to the large number of instances at the next best site.

[0064] Figure 1 is a schematic diagram of a network 100 illustrating an embodiment of the present disclosure. The network 100 includes a data center 102 (commonly referred to as a cloud data center or cloud) that hosts and provides multiple services. In an embodiment, the data center 102 is an edge DC managed by a cloud DC operator. An edge DC is a DC that is positioned closer to the edge of the network so that computing and storage resources are closer to end users or edge devices, thereby reducing latency and improving the performance of services and applications. For example, in an embodiment, the network 100 represents a single domain such as a 5G local data network, which is a limited domain with edge services a few hops away from the entry node. For example, the data center 102 can be connected to multiple customer premises equipment (CPE) via an edge gateway 106 or a virtual private network (VPN) gateway 108, labeled as C1 peers 104A to C N The edge gateway 106 and VPN gateway 108 are devices or routers that serve as gateways or entry points for network traffic flowing into or out of the data center 102. In an embodiment, the C1 peers 104A to C N Peers 104N may be network devices (eg, routers or customer premises gateway devices) located at various branch offices of an enterprise, which are connected to data center 102 to receive services provided by data center 102. CPE (C1 peers 104A to C N One or more CPEs in peers 104A to 104N may be connected to edge gateway 106 and VPN gateway 108 via one or more provider networks 1010A and 1010B. N One or more CPEs in peer 104N may connect to VPN gateway 108 via public Internet 1012 using a secure tunnel (e.g., using Internet protocol security (IPsec)) to establish a secure connection with VPN gateway 108 (using Figure 1 ). Although Figure 1Only one edge gateway 106 and VPN gateway 108 are shown in the figure, but the data center 102 can have multiple edge gateways 106 and VPN gateways 108. In the present disclosure, CPE (C1 peers 104A to C N Peers 104N) are referred to as ingress routers (or ingress nodes) because they are connected to the data center 102 and represent client-side connections. One or more edge gateways 106 and VPN gateways 108 are referred to as egress routers (or egress nodes) and represent server-side / DC connections. In an embodiment, the ingress router and the egress router can establish a border gateway protocol (BGP) session (e.g., using a BGP route reflector (RR) 1014) to exchange routing information. BGP is an autonomous system (AS) inter-routing protocol for exchanging network reachability information with other BGP systems. Specifically, BGP UPDATE messages are used to announce and exchange routing information between BGP neighbors. Routing information is used to determine the path for transmitting data packets between the ingress router and the egress router.

[0065] Data center 102 may include multiple server racks that house all of the servers at data center 102. Data center 102 may include multiple sites (i.e., groups of hosts located in different locations). Each site may be composed of multiple sections, called container sets, which are easier to cool than one large space. Sometimes, one or more servers, server racks, container sets, or sites may fail, which may cause the site's operating capacity to be degraded or the entire site to go down completely. Failures may be caused by a variety of reasons, including but not limited to fiber cuts connecting to sites or container sets within sites, cooling failures, insufficient backup power, cyber threat attacks, and excessive changes outside of maintenance windows. Currently, when a failure occurs at data center 102, client / entry routers (e.g., C1 peers 104A to C N The egress routers (e.g., edge gateway 106 and / or VPN gateway 108) of data center 102 that are visible to peer 104N may operate normally. Therefore, the ingress routers that have a path to the egress routers cannot detect the failure at the data center using traditional BFD, which is a network protocol for detecting failures between two routers or switches connected by a link.

[0066] As described above, when site capacity degrades or becomes zero, a large number of routes need to be changed. In addition, a large number of routes switched to another site can also cause overload, thereby triggering more failures. In addition, the routes (internet protocol (IP) addresses) in data center 102 are not well aggregated, thereby triggering a very large number of BGP UPDATE messages when a failure occurs. For example, currently, if 10,000 servers / hosts in data center 102 fail (i.e., are no longer reachable), the egress router must send 10,000 BGP UPDATE messages to the affected ingress routers to notify the affected ingress routers that the routes to the hosts are no longer reachable so that the ingress routers can switch to different sites or perform other corrective actions.

[0067] To address the above issues, the present disclosure introduces a new metadata path attribute called a site degradation index, which indicates the degree of degradation that a site of a data center may be experiencing. By applying the disclosed embodiments, when a failure occurs at a site, resulting in a partial or complete loss of operating capacity, the egress router only sends a single BGP UPDATE message for all affected routes. The ingress router that receives the BGP UPDATE message can adjust the amount of traffic to the affected site based on the degree of degradation occurring at the site as indicated by the site degradation index in the received BGP UPDATE message and other factors.

[0068] Figure 2 2 is a flow chart illustrating a process 200 for announcing the operating capacity of a site according to an embodiment of the present disclosure. The operating capacity of a site is the amount by which the site is operating normally or as expected. For example, an operating capacity of 100 means that the site has full operating capacity (i.e., no problems), while an operating capacity of 0 means that the site is completely down. Process 200 may be performed by an egress router (e.g., Figure 1, edge gateway 106 and / or VPN gateway 108 in the data center). In step 202, the egress router sends a BGP UPDATE message including a site reference identifier (ID). In some embodiments, the site reference ID represents a set of routes within a site / container set in a data center. In some embodiments, the site reference ID can be a locally valid site / container set ID that represents the operating capacity of all routes (instances) in the site / container set. There can be many sites / container sets connected to the egress router to link the site reference ID with the client routes. In some embodiments, the site reference ID is included in the site capacity opaque extended community attribute of the BGP UPDATE message periodically sent by the egress router. The site capacity opaque extended community attribute is an opaque extended community attribute with a capacity index subtype, as described below and in Figure 4 The opaque extended community attribute is defined in the Internet Engineering Task Force (IETF) document request for comment (RFC) 4360, entitled "BGP Extended Communities Attribute", by S. Sangli et al., published in February 2006. Alternatively, in some embodiments, the site reference ID is included in a metadata path attribute in a BGPUPDATE message sent by the egress router. The metadata path attribute is a BGP path attribute of type metadata, as described below and in Figure 5 . BGP path attributes are used to provide more information (i.e., attributes) about each route, such as weight, local preference, autonomous system (AS) path, etc. In some embodiments, when the site reference ID of the client route has not changed from the site reference ID sent in the previous BGP UPDATE message, the egress router does not include the site reference ID in the BGP UPDATE message. As described below, the receiving ingress router appends the site reference ID to the route in the routing table stored by the ingress router.

[0069] In step 204, the egress router monitors the operating capacity of the site of the edge cloud to determine in step 206 whether the operating capacity of the site in the edge cloud has changed (e.g., degraded or failed, or recovered from a previous failure) when the egress router is operating as expected. For example, the egress router can determine that a portion of the site may be unreachable by periodically performing an echo check on a node in the edge cloud (and not receiving a response) or by monitoring the status of a link connecting the egress router to the node in the edge cloud. Other methods for determining the operating capacity of the site can be used with the disclosed embodiments. In some embodiments, the egress router monitor does not actively perform the monitoring step 204, but instead discovers (e.g., a node in the site cannot be reached) or obtains / identifies information indicating that the operating capacity of the site has changed.

[0070] When the egress router determines that an operational capacity change affecting a set of routes in the edge cloud has occurred (e.g., a degradation or failure has occurred), at step 208, the egress router sends a BGP UPDATE message to notify the site of operational capacity information reflecting the operational capacity change. Figure 4 The site capacity opaque extended community attribute includes the operating capacity information in the BGP UPDATE message. Alternatively, in some embodiments, the operating capacity information is included in the Figure 5 The metadata path attribute includes a degradation index sub-TLV that specifies the degree / level of degradation, as described below and in Figure 6 In some embodiments, the BGP UPDATE message may include additional path attributes, such as, but not limited to, a site preference index or load measurement information. For example, a site preference index may be included in a BGP UPDATE message using a preference index sub-TLV, as described below and in Figure 7 It should be noted that the site reference ID and / or the operating capacity information may be encoded in the BGP UPDATE message in other ways, such as in other types of BGP communities (eg, BGP wide communities). The present disclosure is not limited to a specific encoding scheme.

[0071] Furthermore, in some embodiments, the site reference ID may be included in the same BGP UPDATE message that includes the operational capacity information. In these embodiments, the egress router does not need to issue a separate BGP UPDATE message that includes the site reference ID. Thus, in some embodiments, when the egress router (i.e., a gateway of a cloud data center) determines that the operational capacity of a site has changed, the egress router sends a BGP UPDATE message that includes the site reference identifier (ID) and the operational capacity information of the site.

[0072] The ingress router that receives the BGP UPDATE message uses the operating capacity information, along with any other path attribute information, to reroute traffic as follows Figure 3 This single BGP UPDATE message effectively implements switching of a large number of client routes from a degraded site to other sites, and thus eliminates the need to send a large number of BGP UPDATE messages to announce capacity reduction, and thus optimizes a large number of switches triggered by site failure or degradation of a cloud DC.

[0073] In some embodiments, process 200 returns to step 204 and continues to monitor the operating capacity of the edge cloud site. In step 206, when the egress router determines that the operating capacity of the edge cloud site has changed (e.g., a further degradation of a previously reported site or a degradation of a different site, or the degraded capacity of a previously reported site has increased), in step 208, the egress router sends a BGP UPDATE message to announce the new operating capacity information.

[0074] Figure 3 300 is a flow chart illustrating a process 300 performed by an ingress router according to an embodiment of the present disclosure. At step 302, the ingress router receives a BGP UPDATE message including a site reference ID representing a set of routes within a site / container set of a data center or all routes (instances) within the site / container set. At step 304, the ingress router appends the site reference ID to the routes stored in the routing table at the ingress router.

[0075] In step 306, the ingress router receives a BGP UPDATE message including operating capacity information from the egress router of the edge cloud DC. The ingress router can receive BGP UPDATE messages (with or without operating capacity information) from multiple egress routers of the edge cloud DC or egress routers of other cloud DCs, because most applications currently have multiple server instances instantiated in different regions or on different edge DCs. The ingress router will typically have multiple paths to reach the required service instance. When the ingress router receives BGP UPDATE messages for the same IP address from multiple egress routers, all of these egress routers are considered as potential paths (or next hops) for the IP address (i.e., if BGP add paths are supported).

[0076] In some embodiments, for the selected or affected service, in step 308, the ingress router uses the operating capacity information from one or more BGP UPDATE messages and other factors to determine a path for forwarding traffic corresponding to all routes, all of which are associated with the site reference ID specified in the BGP UPDATE message. For example, in some embodiments, the ingress router can call a function (referred to herein as a cost calculation engine) that can select a path based on a cost associated with the route, the cost being based on, but not limited to, a site degradation index, a site preference index, and other network cost factors. For example, assume that the destination address of S1:aa08::4450 can be reached via three next hops (R1, R2, R3). In addition, assume that the cost calculation engine identifies R1 as the best next hop for the flow to be sent to this destination (S1:aa08::4450). The cost calculation engine can insert a higher weight for the tunnel associated with R1 for the prefix via the tunnel.

[0077] As a non-limiting example, in some implementations, the cost calculation engine calculates the cost of reaching an application server attached to Site-i relative to a reference site (eg, Site-b) based on the following formula.

[0078]

[0079] Load-i represents a load index at Site-i, which is a weighted combination of total packets or / and bytes sent to and received from the application server at Site-i during a fixed time period.

[0080] CP-i represents the operating capacity index at Site-i. A higher CP-i value means a higher operating capacity.

[0081] Delay-i represents the network delay measurement (RTT) of the egress router associated with the application server at Site-i.

[0082] Pref-i represents the preference index of Site-i. A higher preference index value means a higher preference.

[0083] w represents the weight of load and site information, which is a value between 0 and 1. For example, if w is less than 0.5, network latency and site preference have a greater impact; if w is greater than 0.5, server load and operating capacity have a greater impact; and if w is equal to 0.5, network latency and site preference have an equal impact to server load and operating capacity.

[0084] When the reference site Site-b is calculated by the above formula, the cost is 1. Therefore, if the formula returns a value less than 1 for Site-i, the cost of reaching Site-i is less than the cost of reaching Site-b.

[0085] At step 310, the ingress router forwards traffic for the selected service along the selected path. For example, when the ingress router receives a packet, the ingress router performs a route lookup in the forwarding information base (FIB) to obtain the entire path to the destination prefix. The ingress router then encapsulates the packet to the optimal egress node. For subsequent packets belonging to the same flow, the ingress router forwards them to the same egress router unless the selected egress router is no longer reachable. Many commercial routers support keeping packets from a flow to the same egress router, also known as flow affinity.

[0086] As described above, in some embodiments, the site reference ID can be included in the same BGP UPDATE message that includes the operational capacity information. In these embodiments, the ingress router receives a BGP UPDATE message that includes the site reference ID and the operational capacity information of the site. The ingress router appends the site reference ID to a set of routes in the routing table. The ingress router then selects a path for forwarding traffic corresponding to the set of routes based on the operational capacity information. For the selected service, the ingress router forwards traffic along the selected path.

[0087] Figure 4is a site capacity opaque extended community attribute 400 according to an embodiment of the present disclosure. The site capacity opaque extended community attribute 400 includes a type field 402, a subtype field 404, a reserved field 406, a usage index field 408, a site reference field 410, and a site capacity index field 412. The type field 402 is a 1-octet field containing a value = 0x03 to indicate that the type is an opaque extended community type. The subtype field 404 is a 1-octet field indicating the capacity index subtype. The reserved field 406 is a 1-octet field reserved for future use. The usage index field 408 is a 1-octet field indicating whether the site capacity index is an absolute value or a percentage, etc. relative to the set of all sites / containers attached to the BGP speaker. The site reference field 410 is a 2-octet field that specifies a site reference ID as described above. The site capacity index field 412 specifies an operational capacity index that represents a percentage of the operational capacity of the site. In an embodiment, the capacity index is a value between 0 and 100. For example, when a site goes dark, the operational capacity index is set to 0. Likewise, an operational capacity index of 50 means that the site is at 50% functional capacity, while an operational capacity index of 100 means that the site is at 100% functional capacity (i.e., no issues). Unless a site goes dark (i.e., its operational capacity is zero), not all traffic to that site needs to be rerouted.

[0088] Figure 5 It is a metadata path attribute 500 according to an embodiment of the present disclosure. In some embodiments, the metadata path attribute 500 is an optional transitive BGP path attribute that carries edge service metadata. The metadata path attribute 500 includes a service metadata type field 502, a length field 504, and a value field 506. The service metadata type field 502 is a two-octet size field that carries a value indicating that the type-length-value (TLV) is a type code of the metadata path attribute (to be determined (TBD) by the Internet Assigned Numbers Authority (IANA)). The length field 504 is a two-octet size field that carries a value indicating the total number of octets (i.e., size) of the value field 506. The value field 506 includes a set of sub-TLVs (i.e., one or more sub-TLVs), wherein each sub-TLV contains information corresponding to a different / specific metric of the edge service metadata. In an embodiment, all values ​​in the sub-TLVs are unsigned 32-bit integers. Examples of metrics of edge service metadata include, but are not limited to, capacity index values, site preference index values, and load measurement values.

[0089] Figure 6 is a site preference index sub-TLV 600 according to an embodiment of the present disclosure. In an embodiment, the site preference index sub-TLV 600 may be Figure 5 The sub-TLV for specifying the site preference index of the edge cloud site carried in the value field 506 of the metadata path attribute 500 in . The site preference index sub-TLV 600 includes a site preference subtype field 602, a length field 604, and a preference index value field 606. The site preference subtype field 602 is a two-octet size field that carries a value indicating that the sub-TLV is a type code (TBD) of the site preference index. The length field 604 is a two-octet size field that carries a value indicating the total number of octets (i.e., size) of the preference index value field 606. The preference index value field 606 carries a site preference index value. In an embodiment, the site preference index value is a value between 1 and 100, where 1 is the least preferred and 100 is the most preferred.

[0090] The site preference index can be based on various factors. For example, one edge cloud site may have fewer computing servers, less power, or lower internal network bandwidth than another edge cloud site. In an embodiment, an edge site located at a remote cell site may have a lower preference index value than an edge site in a metropolitan area that hosts management systems, analytical functions, and security functions. As described above, in some embodiments, the site preference index is one of the factors integrated into the total cost of path selection.

[0091] Figure 7 is a degradation index sub-TLV 700 according to an embodiment of the present disclosure. The degradation index sub-TLV 700 includes a degradation sub-type field 702, a reserved field 704, a site ID field 706, and a site degradation field 708. The degradation sub-type field 702 is a two-octet size field that carries a value of a type code (TBD) indicating that the sub-TLV is a degradation index type. The reserved field 704 is a two-octet size field reserved for future use. The site ID field 706 is a two-octet size field that carries a site reference ID of a group of routes whose operating capacity is indicated by the operating capacity value carried in the BGP UPDATE message. The site degradation field 708 is a two-octet size field that carries an operating capacity value (e.g., between 1 and 100) representing the operating percentage of the site. For example, a value of 100 represents 100% or full capacity, 50 represents half capacity (eg, 50% degraded), 25 represents one quarter capacity, and 0 represents no capacity or a complete site failure (ie, the site is completely out of service).

[0092] In some embodiments, when the ingress router receives a BGP UPDATE message with the Demotion Index sub-TLV 700 and no additional routes from Router X, the site demotion value indicated in the Site Demotion field 708 is applied to all routes that have Router X as their next hop and are associated with the site reference ID specified in the Site ID field 706.

[0093] Figure 8 800 (e.g., a network node, a network router, a router, etc.). The network device 800 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the network device 800 can be an egress router (e.g., an edge gateway 106) or an ingress router (e.g., a C1 peer 104A to a CN peer 104N). The network device 800 includes an ingress port / ingress device 810 (also known as an upstream port) and a receiver unit (Rx) / receiving device 820 for receiving data; a processor, a logic unit or a central processing unit (CPU) / processing device 830 for processing data; a transmitter unit (Tx) / transmitting device 840 and an egress port / egress device 850 (also known as a downstream port) for sending data; and a memory / storage device 860 for storing data. The network device 800 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the inlet port / inlet device 810, the receiver unit / receiving device 820, the transmitter unit / transmitting device 840, and the outlet port / exit device 850 for the entry and exit of optical or electrical signals.

[0094] The processor / processing device 830 is implemented by hardware and software. The processor / processing device 830 can be implemented as one or more CPU chips, cores (for example, as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing device 830 communicates with the ingress port / ingress device 810, the receiver unit / receiving device 820, the transmitter unit / transmitting device 840, the egress port / egress device 850, and the memory / storage device 860. The processor / processing device 830 includes a site capacity degradation module 870. The site capacity degradation module 870 can implement the methods disclosed herein. Therefore, including the site capacity degradation module 870 provides substantial improvements to the functionality of the network device 800 and implements the transformation of the network device 800 to different states. Alternatively, the site capacity degradation module 870 is implemented as instructions stored in the memory / storage device 860 and executed by the processor / processing device 830.

[0095] The network device 800 may also include input and / or output (I / O) devices or I / O devices 880 for transmitting data to and from a user. The I / O devices or I / O devices 880 may include output devices, such as a display for displaying video data, a speaker for outputting audio data, etc. The I / O devices or I / O devices 880 may also include input devices, such as a keyboard, a mouse, a trackball, etc., and / or corresponding interfaces for interacting with such output devices.

[0096] The memory / storage device 860 includes one or more disks, tape drives, and solid-state hard disks, and can be used as an overflow data storage device to store such programs when such programs are selected for execution, as well as to store instructions and data read during program execution. The memory / storage device 860 can be volatile and / or non-volatile, and can be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0097] Although several embodiments have been provided in the present disclosure, it is understood that the disclosed systems and methods may be implemented in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are considered illustrative rather than restrictive, and are not intended to be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be omitted or not implemented.

[0098] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and shown as discrete or separate in various embodiments may be combined or integrated with other systems, components, techniques or methods. Other examples of changes, substitutions and modifications may be determined by those skilled in the art, and changes, substitutions and modifications may be made without departing from the spirit and scope disclosed herein.

[0099] Although several embodiments have been provided in the present disclosure, it should be understood that the disclosed systems and methods may be implemented in many other specific forms without departing from the spirit or scope of the present disclosure. For example, the disclosed embodiments include a computer program product including computer executable instructions stored on a non-transitory computer readable storage medium, which, when executed by a processor of a device, causes the device to perform the methods disclosed herein. Those skilled in the art will understand how to combine any or all of the above techniques in a variety of permutations and combinations.

[0100] It should be noted that the disclosed embodiments can be applied not only to 5G edge networks, but also to other environments, such as but not limited to storage clusters at remote sites, data centers, cloud DCs, container sets, and enterprise networks with a large number of device failures that cannot be detected from the source.

[0101] In addition, without departing from the scope of the present disclosure, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or discussed as coupled to each other, or directly coupled, or communicating with each other may be indirectly coupled or communicated electrically, mechanically or otherwise through some interface, device or intermediate component. Other examples of changes, substitutions and modifications may be determined by those skilled in the art, and changes, substitutions and modifications may be made without departing from the spirit and scope disclosed herein.

Claims

1. A method implemented by a gateway of a cloud data center, the method comprising: sending a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; determining that an operational capacity change affecting the set of routes within the site has occurred; as well as A second BGP UPDATE message is sent, where the second BGP UPDATE message includes the operation capacity information of the site reflecting the change in the operation capacity.

2. The method according to claim 1, wherein: The operating capacity information indicates an operating capacity percentage of the site.

3. The method according to any one of claims 1 to 2, wherein: The set of routes is all routes within the site.

4. The method according to any one of claims 1 to 3, wherein: The site reference ID is included in the site capability opaque extended community attribute in the first BGPUPDATE message.

5. The method according to any one of claims 1 to 3, wherein: The site reference ID is included in the metadata path attribute in the first BGPUPDATE message.

6. The method according to any one of claims 1 to 3, wherein: The operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

7. The method according to any one of claims 1 to 3, wherein: The operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

8. The method according to any one of claims 1 to 7, further comprising: monitoring subsequent operational capacity changes of said site; Determining that the subsequent operation capacity change occurs; as well as A subsequent BGP UPDATE message is sent, wherein the subsequent BGP UPDATE message includes subsequent operating capacity information of the site corresponding to the subsequent operating capacity change.

9. The method according to claim 8, wherein: The subsequent operating capacity change further reduces the operating capacity of the site.

10. The method according to claim 8, wherein: The subsequent operating capacity change increases the operating capacity of the site.

11. A method implemented by an ingress router of a cloud data center, the method comprising: receiving a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; appending the site reference ID to the set of routes in a routing table; receiving a second BGP UPDATE message, wherein the second BGP UPDATE message includes operating capacity information; selecting, based on the operational capacity information, a path for forwarding traffic corresponding to the set of routes; as well as Traffic for the selected service is forwarded along the path.

12. The method according to claim 11, wherein: Selecting a path for forwarding traffic corresponding to the set of routes includes: calculating a first cost for the path based on a plurality of factors including the operating capacity information; and The first cost of the path is compared to a second cost of a second path.

13. The method according to claim 12, wherein: The multiple factors include a load index, a capacity index, a network latency measurement, and a preference index.

14. The method according to any one of claims 11 to 13, wherein: Forwarding traffic for the selected service along the path includes performing a lookup of the set of routes in a forwarding information base (FIB) to obtain a destination prefix.

15. The method according to any one of claims 11 to 14, wherein: Forwarding traffic for the selected service along the path includes forwarding packets from the same flow to the same egress router.

16. The method according to any one of claims 11 to 15, wherein: The operating capacity information indicates an operating capacity percentage of the site.

17. The method according to any one of claims 11 to 16, wherein: The set of routes is all routes within the site.

18. The method according to any one of claims 11 to 17, wherein: The site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

19. The method according to any one of claims 11 to 17, wherein: The site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

20. The method according to any one of claims 11 to 19, wherein: The operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

21. The method according to any one of claims 11 to 19, wherein: The operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

22. A gateway router of a cloud data center, the gateway comprising: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the gateway to: sending a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; determining that an operational capacity change affecting the set of routes within the site has occurred; as well as A second BGP UPDATE message is sent, where the second BGP UPDATE message includes the operation capacity information of the site reflecting the change in the operation capacity.

23. The gateway router according to claim 22, wherein: The operating capacity information indicates an operating capacity percentage of the site.

24. The gateway router according to any one of claims 22 to 23, wherein: The set of routes is all routes within the site.

25. The gateway router according to any one of claims 22 to 24, wherein: The site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

26. The gateway router according to any one of claims 22 to 24, wherein: The site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

27. The gateway router according to any one of claims 22 to 24, wherein: The operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

28. The gateway router according to any one of claims 22 to 24, wherein: The operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

29. The gateway router according to any one of claims 22 to 28, wherein: The one or more processors are configured to execute the instructions to further cause the gateway router to: monitoring subsequent operational capacity changes of said site; determining that the subsequent operating capacity change occurs; and A subsequent BGP UPDATE message is sent, wherein the subsequent BGP UPDATE message includes subsequent operating capacity information of the site corresponding to the subsequent capacity change.

30. The gateway router according to claim 29, wherein: The subsequent operating capacity change further reduces the operating capacity of the site.

31. The gateway router according to claim 29, wherein: The subsequent operating capacity change increases the operating capacity of the site.

32. A router, comprising: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the instructions to cause the router to: receiving a first Border Gateway Protocol (BGP) UPDATE message, the first BGP UPDATE message including a site reference identifier (ID) corresponding to a set of routes within a site of the cloud data center; appending the site reference ID to the set of routes in a routing table; receiving a second BGP UPDATE message, wherein the second BGP UPDATE message includes operating capacity information; selecting, based on the operational capacity information, a path for forwarding traffic corresponding to the set of routes; as well as Traffic for the selected service is forwarded along the path.

33. The router according to claim 32, wherein: Selecting a path for forwarding traffic corresponding to the set of routes includes: calculating a first cost for the path based on a plurality of factors including the operating capacity information; and The first cost of the path is compared to a second cost of a second path.

34. The router according to claim 33, wherein: The multiple factors include a load index, a capacity index, a network latency measurement, and a preference index.

35. The router according to any one of claims 32 to 34, wherein: Forwarding traffic for the selected service along the path includes performing a lookup of the set of routes in a forwarding information base (FIB) to obtain a destination prefix.

36. The router according to any one of claims 32 to 35, wherein: Forwarding traffic for the selected service along the path includes forwarding packets from the same flow to the same egress router.

37. The router according to any one of claims 32 to 36, wherein: The operating capacity information indicates an operating capacity percentage of the site.

38. The router according to any one of claims 32 to 37, wherein: The set of routes is all routes within the site.

39. The router according to any one of claims 32 to 38, wherein: The site reference ID is included in a site capacity opaque extended community attribute in the first BGP UPDATE message.

40. The router according to any one of claims 32 to 38, wherein: The site reference ID is included in a metadata path attribute in the first BGP UPDATE message.

41. The router according to any one of claims 32 to 40, wherein: The operating capacity information is included in a site capacity opaque extended community attribute in the second BGP UPDATE message.

42. The router according to any one of claims 32 to 40, wherein: The operating capacity information is included in a metadata path attribute in the second BGP UPDATE message.

43. A computer program product comprising computer executable instructions stored on a non-transitory computer readable storage medium, which, when executed by a processor of an apparatus, causes the apparatus to perform the method of any one of claims 1 to 21.

44. A method implemented by a gateway of a cloud data center, the method comprising: Determine that a change in the site's operating capacity has occurred; as well as A BGP UPDATE message is sent, wherein the BGP UPDATE message includes a site reference identifier (ID) of the site and operating capacity information, wherein the site reference ID corresponds to a set of routes within the site of the cloud data center.

45. A method implemented by an ingress router of a cloud data center, the method comprising: receiving a border gateway protocol (BGP) UPDATE message, the BGP UPDATE message including a site reference identifier (ID) and operational capacity information, wherein the site reference ID corresponds to a set of routes within a site of the cloud data center; appending the site reference ID to the set of routes in a routing table; selecting, based on the operational capacity information, a path for forwarding traffic corresponding to the set of routes; and Traffic for the selected service is forwarded along the path.

46. ​​A network device comprising means for performing the method according to any one of claims 1 to 21 and 44 to 45.

47. A computer program product comprising computer executable instructions stored on a non-transitory computer readable storage medium, which, when executed by one or more processors of a device, causes the device to perform the method of any one of claims 1 to 21 and 44 to 45.

Images