Service chaining in multi-architecture cloud networks

Abstract

A technology for routing data packets through service chains within and between public cloud networks of a multi-cloud structure. A router in a network (e.g., a public cloud network) receives a data packet from a node in the network through a segment of the network. Based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the segment of the network receiving the data packet, the router determines a next node in the network to which the data packet is to be forwarded. The router can then forward the data packet to the next node through another segment of the network and then receive the data packet from the next node through another segment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This patent application claims priority to U.S. utility patent application serial number 16 / 801,500, filed on February 26, 2020, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure generally relates to routing data packets through service chaining within and between public cloud networks in a multi-cloud architecture. Background Art

[0004] As the proliferation and use of devices with Internet accessibility continues to increase, the demand for Internet services and content continues to increase. Providers of Internet services and content continue to scale the computing resources required to serve the growing user demand without falling short of user performance expectations. For example, providers typically utilize large and complex data centers to manage network and content demands from users. Data centers typically include server farms that host workloads that support services and content, and also include network equipment (e.g., switches and routers) to route traffic through the data center and enforce security policies.

[0005] Typically, the networks of these data centers are one of the following two types: private networks (e.g., local networks) owned by entities (e.g., enterprises or organizations); and public cloud networks owned by cloud providers, which provide computing resources for users to purchase. Enterprises often own, maintain and operate local computing resource networks to provide Internet services and / or content to users or customers. However, as mentioned above, it may become difficult to meet the growing demand for computing resources while maintaining acceptable performance for users. Therefore, private entities often purchase or otherwise order the use of computing resources and services from public cloud providers. For example, a cloud provider can create a virtual private cloud (also referred to as a "private virtual network" herein) on a public cloud and connect the virtual private cloud or network to a local network to increase the available computing resources and capabilities of the enterprise. Therefore, an enterprise can interconnect its private, or local network's data center with a remote, cloud-based data center hosted on a public cloud, thereby extending its private network.

[0006] However, because local networks and public cloud networks are typically developed and maintained by different entities, there is a lack of uniformity in policy management and configuration parameters between data centers in local networks and public cloud networks. This lack of consistency can significantly limit an enterprise's ability to integrate its local network with a public cloud network (e.g., by failing to apply consistent policies, configuration parameters, routing models, etc.). Various entities have developed software-defined networking (SDN) and data center management solutions that can translate an enterprise or organization's intent from its local network into its virtual private cloud network for applications or services deployed across a multi-cloud structure or environment. Therefore, these multi-cloud SDN solutions must constantly adapt to changes occurring in local networks and public cloud networks while maintaining the business and user intent of the enterprise or organization to supplement its local network with computing resources from the public cloud network.

[0007] For example, enterprises that manage local networks in data centers often isolate and segment their local networks to improve the scalability, resilience, and security of their local networks. In order to meet the entity's needs for isolation and segmentation, endpoints in the local network can be grouped into endpoint groups (Endpoint Grouping, EPG), for example, using isolated virtual networks (the isolated virtual networks can be used to containerize endpoints to allow individualized routing models, policy models, etc. to be applied across endpoints in EPGs). Typically, each subnet in an EPG or other virtual endpoint grouping is associated with an address range that can be defined in a routing table for controlling subnet routing. Since a large number of routing tables are implemented to route traffic through the local network, the entity that manages the local network utilizes virtual routing and forwarding (VRF) technology so that multiple instances of the VRF routing table can exist in one router and work simultaneously. Therefore, the subnets of the EPG in the entity's local network are associated with their respective VRF routing tables, and the router is able to store and utilize multiple instances of the VRF routing table at the same time.

[0008] A service can be initiated and inserted into a cloud network and / or between cloud networks. Examples of such services include, for example, firewalls, application load balancers (ALBs), mirroring / analysis services, and the like. Such services are often deployed as service chains inserted into cloud networks. Such service chains should support multiple nodes in the chain in a flexible manner. Services should be placed in the chain in any order, and traffic may or may not be sent to an Internet Protocol (IP) address hosted by any of these services in the chain. High availability should be provided for each service node in the service chain. Therefore, each service node of the service chain should include multiple instances of the service. It is desirable to have specific flows (e.g., a combination of source to destination) for fine control. It is also desirable to replicate service chains across regions in order to meet traffic in any part of the public cloud in a cost-effective manner. In addition, based on traffic, it is desirable to be able to elastically scale service nodes so that sufficient services can be provided for traffic through service nodes. Currently, cloud providers themselves do not provide complex policy redirection or routing to allow these previously described functions. Routing is based only on the destination, and the context of the service chain legs that have been traversed is not available in the data packets traversing the service chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The following detailed description is made with reference to the accompanying drawings. In the drawings, the leftmost digit(s) of a reference number indicates the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical items. The systems depicted in the drawings are not drawn to scale, and the components in the drawings may not be drawn to scale with each other.

[0010] Figures 1A-1C Schematically illustrates a multi-cloud structure including two public cloud networks;

[0011] Figure 2 The diagram shows the Figures 1A-1C A flowchart of an example method for routing data packets through a service chain within a public cloud network of a multi-cloud structure;

[0012] Figure 3 illustrates a computing system diagram illustrating a configuration of a data center that may be used to implement aspects of the technology disclosed herein;

[0013] Figure 4 is a computer architecture diagram showing an illustrative computer hardware architecture for implementing a server device that can be used to implement aspects of the various techniques presented herein. DETAILED DESCRIPTION

[0014] Overview

[0015] The present disclosure describes a method for routing data packets through a service chain within and between public cloud networks of a multi-cloud structure. The method may include: a router in a network (e.g., a public cloud network) receives a data packet from a first node in the network through a first segment of the network. Based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the first segment of the network, the router determines a second node in the network to which the data packet will be forwarded. Based at least in part on the determination, the router forwards the data packet to the second node through a second segment of the network. The router receives a data packet from the second node through the second segment. Based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the second segment of the network, the router may determine a third node in the network to which the data packet will be forwarded. Based at least in part on the determination, the router may forward the data packet to a third node through a third segment of the network.

[0016] In addition, the network may include multiple subnets, for example, the network may be in the form of a multi-cloud network including multiple public cloud networks and / or multiple regions or partitions of a public cloud network. Thus, the router may be a first router and part of a first subnet, and the method further includes: the first router receives a data packet from a second node via a second segment. Based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the second segment of the network, the first router may determine a third node in the network to which the data packet will be forwarded, the third node including a service. The first router may receive any IP addresses of other nodes in one or more subnets for including services from one or more routers in one or more subnets of the multiple subnets. The router may select a fourth node in a second subnet of the one or more subnets, wherein the fourth node includes a service. The first router may forward the data packet to a second router in a second subnet via the third segment of the network. The second router may forward the data packet to a fourth node via a fourth segment of the second subnet. The second router may receive the data packet from the fourth node via the fourth segment. The second router may determine a next destination to which the data packet will be forwarded. The second router may forward the data packet to the next destination.

[0017] Furthermore, the techniques described herein may be performed by a system and / or device having a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors, perform the above-described methods.

[0018] Exemplary Embodiments

[0019] As described above, businesses and other organizations may own, maintain, and operate local networks of computing resources for users or customers (and also for supporting the internal computing needs of running their organizations). However, due to the difficulty in meeting the growing demand for computing resources while maintaining acceptable performance for users, these businesses may order the use of computing resources and services from public cloud providers in other ways. For example, a cloud provider may create a virtual private cloud (also referred to herein as a "private virtual network") on a public cloud and connect the virtual private cloud or network to a local network to increase the computing resources and capabilities available to the enterprise. Therefore, an enterprise can interconnect its private, or local network's data center with a remote, cloud-based data center hosted on a public cloud, thereby extending its private network.

[0020] However, there is a lack of consistency between local networks and public cloud networks across various dimensions (e.g., policy management, configuration parameters, etc.), which may significantly limit the ability of enterprises to integrate their local networks with public cloud networks (e.g., by failing to apply consistent policies, configuration parameters, routing models, etc.). Various SDN solutions have been developed to transform the intentions of an enterprise or organization from its local network to its virtual private cloud network for applications or services deployed across a multi-cloud structure or environment. For example, Cisco's software-defined networking and data center management solution (Application-Cenric Infrastructure (ACI)) provides a comprehensive solution for automated network connectivity, consistent policy management, and simplified operations for multi-cloud environments. The Cisco Cloud ACI solution allows enterprises to extend their local networks to various public clouds, such as Amazon Web Services (AWS), Google Cloud, Microsoft Azure, etc. The Cisco Cloud ACI solution provides an architectural approach for interconnecting and managing multiple regions and / or sites, for example by defining inter-cloud policies, providing a scalable architecture with full fault domain isolation and change domain isolation, and ensuring that problems cannot cascade and shut down the entire distributed environment.

[0021] SDN solutions (e.g., Cisco Cloud ACI) face various challenges when attempting to interconnect a data center's local network with a public cloud network in the data center. For example, cloud providers may impose different restrictions on networking configurations and policies, routing and policy models, and / or other restrictions on their public clouds. These restrictions may differ from the restrictions or permissions implemented by an enterprise that has developed its data center's local network. However, in order to interconnect local networks with public cloud networks, SDN solutions in the multi-cloud fabric space must typically reconcile these differences to seamlessly extend local networks across public cloud networks.

[0022] As an example, a VPC (virtual private cloud in AWS) or Vnet (virtual network in Microsoft Azure) in a public cloud network usually needs to be connected to a router to route traffic between endpoints in the VPC or Vnet of the public cloud network and endpoints (or other devices) in the local network. SDN solutions attempt to automate this connection between the local network and the public cloud network (for example, by using a solution given by a public cloud network provider). For example, AWS provides a transit gateway (TGW) for automating this connection. Typically, a TGW (or just a gateway) includes a distributed router connected to multiple VPCs. Instead of establishing a VPN connection from each VPC to a router, the gateway is able to connect multiple VPCs to a single gateway and connect their local networks to a single gateway. Connecting a VPN to each VPC is a cumbersome and costly task, and the transit gateway provides a single connection from a local network to multiple VPCs in the AWS public cloud, which has a relatively high bandwidth compared to a VPN connection.

[0023] While these gateways are advantageous for a variety of reasons, the different limitations of using these gateways are surface issues that SDN controllers need to address when automating interconnections across multi-cloud architectures. For example, a gateway may require that each VPC connected to a specific gateway have no overlapping subnets. In other words, all VPCs connected to a given gateway may need to have non-overlapping unique address spaces or ranges, such as Classless Inter-domain Routing (CIDR) blocks. However, enterprises that manage on-premises networks often define address ranges, such as VRFs, with overlapping address spaces (e.g., overlapping prefixes). In fact, one of the advantages of VRFs is that it allows overlapping subnets while providing segmentation and isolation for network paths.

[0024] One of the infrastructure blocks in a cloud networking / transport solution (e.g., Cisco Cloud ACI) is a cloud router that performs fine-grained routing between VPCs / VNets, or between a VPC / VNet and an on-premises network. The term VPC is used herein to describe examples, but it should be understood that a VNet can actually be used in place of a VPC based on a public cloud network, and thus, the use of the term VPC is not meant to be limiting.

[0025] In a configuration, a distributed cloud router may be provided that is used as a networking block that connects a source VPC and a destination VPC of a data packet, and also connects services provided within the VPC (e.g., a service chain between the source VPC and the destination VPC). The distributed cloud router may span multiple regions of a public cloud network, and may span multiple public cloud networks (e.g., AWS cloud networks and Microsoft Azure public cloud networks). Each instance of the cloud router may run or operate in one infra VPC within one region.

[0026] In a configuration, data packet traffic (referred to herein as "traffic") can be routed between VPCs through a tunnel running between a user VPC (e.g., a source VPC and a destination VPC) and an infra VPC. Similarly, traffic between any two networks (cloud to cloud, or cloud to local) can also be routed via a tunnel. In addition, in a configuration, traffic can be routed from one infra VPC to another infra VPC through a tunnel. For example, an Internet Protocol Security (IPSec) / Generic Routing Encapsulation (GRE) tunnel can span between two CSR instances across regions.

[0027] In the configuration, different tunnels are provided within each CSR (cloud router) to represent different branches / combinations (e.g., segments) of the service chain. Therefore, based on the ingress tunnel interface of the data packet in the CSR, the previous node traversed in the service chain can be inferred, so that the data packet can be directed to the next service node in the service chain. Therefore, the same data packet can traverse the same CSR multiple times. The data packet can be forwarded to a different service node in the service chain each time without changing any part of the original data packet (e.g., source IP address and destination IP address).

[0028] For example, assume that a service chain includes a service node A and a service node B (e.g., a service A VPC and a service B VPC) in the service chain. A tunnel is created for each service node in the CSR. For example, tunnel one may be located between the source user VPC and the CSR (so there is no previous service). Tunnel two may extend between service node A and the CSR, for example, the previous service is service A. Tunnel three may extend between service node B and the CSR, for example, the previous service is service B. In the configuration, when a data packet is returned from the service to the CSR, the previous service is determined. Tunnel four may extend between the CSR and the destination user VPC, for example, no further service is performed before delivery to the destination VPC. Each of the tunnels may be in a different VRF so that the current branch may be identified as a segment (or VRF) in the CSR. This may be helpful to customize a forwarding lookup table, a routing table for the same pair of source IP addresses and destination IP addresses (e.g., a combination of source IP addresses and destination IP addresses) of a data packet, which jumps through the same CSR to reach different services without any changes to the data packet address information. Alternatively, all tunnels in a CSR may belong to the same VRF, but the policy-based routing (PBR) rules may differ based on the (tunnel, source, destination) combination.

[0029] Therefore, a routing table can be defined at the CSR, which provides rules for data packets from the first user VPC (source) and the second user VPC (destination) combination. For example, based on the first source VPC and destination VPC combination, the CSR knows to route the data packet to the destination VPC through the service chain including services A and B on the way, and based on the second source VPC and destination VPC combination, the CSR knows to route the data packet to the destination VPC through the service chain including services A, B, and C on the way. Any number of services can be included in a service chain.

[0030] Thus, for example, based on the first source VPC and destination VPC combination, data packets received by the CSR from the source VPC via tunnel one can be routed to the service A node using tunnel two. After service A is executed, the data packets can be routed back to the CSR via tunnel two. Since the CSR receives the data packets via tunnel two, based on the routing table (e.g., source VPC and destination VPC), the CSR routes the data packets on tunnel three to the service B node. After service B is executed, the data packets are routed back to the CSR via tunnel three. Based on the routing table, the CSR knows to now route the data packets to the target user VPC via tunnel four because there are no remaining services that need to be executed.

[0031] In a configuration, multiple regions of a public cloud network may each have the same service VPC, which may also include multiple instances that provide services. For example, a first public cloud network region and a second public cloud network region may each have a service A VPC. The service A VPCs of the two regions may carry the same anycast IP address. The CSR of the region may advertise the anycast IP address to other CSRs and other regions via the Border Gateway Protocol (BGP). The anycast IP of the service node is the destination of the PBR rule. When multiple cloud router CSRs advertise anycast IPs in regions of (one or more) public cloud networks, in a configuration, the service VPC closest to the CSR (receiving the broadcast) may be selected. Typically, if the service A VPC of the same region receiving the broadcast CSR is operating, the service A VPC is selected (e.g., a locally connected VPC in the same partition / region receiving the CSR). However, if the service VPC fails or is overloaded, the anycast IP address of the service A VPC of the adjacent region (or adjacent partition) may be automatically selected.

[0032] Thus, the data packet can be routed from the first CSR (receiving CSR) to the second CSR (CSR in the adjacent region) via a tunnel. The second CSR can then route the data packet to the service A VPC along the appropriate tunnel in the corresponding region. When the service is completed, the data packet can be routed back to the second CSR, which can forward the data packet back to the first CSR, which can then continue to move the data packet to the destination user VPC through the service chain. In the configuration, the second CSR can complete the process of moving the data packet to the destination VPC through the service chain within its region.

[0033] Therefore, since all CSRs in various regions and / or partitions advertise the same anycast IP, e.g., service A, even if the local instance of service A fails, the CSRs can forward data packets to the remote instance of service A in another region / partition via the CSRs in the region / partition.

[0034] To provide service node load balancing and redundancy, multiple instances of a service can be instantiated in a service node. For example, a service A VPC can include multiple instances that can provide service A. If one of the instances fails, the other instances can provide service A. In addition, to provide load balancing within a service A VPC, a CSR can route data packets to a specific service A instance.

[0035] In the configuration, autoscaling can be provided by adding new instances in either of the service VPCs. Routes (tunnels) can be created for new instances from the CSR in the infra VPC. Additionally, BGP weights can be controlled to favor one instance over another in a remote location (e.g., a different region) so that traffic can be load balanced accordingly. BGP weights can cater to the health or load of a specific instance of a service node.

[0036] Certain embodiments and examples of the present disclosure will now be described more fully below with reference to the accompanying drawings, in which various aspects are shown. However, various aspects may be implemented in many different forms, and various aspects should not be construed as being limited to the implementations set forth herein. As described herein, the present disclosure encompasses variations of the embodiments described herein. Throughout the text, similar numbers represent similar elements.

[0037] Figure 1A A multi-cloud structure 100 is schematically illustrated that includes two public cloud networks 102a and 102b (collectively referred to as public cloud networks 102). The public cloud networks 102a, 102b include two regions 104a, 104b and 104c, 104d (collectively referred to as regions 104), respectively. The public cloud network 102 may include more or fewer regions 104. Each region 104 includes a VPC / Vnet 106 that can be configured as a user VPC / VNet or a service VPC / VNet. For simplicity, the VPC / Vnet 106 is collectively referred to as VPC 106 herein. The multi-cloud structure 100 also includes a local network 108. In the configuration, the public cloud networks 102a, 102b, the regions 104a, 104b, 104c, 104d and / or the local network 108 can be considered as subnets.

[0038] The multi-cloud fabric 100 also includes a multi-site controller 110 (e.g., a multi-site application policy infrastructure controller (APIC)) that communicates with cloud controllers 112a, 112b (e.g., cloud APIC) in the public cloud networks 102a, 102b, and controllers in the local network 108. The multi-site controller 112 can work with the controller to manage and enforce policies and configurations on both the public cloud networks 102a, 102b and the local network 108. The multi-site controller 112 can implement the same policy model in the public cloud networks 102a, 102b and the local network 108 (e.g., through conversion), which can be based on a specific SDN or data center solution (e.g., Cisco's ACI). For example, the multi-site controller 112 can implement VRFs, endpoints (EPs), endpoint groups (EPGs), and associated policies in the public cloud networks 106a, 106b and the local network 108.

[0039] In the configuration, data packet traffic (referred to herein as "traffic") may be routed between VPCs 106 via tunnel 114, which operates between user VPCs 106a-106h (e.g., source and destination VPCs) and infra VPCs 116a-116d (collectively, infra VPCs 116), which include CSRs 118a-118d (collectively, CSRs 118). Similarly, traffic may be routed between service VPCs 106i-106p and infra VPCs 116a-116d to provide service chaining. Traffic between any two networks (cloud to cloud 102a-102b or cloud to local 102a, 102b-108) may also be routed via tunnels 120a-120c. Additionally, in the configuration, traffic may be routed from one infra VPC 116 (or infra VNET) to another infra VPC 116 (or infra VNET) via tunnels 120a-120c. For example, an Internet Protocol Security (IPSec) / Generic Routing Encapsulation (GRE) tunnel may span between two CSR instances 118 across a region 104. As will be discussed further herein, CSRs 118a-118d include routing tables 122a-122d, respectively.

[0040] Figure 1B A larger scale of region 104a is schematically illustrated. As previously described, different tunnels are provided within each CSR (cloud router) to represent different branches / combinations (e.g., segments) of the service chain. Specifically, tunnel 114a couples infra VPC 116a and CSR 118a with user VPC 106a, tunnel 114b couples infra VPC 116a and CSR 118a with user VPC 106b, tunnel 114i couples infra VPC 116a and CSR 118a with service VPC 106i, and tunnel 114j couples infra VPC 116a and CSR 118a with service VPC 106j.

[0041] Thus, based on the ingress tunnel interface of the data packet in CSR 118a, the previous node traversed in the service chain (e.g., service VPC 106i) can be inferred, and therefore, the data packet can be directed to the next service node in the service chain (e.g., service VPC 106j). Thus, the same data packet can traverse CSR 118a multiple times. Each time, the data packet can be forwarded to a different service node in the service chain without changing any part of the original data packet (e.g., source IP address and destination IP address).

[0042] For example, assume that a service chain includes a service VPC 106i that executes service A in the service chain, and a service VPC 106j that executes service B in the service chain. Tunnels 114a, 114b, 114i, and 114j are created in CSR 118a for each VPC 106a, 106b, 106i, and 106j. For example, tunnel 114a between user VPC 106a (which user VPC 106a may be the source of the data packet) and CSR 118a (therefore there is no previous service in the service chain). Tunnel 114i may extend between service VPC 106i and CSR 118a, for example, the previous service is service A. Tunnel 114j may extend between service VPC 106j and CSR 118a, for example, the previous service is service B. Tunnel 114b may extend between CSR 118a and user VPC 106b (which may be the destination of the data packet), e.g., without performing further services before delivery to the destination VPC. Each of tunnels 114a, 114b, 114i, and 114j may be in a different VRF, such that the current leg may be identified as a segment (or VRF) in CSR 118a. This may help to customize the forwarding lookup table (e.g., routing table 122) for the same pair of source IP address and destination IP address (e.g., source IP address and destination IP address combination) of the data packet, which jumps from CSR 118a to different services without changing the data packet address information. Alternatively, all tunnels 114 in CSR 118a may belong to the same VRF, but the policy-based routing (PBR) rules may be different based on the (tunnel, source, destination) combination. As is well known, PBR is a technique for making routing decisions based on policies set by network administrators. Specifically, when a router receives a packet, the router will usually decide where to forward the packet based on the destination address in the packet, and then use the destination address to find an entry in the routing table. However, in some cases, it may be necessary to forward packets based on other criteria. For example, a network administrator may want to forward packets based on the source address (rather than the destination address). This allows packets originating from different sources to be routed to different networks (even if the destination is the same), and this may be useful when interconnecting multiple private networks. PBR can also be based on: the size of the packet, the protocol of the payload, or other information available in the packet header or payload.

[0043] Therefore, a routing table 122a may be defined at CSR 118a that provides rules for data packets from a combination of a first user VPC (source) 106a IP address and a second user VPC (destination) 106b IP address. For example, based on the source VPC 106a and destination VPC 106b IP address combination, CSR 118a knows that data packets from VPC 106a to VPC 106b are routed through a service chain that includes service VPC 106i and service VPC 106j (the service VPC 106i and service VPC 106j are on the way to the destination VPC). Although only two service VPCs 106i, 106j are shown, any number of service VPCs 106 may be included in the service chain.

[0044] Thus, for example, based on the combination of source VPC 106a and destination VPC 106b, a data packet (which is received by CSR 118a from source VPC 106a via tunnel 114a) may be routed to service VPC 106i using tunnel 114i. After executing service A, the data packet may be routed back to CSR 118a via tunnel 114i. Since CSR 118a receives the data packet via tunnel 114i, based on routing table 122 (e.g., the combination of source VPC 106a and destination VPC 106b in routing table 122), CSR 118a routes the data packet on tunnel 114j to service VPC 106j. After executing service B, the data packet is routed back to CSR 118a via tunnel 114j. Based on routing table 122, CSR 118a knows to now route the data packet via tunnel 114b to destination customer VPC 106b via tunnel four since there are no services left to perform.

[0045] Reference Figure 1C In a configuration, multiple regions 104a, 104b of a public cloud network 102a and / or multiple regions 104c, 104d of a public cloud network 102b may each have some or all of the same service VPCs (e.g., service VPCs 106i, 106j that provide services A and B, respectively). The service VPCs 106a, 106j may also include multiple instances that provide services A and B, respectively. In a configuration, the service VPC 106i of the regions 104a-104d may carry the same anycast IP address A, and the service VPC 106j of the regions 104a-104d may carry the same anycast IP address B. The CSR 118 of a region may advertise the anycast IP address to other CSRs 118 and other regions 104 via BGP.

[0046] As an example, anycast IP address A of service VPC 106i may be broadcast by CRS 118 and be the destination of the PBR rule. When multiple CSRs 118 advertise anycast IP address A in regions 104 of public cloud networks 102a, 102b, in a configuration, a service VPC 106i that is closest to the CSR (e.g., CSR 118a) receiving the broadcast may be selected. Typically, if a service VPC 106i in the same region 104a receiving the broadcasted CSR 118a is operational, that service VPC 106i is selected, e.g., a locally connected VPC 106i in the same partition / region 104a receiving the CSR 118a. However, if service VPC 106i fails or is overloaded, anycast IP address A of service VPC 106i in an adjacent region (or adjacent partition) (e.g., region 104b) may be automatically selected.

[0047] Therefore, refer to Figure 1C , the data packet may be routed from CSR 118a (receiving CSR) to CSR 118b (the CSR of the adjacent region) of region 104b via tunnel 120a. CSR 118b may then route the data packet to service VPC 106i2 along the appropriate tunnel (e.g., tunnel 114i2 in the corresponding region 104b). When service A is completed, the data packet may be routed back to CSR 118b, which may forward the data packet back to CSR 118a. CSR 118a may then continue to move the data packet through the service chain to, for example, service VPC 106j1 as previously described, and on to destination user VPC 106b. In the configuration, CSR 118b may complete the process of moving the data packet (as previously described) through the service chain within its region (e.g., service VPC 106j2 in region 104b), and then the data packet returns to CSR 118a (which forwards the data packet to destination VPC 106b in region 104a).

[0048] Therefore, because all CSRs 118 in each region 104 advertise the same anycast IP address (e.g., IP service address A), even if the local instance of service VPC 106i providing service A fails, the CSR can forward data packets to the remote service VPC, which provides service A in another region / partition via the CSR in the region / partition.

[0049] In order to provide load balancing and redundancy to the service VPC 106, multiple instances of a service may be instantiated in the service VPC 106. For example, the service VPC 106i may include multiple instances that may provide service A. If one of the instances fails, the other instances may provide service A. In addition, in order to provide load balancing within the service VPC 106i, the CSR 118a may route data packets to a specific service A instance in the VPC 106i. The cloud controller 112 may manage load balancing between instances within the service VPC. In a configuration, automatic scaling may be provided by instantiating a new service instance in any one of the service VPCs 106 by the cloud controller 112. A route (tunnel) for the new instance may be created by the cloud controller 112, which comes from the CSR 118 in the corresponding infra VPC 114. In addition, the BGP weight may be controlled by the cloud controller 112 to favor one instance or another in the service VPC 106, so that traffic may be load balanced accordingly. For example, if a first instance in VPC 106i in region 104a is overloaded, the BGP weight can be adjusted by cloud controller 112a to favor a second instance that is not as loaded. The BGP weight can be tailored to the health or load of a particular instance within service VPC 106.

[0050] In addition, the cloud controller 112 can load balance between service VPCs 106 across regions 104 and / or networks 102. For example, if a local service VPC 106i in region 104a is overloaded, the BGP weight can be adjusted by the cloud controller 112a so that BGP favors a remote service VPC 106i in another region (e.g., region 104b) that is not as loaded.

[0051] Figure 2 A flow chart of an example method 200 is illustrated, and aspects of the functionality that may be provided at least in part by: Figures 1A-1C One or more devices in the multi-cloud architecture 100 described herein may be executed. Figure 2 The logical operations described may be implemented as: (1) a series of computer-implemented acts or program modules running on a computing system, and / or (2) interconnected machine logic circuits or circuit modules within a computing system.

[0052] The implementation of the various components described herein is a matter of choice, the choice depending on the performance and other requirements of the computing system. Therefore, the logical operations described herein are referred to (variously) as operations, structural devices, actions, or modules. These operations, structural devices, actions, and modules may be implemented in software, firmware, dedicated digital logic, and any combination thereof. It should also be understood that more than one operation may be performed. Figure 2 The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention. The present invention also can be used to describe the present invention in an embodiment of the present invention.

[0053] Figure 2 A flow chart of an example method 200 is illustrated for routing data packets through service links within a public cloud network of a multi-cloud structure. In some examples, the method 200 can be performed by a system including one or more processors and one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform the method 200.

[0054] At 202, a router in a network receives a data packet over a first segment of the network. At 204, the router determines a second node in the network to which the data packet is to be forwarded based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the first segment of the network. At 206, based at least in part on the determination, the router forwards the data packet to the second node over a second segment of the network.

[0055] At 208, the router receives a data packet from the second node via the second segment. At 210, the router determines a third node in the network to which the data packet is to be forwarded based at least in part on (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of the second segment of the network. At 212, based at least in part on the determination, the router forwards the data packet to the third node via the third segment of the network.

[0056] Figure 3 is a computing system diagram illustrating the configuration of a data center 300 , which may be used to implement aspects of the techniques disclosed herein. Figure 3 The example data center 300 shown in includes several server computers 302A-302F (which may be referred to herein as a single "server computer 302" (in the singular) or multiple "server computers 402" (in the plural) for providing computing resources. In some examples, the resources and / or server computers 302 may include or correspond to the VPC 106 described herein.

[0057] The server computer 302 can be a standard tower, rack or blade server computer, which is appropriately configured to provide the computing resources described herein. As described above, the computing resources provided by the cloud computing networks 102a and 102b can be data processing resources, such as VM instances or hardware computing systems, database clusters, computing clusters, storage clusters, data storage resources, database resources, networking resources, etc. Some servers 302 can also be configured to execute resource managers that can instantiate and / or manage computing resources. For example, in the case of VM instances, the resource manager can be a hypervisor or another type of program configured to allow multiple VM instances to be executed on a single server computer 302. The server computers 302 in the data center 300 can also be configured to provide network services and other types of services.

[0058] exist Figure 3 In the example data center 300 shown, an appropriate LAN 308 is also used to interconnect the server computers 302A-302F. It should be understood that the configuration and network topology described herein have been greatly simplified, and that many more computing systems, software components, networks, and networking devices may be utilized to interconnect the various computing systems disclosed herein and provide the functionality described above. Appropriate load balancing devices or other types of network infrastructure components may also be used to balance the load between the data centers 300, between each server computer 302A-302F in each data center 300, and potentially between the computing resources in each server computer 302. It should be understood that reference Figure 3 The described configuration of data center 300 is illustrative only, and other implementations may be utilized.

[0059] In some examples, the server computers 302 can each execute one or more virtual resources supporting services or applications (e.g., provided across a collection or cluster of servers 302). The virtual resources on each server computer 302 can support a single application or service or multiple applications or services (for one or more users).

[0060] In some cases, cloud computing networks 102a and 102b may provide computing resources, such as application containers, VM instances, and storage, either permanently or on demand. Among other types of functions, the computing resources provided by the cloud computing network may be used to implement the various services described above. The computing resources provided by the cloud computing network may include various types of computing resources, such as data processing resources like application containers and VM instances, data storage resources, networking resources, data communication resources, network services, etc.

[0061] Each type of computing resource provided by a cloud computing network can be general or can be used in a variety of specific configurations. For example, in many different configurations, data processing resources can be used as physical computers or VM instances. VM instances can be configured to execute applications, including web servers, application servers, media servers, database servers, some or all of the above network services and / or other types of programs. Data storage resources can include file storage devices, block storage devices, etc. The cloud computing network can also be configured to provide other types of computing resources not specifically mentioned herein.

[0062] In one embodiment, the computing resources provided by the cloud computing network may be enabled by one or more data centers 300 (which may be referred to herein as a single "data center 300" (in the singular) or multiple "data centers 300" (in the plural)). A data center 300 is a facility for housing and operating computer systems and associated components. A data center 300 typically includes redundant and backup power, communications, cooling, and security systems. Data centers 300 may also be located in geographically different locations. Figure 4 One exemplary embodiment of a server computer 302 for implementing the techniques disclosed herein is described.

[0063] Figure 4 An example computer architecture is shown for a server computer 302 capable of executing program components for implementing the functionality described above. Figure 4 The computer architecture shown shows a traditional server computer, workstation, desktop computer, laptop computer, tablet computer, network device, e-reader, smart phone or other computing device, and the computer architecture can be used to execute any software components proposed in this article.In some examples, server computer 302 can correspond to physical devices or resources described in this article.

[0064] The server computer 302 includes a baseboard 402 or "motherboard," which is a printed circuit board to which multiple components or devices may be connected via a system bus or other electrical communication paths. In one illustrative configuration, one or more central processing units ("CPUs") 404 operate in conjunction with a chipset 406. The CPU 404 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of the server computer 302.

[0065] The CPU 404 performs operations by manipulating switching elements from one discrete physical state to the next, which distinguish and change these states. Switching elements typically include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide output states based on logical combinations of the states of one or more other switching elements, such as logic gates. These basic switching elements can be combined to create more complex logic circuits, including registers, adders-subtractors, arithmetic logic units, floating point units, and the like.

[0066] Chipset 406 provides an interface between CPU 404 and the rest of the components and devices on baseboard 402. Chipset 406 may provide an interface to RAM 408, which is used as main memory in server computer 302. Chipset 406 may also provide an interface to a computer-readable storage medium such as read-only memory ("ROM") 410 or non-volatile RAM ("NVRAM"), which is used to store basic routines that help boot server computer 302 and transfer information between various components and devices. ROM 410 or NVRAM may also store other software components necessary for the operation of server computer 302 according to the configuration described herein.

[0067] The server computer 302 can operate in a networked environment using logical connections to remote computing devices and computer systems via a network, such as network 308. Chipset 406 may include functionality to provide network connectivity via NIC 412, such as a Gigabit Ethernet adapter. NIC 412 is capable of connecting server computer 302 to other computing devices via network 308. It should be appreciated that multiple NICs 412 may be present in server computer 302, connecting the computer to other types of networks and remote computer systems.

[0068] The server computer 302 can be connected to a storage device 418 that provides non-volatile storage for the computer. The storage device 418 can store an operating system 420, programs 422, and data, which have been described in more detail herein. The storage device 418 can be connected to the server computer 302 via a storage controller 414 connected to the chipset 406. The storage device 418 can be composed of one or more physical storage units. The storage controller 414 can be connected to the physical storage unit via a serial attached SCSI ("SAS") interface, a serial advanced technology connection ("SATA") interface, a fiber channel ("FC") interface, or other types of interfaces for physically connecting and transmitting data between a computer and a physical storage unit.

[0069] The server computer 302 can store data on the storage device 418 by converting the physical state of the physical storage unit to reflect the stored information. In different embodiments of the present specification, the specific conversion of the physical state can depend on various factors. Examples of these factors can include, but are not limited to, the technology used to implement the physical storage unit, whether the storage device 418 is characterized as primary storage or auxiliary storage, etc.

[0070] For example, the server computer 302 can issue instructions through the storage controller 414 to store information to the storage device 418 to change the magnetic properties of a specific location in a disk drive unit, the reflective or refractive properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present description, and the foregoing examples are provided only for ease of description. The server computer 302 can also read information from the storage device 418 by detecting the physical state or properties of one or more specific locations within the physical storage unit.

[0071] In addition to the mass storage device 418 described above, the server computer 302 can access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be understood by those skilled in the art that computer-readable storage media is any available medium that provides non-transitory storage of data and can be accessed by the server computer 302. In some examples, the operations performed by the cloud computing network and / or any components included therein can be supported by one or more devices similar to the server computer 302. In other words, some or all of the operations performed by the cloud computing networks 102a and 102b, and / or any components included therein, can be performed by one or more server computers 302 operating in a cloud-based arrangement.

[0072] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state memory technology, compact disk ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), Blu-ray or other optical storage, cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information in a non-transitory manner.

[0073] As briefly described above, the storage device 418 may store an operating system 420 for controlling the operation of the server computer 302. According to one embodiment, the operating system comprises a LINUX operating system. According to another embodiment, the operating system comprises a LINUX operating system from Microsoft Corporation of Redmond, Washington. Server operating system. According to further embodiments, the operating system may include a UNIX operating system or one of its variants. It should be understood that other operating systems may also be utilized. Storage device 418 may store other systems or applications and data utilized by server computer 302.

[0074] In one embodiment, the storage device 418 or other computer-readable storage medium is encoded with computer-executable instructions that, when loaded into the server computer 302, convert the computer from a general-purpose computing system to a special-purpose computer capable of implementing the embodiments described herein. As described above, these computer-executable instructions convert the server computer 302 by specifying how the CPU 404 transitions between states. According to one embodiment, the computer 302 can access a computer-readable storage medium storing computer-executable instructions that, when executed by the server computer 302, perform the above-referenced Figure 1A , 1B , 1C and Figure 2 The server computer 302 may also include a computer-readable storage medium having stored thereon instructions for performing any other computer-implemented operations described herein.

[0075] The server computer 302 may also include one or more input / output controllers 416 for receiving and processing input from a plurality of input devices (e.g., a keyboard, a mouse, a touch pad, a touch screen, an electronic pen, or other types of input devices). Similarly, the input / output controller 416 may provide output to an output device (e.g., a computer monitor, a flat panel display, a digital projector, a printer, or other types of output devices). It should be understood that the server computer 302 may not include Figure 4 All components shown may include Figure 4 Other components not explicitly shown may be used in conjunction with Figure 4 Completely different architecture shown.

[0076] The server computer 302 may support a virtualization layer, such as one or more virtual resources executing on the server computer 302. In some examples, the virtualization layer may be supported by a hypervisor that provides one or more virtual machines running on the server computer 302 to perform the functions described herein. The virtualization layer may generally support virtual resources that perform at least some of the techniques described herein.

[0077] Although the present disclosure is described with respect to specific examples, it should be understood that the scope of the present disclosure is not limited to these specific examples. Since other modifications and changes made to adapt to specific operating requirements and environments are obvious to those skilled in the art, the present disclosure is not limited to the examples selected for the purpose of disclosure, but covers all changes and modifications that do not depart from the true spirit and scope of the present disclosure.

[0078] Although the present application describes embodiments with specific structural features and / or methodological acts, it should be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are merely some illustrative embodiments falling within the scope of the claims of the present application.

Claims

1. A method for routing data packets, comprising: receiving, at a router in a network, a data packet from a first node in the network through a first tunnel in the network; determining, by the router, a second node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a first tunnel of the network, wherein the second node comprises a plurality of instances of a service, and determining the second node in the network to which the data packet is to be forwarded further comprises determining to which of the plurality of instances the data packet is to be forwarded; forwarding, by the router, the data packet to the second node through a second tunnel of the network based on the determination; At the router, receiving a data packet from the second node through the second tunnel; determining, by the router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network; and Based on the determination, the data packet is forwarded by the router to the third node through a third tunnel of the network.

2. The method according to claim 1, wherein: Determining which instance of the plurality of instances the data packet is to be forwarded to is based on load balancing of the plurality of instances.

3. The method according to claim 1, wherein: Determining which of the multiple instances the data packet will be forwarded to is based on a first instance of the service failure, and determining which of the multiple instances the data packet will be forwarded to includes determining to forward the data packet to a second instance of the service based on the first instance of the service failure.

4. The method according to claim 1, wherein: The method further comprises: determining that a first instance of the service has exceeded a load threshold; and A second instance of the service is generated at the second node.

5. The method according to any one of claims 1 to 4, wherein: Determining a second node in the network to which the data packet is to be forwarded includes determining the second node in the network using a routing map.

6. The method according to any one of claims 1 to 4, wherein: The network includes a plurality of subnets, the router is a first router, and the router is part of the first subnet, and the method further includes: receiving, at the first router, the data packet from the second node through the second tunnel; determining, by the first router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network, the third node comprising a service; receiving, by the first router from one or more routers in one or more of the plurality of subnets, any IP addresses for other nodes in the one or more subnets that include the service; selecting, by the router, a fourth node in a second subnet of the one or more subnets, the fourth node comprising the service; forwarding the data packet to a second router in the second subnet through a third tunnel of the network by the first router; forwarding the data packet to the fourth node by the second router through a fourth tunnel of the second subnet; receiving, at the second router, the data packet from the fourth node through the fourth tunnel; determining, by the second router, a next destination to which the data packet is to be forwarded; and The data packet is forwarded by the second router to the next destination.

7. The method according to claim 6, wherein: Determining, by the second router, a next destination to which the data packet is to be forwarded includes determining that the next destination is the first router.

8. The method according to claim 6, wherein: Determining, by the second router, a next destination to which the data packet is to be forwarded includes determining that the next destination is a fifth node in the second subnet.

9. A system for routing data packets, comprising: one or more processors; as well as One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed by the one or more processors, cause the one or more processors to perform the following operations, the operations comprising: receiving, at a router in a network, a data packet from a first node in the network through a first tunnel in the network; determining, by the router, a second node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a first tunnel of the network, wherein the second node comprises a plurality of instances of a service, and determining the second node in the network to which the data packet is to be forwarded further comprises determining to which of the plurality of instances the data packet is to be forwarded; forwarding, by the router, the data packet to the second node through a second tunnel of the network based on the determination; At the router, receiving a data packet from the second node through the second tunnel; determining, by the router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network; and Based on the determination, the data packet is forwarded by the router to the third node through a third tunnel of the network.

10. The system according to claim 9, wherein: Determining which instance of the plurality of instances the data packet is to be forwarded to is based on load balancing of the plurality of instances.

11. The system according to claim 9, wherein: Determining which of the multiple instances the data packet will be forwarded to is based on a first instance of the service failure, and determining which of the multiple instances the data packet will be forwarded to includes determining to forward the data packet to a second instance of the service based on the first instance of the service failure.

12. The system according to claim 9, wherein: The operations also include: determining that a first instance of the service has exceeded a load threshold; A second instance of the service is generated at the second node.

13. A system according to any one of claims 9 to 12, wherein: Determining a second node in the network to which the data packet is to be forwarded includes determining the second node in the network using a routing map.

14. A system according to any one of claims 9 to 12, wherein: The network includes a plurality of subnets, the router is a first router, and the router is part of the first subnet, and the operations further include: receiving, at the first router, the data packet from the second node through the second tunnel; determining, by the first router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network, the third node comprising a service; receiving, by the first router from one or more routers in one or more subnets of the plurality of subnets, any IP addresses for other nodes in the one or more subnets that include the service; selecting, by the router, a fourth node in a second subnet of the one or more subnets, the fourth node comprising the service; forwarding the data packet to a second router in the second subnet through a third tunnel of the network by the first router; forwarding the data packet to the fourth node by the second router through a fourth tunnel of the second subnet; receiving, at the second router, the data packet from the fourth node through the fourth tunnel; determining, by the second router, a next destination to which the data packet will be forwarded; The data packet is forwarded by the second router to the next destination.

15. The system of claim 14, wherein: Determining, by the second router, a next destination to which the data packet is to be forwarded includes determining that the next destination is the first router.

16. The system of claim 14, wherein: Determining, by the second router, a next destination to which the data packet is to be forwarded includes determining that the next destination is a fifth node in the second subnet.

17. One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform the following operations, the operations comprising: receiving, at a router in a network, a data packet from a first node in the network through a first tunnel in the network; determining, by the router, a second node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a first tunnel of the network, wherein the second node comprises a plurality of instances of a service, and determining the second node in the network to which the data packet is to be forwarded further comprises determining to which of the plurality of instances the data packet is to be forwarded; forwarding, by the router, the data packet to the second node through a second tunnel of the network based on the determination; At the router, receiving a data packet from the second node through the second tunnel; determining, by the router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network; and Based on the determination, the data packet is forwarded by the router to the third node through a third tunnel of the network.

18. One or more non-transitory computer-readable media according to claim 17, wherein: The network includes a plurality of subnets, the router is a first router, and the router is part of the first subnet, and the operations further include: receiving, at the first router, the data packet from the second node through the second tunnel; determining, by the first router, a third node in the network to which the data packet is to be forwarded based on a combination of (i) a source address of the data packet, (ii) a destination address of the data packet, and (iii) an identification of a second tunnel of the network, the third node comprising a service; receiving, by the first router from one or more routers in one or more of the plurality of subnets, any IP addresses for other nodes in the one or more subnets that include the service; selecting, by the router, a fourth node in a second subnet of the one or more subnets, the fourth node comprising the service; forwarding the data packet to a second router in the second subnet through a third tunnel of the network by the first router; forwarding the data packet to the fourth node by the second router through a fourth tunnel of the second subnet; receiving, at the second router, the data packet from the fourth node through the fourth tunnel; determining, by the second router, a next destination to which the data packet is to be forwarded; and The data packet is forwarded by the second router to the next destination.

Images