Tag-based cross-region segment management

Abstract

Systems and methods are provided for managing network segments that span geographical regions and / or other types of network partitioning in a cloud-based network environment. The geographically distributed network infrastructure of a cloud-based network provider can serve as the core of a client's private wide area network (WAN), and the client can define isolated segments to which other networks (VPNs, VLANs, etc.) can be attached. Even when spanning some or all of the same regions as other segments of the same client and / or other clients—and implemented using the same physical and / or virtual routing components—the segments can remain logically isolated from each other.

Description

[0001] This application is a divisional application of patent application No. 2022800872883, filed on November 17, 2022, entitled "Cross-regional segment management based on tags".

[0002] References merged

[0003] The following applications are incorporated herein by reference and form part of this specification:

[0004] Background Technology

[0005] Computing devices can use communication networks to exchange data. Companies and organizations operate computer networks that interconnect several computing devices to support operations or provide services to third parties. Computing devices may be located in a single geographic location or in multiple different geographic locations (e.g., interconnected via private or public communication networks). Specifically, a data center or data processing center (generally referred to herein as a "data center") may include several interconnected computing systems for providing computing resources to users of the data center. A data center may be a private data center operating on behalf of an organization or a public data center operating on behalf of or for the benefit of the general public.

[0006] The emergence of virtualization technology for commodity hardware has offered benefits in managing large-scale computing resources for numerous clients with diverse needs, thereby allowing multiple clients to share various computing resources efficiently and securely. For example, virtualization technology can allow a single physical virtualization host to be shared among multiple users by providing each user with one or more "guest" virtual machines hosted by a single virtualization host. Attached Figure Description

[0007] Embodiments of various inventive features will now be described with reference to the following accompanying drawings. Throughout the drawings, reference numerals may be used repeatedly to indicate the correspondence between the referred elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure.

[0008] Figure 1 It is a block diagram depicting an exemplary network environment that enables cross-regional segment management and routing according to some implementation schemes.

[0009] Figure 2 It is a block diagram depicting an exemplary cloud-based virtual private wide area network (VPN) comprising multiple regions, segments, and attached isolated networks according to some implementation schemes.

[0010] Figure 3 It is a block diagram illustrating the data flow and interactions used for configuring and managing segments according to some implementation schemes.

[0011] Figure 4 It is a flowchart of exemplary routines that can be executed according to some implementation schemes to configure and manage networks and segments.

[0012] Figure 5 This is a block diagram of an exemplary cloud-based virtual private wide area network (VPN) with multiple segments already configured, based on some implementation schemes.

[0013] Figure 6 It is a diagram illustrating an exemplary user interface for viewing and interacting with a cloud-based virtual private wide area network (VPN) with segments, attachments, and shared routes, based on some implementation schemes.

[0014] Figure 7 It is a block diagram illustrating the exemplary data flow and interactions between gateway nodes and endpoints during policy-based routing of traffic within virtual cross-regional segments, according to some implementation schemes.

[0015] Figure 8 It is a block diagram illustrating the exemplary data flow and interaction between gateway nodes during signaling of segment-specific routing data according to some implementation schemes.

[0016] Figure 9 It is a flowchart of an exemplary routine that a gateway node, according to some implementation schemes, can execute to select the best path from multiple paths to the destination.

[0017] Figure 10 It is a block diagram illustrating exemplary data flows and interactions during traffic routing within an area having multiple availability zones, to or through the area, according to some implementation schemes.

[0018] Figure 11 This is a block diagram of an exemplary gateway node configured to implement features of some implementation schemes. Detailed Implementation

[0019] Generally speaking, this disclosure relates to managing network segments across geographic regions and / or other types of network partitioning in a cloud-based network environment. Advantageously, this enables the geographically distributed network infrastructure of a cloud-based network provider to serve as the core of a distributed network for multiple clients, where those clients can define extensive but isolated segments to which other networks (Virtual Private Cloud, Virtual Private Network, etc.) can be attached. Even when spanning some or all of the same regions as other segments of the same client and / or other clients—and implemented using the same physical and / or virtual routing components—the segments can remain logically isolated from each other.

[0020] Some conventional cloud-based network providers offer clients access to shared, geographically dispersed network infrastructure. For example, virtualized compute instances, distributed storage systems, and virtual private clouds (“VPCs”) can be hosted within this shared network infrastructure. As another example, clients can connect their local area networks to a cloud-based network provider’s shared network infrastructure via direct physical links (e.g., “direct connection”), software-defined wide area networks (“SD-WAN”), or virtual private network (“VPN”) tunnels. Within this shared network infrastructure, networking and interconnectivity features can be provided to meet the requirements of applications using services hosted within the infrastructure. For example, routers or other gateways in different, distinct geographical regions or network architectures in other physical and / or logical partitions can be peered, allowing them to share routing data and provide end-to-end routing across the shared network infrastructure in a manner transparent to clients. However, because the network is physically and / or logically separated into regions or other partitions, it is difficult or impossible to provide cross-regional isolated segmentation of traffic using conventional routers and configurations.

[0021] Some aspects of this disclosure address some or all of the problems identified above by providing systems and methods for implementing isolated segments across regions or other network partitions. A Virtual Private Cloud-based Global Wide Area Network (also known as a “VPC Global WAN” or, for brevity, simply a “VPC-WAN”) can be configured within a shared network infrastructure (also known as a “cloud provider network”) of a cloud-based network provider. A VPC-WAN can be considered “private” because it is separate from any other traffic and / or clients (including, but not limited to, other VPC-WANs) sharing the same cloud provider network. Therefore, a VPC-WAN can also be more generally referred to as a “private wide area network” or as an example of a “private network.” A VPC-WAN can be considered both “virtual” and “cloud-based” because it is implemented on top of a shared network infrastructure of a cloud-based network provider, rather than on a separate infrastructure for the client.

[0022] VPC-WAN can be configured using network policy data that defines various aspects. For example, network policy data can define the regions covered by the VPC-WAN, segments that can span multiple regions within the VPC-WAN but remain isolated or substantially isolated from each other, and how isolated networks (VPCs, VPNs, SD-WANs, direct connections to client-local networks, etc.) are attached to segments. Therefore, a VPC-WAN can span multiple regions of a cloud provider network and can include any number of isolated networks, which may be hosted within the cloud provider network's physical data center (e.g., a VPC) or physically located outside the cloud provider network's data center (e.g., client-local networks or third-party networks communicating with the cloud provider network via VPNs, SD-WANs, direct connections, etc.). This allows client traffic originating from one endpoint to be routed to another endpoint of the VPC-WAN, regardless of whether one or both endpoints are within or outside the cloud provider network's physical data center. Furthermore, clients can segment VPC-WAN traffic by defining the segments within the network policy data using one or more rules for attaching isolated networks to segments. For example, rules for a specific segment can specify that attachments associated with a specific tag will be automatically associated with the segment. In some implementations, other rules, attributes, etc., can be specified in the policy data. For example, rules that require authorization to attach an isolated network to a given segment, rules that maintain isolation for all attachments to a given segment, rules regarding resources to be shared between attachments or segments that are to be isolated in other ways, etc., can be defined.

[0023] Additional aspects of this disclosure relate to methods for signaling segment-specific routing data between gateways in different areas or other network partitions. These methods facilitate the formation of cross-area segments from a set of area-specific segments while ensuring that traffic in one such cross-area segment remains isolated from traffic in another, even if the two segments share some or all of the same areas, gateways, etc. When routing tables or other routing data are generated within an area of ​​an area-specific segment, and where that area-specific segment will become part of a cross-area segment, routes in the area-specific routing table can be signaled to other areas associated with the area-specific segment identifier. Other gateways in other areas can obtain the routing table data and add it to their own segment-specific routing tables. In this way, gateways can build routing data tables for all areas of a given segment. Using segment identifiers allows routers to maintain different routing tables for different segments. Thus, within a given area, a single gateway (e.g., a router or a system of routers) can route traffic for multiple segments within that area while also maintaining isolation between segments (unless otherwise permitted or specified).

[0024] Another aspect of this disclosure relates to policy-based routing using packet header metadata. At Layer 3 of the Open Systems Interconnection (“OSI”) model, some networks use packet headers to perform routing operations, the packet header including 5-tuple metadata: source address, such as the Internet Protocol (“IP”) address of the packet sender; destination address, such as the IP address of the packet’s intended destination; source port, such as the Transmission Control Protocol (“TCP”) or User Datagram Protocol (“UDP”) port from which the packet originated; destination port, such as the TCP or UDP port of the packet’s intended destination; and the protocol to be used. In some implementations, to facilitate routing traffic across areas within a given segment while maintaining isolation between different segments, additional metadata can be added to the packet header to indicate the segment from which the packet originated. For example, a Layer 3 packet header can be extended to 6-tuple metadata for routing, where the additional metadata item is a segment identifier. The additional metadata item can be added to the header by a gateway or the packet sender (e.g., by the host device from which the packet originated, by a virtual machine instance or hypervisor running on the host device, etc.). Policies can be implemented at the gateway so that when a packet is received, the gateway can evaluate the segment identifier and determine which routing data (e.g., a segment-specific routing table) to use to route the packet. In this way, a single gateway or a system of gateways within a given area can resolve the segment to which a packet belongs and route traffic across multiple segments encompassing the area, while maintaining isolation between segments using segment-specific routing information.

[0025] Additional aspects of this disclosure involve using metadata and / or metrics to select a specific path to a particular destination from all available paths to that destination. In some implementations, when a gateway is selecting a path to update its routing table, it may use dynamic optimization to select the best path between any two points based on a variety of factors such as: number of hops, physical distance between hops, network latency between hops, packet loss between hops, jitter between hops, link utilization between hops, etc.

[0026] The present disclosure will now describe various aspects of this disclosure with reference to certain examples and embodiments, which are intended to be illustrative and not limiting. While aspects of some embodiments described herein will focus on specific examples of network zones, segments, sections, metadata, gateways, and signaling protocols for illustrative purposes, these examples are merely illustrative and not intended to be limiting. In some embodiments, the techniques described herein may be applied to additional or alternative network zones, segments, sections, metadata, gateways, signaling protocols, etc. Any feature used in any embodiment described herein may be used in any combination with any other feature without limitation.

[0027] Refer to the exemplary implementation plan, Figure 1 An example computing environment in which the features of this disclosure can be implemented is illustrated. As shown, the computing environment includes a cloud provider network underlying 100 (also referred to herein as a "cloud provider network," "provider network," "cloud provider system," or simply "cloud" for convenience), any number of client-side locale networks 150 located outside the cloud provider network 100 (also simply referred to herein as "locale networks" for convenience), and any number of third-party networks 160 located outside the cloud provider network 100. The cloud provider network 100, locale networks 150, and third-party networks 160 can communicate with each other via a communication network 180 (such as an intranet or the Internet).

[0028] Cloud provider network 100 is a pool of network-accessible computing resources (such as compute, storage, and networking resources, applications, and services), which may be virtualized or bare metal. Cloud provider network 100 provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provided and released in response to client commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads. Therefore, cloud computing can be considered both applications delivered as a service over a publicly accessible network (e.g., the Internet, cellular communication networks) and the hardware and software in the cloud provider's data center that provide those services.

[0029] Cloud provider network 100 can provide on-demand scalable computing platforms to users via the network, thereby allowing users, for example, to have scalable "virtual computing devices" at will through their use of computing server 122 (which provides computing instances by using one or both of a central processing unit ("CPU") and a graphics processing unit ("GPU"), optionally with local storage devices) and block storage server 124 (which provides virtualized persistent block storage devices for specified computing instances). These virtual computing devices have the attributes of personal computing devices, including hardware (various types of processors, local memory, random access memory ("RAM"), hard disk and / or solid-state drive ("SSD") storage devices), operating system selection, networking capabilities, and pre-loaded application software. Each virtual computing device can also virtualize its console input and output (e.g., keyboard, monitor, and mouse). This virtualization allows users to connect their virtual computing devices using computer applications (such as browsers, application programming interfaces, software development kits, etc.) so that they can configure and use their virtual computing devices as they would do with personal computing devices. Unlike personal computing devices with a fixed number of hardware resources available to users, the hardware associated with a virtual computing device can be scaled up or down according to the resources required by the user. An Application Programming Interface (“API”) is an interface and / or communication protocol between a client and a server, enabling the client to receive a response in a specific format or initiate a defined action if the client makes a request in a predefined format. In the context of a cloud provider network, an API provides a gateway for clients to access cloud infrastructure by allowing clients to obtain data from or initiate actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted within the cloud provider network. APIs also enable different services within the cloud provider network to exchange data with each other. Users may choose to deploy their virtual computing systems to provide network-based services for their own use and / or for use by their clients or other clients.

[0030] The cloud provider network 100 can provide various computing resources or services, including virtual computing services, data processing services (e.g., mapping simplification, data streaming, and / or other large-scale data processing technologies), data storage services (e.g., object storage services, block-based storage services, or data warehouse storage services), and / or any other type of network-based service (which may include various other types of storage, processing, analytics, communication, event handling, visualization, and security services not illustrated). The resources required to support the operation of such services (e.g., computing and storage resources) may be provided in an account associated with the cloud provider, unlike the resources that can be provisioned in a user account upon request by a user of the cloud provider network.

[0031] The cloud provider network 100 may be formed into several regions, where a region is a separate geographical area in which the cloud provider clusters data centers. In some implementations, each region may be implemented as or otherwise regarded as a region-based autonomous system (“AS”). Each region may include two or more availability zones connected to each other via a private high-speed network (e.g., fiber optic communication connection). An availability zone (“AZ”) refers to a fault isolation domain that includes one or more data center facilities with separate power, networking, and cooling from those data center facilities in another availability zone. Preferably, availability zones within a region are positioned sufficiently far apart from each other that the same natural disaster will not cause more than one availability zone to go offline simultaneously. Clients can connect to the availability zones of the cloud provider network via a publicly accessible network (e.g., the Internet, cellular communication network). A switching center (“TC”) is the main backbone location linking clients to the cloud provider network and may be co-located at other network provider facilities (e.g., Internet service provider, telecommunications provider). For redundancy, two TCs may operate in each region.

[0032] The cloud provider network 100 may include a physical network (e.g., metal enclosures, cabling, rack hardware) referred to as the underlying layer. The underlying layer can be considered a network structure containing the physical hardware running the provider network's services, and may include networking devices (such as routers, switches, network address translators (“NAT”), etc.), and physical connections between devices. The underlying layer may be isolated from the rest of the cloud provider network 100; for example, it may not be possible to route network addresses from the underlying layer to addresses in the production network running the cloud provider's services, or to client networks hosting client resources.

[0033] The cloud provider network 100 may also include an overlay network of virtualized computing resources running on the underlying layer. In at least some embodiments, a hypervisor or other device or process on the network layer may use encapsulation protocol technology to encapsulate and route network packets (e.g., client IP packets) between client resource instances on different hosts within the provider network via the network layer. Encapsulation protocol technology may be used on the network layer to route encapsulated packets (also referred to as network layer packets) between endpoints on the network layer via overlay network paths or routes. Encapsulation protocol technology can be viewed as providing a virtual network topology overlayed on the network layer. Therefore, network packets can be routed along the underlying network according to the constructs (e.g., VPCs, security groups) in the overlay network. A mapping service may coordinate the routing of these network packets. The mapping service may be a regionally distributed lookup service that maps combinations of overlay IPs and network identifiers to underlying IPs, enabling distributed underlying computing devices to find out where to send packets.

[0034] For illustration, each physical host (e.g., compute server 122, block storage server 124, object storage server 126, control server 112, etc.) may have an IP address in the underlying network. Hardware virtualization technology enables multiple operating systems to run simultaneously on a host computer, for example, as virtual machines (VMs) on a compute server. A hypervisor or virtual machine monitor (VMM) on the host allocates the host's hardware resources among the various VMs on the host and monitors the execution of the VMs. Each VM may be configured with one or more IP addresses in the overlay network, and the VMM on the host is aware of the IP addresses of the VMs on the host. The VMM (and / or other devices or processes at the network layer) may use encapsulation protocol technology to encapsulate network packets (e.g., client IP packets) and route said network packets between virtualized resources on different hosts within the cloud provider network 100 via the network layer. Encapsulation protocol technology can be used at the network layer to route encapsulated packets between endpoints at the network layer via overlay network paths or routes. Encapsulation protocol technology can be viewed as providing a virtual network topology overlayed at the network layer. Encapsulation protocol technology may include a mapping service that maintains a mapping directory that maps IP overlay addresses (e.g., public IP addresses) to underlying IP addresses (private IP addresses), the mapping directory being accessible by various processes on a cloud provider network for routing packets between endpoints.

[0035] In various implementations, the traffic and operations at the underlying layer of the cloud provider network can be broadly categorized into two types: control plane traffic carried over the logical control plane and data plane operations carried over the logical data plane. The data plane represents the movement of user data through the distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. The control plane typically includes one or more control plane components 102 distributed across and implemented by one or more control servers 112. Control plane traffic typically includes administrative operations such as establishing isolated virtual networks for various clients, monitoring resource usage and health, identifying the specific host or server where a requested compute instance will be launched, and configuring additional hardware as needed. The data plane typically includes one or more data plane components 104 distributed across and implemented by one or more data plane servers. The data plane includes client resources implemented on the cloud provider network 100 (e.g., compute instances, containers, block storage volumes, databases, or file storage devices, as described in more detail below). Data plane traffic typically includes non-administrative operations such as transferring data to and from client resources.

[0036] Some control plane components 102 (e.g., first-level control plane components, such as the control plane for virtualized computing services) are typically implemented on a separate set of servers from the data plane component 104, while other control plane components 102 (e.g., second-level control plane components, such as analytics services) may share virtualized servers with the data plane component 104. Control plane resources can be provisioned in one (or multiple) account with the cloud provider, while data plane resources can be provisioned in the respective user accounts.

[0037] Control plane traffic and data plane traffic can be sent over separate / different networks. In some implementations, control plane traffic and data plane traffic may be supported by different protocols. In some implementations, messages (e.g., packets) sent over a provider network include flags indicating whether the traffic is control plane traffic or data plane traffic. In some implementations, the traffic payload can be examined to determine its type (e.g., control plane or data plane). Other techniques for distinguishing traffic types are possible.

[0038] As shown in the figure, data plane component 104 may include one or more compute servers 122, which may be bare metal (e.g., a single tenant) or multiple VMs (sometimes referred to as "instances") that can be virtualized by a hypervisor to run one or more clients. These compute servers 122 may support virtualized computing services of cloud provider network 100. Cloud provider network 100 may provide virtual computing instances with different compute and / or memory resources. In one embodiment, each of the virtual computing instances may correspond to one of several instance types. Instance types can be characterized by their hardware type, compute resources (e.g., the number, type, and configuration of central processing units (CPUs) or CPU cores), memory resources (e.g., the capacity, type, and configuration of local memory), storage resources (e.g., the capacity, type, and configuration of locally accessible storage devices), network resources (e.g., the characteristics of their network interfaces and / or network capabilities), and / or other suitable descriptive characteristics. Using instance type selection functionality, the instance type can be selected for the client, for example (at least in part) based on input from the client. For example, the client may select an instance type from a predefined set of instance types. As another example, the client can specify the expected resources for the instance type and / or the workload requirements that the instance will run, and the instance type selection functionality can select the instance type based on this specification.

[0039] The data plane may also include one or more block storage servers 124, which may include persistent storage devices for storing client data and software for managing these volumes. These block storage servers may support managed block storage services for the cloud provider network 100. Block storage servers 124 include one or more servers on which data is stored as blocks. A block is a sequence of bytes or bits, typically containing a full number of records with a maximum length of block size. Blocked data is typically stored in a data buffer and read or written as a whole block at a time. Generally, a volume may correspond to a logical collection of data, such as a dataset maintained on behalf of a user. A user volume may be considered as an individual hard drive ranging in size from, for example, 1 GB to 1 terabyte (TB) (or larger), consisting of one or more blocks stored on a block storage server. Although considered as an individual hard drive, it should be understood that a volume may be stored as one or more virtualized devices implemented on one or more underlying physical host devices. A volume may be partitioned a small number of times (e.g., up to 16 times), with each partition hosted by a different host. The data of a volume may be replicated among multiple devices within the provider network to provide multiple copies of the volume (where such copies may collectively represent the volume on a computing system). In a distributed computing system, volume replicas can beneficially provide automatic failover and recovery, for example, by allowing users to access a primary copy of the volume or a secondary copy of the volume that is block-level synchronized to the primary copy, ensuring that failure of the primary or secondary copy does not inhibit access to information about the volume. The primary copy can facilitate reads and writes (sometimes referred to as "input / output operations" or simply "I / O operations") on the volume and propagate any writes to the secondary copy (preferably synchronously along the I / O path, but asynchronous replication can also be used). The secondary copy can be updated synchronously with the primary copy and provides a seamless transition during failover operations, where the secondary copy assumes the role of the primary copy, and the previous primary copy is designated as the secondary copy or a new replacement secondary copy is provisioned. Compute instances can virtualize their I / O as volumes via clients. A client represents instructions that enable a compute instance to connect to a remote data volume (e.g., a data volume stored on a physically separate computing device accessed over a network) and perform I / O operations at said remote data volume. Clients can be implemented on off-board cards of servers that include the processing units (e.g., CPUs or GPUs) of the compute instances.

[0040] The data plane may also include one or more object storage servers 126, which represent another type of storage device within the cloud provider network 100. Object storage servers 126 include one or more servers on which data is stored as objects within resources called buckets, and can be used to support managed object storage services for the cloud provider network 100. Each object typically includes stored data, variable metadata implementing the object storage service server regarding various capabilities for analyzing the stored object, and a globally unique identifier or key that can be used to retrieve the object. Each bucket is associated with a given user account. Clients can store any desired number of objects in their buckets, can write, read, and delete objects in their buckets, and can control access to their buckets and the objects contained therein. Additionally, in embodiments with several different object storage service servers distributed across different regions described above, users can select the region (or regions) where the buckets are stored, for example, to optimize for latency. Clients can use buckets to store various types of objects, including machine images that can be used to launch VMs and snapshots that can be used to restore volumes.

[0041] In some embodiments, the data plane may include one or more gateway nodes 140 configured to implement aspects of this disclosure for routing data packets from source to destination over a cloud provider network 100. Gateway nodes 140 may be implemented on devices (e.g., routers, servers, etc.) or sets of devices separate from the storage and compute servers of the data plane 104. In some embodiments, gateway nodes 140 may share one or more virtualized servers with storage or compute servers. In some embodiments, gateway nodes 140, or certain modules or components thereof, may be part of the control plane, making them control plane components 102.

[0042] Some clients may expect to use the resources and services of Cloud Provider Network 100, but for various reasons (e.g., latency of communication with client devices, legal compliance, security, or other reasons), prefer to provision these resources and services within their own network (e.g., at the client's location). A small portion of the Cloud Provider Network—referred to herein as a “Provider Underlying Extension” or PSE—can be provisioned within the client's network. Clients can access their PSE through Cloud Provider Network 100 or their own network and can use the same API they will use to create and manage resources in the PSE.

[0043] PSEs can be pre-configured by the provider network operator, for example, with an appropriate combination of hardware and software and / or firmware components to support various types of compute-related resources, and in a manner that reflects the experience of using the cloud provider network 100. For example, one or more PSE servers can be pre-configured by the cloud provider within the client network. As described above, the cloud provider network 100 can provide a set of predefined instance types, each with different types and numbers of underlying hardware resources. Each instance type can also be offered in various sizes. To enable clients to continue using the same instance types and sizes in their PSEs as they use in the cloud provider network 100 region, the PSE servers can be heterogeneous servers. Heterogeneous servers can simultaneously support multiple instance sizes of the same type and can also be reconfigured to host any instance type supported by its underlying hardware resources. Reconfiguration of heterogeneous servers can be done on the fly using the available capacity of the PSE servers, meaning while other VMs are still running and consuming additional capacity of the PSE servers. This improves resource utilization within the PSE by allowing for better packaging of instances running on physical hosts and provides a seamless experience regarding instance usage across the cloud provider network 100 and the PSE.

[0044] As described above, a PSE (Portable Sockets Area) forms an edge location by providing cloud provider network resources and services outside of traditional cloud provider data centers and closer to client devices. Edge locations, as referred to herein, can be structured in several ways. In some implementations, an edge location can be an extension of the underlying cloud provider network, comprising a limited number of capacities managed by the cloud provider but provided outside of traditional availability zones (e.g., in a small data center of the cloud provider or in other facilities located close to client workloads and far from any availability zone). Such edge locations may be referred to as local regions (due to their greater locality or proximity to a set of users compared to traditional availability zones). Local regions can be connected to publicly accessible networks such as the Internet in various ways (e.g., directly, via another network, or via a private connection to the region). While local regions typically have more limited capacity than regions, in some cases, remote regions can have considerably larger capacities, such as thousands of racks or more.

[0045] In some implementations, an edge location can be an extension of the underlying layer of a cloud provider network, consisting of one or more servers located in client or partner facilities in a localized manner. These servers communicate with nearby availability zones or areas of the cloud provider network via a network (e.g., a publicly accessible network such as the Internet). This type of underlying extension located outside the cloud provider network's data centers can be referred to as an "outpost" of the cloud provider network. Some outposts can be integrated into a communications network, for example, as a multi-edge cloud with physical infrastructure scattered across telecommunications data centers, telecommunications aggregation sites, and / or telecommunications base stations within a telecommunications network. In the localized example, the limited capacity of the outpost may be available only to the client that owns the localized location (and any other accounts permitted by the client). In the telecommunications example, the limited capacity of the outpost may be shared among several applications (e.g., games, virtual reality applications, healthcare applications) that send data to users on the telecommunications network.

[0046] Edge locations may include data plane capacity that is at least partially controlled by the control plane of a nearby availability zone. Therefore, an availability zone group may include a “parent” availability zone and any “child” edge locations belonging to the parent availability zone (e.g., at least partially controlled by its control plane). Certain limited control plane functionality (e.g., features requiring low-latency communication with customer resources and / or features enabling the edge location to continue functioning when disconnected from the parent availability zone) may also exist in some edge locations. Thus, in the example above, an edge location refers to an extension of at least the data plane capacity located at the edge of the cloud provider’s network, close to client devices and / or workloads.

[0047] Figure 2 An example of a VPC-WAN 200 implemented using at least part of the hardware and services of a cloud provider network 100 is illustrated. The example VPC-WAN 200 covers multiple regions of the cloud provider network 100: Western Region 202, Central Region 204, and Eastern Region 206. As described herein, each region can be implemented as an autonomous system located in a geographically distinct region from each of the other regions. For example, the hardware components of each region may be located in physically separate data centers in regions of the world where the hardware components of each of the other regions are geographically different.

[0048] To facilitate cross-regional communication, each region may include one or more gateway nodes. A gateway node may act as a router for forwarding packet traffic originating within its corresponding region to another region where the packet destination may be located or which, from a networking perspective, may be closer to the destination. A gateway node may also act as a router for forwarding packet traffic originating outside its corresponding region (e.g., traffic originating from another region) to a destination within its region or to another region.

[0049] In the illustrated example, Western Region 202 is shown with gateway node 140A, Central Region 204 is shown with gateway node 140B, and Eastern Region 206 is shown with gateway node 140C. However, these examples are for illustrative purposes only and are not intended to be limiting, necessary, or exhaustive. In some embodiments, VPC-WAN 200 may have additional, fewer, and / or alternative regions. In some embodiments, a given region may have two or more gateway nodes (e.g., one or more gateway nodes in each availability zone of the region). In some embodiments, a region may have a different number of gateway nodes and / or AZs.

[0050] The VPC-WAN 200 has been segmented into different segments: Development Segment 210, Test Segment 212, and Production Segment 214. Although each segment covers the same area, they are still isolated from each other. Conventionally, because segments cross area boundaries and cover the same area, it would be difficult or impossible to maintain segment traffic isolation without using a separate router or other gateway for each segment. Advantageously, by using segment identifiers to label packets and using separate routing data for each segment, as described in more detail below, a single gateway node can maintain the isolation of segment-specific traffic across multiple segments within an area, and segment traffic can remain isolated from other traffic in other segments even when crossing areas.

[0051] Segments can be used to isolate different parts of a VPC-WAN 200 from each other, and / or facilitate communication between different parts of the VPC-WAN 200, even across regions. Segment construction can be particularly useful when incorporating isolated networks into a VPC-WAN. The process of incorporating an isolated network into a VPC-WAN can be called “attaching” the isolated network, and the attached isolated network can be called an “attachment.”

[0052] The isolated network can be implemented within the cloud provider network 100 (e.g., in one or more regions) or outside the cloud provider network 100 (e.g., at the site client network 150 or a third-party network 160). For example, a client managing VPC-WAN 200 may implement VPC 230 within Western Region 202. As another example, a second client may implement VPC 232 within Central Region 204 and may be permitted to attach VPC 232 to a segment of the first client's VPC-WAN 200 (e.g., Development Segment 210). As another example, a client may attach to the site network 150 via VPN 234 (shown as attached to Development Segment 210), Direct Connection 236 (shown as attached to Test Segment 212), and / or SD-WAN 238 (shown as attached to Production Segment 214). As yet another example, a second client may attach to a third-party network 150 via VPN 234, Direct Connection 236, or SD-WAN 238.

[0053] Figure 3 Example data flows and interactions between the control server 112 and other components of the cloud provider network 100 for configuring VPC-WAN, defining VPC-WAN segments, and attaching isolated networks to VPC-WAN are illustrated.

[0054] At [A], control server 112 may receive or otherwise obtain policy data 300 for configuring the VPC-WAN. Policy data 300 may include metadata describing the attributes of the VPC-WAN to be configured. For example, policy data 300 may include details about: the regions of the cloud provider network to be covered by the VPC-WAN, gateway node configuration information, segments to be defined within the VPC-WAN, the routing behavior of the segments, and / or attachment policies defining how attachments will be mapped to the segments. In some implementations, policy data 300 may be obtained in a standardized format such as a JavaScript Object Notation (“JSON”) document, an Extensible Markup Language (“XML”) document, YAML-not-a-Markup Language (“YAML”) data, etc.

[0055] Table 1 below provides an example of policy data 300 represented as a JSON document. Some parts of the example document will use placeholders " <data>"Used for the value of the entire section or an individual key within a section."

[0056]

[0057] Table 1

[0058] At [B], control server 112 may evaluate policy data 300 to determine the actions required to configure VPC-WAN. Using the example policy data 300 in Table 1, control server 112 may evaluate the "core-network-configuration" section, "segments" section, "segment-actions" section, "attachment-policies" section, or some subset thereof, as defined within policy data 300 and required for configuring VPC-WAN.

[0059] The "core-network-configuration" section defines high-level definitions of the core network (e.g., the portion of VPC-WAN within the cloud provider's network). For example, it can specify one or more regions where operations take place, as well as region routing settings.

[0060] The "segments" section defines the segments to be implemented. Each segment can be implemented as a completely separate routing domain. This section can be used to provide a description for each segment in a VPC-WAN, change default values, provide explicit area action filters and routing filters, etc. The name defined for each segment can be used in other sections, such as the "segment-actions" section and the "attachment-policies" section. In some implementations, overridable default values ​​and other configuration settings that can be set for a given segment include: an optional definition of the segment's "edge-locations" when the segment is limited to fewer than all available edge locations (e.g., areas) defined by the "core-network-configuration" segment; an optional "isolate-attachments" setting that specifies whether attachments on the same segment can communicate with each other (e.g., sandbox segments, such as development segments where VPCs should be prohibited from communicating with each other); a "require-attachment-acceptance" setting that specifies whether an isolated network is automatically approved or required to be attached to a segment; a "deny-filter" setting that prunes routes from listed segments after routes have been shared, thereby prohibiting two segments from sharing routes with each other; and / or an "allow-filter" setting that allows only routes from listed segments.

[0061] The "segment-actions" section defines routing behavior between segments. In some implementations, attachments may, by default, only communicate with other attachments within the same segment, ensuring complete segment isolation. The "segment-actions" section can be used to create connectivity. In some implementations, segment actions may include a "share" action that enables attachments from two different segments to communicate with each other. For example, if a segment is set to "isolate-attachments" (e.g., in the "segments" section described above), attachments may not be able to communicate with any other attachments on the same segment or with any other segment (segments are isolated from each other by default). The "share" action creates routes between attachments in the segments listed for that action. This can be used, for example, to establish shared resource segments reachable from one or more other segments. In some implementations, segment actions may include a "create-route" action for creating static routes within a segment.

[0062] The "attachment-policies" section can be used to define how attachments are mapped to segments. For example, an isolated network can be attached to a segment by specifying the specificity of the isolated network (such as an identifier). This process can be used to manually manage attachments. In some implementations, instead of manually associating segments with each attachment, defined tags can be used to label attachments to associate them with corresponding segments. The "attachment-policies" section can include one or more rules, each of which can include one or more conditions for automatically determining when to take an action related to attaching a specific isolated network to a specific segment. Some types of conditions can include "type", "operator", "value", and / or "key" for evaluating the rules. For example, a rule could be a string matching rule that specifies that any attachment with a specific key and a tag (e.g., "environment") that satisfies a conditional operator about the value (e.g., "equals" and "production") will be automatically attached to the corresponding segment (e.g., the "production" segment). Additional examples of actions could include whether acceptance from a selected entity (e.g., a VPC-WAN administrator) is required for attachment.

[0063] The example policies, rules, and actions shown in Table 1 and described herein are merely illustrative and are not intended to be restrictive, necessary, or exhaustive. Additional and / or alternative policies, rules, and actions may be implemented in some embodiments.

[0064] At [C], control server 112 can configure VPC-WAN based on evaluation policy data 300. To configure VPC-WAN, control server 112 can allocate one or more gateway nodes in each area covered by VPC-WAN. Within individual areas, control server 112 can generate area-specific segment identifiers for gateway nodes to use in distinguishing packets within the various segments that cover the corresponding area. For example, within area 202, there are three segments 210, 212, and 214. A separate segment identifier can be generated for each segment, and gateway node 140A can be configured with separate routing data (e.g., a separate routing table) for each segment identifier. In some implementations, multiple gateway nodes 140 can be configured in this manner, such as when the area consists of multiple AZs as described herein.

[0065] Across regions, control servers 112 can implement policies within gateway nodes, enabling them to determine the appropriate routing data to use and thus route packets for specific segments. Policy-based routing may rely on metadata within the packet header, which indicates the segment identifier of the segment from which that packet was sent, as described in more detail below. Additional policies, such as applying routing filters (e.g., based on deny-filter and / or allow-filter data), static route sharing, etc., can be implemented based on policy data 300.

[0066] In some implementations, policy data 300 may specify a particular isolated network to be attached to a particular segment, as an alternative to—or supplement to—dynamic label-based attachment, which will be evaluated later. Based on such allocation, control server 112 may configure attachments (e.g., by adding routes to the segment routing table, configuring security settings, etc.).

[0067] Once VPC-WAN has been configured using policy data 300, clients can begin attaching isolated networks to segments and transferring data across networks. For example, data can originate within or outside the cloud provider's network, and / or be routed to destinations located within or outside the cloud provider's network. For instance, site network 150 can transfer data via VPC-WAN to different site networks 150 or to VPC 230 within the cloud, or vice versa.

[0068] At [D], control server 112 may receive attachment data 302 regarding the attachment operation to be performed. In some implementations, a user belonging to a client or a third-party entity (e.g., an administrator) may initiate the attachment of an isolated network to the VPC-WAN via an application programming interface ("API"), a graphical user interface ("GUI"), or some other interface. For example, a client may configure VPC 232 and add a tag to VPC 232, thereby triggering an attachment policy in policy data 300.

[0069] At [E], control server 112 can evaluate data about the attachment operation to be performed (e.g., data identifying the isolated network, labels associated with the isolated network, etc.) with respect to policy data 300 to determine how the attachment will be configured. Control server 112 can then configure the attachment at [F]. For example, routing data for the attachment can be added to the segment-specific routing table of gateway node 140A.

[0070] Figure 4 This is a flowchart of an example routine 400 that a control server 112, according to some implementations, can execute to manage the attachment of isolated networks to segments of a VPC-WAN. Advantageously, routine 400 utilizes labels associated with isolated networks and segments to provide dynamic attachment, which can be changed when the underlying attachment policy changes (e.g., when there is a change in the attachment rules) and / or when the label associated with the isolated network changes.

[0071] Routine 400 begins at box 402. In some implementations, routine 400 may begin in response to an event (such as initiating operation of control server 112, receiving data about initiating attachment) or in response to some other event.

[0072] At box 404, control server 112 or some other module or component may evaluate the attachment metadata associated with the attachment operation. The attachment metadata may be associated with the isolated network to be attached to the VPC-WAN. For example, the attachment metadata may include a unique identifier, tag, other data, or some combination thereof for the isolated network.

[0073] At box 406, control server 112 or some other module or component may identify the segment to which the isolated network will be attached based on tag data associated with the isolated network. For example, control server 112 may evaluate policy data 300 and identify attachment rules satisfied by tag data and / or other attachment metadata.

[0074] At decision box 408, control server 112 or some other module or component may determine whether the attachment needs to be accepted before it is completed. In some implementations, this determination may be based on segment metadata specified in policy data 300. For example, in the policy data shown in Table 1, a segment named "dev" is associated with the attribute "'require-attachment-acceptance' : false". In this case, if the attachment to the development segment 210 is the subject of the current iteration of routine 400, acceptance may not be required, and routine 400 may proceed to box 414. Otherwise, if acceptance is required, routine 400 may proceed to decision box 410, where the entity managing the VPC-WAN clients (e.g., an administrator) is prompted to accept the attachment.

[0075] At box 414, control server 112 or some other module or component may be configured to attach to an associated segment based on policy data 300. Configuring an attachment may involve incorporating routes to destinations within the isolated network into routing data associated with the segment.

[0076] At decision box 416, control server 112 or some other module or component can determine whether the label association of the isolated network has changed. For example, a user can reconfigure the isolated network or otherwise change the label associated with it. Such configuration changes can affect the attachment of the isolated network, such as by changing the segment to which the isolated network is attached or by changing the properties of the connection itself. If the label has changed, routine 400 can return to box 404. Otherwise, if the label has not changed, routine 400 can terminate.

[0077] Returning to box 404, control server 112 can evaluate the attachment metadata, and at box 406, the control server can identify the segment based on the tag data associated with the attachment. At decision box 408, control server 112 can determine whether the changed attachment configuration needs to be accepted, and at decision box 410, the control server can determine whether acceptance data indicating that the attachment has been accepted has been received. In some implementations, if the attachment has not yet been accepted, routine 400 can proceed to box 412, where the isolated network remains attached to its current segment instead of being detached from the VPC-WAN and effectively removed.

[0078] Figure 5 Another example of a VPC-WAN 500 is illustrated, implemented at least in part using the hardware and services of a cloud provider network. The VPC-WAN 500 is implemented as a "global" WAN that covers multiple areas of the cloud provider network 100 and uses the infrastructure of the cloud provider network 100 to transfer data between client-side premises networks 150 and / or third-party networks 160 without necessarily using other compute or storage resources of the cloud provider network 100. For example, this VPC-WAN may not have connectivity to any VPCs hosted within the cloud provider network 100. Instead, it uses the cloud provider network 100 for transport between client-side premises networks 150 and for internet access.

[0079] The example VPC-WAN 500 covers multiple regions of the cloud provider network 100: Region 1 502, Region 2 504, Region 3 506, and Region 4 508. These regions may correspond to different geographical areas, such as different neighborhoods, cities, counties, states, provinces, countries, and / or continents. Furthermore, VPC-WAN 500 includes four segments: Sales Segment 510, Engineering Segment 512, Internet of Things ("IoT") Segment 514, and Internet Segment 516.

[0080] Several isolated networks exist attached to the various segments of the VPC-WAN 500, including: VPNs 530 and 532 attached to the Sales segment 510 and providing tunnels to / from different client premises networks; SD-WAN 534 attached to the Engineering segment 512 and the IoT segment 514 and providing access to one or more client premises networks; and VPN 536 attached to the IoT segment 514 and providing access to one or more client premises networks. Table 2 below includes examples of policy data that can be used to define and implement VPC-WANs such as the VPC-WAN 500.

[0081]

[0082] Table 2

[0083] In this example, due to the first "share" action that causes shared routing between the segments, computing resources in the premises network mapped to VPN 530 or 532 and attached to sales segment 510 are granted internet access. IoT segment 514 is under security review, so attachments within IoT segment 514 cannot communicate with each other (e.g., SD-WAN 534 cannot communicate with VPN 536 through IoT segment 514). However, SD-WAN 534 may have been deployed to parts of the premises engineering network and the premises IoT network. Engineers may need direct access to the premises IoT network, which in this example is achieved using a hybrid of VPN 536 and SD-WAN 534. In some cases, SD-WAN 534 will take direct routing between premises sites. In other cases, such as when traffic will traverse the premises engineering network and the premises IoT network, SD-WAN 534 can use VPC-WAN 500 as a transport.

[0084] To reduce workload, the strategy data uses the "Assign-to" tag to define which segment a new attachment should be mapped to. For example, whenever an attachment includes the tag "Assign-to: sales", the attachment will be automatically mapped to the sales segment 510. Because no value is provided for the "require-attachment-acceptance" attribute, a default value will be used, which requires all attachments to be approved.

[0085] Figure 6 An example user interface 600 is illustrated that can be used to present a graphical view of a VPC-WAN. In some implementations, the user interface 600 may be generated as markup language instructions or graphics for network resources, such as Hypertext Markup Language ("HTML") for web pages.

[0086] As shown in the figure, VPC-WAN can be represented as a diagram, where the displayed objects (such as nodes 610, 612, 614, and 616) represent the corresponding segments of the VPC-WAN. The appendix uses displayed objects (such as node groups 620 and 622 for segments 610 and 614, respectively) for illustration.

[0087] The paths between segments are illustrated by edges 630 and 632, indicating that the segment represented by node 614 has available paths to and / or from the segments represented by nodes 610 and / or 612. For example, the segment represented by node 614 can be used to provide shared services to other segments. Node 616 has no edges to another node representing another segment, thus indicating that there is no shared path between the segment represented by node 616 and any other segment.

[0088] In some implementations, a user can apply one or more filters to the user interface 600. For example, filter control 602 may allow selection of various filters, such as filtering based on specific content about source and / or destination attachments, segments, etc.

[0089] Figure 7 Example data flows and interactions between gateway nodes according to some implementation schemes for routing packets within segments and across regions are illustrated.

[0090] As shown in the figure, compute node 700 may be located within an isolated network (such as VPC 230) that is within a region (such as Western Region 202) and attached to a segment (such as Development Segment 210). Compute node 700 may initiate packet 710 transmissions to destinations within Development Segment 210 but outside Western Region 202.

[0091] In some implementations, a packet 710 transmitted from compute node 700 to gateway node 140A within area 202 may include metadata indicating the segment the packet is transmitting within, or metadata that can be used to identify the segment. For example, to facilitate intra-segment cross-area routing of packet 710, compute node 700 or a component thereof (e.g., a VM or hypervisor) may include a segment identifier 732 in the routing metadata 720 of packet 710. As another example, packet 710 may include metadata items other than the segment identifier, and policies for mapping metadata items to specific segments and / or routing tables used to route packet 710 may be implemented in gateway node 140A.

[0092] In some implementations, routing metadata 720 may be a packet header or included in a packet header. For example, some conventional Layer 3 networks use a packet header that includes 5-tuple routing metadata: source address, source port, destination address, destination port, and protocol. Routing metadata 720 may be enhanced using additional metadata items: a segment identifier. Therefore, the packet header of packet 710 may include 6-tuple routing metadata 720: source address 722, source port 724, destination address 726, destination portion 728, protocol 730, and segment identifier 732.

[0093] Packet 710 may be received by gateway node 140A in the area where it originated (or where it initially entered cloud provider network 100). Gateway node 140A may maintain multiple routing tables, allowing routing data for certain segments to be maintained and used separately from routing data for other segments. For example, if gateway node 140A is configured to route packets for two segments, it may maintain at least a first routing table 750A for the first segment and at least a second routing table 750B for the second segment. Different routing tables for different segments may include segment-specific routing data and may therefore differ from each other. For example, routes specific to the first segment may not be included in routing table 750B for the second segment (unless the segment is configured to share routes across segments, as described elsewhere herein).

[0094] Gateway node 140A can be configured to evaluate routing metadata 720 in packet 710 and determine the routing table to be used for routing the packet. For example, a policy may specify that gateway node 140A will determine which routing table to use based on the packet's source and / or destination, and obtain routing data from the determined routing table. In the illustrated example, gateway node 140A may select a route from routing table 750A to gateway node 140B in a different area (central area 204). In some implementations, routing table 750A may include or be associated with segment identifiers for segments. Gateway node 140A may include segment identifiers in the routing metadata 720, which are shown as segment identifier 732 in the illustrated example.

[0095] In some implementations, packet 710 may include metadata representing segment identifier 732 when received by gateway node 140A, or even when received from a source within an area where gateway node 140A is operating or attached to said area. Gateway node 140A may be configured to evaluate routing metadata 720 in packet 710, determine segment identifier 732 included in said routing metadata, and select routing table 750A or other set of routing data to be used for routing packets in the segment associated with segment identifier 732. In the illustrated example, gateway node 140A may select a route to gateway node 140B in a different area (central area 204).

[0096] Gateway node 140B can be configured to evaluate routing metadata 720 in packet 710 and determine segment identifier 732 included in the routing metadata. Based on the routing policy configured to be applied, gateway node 140B can select the routing table 760A to use based on the segment identifier 732. In the illustrated example, gateway node 140B can select a route to a node in a client's premises network accessible via VPN 234.

[0097] In some implementations, the segment identifier 732 included in the packet header routing metadata 720 may be a region-specific segment identifier. For example, a cross-region segment 210 may consist of a set of region-specific segments. Each region-specific segment may have its own segment identifier. By applying routing policies at gateway nodes 140A and 140B, individual region-specific segments can form the basis of cross-region segments, and in some cases, this can be completely transparent to the user. In these implementations, a gateway node receiving packets from different regions may modify its routing metadata to include the segment identifier of the corresponding region-specific segment in the gateway node's region. For example, gateway node 140B may replace the routing metadata in the original header of packet 710 with its own region-specific segment identifier as segment identifier 740.

[0098] While the illustrated examples include different routing tables for different segments, these examples are merely illustrative and not intended to be limiting. In some implementations, all routing data used by a given gateway node 140 may be maintained in a single routing table, and routes may be tagged with segment identifiers of the segments they can be used in. In some implementations, the VPC-WAN may consist of only a single segment. In such cases, each of one or more routing tables maintained by gateway node 140 may be available for the single segment. Optionally, the segment identifier of the single segment may still be included in the routing metadata 720. In this way, the infrastructure and processing of multiple segments can be scaled for use when only a single segment is used.

[0099] Gateway nodes in different regions can exchange routing data. For example, it may be necessary to include paths with more than one gateway node to deliver traffic between isolated networks, such as in scenarios where traffic must be transmitted across continental, national, state, or regional boundaries. For such traffic, pairs of gateway nodes can be programmatically associated with each other. This association can be called "peer attachment" or more simply "peering," and the associated gateway nodes can be referred to as "peers" to each other. To enable cross-region routing of traffic associated with a specific segment, routing data can be associated with the identifier of the segment from the source region, allowing other regions to be configured to associate traffic from that source region with the correct routing table.

[0100] Figure 8 An example is illustrated where dynamic routing involving the exchange of routing data using an exchange protocol such as Border Gateway Protocol ("BGP") can be implemented for a pair of peer gateway nodes. A pair of gateway nodes 140A and 140B can be configured or established, for example, in response to a programming request from one or more clients of the cloud provider network. Illustratively, gateway node 140A may be established in a first geographic region 800 of the cloud provider network 100, and gateway node 140B may be established in a different geographic region 802 of the cloud provider network 100.

[0101] Gateway nodes 140A and 140B can be programmatically peered to each other and to one or more isolated networks in response to a programmatic attachment request (e.g., submitted by a client, where the gateway node and the isolated network are configured on behalf of the client). A given attachment using a gateway node can be a peering attachment that associates two gateway nodes, or it can fall into one of several isolated network attachment categories, such as: a direct connection attachment, where the isolated network at the client's premises is connected to the cloud provider network via a dedicated physical link and associated with the gateway node; a VPN attachment, where the isolated network at the client's premises is connected to the cloud provider network via one or more VPN tunnels and associated with the gateway node; an SD-WAN attachment, where the client's premises SD-WAN appliance is connected to the cloud provider network and associated with the gateway node; an isolated virtual network attachment, which associates an isolated virtual network of a virtualized computing service (e.g., VPC) with the gateway node; etc.

[0102] exist Figure 8 In the illustrated example, both isolated network 810 (e.g., a VPC hosted in region 800 of cloud provider network 100) and isolated network 812 (e.g., a client-side network accessed using a VPN tunnel) can be programmably attached to gateway node 140A. Furthermore, both isolated network 820 (e.g., a separate VPC hosted in region 802 of cloud provider network 100) and isolated network 822 (a direct connection to the client-side network) can be programmably attached to gateway node 140B. Peer attachments can be established between gateway nodes 140A and 140B. Illustratedly, each of these five attachments can be established in response to one or more programming requests from clients.

[0103] Dynamic routing data (e.g., depending on the BGP version or variant) can be exchanged between gateways 140A and 140B. In scenarios where multiple paths are available for transmitting data packets between isolated networks, routing data allows for the dynamic selection of a better path for application data packets at the virtual router. This type of routing is referred to as dynamic routing. In some implementations, when implementing dynamic routing, any of several different factors can be considered at the virtual router when selecting the next hop or path for a packet, such as bandwidth availability, latency, historical congestion patterns, and / or agreements with intermediary or transit network providers.

[0104] In addition to enabling the delivery of dynamic routing data, in at least some implementations, a client may (e.g., via API, GUI, policy data document, etc.) provide a set of one or more dynamic routing protocol configuration settings for delivery. Such settings may indicate various preferences of the client regarding aspects of route data delivery. One such setting may, for example, include filtering rules to determine whether a route to a particular destination will be delivered from one gateway node to another. Another setting may indicate the appropriate priority to be assigned to individual attributes among a plurality of routing-related attributes to select the next hop to the destination, such as: (a) a local preference attribute, (b) a local prefix origin attribute, (c) an autonomous system path length attribute, (d) a multi-escape discriminator attribute, or (e) a router identifier attribute. Local preferences may use, for example, values ​​propagated in route updates from the gateway node's BGP neighbors within the same autonomous system to indicate the corresponding preference the gateway node will use for different available paths. Clients may use local preferences to influence the preferred exit point among multiple exit points of the autonomous system. In one implementation, the gateway node may select the route with the highest local preference (among available alternative routes) for a packet. In some implementations, when alternative paths involving other gateway nodes are also available, the local prefix origin attribute can be used at the gateway node to preferentially select a path in an isolated virtual network directly attached to the gateway node. In implementations where the AS path length attribute is used, the gateway node can select the path with the shortest AS path length (among available alternative paths). In some implementations, a multi-egress discriminator ("MED") can be obtained at the gateway node from BGP neighbors in different ASs, and when alternative paths with different MEDs are available, the gateway mode can select the path with the lowest MED. In some implementations, a digital router identifier can be assigned to each gateway node as well as to the hardware router, SD-WAN appliance, etc., owned by the client; among alternative paths involving the respective router, if in one implementation, no other attribute considered leads to preference, the path of the router with the lowest router identifier can be selected. In some implementations, client-specified settings can indicate the specific variant and / or version of BGP to be used (e.g., iBGP, eBGP, MP-BGP, etc.) and / or the Classless Inter-Domain Router ("CIDR") blocks from which addresses will be assigned to the BGP processing engine associated with the gateway node. In some implementations, other parameters governing the delivery of routing data can be specified by the client (e.g., via API, GUI, policy data document, etc.).

[0105] In some implementations, to enable dynamic routing data delivery, a BGP processing engine can be established or instantiated for each of the respective gateway nodes. In the depicted implementation, based on configuration settings indicated by the client, one or more BGP sessions can be initiated between two processing engines to exchange dynamic routing data that enables network packets to be forwarded from each of the gateway nodes to the isolated network via the other gateway node. The transfer of routing data from one BGP processing engine to another, relative to various sets of destination endpoints, can be referred to as "advertising" routing data.

[0106] Gateway nodes 140A and 140B may each maintain at least one routing table associated with the peer attachment between the gateway nodes. In some implementations, each gateway node may maintain at least one routing table for each segment spanning the peer attachment (e.g., a segment spanning both geographic areas 800 and 802). For example, a first segment may span areas 800 and 802 and may include four isolated network attachments: isolated networks 810 and 812 in area 800, and isolated networks 820 and 822 in area 802. A second segment may also span areas 800 and 802 and may include different isolated network attachments (not shown).

[0107] To facilitate policy-based routing that maintains the desired level of isolation between segments, gateway node 140A may maintain at least one routing table—routing table 830—for the first segment and at least one separate routing table—routing table 832 (shown in dashed lines)—for the second segment. Similarly, gateway node 140B may maintain at least one routing table—routing table 840—for the first segment and at least one separate routing table—routing table 842 (shown in dashed lines) for the second segment. Entries in a given routing table indicate the next hop for each group of destination endpoints, referred to as the destination prefix. In some implementations, individual routing data items may be specified in CIDR format.

[0108] Referring to a particular exemplary example, isolated network 810 includes a first set of network endpoints where IP version 4 ("IPv4") addresses are in the range ABCD / 16 (expressed in CIDR notation). Isolated network 812 includes a second set of network endpoints where IPv4 addresses are in the range AFCD / 16. Isolated network 820 includes a third set of network endpoints where IPv4 addresses are in the range AGCD / 16. Isolated network 822 includes a fourth set of network endpoints where IPv4 addresses are in the range KLMN / 16. To enable traffic to flow through peering attachments between gateway nodes, gateway node 140A (or a component thereof, such as a BGP processing engine) transmits advertisements for ADCD / 16 and AFCD / 16 to gateway node 140B (or a component thereof, such as a BGP processing engine). In the depicted embodiment, gateway node 140B transmits advertisements for AGCD / 16 and KLMN / 16 to gateway node 140A. Therefore, routing table 830 is filled with entries indicating that gateway node 140B is the next hop or destination prefix for destinations within the AGCD / 16 range, and another entry indicating that gateway node 140B is the next hop for destinations within the KLMN / 16 range. Routing table 840 is filled with entries indicating gateway node 140A as the next hop based on ABCD / 16 and AFCD / 16 announcements received from gateway node 140A.

[0109] As shown in the figures, routing tables 830 and 840 also include next-hop entries for isolated networks directly attached to the corresponding gateway nodes. In some implementations, some or all of isolated networks 810, 812, 820, and 822 may include their own gateway nodes and / or BGP processing engines. For example, advertisements for endpoints in isolated network 822 may be transmitted from another BGP processing engine configured within isolated network 822 to gateway node 140B.

[0110] Referring to a particular non-limiting example, when a first segment is configured (e.g., based on policy data as described in more detail above), routing table 830 may be established in gateway node 140A. Routing table 830 may be marked with a region-specific segment identifier for the first segment in area 800. Routes for any isolated networks 810 and 812 attached to the first segment in area 800 may be added to routing table 830, and routing table 830 may be advertised. The advertisement of routing table 830 marked with a region-specific segment identifier for the first segment in area 800 may trigger the establishment of a policy within gateway node 140A to use routing table 830 to route any packets that include a region-specific identifier for the first segment in area 800 in the routing metadata. Similarly, routing table 840 may be established in gateway node 140B, marked with a region-specific segment identifier for the first segment in area 802. Routes can be added to isolated networks 820 and 822 that are attached to segments in area 802, routing table 840 can be advertised, and policies can be established within gateway 140B to use routing table 840 to route packets that include an area-specific identifier for the first segment in area 802 in the routing metadata.

[0111] When routes from routing table 830 are advertised to other gateway nodes (including gateway node 140B) of a segment, the other gateway nodes can determine the routing table corresponding to the segment marked by the advertised routing data and add those routes to the routing table as appropriate. For example, gateway node 140B can determine that the segment marked by the advertised routing data is the first segment maintained by routing table 840. Gateway node 140B can add the advertised routes to routing table 840 as appropriate (e.g., based on a selection algorithm), and can also establish policies within gateway node 140B to use routing table 840 to route any packets received from gateway 140A that include an area-specific identifier for the first segment in area 800 in the routing metadata.

[0112] Dynamic routing data (e.g., BGP advertisements) passed between gateway nodes can be used to route network packets from one isolated network to another. For example, if a packet originating in isolated network 810 is directed to an address within the range of isolated network 822, the packet may include routing metadata indicating a segment identifier that includes a first segment of isolated network 810. Gateway node 140A may determine, based on the segment identifier, to route the packet using routing table 830 instead of routing table 832, and may forward the packet to gateway node 140B. Gateway node 140B may determine, based on the segment identifier, to route the packet using routing table 840 instead of routing table 842, and may forward the packet to isolated network 822.

[0113] In the illustrated example, the third region—region 804 with gateway node 140C—is also part of the first segment. Therefore, gateway nodes 140A and 140B can advertise routes for the first segment to gateway node 140C, and gateway node 140C can advertise routes for the first segment to gateway nodes 140A and 140B. This implementation can be called a "whole network" implementation.

[0114] In some implementations, a gateway node in one region may not need to advertise every route in its routing table to gateway nodes in other regions for that region. Instead, some routes may be maintained as "local only" or "community" routes. For example, some routes may be tagged with metadata or otherwise indicated as local routes based on requests from network administrators (e.g., via API), rules in policy data used to configure the segment, or some other data. Local routes can be filtered out when a gateway node is ready to advertise routing data. However, the routes can still be used within the region to route data to / from various endpoints. In some implementations, an entire isolated network (e.g., VPN, VPC, etc.) may be considered a "local only" or "community" annex to a segment. For example, a network administrator may define a segment as a local segment via API and / or rules in policy data. Therefore, all routes to endpoints in the annex may be tagged with metadata indicating them as local routes.

[0115] Although only four isolation networks and three zones are shown, and each zone is shown as having one gateway node, the example is merely illustrative and is not intended to be limiting, necessary, or exhaustive. In some implementations, additional or fewer isolation networks, zones, segments, and / or gateway nodes may be used.

[0116] Figure 9 This is a flowchart of an example routine 900 that a gateway node 140, according to some implementation schemes, can execute to manage path selection and the generation of routing data. Advantageously, routine 900 utilizes optimized data (including dynamic metrics) to select the best path to the destination in a more dynamic and optimized manner compared to using conventional methods alone (such as the BGP best path selection algorithm). For example, because cloud network providers use multiple paths between regions (up to and including the entire network between regions) to provide cross-regional networks, there are multiple possible paths between any two regions, and conventional metrics do not necessarily have to select the best path from a latency and / or cost perspective. By combining metrics such as hop count, physical distance between each hop, network latency between each hop, packet loss between each hop, jitter between each hop, link utilization between each hop, etc., the selected path can perform better than in other cases.

[0117] Routine 900 begins at box 902. In some implementations, routine 900 may begin in response to an event such as establishing a dynamic routing data exchange session at gateway node 140.

[0118] At box 904, gateway node 140 (or some other module or component) can obtain routing data. The routing data may be provided by one or more sources, such as clients (e.g., clients managing VPC-WAN and / or its segments), one or more other gateway nodes, some other source of routing data, or any combination thereof. Illustratively, at least a portion of the routing data may be received in conjunction with an advertisement from another gateway node 140 (e.g., a gateway node in a different region than the gateway node performing the current iteration of routing 900).

[0119] At block 906, gateway node 140 (or some other module or component) has access to optimization data. In some implementations, the optimization data may include: the number of hops associated with a path, the physical distance between each hop associated with a path, the network latency between each hop associated with a path, the packet loss rate between each hop associated with a path, the jitter rate between each hop associated with a path, the link utilization rate between each hop associated with a path, other optimization data, or any combination thereof. The optimization data may be obtained from one or more sources, any one, all, or none of which may be located outside of gateway node 140 performing the current iteration of routine 900. For example, the physical distance between each hop may be determined and maintained by a separate management system and provided to the gateway node as optimization data for path selection. As another example, the latency, packet loss, or jitter rate between two hops may be determined by the means involved in the hop (e.g., including gateway node 140) and may be reported to the source, destination, router, or management system.

[0120] At box 908, gateway node 140 (or some other module or component) can use optimization data to select the best path to each destination from among multiple available paths to each destination. In some implementations, using optimization data to select the best path can be performed using a modified version of the BGP best path selection algorithm, or a custom algorithm can be used. For example, certain attributes evaluated during the BGP best path selection algorithm (e.g., weights, local preferences, local prefix origin, AS path length, MED, router identifier) ​​may precede or replace one or more metrics or other attributes (e.g., hop count, physical distance between hops, network latency between hops, packet loss between hops, jitter between hops, link utilization between hops, etc.) in the optimization data.

[0121] At box 910, gateway node 140 (or some other module or component) may signal the selected path to other gateway nodes 140, or otherwise exchange routing data with other gateway nodes 140 (e.g., as part of a dynamic routing data exchange session). By signaling the selected path to other gateway nodes 140, the optimal path selection process is performed in a distributed manner. For example, each gateway node may obtain optimization data, perform optimal path selection locally, and notify all other peers. The entire VPC-WAN can use this distributed routing system to find the optimal path. Due to the distributed nature of this system, it can be highly available without any single point of failure.

[0122] In some conventional systems, cross-region traffic can be routed point-to-point through a specific pair of gateway nodes selected by the client. However, traffic can traverse multiple regions before reaching its destination. Furthermore, while some conventional systems require traffic to have at least one endpoint within the cloud provider network, VPC-WAN 200, as described herein, can have two endpoints outside the cloud provider network 100 (e.g., in different client premises networks 150, third-party networks 160, some combination thereof, etc.). In such implementations, the distributed system implementing the gateway nodes that optimize the best path selection can influence how client premises traffic is sent to the cloud provider network 100 and which is the optimal entry point for the traffic. For example, in the case of a direct connection attachment to a client premises network, traffic is sent via the direct connection by default. However, if multiple attachments to a client premises network exist, optimizing the best path selection can result in optimal paths from different attachments, different client premises networks, etc.

[0123] Referring to illustrative examples, traffic may flow between two different areas of a VPC-WAN, and from a networking and / or geographical perspective, the distance to a specific destination in a third area via either area may be approximately the same. Optimized best-path selection, performed using distributed systems and optimized data, selects the best path under observed network conditions. As another illustrative example, if something disrupts the connection between two areas that were originally configured as a whole network, one area may no longer have routes to / through the other area, but may still have routes for each of the other areas. Optimized best-path selection, performed using distributed systems and optimized data, selects the best path to the best other area as an intermediary. As yet another illustrative example, there may be heavily used segments of physical network infrastructure between two areas that cause congestion. Optimized best-path selection, performed using distributed systems and optimized data, selects the best path from one of the two areas to the other area via an intermediary, rather than using a direct path.

[0124] Routine 900 can terminate at box 910.

[0125] Figure 10 An example of region 1002 is shown, having two availability zones, 1020 and 1022. Each availability zone may have one or more gateway nodes. For example, availability zone 1020 may include gateway node 140A, and availability zone 1022 may include gateway node 140B. For redundancy, gateway nodes in each availability zone within the region may include the same routing data.

[0126] When traffic from another region (such as region 1000) passes through region 1002 on its way to an endpoint in an adjacent region (such as isolated network 1030), multiple communication sessions can be established: one session each in availability zones 1020 and 1022. Source region 1000 can use equal-cost multipath ("ECMP") routing to balance the load to isolated network 1030 across the two availability zones. Furthermore, and separately from the load balancing aspect, multiple redundant availability zones are also used to ensure the availability of computing resources in region 1002. If one availability zone (such as availability zone 1020) becomes unavailable, traffic can still reach isolated network 1030 through another availability zone 1022, as indicated by the dashed lines.

[0127] Similarly, because gateway nodes in a specific area exchange routing data (e.g., using the protocols described above), a complete outage of an area may not prevent traffic from being routed to its destination in the isolated network. For example, if all availability zones in area 1002 are inaccessible, traffic can be routed to different areas (such as area 1004) and / or, as needed, through any other intermediary area, isolated network, etc., to reach isolated network 1030.

[0128] Figure 11 Various components of an example gateway node 140 configured to implement the various functionalities described herein are illustrated. Gateway node 140 may be a physical host computing device, such as a computing server 122, on which a gateway is implemented.

[0129] In some implementations, as shown in the figures, gateway node 140 may include: one or more computer processors 1102, such as a physical central processing unit ("CPU"); one or more network interfaces 1104, such as a network interface card ("NIC"); one or more computer-readable media drives 1106, such as a high-density disk ("HDD"), a solid-state drive ("SSD"), a flash drive, and / or other persistent non-transitory computer-readable media; and one or more computer-readable storage devices 1108, such as random access memory ("RAM") and / or other volatile non-transitory computer-readable media.

[0130] Computer-readable storage 1108 may include computer program instructions that execute on one or more computer processors 1102 and / or data used by one or more computer processes 1102 to implement one or more embodiments. For example, computer-readable storage 1108 may store routing instructions 1110 for routing packets based on routing metadata and data in one or more routing tables 1114. As another example, computer-readable storage 1108 includes dynamic routing data exchange instructions 1112 for exchanging routing data with other gateway nodes.

[0131] Some inventive aspects of this disclosure are set forth in the following clauses: Clause 1: A system comprising: Multiple gateway nodes, wherein individual gateway nodes are configured to route network traffic associated with corresponding region-based autonomous systems (AS) within multiple region-based ASs of the provider network; and A control server, comprising one or more processors and executable instructions, wherein the control server is programmed to at least: Obtain policy data for a Virtual Private Cloud-based Wide Area Network (VPN), wherein the policy data specifies that the VPN will be implemented using at least a first region-based Autonomous System (AS) and a second region-based Autonomous System (AS) of the provider network; The first segment of the Virtual Private Cloud-based Wide Area Network (VPN) is established using at least a first gateway node in the first region-based autonomous system and a second gateway node in the second region-based autonomous system. At least one routing policy for the first segment is established based on the policy data. At least a first portion of the traffic in the first segment is isolated from at least a second portion of the traffic in the second segment of the VPN. Both the first and second portions of the traffic pass through the first and second region-based autonomous systems. Based on a tag associated with a first isolated network of the provider network, communication between the first isolated network and the second isolated network is determined via the first segment, wherein the policy data specifies that the isolated network associated with the tag will be able to communicate via the first segment, and wherein the first isolated network includes one of the following: a Virtual Private Cloud, a Virtual Private Network, a Software-Defined Wide Area Network, or a direct connection to a client's local area network; and The first segment enables communication between the first isolated network and the second isolated network.

[0132] Clause 2: A system as described in Clause 1, wherein the routing policy of the first segment involves at least one of the following: preventing communication between isolated networks of the first segment, restricting the first segment to a subset of the plurality of region-based autonomous systems, or applying a filter to routes accessible from within the first segment by the provider network.

[0133] Clause 3: The system as described in Clause 1, wherein the control server is further programmed to at least: Based on the policy data, it is determined that enabling communication through the first segment of the first isolated virtual network requires the approval of the administrator of the virtual private cloud-based wide area network; and The control server receives acceptance data indicating the administrator's acceptance, wherein the control server determines, in response to receiving the approval data, to enable communication of the first isolated virtual network through the first segment.

[0134] Clause 4: The system as described in Clause 1, wherein the first gateway node is configured to route packets associated with the first segment using a first routing table associated with the first segment, and wherein the first gateway node is configured to route packets associated with the second segment using a second routing table different from the first routing table.

[0135] Clause 5: A computer-implemented method comprising: Under the control of a computing system within a cloud provider's network, the computing system includes memory and one or more computer processors configured to execute specific instructions: Obtain policy data associated with a private network that is at least partially implemented within the cloud provider's network; Based on the policy data, at least a first network management node in a first geographical region of a plurality of geographical regions of the cloud provider network is used to establish a first segment within the private network, wherein at least a first portion of the traffic associated with the first segment will be isolated from at least a second portion of the traffic associated with a second segment of the private network; Obtain attachment metadata indicating the isolated network of the cloud provider network associated with the first segment; and This enables the isolated network to communicate through the first segment, wherein the policy data specifies that the isolated network associated with the attached metadata will be able to communicate through the first segment.

[0136] Clause 6: A computer-implemented method as described in Clause 5, wherein establishing the first segment includes configuring a second gateway node in a second geographic region of the plurality of geographic regions based on the policy data to isolate at least a third portion of traffic associated with the first segment from at least a fourth portion of traffic associated with a different segment of the private network.

[0137] Clause 7: The computer-implemented method as described in Clause 5 further includes: Determine that the policy data indication needs to be received to enable the isolated network to communicate through the first segment; and Receiving indicates approval that enables the isolated network to communicate via the first segment, wherein the isolated network enables communication via the first segment in response to receiving the received data.

[0138] Clause 8: The computer-implemented method as described in Clause 5 further includes: Based on the policy data, it is determined that isolated networks that can communicate through the first segment are prohibited from communicating with each other through the first segment; and Prevent the isolated network from communicating with the second isolated network associated with the first segment.

[0139] Clause 9: The computer-implemented method as described in Clause 8 further includes: The policy data enables communication between the isolated network and the shared resource segment; and The policy data enables communication between the second isolated network and the shared resource segment.

[0140] Clause 10: The computer-implemented method as described in Clause 5 further includes: determining, based on the policy data, a subset of the plurality of geographic regions to be established, wherein the subset of the plurality of geographic regions includes less than all of the plurality of geographic regions.

[0141] Clause 11: The computer-implemented method as described in Clause 10 further includes: determining, based on the policy data, a second subset of the second segment to be established in the plurality of geographic regions, wherein the second subset of the plurality of geographic regions is different from the subset of the plurality of geographic regions.

[0142] Clause 12: The computer-implemented method as described in Clause 5 further includes: determining, based on the policy data, to refuse to share routes from the second segment with the first segment.

[0143] Clause 13: The computer-implemented method as described in Clause 5 further includes: determining, based on the policy data, permission to share routes from the second segment with the first segment.

[0144] Clause 14: The computer-implemented method as described in Clause 8 further includes: generating a graphical user interface, the graphical user interface comprising: This represents the first display object of the first segment; This indicates the second display object in the second section; This indicates a third display object attached to the isolated network to the first segment; and The fourth display object represents the path shared between the first segment and the second segment.

[0145] Clause 15: A system comprising: A computer-readable storage medium storing executable instructions; and One or more processors, the one or more processors communicating with the computer-readable storage and programmed by the executable instructions to at least: Obtain policy data associated with a private network that is at least partially implemented within a cloud provider's network; Based on the policy data, at least a first gateway node in a first geographic region and a second gateway node in a second geographic region of multiple geographic regions of the cloud provider network are used to establish a first segment within the private network, wherein at least a first portion of the traffic associated with the first segment will be isolated from at least a second portion of the traffic associated with the second segment of the private network. Obtain data indicating the isolation network of the cloud provider's network and its association with the tag; and This enables the isolated network to communicate through the first segment, wherein the policy data specifies that the isolated network associated with the tag will be able to communicate through the first segment.

[0146] Clause 16: A system as described in Clause 15, wherein the one or more processors are programmed by additional executable instructions to: The policy data indicates that acceptance is required to enable the isolated network to communicate through the first segment; and Receiving indicates approval that enables the isolated network to communicate via the first segment, wherein the isolated network is able to communicate via the first segment in response to receiving the received data.

[0147] Clause 17: A system as described in Clause 15, wherein the one or more processors are programmed by additional executable instructions to: Based on the policy data, it is determined that isolated networks capable of communicating through the first segment are prohibited from communicating with each other through the first segment; and Prevent the isolated network from communicating with the second isolated network associated with the first segment.

[0148] Clause 18: A system as described in Clause 17, wherein the one or more processors are programmed by additional executable instructions to: Communication between the isolated network and the shared resource segment is achieved based on the policy data; and Communication between the second isolated network and the shared resource segment is achieved based on the policy data.

[0149] Clause 19: A system as described in Clause 15, wherein the one or more processors are programmed by additional executable instructions to: Based on the strategy data, a first subset of the first segment will be established from among the plurality of geographic regions, wherein the first subset of the plurality of geographic regions includes less than all of the plurality of geographic regions; and Based on the strategy data, a second subset of the second segment will be established from among the plurality of geographic regions, wherein the second subset of the plurality of geographic regions is different from the first subset of the plurality of geographic regions.

[0150] Clause 20: A system as described in Clause 15, wherein one or more processors are programmed by additional executable instructions to determine, based on the policy data, to refuse to share routes from the second segment with the first segment.

[0151] Clause 21: A system comprising: A first gateway node, associated with a first geographical region of a provider network, wherein a Virtual Private Cloud-based Wide Area Network (VPN) is implemented on the provider network, and wherein the first gateway node is configured to use a first routing table to route traffic associated with a first segment of the VPN, and a second routing table to route traffic associated with a second segment of the VPN; and A second gateway node, which is associated with a second geographical region of the provider network, is configured to use a third routing table to route traffic associated with the first segment and a fourth routing table to route traffic associated with the second segment. The first gateway node is further configured as follows: Send first routing data, representing at least a portion of the routes identified in the first routing table, to the second gateway node; The first routing table is updated based on second routing data received from the second gateway node, which represents at least a portion of the routes identified in the third routing table; Receive packets associated with routing metadata, the routing metadata representing at least a first source address in the first geographic region, a first destination address in the second geographic region, and an identifier for the first segment; and The packet is routed to the second gateway node using the first routing table, at least in part, based on the identifier of the first segment; The second gateway node is further configured as follows: Send the second routing data to the first gateway node; The third routing table is updated based on the first routing data received from the first gateway node; Receive the packet from the first gateway node; and The third routing table is used, at least in part, based on the identifier of the first segment, to route the packet to the destination address.

[0152] Clause 22: The system as described in Clause 21, wherein the second gateway node is further configured to modify the routing metadata associated with the packet to generate modified routing metadata, wherein the modified routing metadata represents a second identifier associated with the first segment.

[0153] Clause 23: The system as described in Clause 21, wherein the first gateway node is further configured to select a path to the destination address from a plurality of paths to the destination address, wherein the path is selected based on at least one of the following: the number of hops associated with the path, the physical distance between each hop associated with the path, the network latency between each hop associated with the path, the packet loss rate between each hop associated with the path, the jitter rate between each hop associated with the path, or the link utilization rate between each hop associated with the path.

[0154] Clause 24: The system as described in Clause 21 further includes: a third gateway node associated with the first geographic region, wherein the third gateway node is configured to use a fifth routing table to route traffic associated with the first segment and a sixth routing table to route traffic associated with the second segment, wherein the first gateway node is located in a first availability zone of the first geographic region, and wherein the third gateway node is located in a second availability zone of the first geographic region.

[0155] Clause 25: A computer-implemented method comprising: Under the control of the first gateway node associated with the first geographical region of the provider network: Receive a packet including first routing metadata representing a destination address and a first segment identifier, wherein the first segment identifier is associated with a first segment of a plurality of segments of a private wide area network implemented on the provider network; The packet is routed using a first routing table from a plurality of routing tables maintained by the first gateway node, at least in part based on the first segment identifier, wherein the first routing table is associated with the first segment, and wherein a second routing table from the plurality of routing tables is associated with a second segment of the plurality of segments; and The packet is transmitted at least in part based on first routing data from the first routing table, wherein the first routing data is associated with the destination address.

[0156] Clause 26: A computer-implemented method as described in Clause 25, wherein transmitting the packet includes transmitting the packet to a second gateway node associated with a second geographic region of the provider network.

[0157] Clause 27: A computer-implemented method as described in Clause 25, wherein transmitting the packet includes transmitting the packet to a second gateway node associated with an isolated network in the first geographic region.

[0158] Clause 28: The computer-implemented method as described in Clause 25 further includes: modifying the first routing metadata to generate second routing metadata, wherein the second routing metadata includes a second segment identifier associated with the first segment, wherein the second segment identifier is a region-specific segment identifier associated with the first geographic region.

[0159] Clause 29: The computer-implemented method as described in Clause 25 further includes: Receive a second packet including second routing metadata representing a second destination address and a second segment identifier associated with the first segment; The routing of the second packet based on the first routing table is determined at least in part based on the second segment identifier; and The second packet is transmitted based at least in part on second routing data from the first routing table, wherein the second routing data is associated with the second destination address.

[0160] Clause 30: The computer-implemented method as described in Clause 25 further includes: Receive a second packet including second routing metadata representing a second destination address and a second segment identifier associated with the second segment; The routing of the second packet based on the second routing table is determined at least in part based on the second segment identifier; and The second packet is transmitted at least in part based on second routing data from the second routing table, wherein the second routing data is associated with the second destination address.

[0161] Clause 31: The computer-implemented method as described in Clause 25 further includes: sending second routing data representing at least a portion of the routes identified in the first routing table to one or more other areas of the provider network, wherein the one or more other areas are associated with the first segment.

[0162] Clause 32: The computer-implemented method as described in Clause 25 further includes: updating the first routing table based on second routing data received from a second gateway node in a second area of ​​the provider network, representing at least a portion of routes in a third routing table maintained by the second gateway node, wherein the third routing table is associated with the first segment.

[0163] Clause 33: A computer-implemented method as described in Clause 32, wherein updating the first routing table includes selecting a path to the destination address from a plurality of paths to the destination address, wherein the path is selected based on at least one of: the number of hops associated with the path, the physical distance between each hop associated with the path, the network latency between each hop associated with the path, the packet loss rate between each hop associated with the path, the jitter rate between each hop associated with the path, or the link utilization rate between each hop associated with the path.

[0164] Clause 34: A computer-implemented method as described in Clause 25, wherein transmitting the packet based at least in part on the first routing data includes transmitting the packet to a second gateway node in a first availability zone of a plurality of availability zones of a second region of the vendor network, wherein the second packet is transmitted at least in part on a third gateway node in a second availability zone of the plurality of availability zones based on the second routing data.

[0165] Clause 35: A system comprising: A computer-readable storage medium storing executable instructions; and One or more processors, the one or more processors communicating with the computer-readable storage and programmed by the executable instructions to at least: Receive a packet including first routing metadata representing a destination address and a first segment identifier, wherein the first segment identifier is associated with a first segment among a plurality of segments of a private wide area network implemented within a shared network infrastructure; The packet is routed using a first routing table from a plurality of routing tables maintained by the first gateway node, at least in part based on the first segment identifier, wherein the first routing table is associated with the first segment, and wherein a second routing table from the plurality of routing tables is associated with a second segment of the plurality of segments; and The packet is transmitted at least in part based on first routing data from the first routing table, wherein the first routing data is associated with the destination address.

[0166] Clause 36: A system as described in Clause 35, wherein, in order to transmit the packet, the one or more processors are programmed by additional executable instructions to transmit the packet to a second gateway node associated with a second geographic region of the provider network.

[0167] Clause 37: A system as described in Clause 35, wherein, in order to transmit the packet, the one or more processors are programmed by additional executable instructions to transmit the packet to a second gateway node associated with an isolated virtual network in the first geographic region.

[0168] Clause 38: A system as described in Clause 35, wherein one or more processors are programmed via additional executable instructions to modify the first routing metadata to generate second routing metadata, wherein the second routing metadata includes a second segment identifier associated with the first segment, wherein the second segment identifier is a region-specific segment identifier associated with the first geographic region.

[0169] Clause 39: A system as described in Clause 35, wherein the one or more processors are programmed by additional executable instructions to: Send second routing data representing at least a portion of the routes identified in the first routing table to one or more other areas of the provider network, wherein the one or more other areas are associated with the first segment; and The first routing table is updated based on third routing data received from a second gateway node in a second area of ​​the provider network, representing at least a portion of routes in a third routing table maintained by the second gateway node, wherein the third routing table is associated with the first segment.

[0170] Clause 40: A system as described in Clause 39, wherein the one or more processors are programmed by additional executable instructions to select a path to the destination address from a plurality of paths to the destination address, wherein the path is selected based on at least one of: the number of hops associated with the path, the physical distance between each hop associated with the path, the network latency between each hop associated with the path, the packet loss rate between each hop associated with the path, the jitter rate between each hop associated with the path, or the link utilization rate between each hop associated with the path.

[0171] All methods and tasks described herein can be performed by a computer system and are fully automated. In some cases, the computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate via a network to perform the described functions. Each such computing device typically includes a processor (or multiple processors) that executes program instructions or modules stored in memory or other non-transitory computer-readable storage media or devices (e.g., solid-state storage devices, disk drives, etc.). The various functions disclosed herein may be embodied in such program instructions or implemented in the computer system's dedicated circuitry (e.g., ASICs or FPGAs). In cases where the computer system includes multiple computing devices, these devices may, but need not, be located in the same location. The results of the disclosed methods and tasks can be persistently stored by transforming physical storage devices, such as solid-state memory chips or disks, into different states. In some embodiments, the computer system may be a cloud-based computing system whose processing resources are shared by multiple different business entities or other users.

[0172] Depending on the implementation, certain actions, events, or functions of any process or algorithm described herein may be performed in a different order, and may be added, combined, or omitted entirely (e.g., not all described operations or events are necessary for the practical algorithm). Furthermore, in some implementations, operations or events may be performed simultaneously rather than sequentially, for example, through multithreading, interrupt handling, or on multiple processors or processor cores or other parallel architectures.

[0173] The various exemplary logic blocks, modules, routines, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware or a combination of electronic hardware and computer software. To clearly illustrate this interchangeability, the various exemplary components, blocks, modules, and steps have been generally described above in accordance with their functionality. Whether such functionality is implemented as hardware or as software running on hardware depends on the specific application and design constraints imposed on the system as a whole. For each specific application, the described functionality may be implemented in different ways, but such implementation decisions should not be interpreted as departing from the scope of this disclosure.

[0174] Furthermore, the various exemplary logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by machines designed to perform the functions described herein, such as processor devices, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or any combination thereof. The processor device may be a microprocessor, but alternatively, it may be a controller, microcontroller, or state machine, a combination thereof, etc. The processor device may include electronic circuitry configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. The processor device may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors along with a DSP core, or any other such configuration. Although this document primarily describes digital technologies, the processor device may also primarily include analog components. For example, some or all of the algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. Computing environments can include any type of computer system, including, but not limited to, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances.

[0175] Elements of the methods, processes, routines, or algorithms described in conjunction with the embodiments disclosed herein can be embodied directly in hardware, software modules executed by a processor device, or a combination of both. The software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of non-transitory computer-readable storage medium. An exemplary storage medium can be coupled to a processor device such that the processor device can read information from and write information to the storage medium. Alternatively, the storage medium can be integrated with the processor device. The processor device and storage medium can reside in an ASIC. The ASIC can reside in a user terminal. Alternatively, the processor device and storage medium can reside as discrete components in the user terminal.

[0176] Unless otherwise specifically stated, or otherwise understood as used in the context herein, the conditional language used herein (such as, in addition to, "can," "may," "can," "may," "for example," etc.) is generally intended to convey that certain embodiments include certain features, elements, or steps that are not included in other embodiments. Therefore, such conditional language is generally not intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include a function to determine, with or without further input or prompting, whether such features, elements, and / or steps are included or to be performed in any particular embodiment. The terms "comprising," "including," "having," etc., are synonymous and used inclusively in an open-ended manner, without excluding additional elements, features, actions, operations, etc. Additionally, the term "or" is used in its inclusive sense (and not in an exclusive sense) such that, when used, for example, to connect a list of elements, the term "or" means one, some, or all of the elements in the list.

[0177] Unless otherwise specifically stated, disjunctive language such as the phrase "at least one of X, Y, or Z" is understood in context to generally represent that items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended and should not imply that certain embodiments require the presence of at least one of X, at least one of Y, or at least one of Z.

[0178] Unless otherwise expressly stated, articles such as 'a' or 'an' should generally be understood to include one or more of the described items. Thus, phrases such as "a device configured to..." are intended to include one or more of the described devices. Such one or more of these devices may also be collectively configured to perform the stated description. For example, "a processor configured to perform descriptions A, B, and C" may include a first processor configured to perform description A, which operates in conjunction with a second processor configured to perform descriptions B and C.

[0179] While the above detailed description has shown, described, and pointed out novel features applicable to various embodiments, it is understood that various omissions, substitutions, and changes in the form and details of the illustrated apparatus or algorithm may be made without departing from the spirit of this disclosure. It will be appreciated that some embodiments described herein may be embodied in forms that do not provide all the features and benefits set forth herein, as some features may be used or practiced separately from other features. The scope of certain embodiments disclosed herein is indicated by the appended claims rather than by the foregoing description. All changes falling within the meaning and scope of the equivalents of the claims will be included within the scope of the claims.< / data>

Claims

1. A system comprising: A computer-readable storage medium storing executable instructions; as well as One or more processors, the one or more processors communicating with the computer-readable storage and programmed by the executable instructions to at least: Obtain policy data associated with a private network that is at least partially implemented within a cloud provider's network; Based on the policy data, at least a first gateway node in a first geographic region and a second gateway node in a second geographic region of multiple geographic regions of the cloud provider network are used to establish a first segment within the private network, wherein at least a first portion of the traffic associated with the first segment will be isolated from at least a second portion of the traffic associated with the second segment of the private network. Obtain data indicating the isolation network of the cloud provider's network and its association with the tag; and This enables the isolated network to communicate through the first segment, wherein the policy data specifies that the isolated network associated with the tag will be able to communicate through the first segment.

2. The system of claim 1, wherein, in order to establish the first segment, the one or more processors are further programmed via the executable instructions to: configure a second gateway node in a second geographic region of the plurality of geographic regions based on the policy data to isolate at least a third portion of traffic associated with the first segment from at least a fourth portion of traffic associated with a different segment of the private network.

3. The system of claim 1, wherein the one or more processors are programmed via additional executable instructions to: The policy data indicates that acceptance is required to enable the isolated network to communicate through the first segment; and Receiving indicates approval that enables the isolated network to communicate via the first segment, wherein the isolated network is able to communicate via the first segment in response to receiving the received data.

4. The system of claim 1, wherein the one or more processors are programmed via additional executable instructions to: Based on the policy data, it is determined that isolated networks capable of communicating through the first segment are prohibited from communicating with each other through the first segment; and Prevent the isolated network from communicating with the second isolated network associated with the first segment.

5. The system of claim 4, wherein the one or more processors are programmed via additional executable instructions to: Communication between the isolated network and the shared resource segment is achieved based on the policy data; and Communication between the second isolated network and the shared resource segment is achieved based on the policy data.

6. The system of claim 1, wherein the one or more processors are programmed via additional executable instructions to: Based on the strategy data, a first subset of the first segment will be established from among the plurality of geographic regions, wherein the first subset of the plurality of geographic regions includes less than all of the plurality of geographic regions; and Based on the strategy data, a second subset of the second segment will be established from among the plurality of geographic regions, wherein the second subset of the plurality of geographic regions is different from the first subset of the plurality of geographic regions.

7. The system of claim 1, wherein the one or more processors are programmed by additional executable instructions to: determine, based on the policy data, to refuse to share routes from the second segment with the first segment.

8. A computer-implemented method, comprising: Under the control of the first gateway node associated with the first geographical region of the provider network: Receive a packet including first routing metadata representing a destination address and a first segment identifier, wherein the first segment identifier is associated with a first segment of a plurality of segments of a private wide area network implemented on the provider network; The packet is routed using a first routing table from a plurality of routing tables maintained by the first gateway node, at least in part based on the first segment identifier, wherein the first routing table is associated with the first segment, and wherein a second routing table from the plurality of routing tables is associated with a second segment of the plurality of segments; as well as The packet is transmitted at least in part based on first routing data from the first routing table, wherein the first routing data is associated with the destination address.

9. The computer-implemented method of claim 8, wherein transmitting the packet comprises transmitting the packet to one of: a second gateway node associated with a second geographic region of the provider network or a second gateway node associated with an isolated virtual network in the first geographic region.

10. The computer-implemented method of claim 8, further comprising: The first routing metadata is modified to generate second routing metadata, wherein the second routing metadata includes a second segment identifier associated with the first segment, wherein the second segment identifier is a region-specific segment identifier associated with the first geographic region.

11. The computer-implemented method of claim 8, further comprising: Receive a second packet including second routing metadata representing a second destination address and a second segment identifier associated with the first segment; The routing of the second packet based on the first routing table is determined at least in part based on the second segment identifier. as well as The second packet is transmitted based at least in part on second routing data from the first routing table, wherein the second routing data is associated with the second destination address.

12. The computer-implemented method of claim 8, further comprising: Receive a second packet including second routing metadata representing a second destination address and a second segment identifier associated with the second segment; The routing of the second packet is determined, at least in part, based on the second segment identifier, according to the second routing table; as well as The second packet is transmitted at least in part based on second routing data from the second routing table, wherein the second routing data is associated with the second destination address.

13. The computer-implemented method of claim 8, further comprising: Send second routing data representing at least a portion of the routes identified in the first routing table to one or more other areas of the provider network, wherein the one or more other areas are associated with the first segment.

14. The computer-implemented method of claim 8, further comprising: The first routing table is updated based on second routing data received from a second gateway node in a second area of ​​the provider network, representing at least a portion of routes in a third routing table maintained by the second gateway node, wherein the third routing table is associated with the first segment.

15. The computer-implemented method of claim 8, wherein transmitting the packet based at least in part on the first routing data includes transmitting the packet to a second gateway node in a first availability zone of a plurality of availability zones in a second region of the vendor network, wherein the second packet is transmitted at least in part on a third gateway node in a second availability zone of the plurality of availability zones based on the second routing data.

Images