Wide area networking services using provider network backbone networks

Abstract

A representation of a set of facilities between which network traffic is routed via a private fiber backbone of a provider network is obtained. Respective virtual routers are configured for the first facility and the second facility, and connections are established between the virtual routers and a routing information source for the facilities. Contents of at least one network packet originating at the first facility are transmitted over the private fiber backbone to the second facility using routing information obtained by the virtual router from the routing information of the second facility.

Description

[Background technology]

[0001] Many businesses and other organizations operate computer networks that interconnect numerous computing systems to support their operations, e.g., the computing systems are co-located (e.g., as part of a local network) or located in multiple separate geographic locations (e.g., connected through one or more private or public intermediate networks). Data centers that house significant numbers of interconnected computing systems have become commonplace, e.g., private data centers operated by a single organization for its own use, and public data centers operated by business entities to provide computing resources to customers. Some public data center operators provide network access, power, and secure installation facilities for various customer-owned hardware, while others provide "full-service" facilities that also include hardware resources made available for use by customers.

[0002] The advent of virtualization technology for commodity hardware has brought benefits in managing large-scale computing resources for many customers with diverse needs, allowing various computing resources to be shared more efficiently and securely among multiple customers. For example, virtualization technology may allow a single physical virtualization host to be shared among multiple users by providing each user with one or more "guest" virtual machines hosted by the single virtualization host. Each such virtual machine may represent a software simulation that functions as a separate logical computing system that gives the user the illusion of being the sole operator of a given hardware computing resource, and also provides application isolation and security among the various virtual machines. Creating instances of several different virtual machines on the same host may also help increase the overall hardware utilization level in a data center, resulting in a higher return on investment.

[0003] As the demand for virtualization-based services in a provider network grows, more networking and interconnection-related functionality may need to be added to meet the requirements of the applications implemented using the services. Many such functionality may require some form of network packet address manipulation, for example at level 3 or level 4 of the Open Systems Interconnection stack. Some clients of virtualization computing services may want to employ customized policy-based packet processing for application traffic flowing between specific sets of endpoints. Using ad-hoc solutions for all the different types of packet transformation requirements may not scale in large provider networks where traffic associated with hundreds of thousands of virtual or physical machines may be processed simultaneously. [Brief description of the drawings]

[0004] [Figure 1] 1 illustrates an exemplary system environment in which a scalable virtual router for traffic flowing between isolated networks may be implemented, according to at least some embodiments. [Diagram 2] 1 illustrates exemplary categories of packet processing applications implemented with the aid of a virtual router, according to at least some embodiments, and auxiliary tasks that may be performed for some of the categories. [Diagram 3] 1 illustrates an overview of example interactions between exception path nodes of a virtual router, fast path nodes of a virtual router, and auxiliary task offload resources associated with a virtual router, according to at least some embodiments. [Figure 4] FIG. 1 illustrates an example scenario in which resources used for a virtual router may be automatically scaled independently of resources used for auxiliary tasks related to traffic routed through the virtual router, in accordance with at least some embodiments. [Diagram 5] 1 illustrates an example use of independently managed packet processing cells for a virtual router, according to at least some embodiments. [Figure 6] 1 illustrates an example use of independently managed auxiliary task offload cells for a virtual router, according to at least some embodiments. [Figure 7] 1 illustrates an example technique for connecting nodes of a virtual router with an auxiliary task offloader, according to at least some embodiments. [Figure 8] 1 illustrates an example of programmatic interaction between a client and a packet processing service related to configuration and use of a virtual router and associated auxiliary task offload resources, according to at least some embodiments. [Figure 9] 1 illustrates an example of programmatic interaction between a client and a packet processing service related to configuration and use of a virtual router and associated auxiliary task offload resources, according to at least some embodiments. [Figure 10] FIG. 1 is a flow diagram illustrating aspects of operations that may be performed to offload certain types of tasks from a virtual router, according to at least some embodiments. [Figure 11] 1 illustrates an example system environment in which a protocol stack multiplexer and multiple protocol stack instances may be configured to offload auxiliary tasks of a scalable virtual router, according to at least some embodiments. [Figure 12] 1 illustrates an example set of protocols, each of which may be executed on a device equipped with a protocol stack multiplexer, according to at least some embodiments. [Figure 13] 1 illustrates an example set of interactions between components of an auxiliary task offload device that may configure a protocol stack multiplexer for a virtual router, according to at least some embodiments. [Figure 14]1 illustrates an example scenario in which protocol stack instances developed in several different programming languages ​​may be executed within respective software containers of an auxiliary task offload device, according to at least some embodiments. [Figure 15] 1 illustrates an example scenario in which multiple independent instances of a given protocol stack may run simultaneously on an auxiliary task offload device, according to at least some embodiments. [Figure 16] 1 illustrates an example of an alternative approach for storing protocol state information related to auxiliary tasks of a virtual router, according to at least some embodiments. [Figure 17] FIG. 11 is a flow diagram illustrating aspects of operations that may be performed to offload certain types of tasks from a virtual router using a protocol stack multiplexer and independent instances of protocol stacks, according to at least some embodiments. [Figure 18] 1 illustrates an exemplary system environment in which dynamic routing, including the exchange of routing information using a Border Gateway Protocol (BGP) processing engine, according to at least some embodiments, may be enabled for a peering pair of virtual routers upon request of a client of a packet processing service. [Figure 19] 1 illustrates an example scenario in which dynamic routing information exchange is enabled for several different types of programmatic attachment of virtual routers, according to at least some embodiments. [Figure 20] 1 illustrates an example scenario in which a custom protocol for routing information transfer may be used by a virtual router to exchange information originally sent to the virtual router using BGP, according to at least some embodiments. [Figure 21] 1 illustrates an example scenario in which multiple peering attachments may be established between pairs of virtual routers, according to at least some embodiments. [Figure 22]1 illustrates an example set of programmatic interactions for configuring dynamic routing for peered virtual routers, according to at least some embodiments. [Diagram 23] FIG. 1 is a flow diagram illustrating aspects of operations that may be performed to enable and utilize dynamic routing for peered virtual routers, according to at least some embodiments. [Figure 24] 1 illustrates an example environment in which a wide area network linking geographically separate facilities of an organization may be managed by the organization using leased fiber lines and equipment from various vendors, according to at least some embodiments. [Diagram 25] 1 illustrates an exemplary system environment in which traffic between remote facilities of clients of a provider network is transmitted using the provider network's wide area network (WAN) service, which employs an internal fiber backbone network and a collection of virtual routers that enable dynamic routing, according to at least some embodiments. [Figure 26] 1 illustrates an example web-based interface that may be used to provide WAN quality of service metrics for traffic between client-specified locations, according to at least some embodiments. [Figure 27] 1 illustrates an example web-based interface that may be used to present status information for traffic flowing between client-specified locations, according to at least some embodiments. [Figure 28] 1 illustrates an example scenario in which required intermediary devices for traffic flowing between specified locations may be configured on behalf of a client of a WAN service, according to at least some embodiments. [Figure 29] 1 illustrates an example set of programmatic interactions related to the use of private provider network backbone network links for traffic between client premises, according to at least some embodiments. [Diagram 30]1 is a flow diagram illustrating aspects of operations that may be performed in a wide area networking service of a provider network that transmits traffic between client premises over a private fiber backbone, according to at least some embodiments. [Diagram 31] FIG. 1 is a block diagram illustrating an example computing device that may be used in at least some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0005] Although the embodiments are described herein by way of example in several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. The drawings and detailed description thereof are not intended to limit the embodiments to the particular disclosed forms, but on the contrary, it should be understood that the intent is to cover all modifications, equivalents, and alternatives that are within the spirit and scope defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the detailed description or the claims. As used throughout this application, the term "may" is used in an permissive sense (i.e., meaning having the potential to), rather than a required sense (i.e., meaning necessary). Similarly, the words "include," "including," and "includes" are used in the sense of including, not limited. When used in the claims, the word "or" is used as an inclusive "or" and not as an exclusive "or." For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, and any combination thereof.

[0006] The present disclosure relates to a method and apparatus for efficiently implementing several categories of auxiliary tasks (such as routing configuration information exchange tasks, encryption / decryption tasks, or custom client request tasks) related to the transmission of network packets containing application data between isolated networks via a virtual router implemented in a provider network or cloud computing environment. A given virtual router includes a collection of nodes of a multi-layer packet processing service, including fast path nodes configured to quickly execute locally cached routing or forwarding actions, and exception path nodes that determine the actions to be taken for various packets based on client-specified policies and connectivity requirements of the isolated networks. In a scenario where dynamic routing is performed for application data traffic between a pair of isolated networks using a routing information exchange protocol (such as a protocol similar to the Border Gateway Protocol (BGP)), processing of messages containing dynamic routing information can be offloaded from the virtual router node to a protocol processing engine running on another device, allowing the virtual router node to remain dedicated to its primary task of rule-based packet forwarding. For similar reasons, other types of auxiliary tasks can also be handed off from the virtual router to an offload device, such as multicast configuration management via a protocol similar to IGMP (Internet Group Management Protocol), encryption / decryption of packet content, or custom packet analysis / processing tasks requested by clients.

[0007] The offload tasks may differ from the baseline or core forwarding-related tasks that run on the virtual router in several important ways. For example, at least some offload tasks may typically need to be executed much less frequently than application data packet forwarding tasks, and the amount of computation or other resources (e.g., memory or storage for saving state information) required for some offload tasks may be much greater than the amount of the same resources required for application data packet forwarding, etc. Offloading auxiliary tasks may be beneficial not only for performance reasons (e.g., to avoid bypassing the resources of a virtual router where demand for application data forwarding may already be high), but also to allow the forwarding-related components of the virtual router to be enhanced and developed independently of the auxiliary task processing components. In some cases, the auxiliary tasks may be executed using protocol stack instances (e.g., instances of a BGP processing engine or an IGMP processing engine) that run in user mode on the offload device (e.g., a virtualization host for a computing service), and a protocol stack multiplexer distributes the auxiliary tasks among the protocol stack instances of a given offload device as needed.

[0008] Offloading techniques make several types of packet processing applications more practical and performant. Virtual routers implemented in different geographic regions (e.g., using resources located in datacenters in different states, countries, or continents) can be programmatically attached (or "peered") to each other and configured to obtain and automatically exchange dynamic routing information about isolated networks in different regions using a protocol similar to BGP, freeing clients from having to painstakingly configure static routes for traffic flowing between isolated networks. Clients can control how routing information is forwarded between virtual routers by specifying various parameters and settings (e.g., the particular protocol version to use, rules to filter routing information advertised to and from the virtual router, etc.). Wide area networking (WAN) services can be implemented in provider networks using programmatically attached virtual routers with dynamic routing enabled, allowing clients to utilize the provider network's private fiber backbone links (already used for traffic between the provider network's datacenters on behalf of users of various other services) for traffic between client facilities distributed around the world, and to manage their long-distance traffic using easy-to-use tools with visualization interfaces. In general, such WAN services or dynamic routing of peered virtual router pairs may benefit from the use of offload resources, although it should be noted that, at least in some embodiments, such offloading is not necessarily required for all virtual routers used in such scenarios.

[0009] As will be appreciated by those skilled in the art in light of this disclosure, certain embodiments may be capable of achieving various advantages, including some or all of the following: (a) enabling processing associated with various configuration management protocols used for certain types of packet processing applications, such as BGP, IGMP, etc., to be performed efficiently using dedicated resources and without consuming computational resources primarily reserved for high speed packet forwarding / routing actions; (b) scaling the set of resources dedicated to high speed packet forwarding / routing independently from resources used for auxiliary tasks, such as routing table configuration management, cryptographic transcoding of packet contents, and performance delay measurement, thereby ensuring high performance for the auxiliary tasks as well as the forwarding / routing actions; (c) reducing the number of network configuration resolutions required in organizations with geographically distributed computing resources, for example, by eliminating the need for error-prone specification of static routes between networks in different geographic regions and by eliminating the need to manage traffic over leased fiber lines; and / or (e) improving the user experience of system administrators and / or application owners of applications running in geographically distributed environments by providing configuration information and metrics separately at the intra-regional and inter-regional levels. Due to the multi-tenant nature of the packet processing services used by the virtual router, in at least some embodiments, the total amount of computing and other resources required to route traffic between various isolated networks may also be reduced.

[0010] According to some embodiments, a system may include one or more computing devices. The computing devices may include instructions that, when executed on or across the computing devices, cause a determination of a category of auxiliary tasks related to transmitting at least a subset of network packets between various isolated networks based at least in part on input received from a client of the provider network via one or more programmatic interfaces. Depending on the details of a packet processing application instance that a client wishes to implement using the packet processing services of the provider network, any combination of numerous categories of auxiliary tasks may be required in various embodiments. The set of auxiliary task categories may include, for example, a routing configuration management category (e.g., tasks related to processing BGP messages, provider network custom routing information exchange protocol messages, IGMP messages, provider network custom multicast configuration protocol messages, etc.), (b) a packet content transformation category (e.g., encryption / decryption of packet content according to a protocol such as IPSec or a provider network custom security protocol), (c) a performance management category (e.g., measuring packet delay or packet loss rates between geographically distant resources), (d) custom processing tasks specified by a client (e.g., calculations performed specifically on packets that have been assigned a client-defined tag), etc. A given isolated network of traffic over which the auxiliary tasks are performed may, for example, in various embodiments, include a computing service provider (VPC) in a provider network, or an isolated virtual network (also known as a virtual private cloud or VPC, or virtual network) that includes several computing instances of the network configured in a client-owned facility outside the provider network.Such resources at client-owned facilities may be linked to a provider network data center in any of a number of ways in different embodiments, such as by using one or more virtual private network (VPN) tunnels, by using dedicated private physical network links (sometimes called direct connect links), etc.

[0011] The virtual router may be configured, in various embodiments, to send network packets between a first isolated network and a second isolated network indicated by a client. The virtual router may programmatically attach to at least one of the isolated networks, for example, in response to a request from a client of the provider network. Such an attachment request may indicate that traffic of the isolated network is to be processed by the virtual router. A given virtual router may include multiple packet processing nodes, including fast path nodes and exception path nodes, in some embodiments. The fast path nodes may be configured to (a) obtain executable versions of one or more routing actions for a given packet flow between a pair of isolated networks from the exception path nodes, (b) cache the executable versions, and (c) use the executable versions to implement the routing actions on packets of the flow. The term "exception" may be used to refer to a node at which an executable version of a routing action is generated. This is because, on average, a given executable action may be executed many times (e.g., for each packet of a flow containing thousands of packets), and thus the activity of generating an action may be considered an exceptional or infrequent activity. A virtual router (VR) may also be referred to as a virtual traffic hub (VTH) or a transit gateway (TGW) in various embodiments. In at least some embodiments, individual nodes of a VR may include one or more threads of execution on computing instances (e.g., virtual machines) of a virtualized computing service of a provider network, or one or more threads of execution on a non-virtualized server. Fast path nodes and exception path nodes may be collectively referred to as forwarding plane nodes (or routing plane nodes) in some embodiments, since one of their primary functions may include forwarding packets containing client application data between different isolated networks as quickly as possible.

[0012] In addition to the VR itself, in at least some embodiments, one or more auxiliary task offload resources (ATORs) (also referred to as auxiliary task offloaders (ATOs)) may be configured on behalf of the client to perform auxiliary tasks of identified categories for the client's packet processing application, for example by a management plane or control plane component of the packet processing service in various embodiments. A communication path may be established between the ATOR and one or more packet processing or forwarding plane nodes of the VR, for example using metadata provided to the exception path node of the VR. In some embodiments, establishing such a connection may include setting up a virtual network interface through which packets from the VR forwarding plane node are directed and configuring one or more tunnels of an encapsulation protocol between the virtual network interface and the ATOR. The ATOR may include, for example, one or more threads that execute a computing instance or a non-virtualized host in different embodiments.

[0013] After the ATOR is established and connected to the VR, in various embodiments, a message or packet indicating an auxiliary task may be sent from the VR to the ATOR. When a particular packet is received at the ATOR, the auxiliary task corresponding to that packet may be executed (e.g., updating routing information based on the packet's contents), and the result of the auxiliary task (e.g., an updated route) may be sent back to the VR. The result of the auxiliary task may be used in the VR to send one or more packets between isolated networks whose traffic is assigned to the VR. For example, in one scenario, the auxiliary task may lead to the insertion or deletion of one or more routes in a route table used to select the next hop to send the packet over. In another example, the result of the auxiliary task (e.g., an encrypted version of the packet's application data payload) may itself be sent in the packet.

[0014] In contrast to the VR's forwarding plane actions, which may be performed at high speed for most of the traffic received at the VR, at least some categories of auxiliary tasks may be performed less frequently (e.g., using ATOR) and may be performed asynchronously with respect to the forwarding of application data packets. For example, routing information messages may be received and / or sent relatively infrequently on at least some of the BGP sessions, and the processing of a given BGP message may not be part of the critical path for forwarding application data packets (the results of the processing of the BGP message may change the next hop to which some application packets are sent). Thus, the timing at which the VR's forwarding plane components receive the results of some categories of auxiliary task processing may, in some embodiments, be independent and asynchronous with respect to the individual application packet forwardings performed by the forwarding plane components. For example, a given fast path node may not need to wait for BGP message processing tasks to complete before forwarding a particular packet to its destination. For other types of auxiliary tasks, the forwarding of a given application data packet by a fast path node may depend on the completion of the auxiliary task. For example, if the contents of a forwarded application data packet are encrypted as part of an auxiliary task, or if a log record is generated and stored as part of an auxiliary task before a packet with a particular client-specified tag or label is sent from the VR, the fast path node may need to wait until the auxiliary task is completed.

[0015] In some embodiments, multiple types of auxiliary tasks may be performed on packets of a given flow of application data. For example, the results of BGP message processing may be used to determine the next hop for packets of the flow, encryption / decryption tasks may be performed on packets of the flow, etc. One flow may be distinguished from another by some combination of properties including source and destination Internet Protocol (IP) addresses, source and destination ports, source and destination virtual network interface identifiers, source and destination isolated network identifiers, etc. In one such embodiment, a given ATOR may be used to implement several different categories of auxiliary tasks. In other embodiments, one category of auxiliary tasks may be performed using a first ATOR and another category of auxiliary tasks may be performed using a second ATOR.

[0016] In one embodiment, the ATOR may comprise a hardware card attached to the server via a peripheral interconnect, such as a Universal Serial Bus (USB) or Peripheral Component Interconnect-Express (PCIe). In such an embodiment, at least some categories of auxiliary tasks may be performed entirely on the hardware card (e.g., using a processor and memory embedded within the card). In some embodiments, instead of being implemented using resources external to the VR, the ATOR may be implemented using resources (e.g., auxiliary processing nodes that are logically distinct from the fast path and exception path nodes) that are configured and managed as part of the VR by the packet processing service's control plane.

[0017] According to some embodiments, a system may include one or more computing devices. The computing device may include instructions that, when executed on or across the computing device, cause an offload device (e.g., a virtualization host) to receive a message indicating at least a portion of a first auxiliary task related to transmitting a network packet from a virtual router configured to transmit the network packet between a first isolated network and a second isolated network. A stack multiplexer (e.g., one or more processes or threads executing on a computing instance and / or a virtualization manager) of the offload device may select a particular protocol stack instance from a set of protocol stack instances executing on the offload device to process at least a portion of the message. A given protocol stack instance may, at least in some embodiments, include software that implements, in user mode or user space, one or more layers of a stack of networking protocols, e.g., layers defined in the OSI (Open Systems Interconnection) model. The stack multiplexer may have access to a direct memory access (DMA) buffer of the offload device into which a message may be placed by a network interface card where the message is retrieved from the virtual router. A particular protocol stack instance may be selected based at least in part on metadata included in or associated with the message, such as the protocol used for the underlying packets encapsulated within the message, an identification of the isolated virtual network for which the virtual router is configured, an identifier of the client or clients on whose behalf traffic is being sent by the virtual router, etc. At least some of the protocol stack instances may execute in user space in various embodiments.The selected protocol stack instance (which may include, for example, a BGP processing engine, an IGMP processing engine, and logic for lower layer protocols utilized by BGP or IGMP) may analyze the message and perform the necessary auxiliary tasks. The results of the auxiliary tasks may then be sent via a multiplexer to the virtual router where they may be utilized to transmit at least some packets between the first and second isolated networks. According to at least some embodiments, a given protocol stack instance may be configured for use in a single-tenant mode (on behalf of only one client, or on behalf of only one virtual router) or in a multi-tenant mode for auxiliary tasks performed on behalf of multiple clients or multiple virtual routers. The tenant mode, in some embodiments, may be selected based at least in part on program input from a client for which a virtual router is configured.

[0018] According to some embodiments, a system may include one or more computing devices. The computing device may include instructions that, when executed on or across the computing device, cause the computing device to create a plurality of virtual routers, including a first virtual router and a second virtual router, using resources of a provider network. Transfer of routing information between the first virtual router and the second virtual router may be enabled according to a group of dynamic routing protocol control settings indicated by a client of the provider network via one or more programmatic interfaces. At least a portion of the routing information may be associated with a plurality of isolated networks, including a first isolated network programmatically attached to the first virtual router and a second isolated network programmatically attached to the second virtual router. The particular settings of the group of dynamic routing protocol control settings may include, for example, filter rules used to determine whether a route to a particular destination is forwarded, respective priorities associated with various BGP attributes of a group of network addresses, etc. Respective protocol processing engines for dynamic routing protocols may be established for each of the virtual routers, for example, using offload resources of the types described above in some embodiments. Based on client-specified preferences, some embodiments may use any of several variants of BGP (such as internal BGP, external BGP, or multiprotocol BGP) for dynamic routing. In one embodiment, a client may specify configuration settings using BGP attributes, but the routing information may actually be transferred between the virtual router's protocol processing engines using a custom protocol in the provider network.

[0019] In some cases, dynamic routing enabled virtual routers may be utilized to transfer data between geographically distributed facilities of provider network clients using private fiber backbone links of the provider network. According to some embodiments, a system may include one or more computing devices. The computing devices may include instructions, when executed on or across the computing devices, that cause the computing devices to obtain, via one or more programmatic interfaces of a wide area networking service of the provider network, an indication of (a) a plurality of client facilities between which network traffic is routed via the private fiber backbone of the provider network, including a first facility in a first geographic region and a second facility in a second geographic region, and (b) a particular protocol used to obtain dynamic routing information for at least the first client facility and the second client facility. A first virtual router may be configured for the client using at least a first set of resources of a virtualized computing service in a first provider network data center that meets a proximity criterion to the first facility. Similarly, a second virtual router may be configured using at least a second set of resources of a second provider network data center that meets a proximity criterion to the second facility. In various embodiments, connections may be enabled between (a) a first virtual router and a second virtual router, (b) a first virtual router and a first facility, and (c) a second virtual router and a second facility. Contents of at least one network packet originating at the first facility may be forwarded to the second facility over the private fiber backbone using a set of routing information. At least a portion of the set of routing information may be obtained from a second dynamic routing information source by a protocol processing engine associated with the second virtual router. The protocol processing engine may be configured to process messages of a particular protocol specified by a client.

[0020] As mentioned above, in at least some embodiments, the types of virtual routers and / or associated auxiliary task offloaders described above may be implemented using resources of a provider network. A cloud provider network (sometimes simply referred to as a "cloud") refers to a network-accessible pool of computing resources (e.g., compute, storage, and networking resources, applications, and services), which may be virtualized or bare metal. A cloud may provide convenient on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to changing loads. Thus, cloud computing can be viewed as both applications delivered as a service over a publicly accessible network (e.g., the Internet, or a cellular communications network) and the hardware and software in the cloud provider's data center that provides those services.

[0021] A cloud provider network can be formed as multiple regions, where a region is a distinct geographical area in which a cloud provider clusters its data centers. Such regions may also be referred to as provider network-defined regions, since their boundaries may not necessarily coincide with national, state, or other boundaries. Each region may include two or more availability zones interconnected via a private high-speed network (e.g., fiber telecommunication connections). An availability zone (also called an availability domain, or simply a "zone") refers to an isolated failure domain that includes one or more data center facilities with a separate power source, separate network, and separate cooling from another availability zone. A data center refers to a physical building or enclosure that houses and provides power and cooling to the servers of a cloud provider network. Preferably, availability zones within a region are located far enough away from each other so that the same natural disaster does not take multiple availability zones offline at the same time. Customers can connect to the availability zones of the cloud provider network through publicly accessible networks (e.g., the Internet, cellular telecommunication networks) via transit centers (TCs). TCs can be considered as key backbone locations linking customers to the cloud provider network, and may be located in other network provider facilities (e.g., Internet service providers, telecommunications providers) and securely connected to availability zones (e.g., via VPN or direct connection). Each region may operate two or more TCs for redundancy. Regions are connected to a global network that connects each region to at least one other region. The cloud provider network may deliver content from points of presence that are outside of, but networked with, these regions via edge locations and regional edge cache servers (points of presence or PoPs). This partitioning and geographic distribution of computing hardware allows the cloud provider network to offer customers global low-latency resource access with a high degree of fault tolerance and stability.

[0022] In some embodiments, the provider network may implement various computing resources or services, which may include virtual computing services, data processing service(s) (e.g., map-reduce, dataflow, and / or other large-scale data processing techniques), data storage services (e.g., object storage services, block-based storage services, or data warehouse storage services), and / or any other type of network-based services (which may include various other types of storage, processing, analytics, communication, event processing, visualization, and security services not shown). Resources (e.g., computing resources and storage resources) required to support the operation of such services may be provisioned in an account associated with the cloud provider, as opposed to resources that may be requested by a user of the cloud provider network and provisioned in a user account.

[0023] Various network-accessible services may be implemented in one or more data centers of a provider network in various embodiments. The network-accessible computing services may include an elastic computing cloud service (referred to in various implementations as an elastic computing service, a virtual machine service, a computing cloud service, a computing engine, or a cloud computing service), which may provide virtual computing instances (also referred to as virtual machines, or simply "instances") with varying computing and / or memory resources, which are managed by a computing virtualization service (referred to in various implementations as an elastic computing service, a virtual machine service, a computing cloud service, a computing engine, or a cloud computing service). In one embodiment, each of the virtual computing instances may correspond to one of several instance types or families. An instance type may be characterized by its hardware type, computational resources (e.g., the number, type, and configuration of central processing units [CPUs] or CPU cores), memory resources (e.g., the capacity, type, and configuration of local memory), storage resources (e.g., the capacity, type, and configuration of locally accessible storage), network resources (e.g., its network interface characteristics and / or network capabilities), and / or other suitable descriptive characteristics (such as being a “burstable” instance type that has a baseline performance guarantee and can periodically burst above that baseline, or a non-burstable or dedicated instance type that has a fixed amount of resources allocated and guaranteed). Each instance type may have a particular ratio of processing, local storage, memory, and network resources, and different instance families may have different types of these resources. Within a given instance type, multiple sizes of these resource configurations may be available.The instance type selection functionality may be used to select an instance type for a customer based (at least in part) on input from the customer, for example. For example, a customer may select an instance type from a set of predefined instance types. As another example, a customer may specify desired resources for the instance type and / or requirements for the workload the instance will run, and the instance type selection functionality may select an instance type based on such specifications. A suitable host for the requested instance type may be selected based at least in part on factors such as collected network performance metrics, resource utilization levels on various available hosts, etc.

[0024] Computing services of the provider network may also include container orchestration and management services (referred to in various embodiments as container services, cloud container services, container engines, or container cloud services). A container represents a logical packaging of a software application that abstracts the application from the computing environment in which the application runs. For example, a containerized version of a software application includes the software code and any dependencies used by the code, allowing the application to run consistently on any infrastructure that hosts a suitable container engine (e.g., Docker® or Kubernetes® container engines). Compared to a virtual machine (VM), which emulates an entire computer system, a container virtualizes at the operating system level and therefore typically represents a more lightweight package for running an application on a host computing system. An existing software application can be "containerized" by packaging the software application in an appropriate manner and generating other artifacts (e.g., container images, container files, or other configurations) that are used to enable the application to run on a container engine. In some embodiments, the container engine can run on a virtual machine instance, which is selected based at least in part on the described network performance metrics. In some embodiments, other types of network-accessible services, such as packet processing services, database services, wide area networking (WAN) services, etc., may also be implemented in the cloud provider network.

[0025] Cloud provider network traffic and operations, in various embodiments, may be roughly subdivided into two categories: control plane operations carried on a logical control plane, and data plane operations carried on a logical data plane. The data plane represents the movement of user data through a distributed computing system, and the control plane represents the movement of control signals through a distributed computing system. The control plane generally includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic typically includes management operations such as system configuration and management (e.g., resource allocation, hardware capacity management, diagnostic monitoring, system state information). The data plane includes customer resources (e.g., computing instances, containers, block storage volumes, databases, file storage) implemented on the cloud provider network. Data plane traffic generally includes non-management operations such as customer data transfer to and from customer resources. While certain control plane components (e.g., tier 1 control plane components such as the control plane of virtualized computing services) are typically implemented on a set of servers separate from the data plane servers, other control plane components (e.g., tier 2 control plane components such as analytics services) may share virtualization servers with the data plane, and control plane traffic and data plane traffic may be transmitted over separate / separate networks.

[0026] Exemplary System Environment for Offloading Virtual Router Resources FIG. 1 illustrates an exemplary system environment in which a scalable virtual router for traffic flowing between isolated networks may be implemented, according to at least some embodiments. As shown, the system 100 includes an instance 102 of a scalable virtual router (VR) configured, in the illustrated embodiment, using resources of a multi-layer packet processing service (PPS). The VR instance 102 may be used to enable connectivity between multiple isolated networks 140A-140D. The PPS may include, for example, in the illustrated embodiment, a management or control plane 190, as well as a data plane including fast path nodes (FNs) 114 and exception path nodes (ENs) 115. In some embodiments, the FNs and ENs may be collectively referred to as forwarding nodes 111 or forwarding plane nodes. In the illustrated embodiment, the control plane may be responsible for configuring the VR instance and associated routing / forwarding metadata 108, while the data plane resources may be used to generate and implement actions for routing packets originating at (and destined for) the isolated network 140. In various embodiments, multiple VR instances may be configured in response to requests from clients of the provider network. In some cases, multiple VR instances may be configured for a single client, for example to handle traffic between different sets of isolated networks. In some embodiments, some VRs may be configured in single-tenant mode (e.g., to handle application data for a single client), while other VRs may be configured in multi-tenant mode (to handle application data for multiple different clients). In at least one embodiment, the tenancy mode used for a given virtual router may be indicated by the client for which the VR is configured.

[0027] Connections between the many different types of isolated networks 140 may be provided using VR instances 102 in the illustrated embodiment, for example, in response to program requests sent from PPS clients 195 to PPS control plane 190 via interface 170. For example, isolated network 140A may include a set of resources located in a data center or facility outside the provider network's own data center, which may be linked to the provider network using a VPN (Virtual Private Network) tunnel or a connection that, in the illustrated embodiment, utilizes a portion of the public Internet. Isolated network 140B may also include resources located in a facility outside the provider network, which, in the illustrated embodiment, is connected to the provider network via a dedicated physical link (sometimes referred to as a "direct connect" link). Isolated networks 140C and 140D may include respective isolated virtual networks (IVNs) configured using resources located in the provider network's data center in the illustrated exemplary scenario. An isolated virtual network may include a collection of networked resources (including, for example, computing instances such as virtual machines) assigned to a given client of a provider network that are logically isolated (by default inaccessible) from resources assigned to other clients in other isolated virtual networks. The client on whose behalf an IVN is established may be given significant flexibility with respect to the network configuration of the IVN's resources. For example, a private IP address for a virtual machine may be selected by the client without having to consider the possibility that other resources in other IVNs may have been assigned the same IP address, that a subnet selected by the client may have been established in the IVN, that security rules may have been set by the client for traffic to and from the IVN, etc.Similar flexibility may be applied to configurations in VPN-connected isolated networks such as 140A, in the illustrated embodiment, and / or isolated network 140B connected to the provider network via a dedicated link.

[0028] In addition to the baseline or primary task of packet forwarding / routing, one or more types of auxiliary tasks may be performed for traffic between various (or all) pairs of isolated networks 140, depending on the requirements of the client for which the VR instance 102 is configured. For example, if a client indicates that dynamic routing using BGP or a similar protocol is implemented for packets flowing between a given pair of isolated networks 140, BGP messages may need to be processed and the results of the BGP processing may need to be inserted into one or more route tables 109. Similarly, multicast applications may require processing of IGMP messages, some packet flows may require IPSec (Internet Protocol Security) or other security-related processing, etc. To facilitate such auxiliary tasks without adding to the primary workload of the forwarding node 111, in the illustrated embodiment, a set of offload resources 160 may be configured for the VR instance 102, e.g., by the PPS control plane 190. The particular category (or categories) of auxiliary tasks required for traffic between a given pair of isolated networks (or a given packet flow) may be determined based on input provided by the PPS client 195, in various embodiments. Exemplary auxiliary task categories, in various embodiments, may include, among others, routing configuration management, packet content transformation (e.g., using cryptographic protocols), performance monitoring, availability monitoring, etc. After one or more auxiliary task offload resources 160 for the required categories of tasks are provisioned, in various embodiments, connections between at least some of the forwarding nodes 111 and the auxiliary task offload resources may be established. In at least some embodiments, some of the packets received at the VR instance 102 may trigger a request for an offload resource. For example, if a BGP message is received at the VR, the BGP message may be sent to a BGP processing offload resource.In various embodiments, the necessary tasks may be performed on the offload resources 160 and the results (if there are any results to be consumed by the forwarding node) may be sent to the forwarding node 111. Such results may then be used, in various embodiments, to forward at least some packets from one isolated network 140 to another. For example, the preferred next hops of packets of a packet flow, as determined as a result of the BGP offload tasks, may be used to route the packets of the flow.

[0029] In at least some embodiments, the PPS client 195 may provide, e.g., using programmatic interfaces 170 of the PPS control plane 190, at least some of the routing / forwarding metadata 108 of the VTH instance that is used to generate the actions that are ultimately used to forward network packets between the isolated networks 140. In the illustrated embodiment, the routing / forwarding metadata 108 may include multiple route table 109 and / or policy-based routing rule 110 entries indicated by the client. A given isolated network 140 may be programmatically associated with a particular route table 109, e.g., using a first type of programmatic interface in the illustrated embodiment (the interface used for the "associate" verb or action). Such associated route table 109 may be used to direct at least a subset of outbound packets from the isolated network. In another type of programmatic action, route table entries whose destinations are within the given isolated network 140 may be programmatically propagated / installed in one or more route tables (e.g., using a different interface for propagating or installing entries in a particular table) so that traffic from other sources may be received at the isolated network. In at least some embodiments, an entry with a destination in a particular isolated network, such as 140C, may be propagated to one or more route tables 109 associated with other isolated networks, such as 140A or 140B, for example, allowing traffic to flow from the other isolated networks to 140C along paths 155A and 155B. Similarly, one or more entries with a destination in isolated network 140D may be propagated to a route table associated with isolated network 140C, allowing traffic to flow from isolated network 140C to isolated network 140D along path 155C.For traffic forwarded over path 155D, in the illustrated embodiment, an entry with a destination in isolated network 140D may be propagated to a route table associated with isolated network 140B. In general, any desired combination of unidirectional or bidirectional traffic between a given pair of isolated networks programmatically attached to a VR instance 102 may be enabled in various embodiments by using an appropriate combination of route table associations and route table entry propagation. This allows a wide variety of network flow configurations to be supported in different embodiments, as described in further detail below.

[0030] After the routing metadata 108 and auxiliary task offload resources 160 are configured, network packets containing application data may be accepted at the FN 114 of the VR instance (e.g., including one or more action implementation nodes or AINs) from various resources in the different isolated networks 140 in the illustrated embodiment. When a packet is received at an AIN, the AIN may, in various embodiments, attempt to find a matching action in its action cache (e.g., using a key based on various properties of the packet's flow, including a combination of source and destination IP addresses and ports). If the action is not found in the cache, the EN 115 (e.g., including the decision node (DN) of the VR instance) may be consulted by the AIN. In some embodiments, the DN may look in its own cache for a previously generated action appropriate for the received packet. If a pre-generated action is found, it may be provided to the AIN for cache entry and implementation. If no such action is found by the DN, a new action may be generated, for example, in the illustrated embodiment, using one of the route tables 109 associated with the source isolated network where the packet was received. An executable version of the action (e.g., bytecode expressed using instructions of a register-based virtual machine optimized for implementing network processing operations) may be generated, optionally cached at the decision layer, and provided to the AIN, where, in various embodiments, it is implemented for the current packet (and cached and reused for subsequent packets of the same flow).

[0031] In various embodiments, a given flow for which an action is generated may be characterized (or distinguished from other flows) based on one or all of the following attributes or elements of a packet received at a packet processing service (PPS): the network protocol used to send the packet to the PPS, a source network address, a source port, a destination network address, a destination port, and / or an application identifier (e.g., an identifier of a particular virtual network interface configured for communication between the isolated network and the PPS). In some embodiments, the direction in which the packet is sent (e.g., toward or away from the PPS) may also be included as an identifier of the flow. Packets formatted according to a number of different networking protocols may be processed and / or forwarded between isolated networks 140 by forwarding node 111 of VR instance 102 in different embodiments. Protocols include, for example, Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), Internet Control Message Protocol (ICMP), protocols not belonging to or dependent on the TCP / IP protocol suite, etc. Messages formatted according to various additional protocols, such as BGP, IGMP, IPSec, TWAMP (Two-Way Active Measurement Protocol), etc., may be at least partially processed by auxiliary task offload resources 160 in the illustrated embodiment.

[0032] Exemplary Packet Processing Applications and Auxiliary Tasks 2 illustrates exemplary categories of packet processing applications implemented with the aid of a virtual router, according to at least some embodiments, and auxiliary tasks that may be performed for some of the categories. As shown, application categories 200 in the illustrated embodiment may include, for example, scalable cross-IVN (isolated virtual network) channels 206, scalable VPN (virtual private network) connections 208, scalable dedicated link connections 210, multicast 212, address substitution 216, network traffic security / audit applications 218, scalable WAN (wide area networking) using a private backbone network of a provider network, and the like. Other types of packet processing applications may be supported in various embodiments. In general, a virtual router similar to virtual router instance 102 of FIG. 1 may be configured to implement any desired type of packet processing or transformation (or a combination of different types of packet processing or transformation) (e.g., with the aid of auxiliary task offload resources) with virtual router nodes and / or auxiliary resources dynamically allocable as needed to support a wide range of traffic rates in a transparent and scalable manner.

[0033] In some embodiments, as previously described, a virtual router (VR) may be implemented in the provider network that can establish isolated virtual networks. In such embodiments, the VR may act as an intermediary or channel between the private address spaces of two or more different IVNs, in effect setting up a scalable and scalable cross-IVN channel 206. In at least some embodiments, the cross-IVN channel may not require auxiliary tasks. For a scalable VPN connection 208, in the illustrated embodiment, auxiliary tasks such as BGP processing, encryption, etc. may be required, and such auxiliary tasks may be implemented using offloaded resources associated with the VR. The scalable VPN connection may be established, for example, to one or more client-owned facilities outside the provider network, and such facilities may include routers or other equipment with BGP processing engines. A BGP session may be set up between such an external processing engine and a BGP processing engine that is configured with auxiliary tasks that in some embodiments offload resources of the type introduced above.

[0034] In some embodiments, the provider network may support scalable connectivity 210 with external networks via dedicated physical links, referred to as "direct connect" links, and traffic between such external networks (and between such external networks and IVN or VPN connected external networks) may be managed using virtual routers. The auxiliary tasks in such scenarios, at least in some embodiments, may also include BGP processing, including, for example, processing of BGP session messages exchanged between a BGP processing engine at the external facility and a BGP processing engine at the offload resource.

[0035] Multicast 212 is a network technology that, in some embodiments, can be implemented using VR, where the contents (e.g., body) of a single packet sent from a source are replicated to multiple destinations in a specified multicast group. In some embodiments, multicast group membership information may be periodically obtained and / or verified via IGMP messages, so that auxiliary tasks including IGMP message processing can be performed using offload resources of the multicast application.

[0036] Address substitution 216, as the name suggests, may involve substituting source addresses and ports in a consistent manner for packets of a particular flow. Such address substitution techniques may be useful, for example, when overlap exists between the private address ranges of two or more isolated networks, and in some embodiments, a VR may be used as an intermediary to handle such substitution. In the illustrated embodiment, address substitution may not require auxiliary tasks.

[0037] Some clients of the provider network may wish to implement network traffic security or auditing applications 218 for at least a subset of the traffic flowing through the VR between various sets of endpoints. The subset of traffic may be indicated, for example, via client-assigned tags or labels and specified by the client in the form of policy-based routing rules. For such applications, in the illustrated embodiment, the auxiliary tasks may include processing related to IPSec or other client-selected security protocols, audit log generation and management tasks, etc. For scalable wide area networking applications 220, in some embodiments, the auxiliary tasks may include BGP processing, performance measurement including TWAMP processing or custom latency measurement protocols for the provider network, etc.

[0038] It should be noted that in at least some embodiments, a single VR may combine some of the packet processing functions (and / or other packet processing techniques) shown in Figure 2. For example, in some embodiments, a single VR may simultaneously implement (or simultaneously implement in conjunction with other VRs) scalable cross-IVN channels, scalable VPN connections, scalable dedicated link-based connections, etc. Other categories of packet processing may be supported using VRs in different embodiments, although at least some of the types of applications shown in Figure 2 may not be supported in some embodiments.

[0039] Example of interaction between virtual router nodes and offload resources 3 illustrates an overview of an example of interactions between a virtual router's exception path nodes, a virtual router's fast path nodes, and auxiliary task offload resources associated with the virtual router, according to at least some embodiments. In the illustrated embodiment, the virtual router 327 is assigned to process traffic between a client traffic source endpoint 364 and a client traffic destination endpoint 372 of one or more PPS clients 310. The PPS clients 310, in the illustrated embodiment, may send application setting / configuration requests 343 to the PPS control plane 314, for example, via a web-based console, a command line tool, an API, a graphical user interface, etc. The requests 343 may indicate the type of packet processing to be performed with the aid of the VR 327 (e.g., policies to be implemented for packet forwarding / routing), the desired performance, or other goals to be met, etc. In the illustrated embodiment, based on the client's requirements and / or the availability and current resource consumption levels of various resources of the PPS, the PPS control plane 314 may identify or configure a set of exception path nodes (ENs, also referred to as decision nodes) 325 and a set of fast path nodes (FNs, also referred to as action implementation nodes) 368 of the VR 327. Additionally, in at least some embodiments, resources for performing auxiliary tasks (e.g., tasks of the type shown in FIG. 2) related to traffic between the endpoints 364 and 372 may also be similarly configured in the illustrated embodiment, such as auxiliary task offloaders 373A and 373B. In various embodiments, a connection may be established between the auxiliary task offloader 373 and at least some nodes of the virtual router 327. In some embodiments, a connection may be established only between the FNs 368 and the auxiliary task offloaders. In other embodiments, a connection may also or instead be established between the ENs 325 and the auxiliary task offloaders.

[0040] Configuration metadata 305, such as client-provided Forwarding Information Base (FIB) entries, client-indicated Policy Based (PBR) routing rules used to make packet processing decisions, etc., may be transmitted from the PPS control plane 314 to one or more ENs 325 in the illustrated embodiment. In some embodiments where a given VR 327 includes multiple ENs, all ENs may be provided with all metadata regarding the application or applications to which the VR 327 is assigned. In other embodiments, respective subsets of the metadata may be provided to individual ENs.

[0041] When a packet is received from an application's traffic source endpoint 364 at the FN 368, an attempt may be made to find a corresponding action in the action cache 397. If such an action is found, e.g., by lookup using a key based on some combination of packet header values, client identifiers, etc., the action is implemented, which results in, in the illustrated embodiment, sending at least some content of the received packet to one or more destination endpoints 372. This "fast path" 308 processing, where a cache hit occurs and the EN is not directly involved, may in various embodiments be encountered in practice much more frequently than the slower case of a cache miss (or when some auxiliary task needs to be performed). Note that in at least some applications, the total number of packets for which the same logical action is implemented may be very large. For example, hundreds or thousands of packets may be sent from one source endpoint to a destination endpoint using the same long-lived TCP connection.

[0042] In a scenario in which the arrival of a packet causes a cache miss at the FN 368, in the illustrated embodiment, a request-response interaction 307 with the EN 325 may be initiated by the AIN. An action query (which in some implementations may include the entire received packet, and in other implementations may include a representation or portion of the packet, such as some combination of its header values) may be sent from the FN 368 to the EN 325. The EN 325, for example, inspects the action query and the contents of the configuration metadata 305 (including PBR rules 337) and, in the illustrated embodiment, determines an action to be implemented for the packet causing the cache miss and related packets (e.g., packets that belong to the same flow, where the flow is defined at least in part by some combination of packet header values). In at least some embodiments, the EN 325 may include an action code generator 326 that generates an executable version of an action that (a) can be executed quickly at the FN and (b) does not necessarily need to be interpreted or "understood" at the FN. In at least one embodiment, the generated action may include instructions of an in-kernel register-based virtual machine instruction set that can be used to perform operations similar to those of an extended Berkeley Packet Filter (eBPF) interface, which, in at least some embodiments, may be returned to the FN for caching and for implementation on packets that cause cache misses.

[0043] In at least some embodiments, this type of cache-miss request-response path may also be used for auxiliary tasks. For example, configuration metadata 305 may indicate to the EN (e.g., using encapsulation techniques described below) that for a particular type of packet (such as a BGP packet received at the FN from a BGP processing engine at the client premises), the action to be taken is to send the packet to an auxiliary task offloader. In some implementations, the FN may send such a packet to the auxiliary task offloader 373A by implementing an action generated at the EN and cached in cache 397. When a response packet containing a result 308 of such an auxiliary task is received from the FN's auxiliary task offloader 373A, in some embodiments, the result 308 is sent to the EN, where it may be used to generate or modify an action used to forward one or more subsequent packets between endpoints 364 and 372. In one embodiment, the EN may be configured to communicate directly with the auxiliary task offloader 373B instead of using the FN as an intermediary. In such embodiments, the results of the auxiliary tasks may be received directly by the EN and may be used to generate / modify the actions used to forward client traffic between endpoints 364 and 372. Note that in at least some embodiments, at least some of the network packets or messages that trigger the auxiliary tasks may be directed to VR 327's own node (e.g., have a destination address assigned to the VR node), as opposed to client application packets that are directed to endpoint 372.

[0044] At the FN 368 that sent the action query, the generated action is stored in the cache 397 and, in various embodiments, may be reused as necessary for other packets in addition to the initial packet that resulted in the identification and generation of the action. Any of a variety of eviction policies may be used to remove entries from the cache 397. For example, if no packets requiring the execution of a given action A1 have been received for some threshold time interval, then in one embodiment, A1 may be removed from the cache. In at least one embodiment, each entry in the cache may have an associated usage timing record that includes, for example, a timestamp corresponding to the last time the action was performed on any packet. In such an embodiment, an entry may be removed from the cache if / when its usage timing record indicates that an eviction criterion has been met (e.g., when an action has not been performed for some threshold seconds / minutes). In some embodiments, cached actions may be periodically rechecked with respect to the current state of the configuration metadata 305. For example, every T seconds (T is a configurable parameter), the FN may send a revalidation query to the EN indicating the cached action, and the EN may verify that the cached action has not been invalidated by a newly updated configuration metadata entry. Note that in various embodiments, as long as the action ultimately taken for a given received packet is correct, from a functional standpoint it may not matter whether the action was cached at the FN or had to be generated at the EN. Thus, even if actions are occasionally evicted from the cache 397 unnecessarily, or as a result of overly pessimistic eviction decisions, in such embodiments the overall impact to the packet processing application may be small (as long as unnecessary evictions do not occur frequently).

[0045] Example independent scaling of offload resources One advantage of separating resources used for auxiliary tasks from resources used for baseline forwarding tasks is that workload changes can be handled independently by the two types of tasks. FIG. 4 illustrates an exemplary scenario in which resources used for a virtual router may be automatically scaled independently of resources used for auxiliary tasks related to traffic routed through the virtual router, according to at least some embodiments. A set of VR scaling managers 477 may be assigned responsibility for collecting and analyzing workload and resource utilization levels for VR resources such as FNs and ENs, and in the illustrated embodiment, initiating acquisition or release of VR resources as needed in response to trends or changes in the collected metrics. Similarly, a set of auxiliary task offloader scaling managers 478 may be assigned responsibility for collecting and analyzing workload and resource utilization levels for a set of auxiliary task offloaders (ATOs) associated with the virtual router, and initiating acquisition or release of resources for the ATOs independently of changes initiated by the VR scaling managers.

[0046] The initial VR resource set 410A for a given VR may, for example, in the illustrated exemplary scenario, include four fast path nodes (FNs 402A, 402B, 402C, and 402D) and a pair of exception path nodes (ENs 403A and 403B), each including, for example, one or more processes or threads executing on a respective computing device. The initial auxiliary task offload resource set 450A may include two ATOs 491A and 491B, each of which may include one or more processes or threads executing on a respective computing device. As the rate of application data packets arriving at the VR changes, in the illustrated embodiment, the resource set 410A may be expanded or contracted by the VR scaling manager, independent of changes in workload or configuration at the ATOs. For example, if the arrival rate of application data traffic increases above some threshold, causing corresponding increases in CPU and / or memory utilization levels at the FNs and ENs, and remaining above the threshold for a selected time, two new FNs (402E and 402F) and a new EN 403C may be instantiated and added to the VR, resulting in a scaled-up VR resource set 410B. Alternatively, if the application data traffic decreases below the threshold and remains below the threshold for a while, one of the FNs (402D) and one of the ENs (403B) of the initial VR resource set 410A may be decommissioned, and the VR resource set 410C is contracted or scaled down.

[0047] ATO scaling manager 478 may decide to add more resources (e.g., ATOs 491C and 491D) to initial ATO resource set 450A, which in the illustrated embodiment may result in a scaled-up ATO resource set 450B if resource utilization levels or other metrics collected from the initial ATO resource set meet scale-up criteria. Alternatively, if metrics collected from initial ATO resource set 450A meet different criteria, ATOs such as 491B may be deactivated, resulting in ATO resource set 450C being scaled down. Changes to the ATO resource set configuration may be made asynchronously with respect to changes to the VR resource set in various embodiments.

[0048] Exemplary Cell-Based Virtual Router Architecture In some embodiments, resources used in the forwarding operation of a virtual router may be arranged in autonomous groups called cells. Figure 5 illustrates an example of the use of independently managed packet processing cells for a virtual router, according to at least some embodiments. As shown, a packet processing service (PPS) 502 that can configure a virtual router in response to client requests may include, in the illustrated embodiment, a set of service level control plane resources 571 including API handlers, metadata stores / repositories, etc., in addition to an action implementation layer 541, a decision layer 542, and a cell management layer 543. Each of layers 541, 542, and 543 may include multiple nodes, such as fast path nodes (FNs) in layer 541, exception path nodes (ENs) in layer 542, and management nodes (ANs) in layer 543. The resources of layers 541, 542, and 543 may be organized into groups called quarantine packet processing cells (IPPCs) 527, in various embodiments, where a given IPPC 527 includes several FNs, several ENs, and several ANs. For example, in the illustrated embodiment, IPPC 527A may include FNs 520A, 520B, and 520C, ENs 522A and 522B, and ANs 525A and 525B, while IPPC 527B may include FNs 520L, 520M, and 520N, ENs 522C and 522D, and ANs 525J and 525K. Individual nodes such as FNs, ENs, and / or ANs may be implemented using any combination of software and hardware in one or more computing devices in different embodiments. For example, in some embodiments, a given FN, EN, or AN may include one or more threads or processes of a virtual machine running on a host managed by a virtualized computing service of the provider network, while in other embodiments, the FNs, ENs, and / or ANs may be implemented using non-virtualized servers.

[0049] The resources of packet processing service 502 may serve as an infrastructure or framework that can be used to build various networking applications that use virtual routers, such as the types of applications described in the context of FIG. 2. In some embodiments, an individual IPPC 527 may be assigned to implement the logic of one or more instances of such an application, and traffic associated with that application is processed (at least under normal operating conditions) without crossing an IPPC boundary. For example, in the illustrated embodiment, IPPC 527A may be assigned to an instance of a VR (VR-A) to transmit packets between at least isolated networks 510A and 510B, while IPPC 527B may be assigned to another VR instance (VR-B) to transmit packets between at least isolated networks 510J and 510K. Each network of the isolated networks 510 may have associated private IP (Internet Protocol) address ranges, such that addresses assigned to resources in a given isolated network 510 may not be visible to resources outside the isolated network, and routes between resources in different isolated networks may not necessarily be available, at least by default (e.g., prior to allocation of IPPCs implementing virtual routing applications).

[0050] In various embodiments, an instance of a VR may be configured in response to a programmatic request received from a customer of the PPS 502. Such a request may be received, for example, at an API handler of the PPS control plane 571. For example, in response to a client request(s) to enable connectivity between isolated networks 510A and 510B, in the illustrated embodiment, VR-A, constructed using IPPC 227A, may be assigned to forward packets between the two isolated networks. Similarly, in response to another client request (or the same client request) to enable multicast connectivity between isolated networks 510J, 510K, and 510L, IPPC 527B may be assigned. In at least some embodiments, a collection of virtual network interfaces may be programmatically configured to enable traffic to flow between endpoints (TEs 512D, 512E, 512J, 512K, 512P, 512Q, 512R, 512S, 512V, 512W, etc.) in the isolated networks and the FNs of the cells assigned to those isolated networks. Clients on whose behalf networking applications are configured may, in some embodiments, provide decision metadata (e.g., Layer 3 metadata 523 such as forwarding information base entries, route table entries, etc.) and / or policies that can be used to determine actions to be taken via the control plane programmatic interfaces of the PPS. Metadata received from the client may be propagated to the appropriate IPPC 527 decision layer node, e.g., from the PPS API handler via AN 525 or directly, in the illustrated embodiment. In at least some embodiments, the metadata initially provided by the client may be transformed by converting the high level information into more specific route table entries that take into account, for example, identifiers of the virtual network interfaces used, locality related information, information about availability zones or availability containers in which various FNs are configured, etc., and the transformed versions may be stored in different ENs 522.

[0051] In the illustrated embodiment, a given packet from a source endpoint, such as TE 512K of isolated network 510A, may be received at a particular FN, such as 520C. The particular FN used may be selected based on a shuffle sharding algorithm in some embodiments, such as, for example, such that packets of a particular flow from a particular endpoint are directed to one of a subset of the cell's FNs. Each FN may include or have access to a respective action cache, such as action cache 521A. The action cache may be indexed by a combination of attributes of the received packet, such as a combination of a sending client identifier, source and destination IP addresses, source and destination ports, etc. In some embodiments, the actions may be stored in the cache in executable form, for example, using bytecode expressed using instructions of a register-based virtual machine optimized for implementing network processing operations. FN 520C may attempt to look up a representation of the action for the received packet in its cache. If such an action is found, in the illustrated embodiment, the packet may be processed using a "fast path" 566. For example, an executable version of the action may be implemented in FN 520C, resulting in the packet contents being sent on a path toward one or more destination endpoints, such as TE 512E in isolated network 510B. The path may include zero or more additional FNs. For example, as indicated using arrows 561 and 562, the packet contents may be sent to TE 512E via FN 520B on the illustrated high speed packet path. For example, FN 520B may have a virtual network interface configured to access TE 512E, while FN 520C may not have such a virtual network interface configured, resulting in the packet contents being sent via FN 520B.Note that in at least some embodiments, one or more header values ​​of the packet may be altered by the action (e.g., in a scenario where overlapping private address ranges happen to be used in the source and destination isolated networks), that is, the packet ultimately received at the destination endpoint 512E may have one or more header values ​​that differ from the packet sent from the source endpoint 512K.

[0052] If the FN's local action cache does not contain the action for the received packet, the workflow may be somewhat longer. Thus, for example, if a packet is received from the TE 512P of the FN 520M (as indicated by arrow 567) and a cache miss occurs in the local cache of the FN 520M when a lookup of the received packet is attempted, the FN 520M may send an action query to a selected EN (EN 522D) in the IPPC 527B, as indicated by arrow 568. The EN 522D may determine, for example, based on a client-provided policy indicating that a multicast operation is to be performed and based on forwarding / routing metadata provided by the client, that the contents of the packet are to be sent to a pair of endpoints 512R and 512V in isolated networks 510K and 510L, respectively, in the illustrated example. A representation of the action to effect such a multicast operation may be sent back to the FN 520M, stored in its local cache, and executed by the FN 520M, resulting in a transmission indicated by arrows 569 and 570. In this example, FN 220M may be able to send outbound packets directly to destinations TE 512R and 512V, without having to use paths involving other FNs in IPPC 527B.

[0053] Depending on the type of packet processing application implemented using a VR, such as VR-A or VR-B, auxiliary tasks may be performed in addition to the baseline forwarding actions in the illustrated embodiment. For example, in the case of multicast, messages formatted according to IGMP containing multicast domain configuration information are received at the FN of the VR (e.g., from an IGMP processing engine in a source-isolated network or a destination-isolated network), and processing of the IGMP messages may constitute one category of such auxiliary tasks. In various embodiments, one or more auxiliary task offloaders 577 of the type introduced above may be associated with a given VR, such as VR-B. In the illustrated embodiment, the auxiliary task offloader(s) 577 may be implemented in another isolated network 510Y (e.g., an isolated virtual network of a virtualized computing service) configured specifically to handle such auxiliary tasks.

[0054] A given IPPC 527 may be referred to as being "isolated" in some embodiments because, at least during normal operating conditions, data plane network traffic may not be expected to flow from that IPPC to other IPPCs. In at least one embodiment, control plane traffic may not flow across cell boundaries under normal operating conditions. Such isolation may result in many benefits. For example, (a) an increase in the workload of one VR implemented using one IPPC may not impact resources used for other VRs in other cells, and (b) in the unlikely event of a failure within a given cell, the failure may not be expected to impact applications to which other VRs are assigned. Because software updates may be applied to one IPPC's nodes at a time, bugs potentially introduced by such updates may not impact applications that use other cells. In some embodiments, while at least one IPPC may be assigned to a given VR instance, a given IPPC 527 may be used in a multi-tenant mode with multiple VRs configured on behalf of multiple customers.

[0055] In at least some embodiments, a shuffle sharding algorithm may be used to assign a subset of the nodes of the IPPC527 (e.g., FN) to a given set of one or more source endpoints or destination endpoints. According to such an algorithm, if the IPPC includes N FNs, a packet from a given source endpoint E1 may be directed to one of a subset S1 of K FNs (K < N) (e.g., based on the hash of the packet header value). And a packet from another source endpoint E2 may be directed to another subset S2 of K FNs, where in this case the maximum overlap between S1 and S2 is limited to L common FNs. Similar parameters may be used in various embodiments for connecting outbound packets to destination endpoints. In at least some embodiments, such shuffle sharding techniques may combine the advantages of hash-based load balancing with higher availability of the individual traffic of source and destination endpoints.

[0056] Exemplary cell-based architecture of the auxiliary task offloader In some embodiments, auxiliary task offloaders configured for a virtual router may also be organized into cells for similar reasons as described above for using a cell-based architecture for virtual router nodes. FIG. 6 illustrates an example of the use of independently managed auxiliary task offload cells for a virtual router, according to at least some embodiments. In the illustrated embodiment, a provider network 602 may include a virtualized computing service (VCS) 605 in which isolated virtual networks may be established on behalf of various customers or clients and / or to implement various functions of the provider network services. In the embodiment illustrated in FIG. 6, an IVN 610A may include resources used for packet processing cells assigned to virtual router VR-A, while an IVN 810B may include resources used for auxiliary task offload cells (ATOCs) associated with VR-A. IVN resources (e.g., including computing instances or virtual machines) may, in at least some embodiments, be logically isolated (by default, inaccessible) from resources assigned to other isolated virtual networks. In the illustrated embodiment, the packet processing service itself may be considered a client or customer of the VCS 605. That is, packet processing services may be built by utilizing the features supported by the VCS 605. As previously mentioned, a client on whose behalf an IVN is established may be given significant flexibility with respect to the network configuration of resources of the IVN. For example, a private IP address of a computing instance may be selected by the client without having to consider the possibility that other resources in other IVNs may be assigned the same IP address, that a subnet selected by the client may be established in the IVN, that security rules may be configured by the client for traffic to and from the IVN, that a virtual network interface may be configured at the client's request to enable connectivity among a particular group of resources at the client's request, etc.

[0057] In at least some embodiments, resources of VCS 605, such as hosts on which various computing instances run, may be distributed among multiple availability containers 650, such as 650A and 650B. An availability container, which may then be referred to as an availability zone, may include some or all of one or more different locations or data centers designed in such a way that resources in a given availability container are isolated from failures in other availability containers (e.g., with independent infrastructure components such as power-related equipment, cooling equipment, or physical security components). A failure in one availability container may not be expected to result in failures in any other availability containers, and thus the availability profile of a given resource is intended to be independent from the availability profiles of resources in different availability containers.

[0058] In the illustrated embodiment, the fast path node (FN) 625, exception path node (EN) 627, and management node (AN) 629 (similar in functionality to those shown in FIG. 5) of a given IPPC configured for VR-A may all be implemented at least in part using respective computing instances (CIs) 620 of the VCS 605. As shown, FNs 625A, 625B, 625P, and 625Q may be implemented in CIs 620A, 620B, 620P, and 620Q. ENs 627A and 627B may be implemented in CIs 620D and 620R, respectively, and ANs 629A and 629B may be implemented in CIs 620L and 620S, respectively. In some embodiments, a given CI 620 may be instantiated on a respective physical virtualization host. In other embodiments, multiple CIs may be configured on a given physical host. The illustrated IPPC implemented in IVN 610A may include at least two data plane subnets 640A and 640B and at least two control plane subnets 642A and 642B. One data plane subnet and one control plane subnet may be implemented in each of at least two availability containers 650. For example, subnets 640A and 642A may be configured in availability container 650A, while subnets 640B and 642B may be configured in availability container 650B. In some embodiments, control plane subnet 642 may include one or more ANs 629 in each CI 620, while data plane subnet 640 may include one or more FNs 625 and one or more ENs 627 in each CI 620. As a result of the use of multiple availability containers, the probability that the entire IPPC (or any VR, such as VR-A, that uses the nodes of the IPPC) will be affected by any given failure event may be minimized in the illustrated embodiment. In various embodiments, using different subnets for control plane nodes and data plane nodes can help to isolate at least a majority of a VR's control plane traffic that uses IPPC from the VR's data plane traffic.

[0059] In the example scenario shown in FIG. 6, an auxiliary task offloading cell (ATOC) including auxiliary task offloaders (ATOs) 691A, 691B, 691J, and 691K configured in IVN 610B may be established for or assigned to VR-A. ATOC may also be implemented using resources distributed across availability containers 650A and 650B in the illustrated embodiment. ATO may also be implemented using compute instances. For example, ATOs 691A and 691B may each include one or more processes or threads in compute instance 690A in data plane subnet 643A, while ATOs 691J and 691K may be implemented in CI 690J in data plane subnet 643B. In the illustrated embodiment, a respective virtual network interface (VNI) 655 may be configured in each data plane subnet 643 to enable connectivity between FNs of VR-A in the same availability container. Thus, VNI 655A may be configured for connectivity between FNs 625A and 625B and ATOs 691A and 691B in VR-A, while VNI 655B may be configured for connectivity between FNs 625P and 625Q and ATOs 691J and 691K in VR-A. A virtual network interface may include a set of network configuration properties or attributes (e.g., IP addresses, subnet settings, security settings, etc.) that can be dynamically associated ("attached") or disassociated ("detached") with a computing instance, without requiring changes to be made at the physical network interface when, for example, a computing instance migrates from one physical host to another. As shown in the example of FIG. 6, the use of ATOC distributed among the same availability containers as used for VR nodes may make the processing of auxiliary tasks resilient to failures that are localized within individual availability containers. In some embodiments, separate control plane resources (not shown in FIG. 6) may also be configured to manage auxiliary task offloaders.

[0060] Example of Using Encapsulation Protocol Tunnels for Auxiliary Task-Related Messages 7 illustrates an example technique for connecting nodes of a virtual router with auxiliary task offloaders, according to at least some embodiments. In the illustrated embodiment, a fast path node FN 702 in a virtual router cell 710 has an associated FN virtual network interface 703. The auxiliary task offload cell (ATOC) 710 includes a VNI (AVNI) 754 associated with one or more computing instances on which ATOs 791A, 791B, and 791C execute.

[0061] For example, as part of the setup operations for auxiliary task processing initiated by the control plane components of the packet processing service in which the VR is implemented, in the illustrated embodiment, respective encapsulation protocol tunnels 755 (e.g., 755A, 755B, and 755C) may be established to transmit messages formatted according to various protocols, such as BGP, IGMP, etc., from the AVNI 754 to the ATO. In some implementations, the Generic Network Virtualization Encapsulation (GENEVE) protocol may be used for the tunnels, allowing packets of a wide variety of standard and / or custom protocols to be transmitted between the FN and the ATO using a common tunneling approach. An FN, such as FN 702, may be configured, for example, by an EN receiving metadata regarding the ATOC selected for the EN and the VR to which the FN belongs, to transmit packets requiring auxiliary task processing to the AVNI via the FN VNI 703 using possible actions similar to those used to forward application data packets between isolated virtual networks (IVNs), as shown in label 750, in the illustrated embodiment. That is, in at least some embodiments, a connection between the FN and the ATO may be enabled using methods similar to those used to enable a connection between the FN and an endpoint in the isolated network in which the VR is configured. Similarly, the FN VNI may be configured as the destination of outbound packets from the ATO (which may include the results of the auxiliary task) as part of the configuration operation in the illustrated embodiment, as indicated by label 757. Other techniques for establishing a bidirectional connection between a VR node and an ATO may be used in different embodiments.

[0062] Example programmatic interaction for offloading auxiliary tasks 8 and 9 illustrate example programmatic interactions between a client and a packet processing service related to configuration and use of a virtual router and associated auxiliary task offload resources, according to at least some embodiments. One or more programmatic interfaces 877 may be implemented by a packet processing service (PPS) 812, over which, in the illustrated embodiment, a virtual router is established. Such interfaces may include, for example, a set of application programming interfaces (APIs), a graphical user interface, a command line tool, a web-based console, etc. Configuration-related messages may be handled, for example, by a control plane component of the PPS.

[0063] A client 810 of a PPS 812 may send a CreateVirtualRouter request 814 to begin the process of configuring a VR in the illustrated embodiment. In response to the CreateVR request, a control plane component of the PPS may, in some embodiments, provide a VRID (virtual router identifier) ​​815 that indicates that the requested VR has been created (e.g., metadata describing the VR has been stored in the PPS's repository).

[0064] In some embodiments, a PktProcessingAppInfo message 817 may be sent over interface 877, e.g., indicating a type of packet processing application implemented using the virtual router. For example, one or more of the types of applications discussed in the context of FIG. 2 may be indicated in the PktProcessingAppInfo message, and / or one or more policy-based routing (PBR) rules may be specified for traffic processed using the VR. Information about the application may be saved in the PPS and an AppInfoSaved message 819 may be sent to the client. In at least one embodiment, information provided by the client regarding the application may be analyzed to identify types of auxiliary tasks that may be required for the application, and one or more auxiliary task offload resources may be configured accordingly. For example, the client may indicate that the VR is to be used for multicast and may configure an auxiliary task offloader that includes an IGMP message processor.

[0065] The client may send a programmatic request (CreateVRAttachment) 821 to, in some embodiments, attach a specified isolated network (e.g., an IVN within the provider network in which the PPS 812 is implemented, a VPN-connected network outside the provider network's data center, or an external network connected to the provider network via a dedicated physical link) or another VR to the specified VR, and receive an attachment identifier (AttachmentID) 823 in response. In various embodiments, a given VH may be programmatically attached to several different isolated networks and / or one or more other VRs with the CreateVRAttachment request. Attachments between pairs of VRs, called VR peering attachments, may be used, for example, for wide area networking applications, as described in more detail below. In some embodiments, a request may be sent to create and / or associate a specific route table with a particular isolated network to which an attachment was previously created, allowing the PPS to determine which specific route table to use for traffic originating in the particular isolated network.

[0066] A DescribeVRConfig request 825 may be sent by client 810 to obtain the current configuration of the specified VR (e.g., the different attachments created, the mapping between route tables and isolated networks, whether an auxiliary task offloader has been configured and, if so, the type of auxiliary task offloader, etc.) in the illustrated embodiment. Configuration information for the specified VR may be provided via one or more VRConfigInfo messages 827 in the illustrated embodiment.

[0067] In some embodiments, a programmatic request (ModifyVRConfig) 829 may be sent by a client to the PPS to modify one or more operational parameters of a specified VR. For example, the client may specify new policy-based routing rules for a subset of traffic handled by the VR or modify existing rules. In the illustrated embodiment, the requested configuration changes may be implemented and a ModComplete response message 831 may be sent to the client.

[0068] In the illustrated embodiment, a client 810 may send a GetVRMetrics request 833 to obtain metrics related to operations performed in a specified VR. Such metrics may include, for example, the number of application data packets processed during a time interval (e.g., per isolated network attached to the VR), the number of messages related to auxiliary tasks processed during a time interval, etc. Metrics collected for a VR may be indicated via one or more VRMetrics messages 835.

[0069] As shown in FIG. 9, according to some embodiments, the client 810 may send a descriptor of a custom auxiliary task to be executed with respect to at least some of the traffic processed in the specified VR in a CustomAuxiliaryTaskDescriptor message 914 directed to the PPS 812. Such a task descriptor may include, for example, in different embodiments, the source code or executable code of the task, a filter used to identify a subset of packets on which the task is implemented (e.g., based on client-defined packet tags or labels, based on source / destination virtual network interfaces, based on source / destination isolated networks, etc.), how the results of the custom auxiliary task are saved or used, etc. The PPS may, in some embodiments, perform one or more validation / validation tests to ensure that the requested custom task can be implemented on the offloader type introduced above. If the tests are successful, in the illustrated embodiment, a TaskDescriptorSaved message 915 may be sent to the client.

[0070] In one embodiment, a client may want to control the tenancy mode of an auxiliary task offloader (e.g., whether an auxiliary task runs on a given device or host only for a single client or VR, or for multiple clients / VRs). In such an embodiment, a SetAuxiliaryTaskResourceTenancy request 917 may be sent via programmatic interface 877 indicating such tenant preferences. A TenancyInfoSaved response message 919 may be sent to the client after the preferences are received and saved in the PPS 812.

[0071] According to some embodiments, a client may want to control or specify the types of metrics collected for the VR's auxiliary tasks (e.g., the number / rate of BGP messages processed by the auxiliary task offloader, the number / rate of IGMP messages processed, the amount of memory or storage used to save state information associated with stateful auxiliary tasks, etc.) In such an embodiment, an AuxiliaryTaskMonitoringRequirements message 921 may be sent by the client indicating the client's monitoring-related preferences, and a MonitoringRequirementsSaved message 923 may be sent to the client after the preferences have been saved.

[0072] A client 810 may send a GetAuxiliaryTaskMetrics request 925 to obtain metrics about auxiliary tasks that are executed using VR and offload resources in the illustrated embodiment. A set of metrics collected over a specified period of time (or over a period of time selected by the PPS) may be provided to the client via one or more AuxiliaryTaskMetricsSet messages 927.

[0073] According to some embodiments, a client 810 may send a ModifyAuxiliaryTasks request 929 to the PPS to modify or disable one or more properties of one or more types of auxiliary tasks being performed on traffic processed by the client's VR. For example, the client may modify custom logic used for a specified subset of packets received at the VR or indicate additional auxiliary tasks to be performed on some subset of packets. In response, the PPS may propagate the requested changes to offloaders configured for the VR and send an AuxiliaryTasksModified message 931 to the client.

[0074] It should be noted that in some embodiments, different combinations of program interactions than those shown in Figure 9 may be supported for configuring and using a VR with an auxiliary task offloader. For example, in one embodiment, some of the described operations may be performed in response to a single request instead of using separate requests. For example, a combined request may be used to create a VR and attach a set of isolated networks to it, and a combined request for the attachment auxiliary task of traffic received via the attachment may be sent.

[0075] How to Offload Workloads from Virtual Routers 10 is a flow diagram illustrating aspects of operations that may be performed to offload certain types of tasks from a virtual router, according to at least some embodiments. As shown in element 1001, the set of isolated networks (INs) whose network traffic will be processed based on client-specified application requirements (such as policy-based routing (PBR) rules, multicast requirements, dynamic routing requirements, etc.) may be determined in the illustrated embodiment based on input received from the client via a programmatic interface at a packet processing service of the provider network, for example.

[0076] The PPS, in various embodiments, may identify one or more categories of auxiliary tasks related to sending network packets between the IN pair, IN1 and IN2 (in addition to baseline packet forwarding), based on, for example, an analysis of application requirements and / or based on additional programmatic interaction with the client (element 1004). The categories may include, for example, routing configuration management tasks using BGP, IGMP, or other protocols, encryption of packet contents using IPSec or other protocols, periodic performance measurements (e.g., using TWAMP), DNS tasks, client-specified custom tasks such as tagging-based packet analysis collection, etc.

[0077] In the illustrated embodiment, one or more virtual routers (VRs) may be configured for the client's application (element 1007) and programmatically attached to IN1 and IN2. A given VR may, in various embodiments, include nodes of two packet processing layers: a fast path layer that efficiently implements routing / forwarding actions, and an exception path layer that is responsible for specifying / generating routing / forwarding actions based on client-provided metadata or rules. The fast path and exception path nodes may be referred to as forwarding plane nodes.

[0078] One or more auxiliary task offloaders (ATOs) may be configured, for example, by the control plane of the PPS in the illustrated embodiment, to perform the necessary auxiliary tasks without adding to the workload of the fast path layer node and / or the exception path layer node (element 1010). For example, an ATO may include one or more processes or threads running on a computing instance running on a host other than the host used for the fast path layer or the exception path layer. A connection between the ATO(s) and one or more forwarding plane nodes (fast path nodes and / or exception path nodes) of the VR may be enabled in various embodiments. For example, such a connection may be established by configuring one or more virtual network interfaces through which the VR forwarding plane node can send packets requiring auxiliary task processing, configuring encapsulation protocol tunnels, etc., as described in the context of FIG. 7. In some embodiments, the PPS control plane may generate and specify one or more policy-based routing rules, which, when implemented in the exception path layer, cause packets requiring auxiliary tasks to be sent from the VR forwarding plane to the ATO, and response packets including the results of the auxiliary tasks to be sent back to the forwarding plane node.

[0079] After the initial configuration of the VR forwarding plane nodes and the ATO is complete, the client's application endpoint (e.g., IN1) may begin to send packets containing application data to a destination endpoint (e.g., IN2) via the VR (element 1013). Optionally, based on the particular category of auxiliary tasks identified for traffic between IN1 and IN2, a communication session for a protocol such as BGP may also be initiated between the protocol processing engines of the IN and the ATO, for example, in some embodiments.

[0080] When a packet is received at the VR configured for the client, in various embodiments, a determination may be made as to whether the packet requires or is about to trigger auxiliary task processing (element 1016). If the packet does not require any auxiliary tasks, an appropriate routing / forwarding action may be identified and implemented for the packet without utilizing ATO (element 1025). If the packet requires the execution of one or more auxiliary tasks, in the illustrated embodiment, an implicit or explicit request for the auxiliary task(s) is sent from the VR forwarding plane to the selected ATO (element 1019), and the results of the corresponding auxiliary task(s) may be obtained in the VR forwarding plane.

[0081] The results of the auxiliary task(s) may be used by the VR forwarding plane to transmit at least a portion of some packets between IN1 and IN2 in the illustrated embodiment (element 1022). Examples of results may include, in another embodiment, a route selected using a selected version or variant of BGP, identities of multicast group members verified using a selected version or variant of IGMP, encrypted / decrypted content of an application data packet obtained using a selected version or variant of IPSec or other security protocol, performance metrics obtained using a version or variant of TWAMP or other performance metrics collection protocol, etc. In some cases, such as encryption / decryption scenarios, the results of the auxiliary tasks may be embedded within packets sent to a network isolated from the VR forwarding plane. In other cases, as in the case of BGP / IGMP / TWAMP, the results may be used to select a preferred next hop or route for at least some of the packets, for example at an exception path node of the VR. At least some types of auxiliary tasks (e.g., BGP message processing, IGMP message processing, or TWAMP message processing) may be performed asynchronously with respect to the forwarding actions of the VR. That is, a fastpath node in VR may not need to wait until a given auxiliary task is completed to implement a forwarding action, even if the forwarding action performed at the fastpath node may be affected by the outcome of the auxiliary task.

[0082] Example System Environment Using Protocol Stack Multiplexing for Auxiliary Tasks 11 illustrates an exemplary system environment in which a protocol stack multiplexer and multiple protocol stack instances may be configured to offload auxiliary tasks of a scalable virtual router, according to at least some embodiments. As illustrated, the system 1100 may include an instance 1102 of a scalable virtual router (VR) of a packet processing service of a provider network, configured to transmit network packets containing application data between several isolated networks according to packet processing requirements programmatically indicated by one or more clients of the packet processing service. The isolated networks (INs) to which traffic is routed / forwarded using the VR instances may include, for example, IN 1140A (including resources at a facility outside the provider network and connected using a VPN tunnel to the provider network), IN 1140B (including resources also at a facility outside the provider network and connected using a dedicated physical link to the provider network), IN 1140C, and IN 1140D (each including a separate isolated virtual network configured within the virtualized computing service of the provider network).

[0083] From the perspective of the client(s) for which the VR instance 1102 is configured, the functionality provided by the VR instance 1102 may be very similar or identical to the functionality provided by the VR instance 102 shown in FIG. 1. Applications of the type described in the context of FIG. 2 may be implemented using the VR instance 1102, for example, and similar types of auxiliary tasks may be required for applications such as those shown in FIG. 2. Routing / forwarding metadata 1108, at least a subset of which may be provided by the client via a programmatic interface of the PPS, may be used to generate routing / forwarding actions to be performed on various packet flows, as described above in the context of the VR instance 102. The routing / forwarding metadata may include, for example, client-specified policy-based routing rules used to route packets originating from an isolated network, as well as one or more route tables set up according to configuration settings selected by the client(s). The VR instance 1102 may include a set of forwarding nodes 1111, including, in the illustrated embodiment, fast path nodes (FNs) 1114 and exception path nodes (ENs) 1115 that are similar in function to the forwarding nodes 111 of FIG. 1. A cell-based architecture similar to that shown in FIG. 5 or FIG. 6 may be employed for the VR instance 1102.

[0084] At least some auxiliary tasks associated with transmitting packets between isolated networks 1140 by VR instances, including processing messages formatted according to protocols such as BGP, IGMP, TWAMP, etc., may be performed using a combination of user space protocol stack instances (each including a separate processing engine for one or more protocols) and a protocol stack multiplexer in the embodiment shown in Figure 11. One or more auxiliary task offload devices (ATODs) 1160 (e.g., virtualization hosts of a virtualized computing service of the provider network, or servers not used for virtualization), such as 1160A and 1160B, may be selected to host one or more auxiliary protocol stack instances (PSIs) for the respective protocol stack multiplexers (PSMs) and virtual router instances 1102 in the illustrated embodiment. The offload devices may be selected, for example, by a control plane component of the PPS and / or the virtualized computing service of the provider network in various embodiments. The PSMs and PSIs may be instantiated, in some embodiments, as part of the configuration of the VR instance 1102 or later during the life of the VR instance 1102 (e.g., in response to programmatic interactions related to auxiliary tasks, similar to the messages / requests discussed in the context of FIGS. 8 and 9). For example, in the illustrated embodiment, PSM 1162A, PSI 1163A, PSI 1163B, and PSI 1163C (with the PSMs and individual PSIs each including one or more threads of execution) may be instantiated in offload device 1160A, while PSM 1162B, PSI 1163K, and PSI 1163L may be instantiated in offload device 1160B. In some embodiments, at least a portion of the PSMs and / or PSIs may be implemented using libraries similar to the Data Plane Development Kit (DPDK).

[0085] The number and type of PSIs configured in the VR 1102 and the number of offload devices configured in the VR 1102 may change over time based on factors such as the amount of application data traffic processed through the VR, the tenant requirements indicated by the client for auxiliary task processing, the various protocols over which messages are processed in the auxiliary tasks of the VR, etc. Indeed, auxiliary task offloaders (ATOs) of the type described above (e.g., ATOs 373A and 373B in FIG. 3) may be implemented using a combination of a PSM 1162 and one or more PSIs 1163 in the embodiment shown in FIG. 11. The connection between the forwarding plane nodes of the VR instance 1102 and the PSM 1162 may be established using techniques similar to those described in the context of FIG. 7, such as via an encapsulation protocol tunnel and one or more virtual network interfaces in various embodiments.

[0086] When a packet requiring an auxiliary task, such as BGP processing, is received at a VR 1102 configured to transmit packets between isolated networks 1140, a corresponding message indicating at least a portion of the auxiliary task may be sent by the VR to an offload device 1160. The message (e.g., an encapsulated packet formatted according to GENEVE or another encapsulation protocol) may be received at a network interface card (NIC) of the offload device, which may store the message in one or more DMA (direct memory access) buffers, in various embodiments. A PSM 1162 of the offload device may access the DMA buffers, allowing, at least one embodiment, the PSM to inspect the message (e.g., including one or more encapsulation headers or other metadata associated with the contents of the message) without copying the message from the DMA buffer.

[0087] Based at least in part on the metadata associated with the message and / or, in the illustrated embodiment, included in the message, the PSM 1162 may select a particular PSI 1163 from among a set of PSIs instantiated on the offload device 1160 for further processing the message and performing the associated auxiliary task. Metadata that may be used for the selection may include, for example, (a) an identifier for a networking protocol (e.g., BGP, IGMP, etc.) used for the encapsulated packet included in the message, (b) a virtual router identifier, for example, of a VR instance 1102, (c) an identifier for a virtual network interface (e.g., a VNI of the VR instance from which the message was sent), (d) an identifier for a client of the provider network on whose behalf the auxiliary task is to be performed, and / or (e) an identifier for an isolated network 1140 whose traffic requires the auxiliary task.

[0088] The selected PSI, which may at least in some cases include one or more execution threads running in user mode or user space (e.g., within a computing instance launched on the offload device or within the operating system of a non-virtualized server being used as an offload device), may in turn inspect at least a portion of the message and associated metadata and perform the auxiliary task. In at least some implementations, it may not be necessary to copy the contents of the message from the DMA buffer to another location to complete the auxiliary task. The results of the auxiliary task may be provided to the PSM 1162 and, in the illustrated embodiment, transmitted by the PSM 1162 to a forwarding node of the VR 1102. There, the results of the auxiliary task may be used, in various embodiments, to transmit at least some contents of one or more packets of application data originating on one of the isolated networks 1140 to another isolated network 1140. In at least some embodiments, the PSM 1162 may implement a socket-level interface (e.g., a UNIX socket interface) for communication with the PSI 1163.

[0089] For some types of auxiliary tasks, such as processing messages for a BGP session, state information generated for one auxiliary task may need to be used when performing a subsequent auxiliary task. According to at least some embodiments, such state information may be stored in storage external to the offload device, for example, to ensure that the state information is accessible from an alternative PSI in the event that the original PSI being used fails. A number of approaches for storing such state information are described in further detail below.

[0090] The PSIs 1163 in a given offload device 1160 may run independently of one another, e.g., in respective software containers in some embodiments. In some cases, multiple protocol processing engines of the same protocol used for auxiliary tasks, such as BGP, may run in respective PSIs 1163, e.g., each PSI processes messages of the same protocol in an independent address space. As a result, the PSM 1162 of the offload device may be able to easily multiplex multiple received encapsulated packets that are apparently destined for the same address, but are being used for auxiliary tasks for different VRs or different clients. For example, a first message may be received at the offload device indicating a particular IP address as the destination of an encapsulated packet in the message. The PSM of the offload device may select the first PSI for processing the message using metadata associated with the first message. If a second message is received at the PSM, also indicating the same IP address as the destination, and the metadata associated with the second message indicates that a different PSI should be used, the PSM may have a different PSI process the second message, even though the destination IP address is identical.

[0091] In at least some embodiments, the PSI and / or PSM may be used in a multi-tenant mode or a single-tenant mode, for example, based on tenant requirements or requests received from a client on whose behalf an associated VR is established. For example, a request similar to SetAuxiliaryTaskResourceTenancy request 917 of FIG. 9 may be sent by a client to indicate tenant preferences. When a PSI is configured in a multi-tenant model, it may be used for auxiliary tasks related to (a) traffic flowing between isolated networks programmatically attached to a particular VR at the request of one client, and (b) traffic flowing between isolated networks programmatically attached to a particular VR or a different VR at the request of another client.

[0092] In various embodiments, one PSI in an offload device may implement the same transport and application layer protocols of the OSI model as another PSI in the same offload device, but the two PSIs may be used on behalf of different clients or different VRs. Of course, in some embodiments, different PSIs in a given offload device may implement entirely different protocols. For example, one PSI may include a BGP processing engine, another PSI may include an IGMP processing engine, and so on. In various embodiments, a client for which a VR such as VR 1102 is configured may be able to send program requests for metrics collected for individual protocols (e.g., BGP, IGMP, etc.) used on behalf of an auxiliary task, and receive the requested metrics. For example, messages similar to AuxiliaryTaskMonitoringRequirements message 921 and GetAuxiliaryTaskMetrics 925 of FIG. 9 may be sent by the client to indicate the types of metrics the client would like to obtain, and the requested metrics may be collected from the PSI.

[0093] Example protocols used for auxiliary tasks The auxiliary tasks of traffic processed through the virtual router may utilize any of a number of different protocols (sometimes referred to as auxiliary task protocols) in various embodiments. FIG. 12 illustrates an example of a set of protocols, each of which may be instantiated on a device with a protocol stack multiplexer, according to at least some embodiments. As shown, protocols 1210 for which a PSI similar to PSI 1163 of FIG. 11 may be instantiated on an offload device may include various versions of BGP 1220 and its variants (e.g., internal BGP or iBGP, external BGP or eBGP, multi-protocol BGP or MP-BGP), which may be used for dynamic routing information exchange in various embodiments or versions / variants of IGMP 1225.

[0094] In some embodiments, a performance measurement protocol 1230 such as TWAMP (Two Way Active Measurement Protocol) or OWAMP (Two Way Active Measurement Protocol) may be used for auxiliary tasks. A security protocol 1240 (e.g., protocols from the IP Security (IPSec) suite or other similar suites) that may include cryptographic calculations for encryption or decryption of application data packet content may be used for some VR-based applications in the illustrated embodiment.

[0095] In at least some embodiments, a PSI may be established for a proprietary routing information exchange protocol 1245 (also referred to as a custom routing information exchange protocol) used within the provider network in which the VR is configured. In one embodiment, a client whose traffic is being sent through a VR may indicate a custom protocol 1250 used for auxiliary tasks for at least a subset of packets flowing between designated isolated networks, and a PSI may also be set for such custom protocol.

[0096] In a scenario in which multiple versions of a given protocol may need to be used, for example depending on preferences expressed by a client of the packet processing service, any of several approaches may be adopted with respect to supporting the various versions. In some cases, a respective PSI may be implemented for each version, while in other cases, a single PSI capable of processing packets of several different versions of the protocol may be used. In one embodiment, a given PSI may include respective protocol processing engines for several protocols, similar to or including those shown in FIG. 12.

[0097] Example Interaction Between a Stack Multiplexer and a Protocol Stack Instance FIG. 13 illustrates an example set of interactions between components of an auxiliary task offload device that may configure a protocol stack multiplexer for a virtual router, according to at least some embodiments. An encapsulation packet 1340 may be received from a virtual router at a network interface card 1330 of the auxiliary task offload device (ATOD) 1310 of the illustrated embodiment, as indicated by arrow 1371. The encapsulation packet may include an encapsulation packet 1342 formatted according to an auxiliary protocol P1 (BGP, IGMP, etc.), as well as packet metadata 1341 (e.g., the format of a header generated during encapsulation of the protocol P1 packet 1342). As previously mentioned, in some embodiments, the GENEVE protocol may be used to prepare the encapsulation packet. In other embodiments, other encapsulation protocols may be used. The protocol P1 packet 1342 may include its own header, which may include additional metadata in at least some embodiments.

[0098] The network interface card 1330 may store the received encapsulated packet 1340 in one or more DMA buffers 1335 of the ATOD 1310 in the illustrated embodiment, as indicated by arrow 1372. A protocol stack multiplexer (PSM) 1320 including one or more execution threads may have been previously instantiated in the ATOD. Depending on the implementation, the PSM may include one or more kernel mode threads, one or more user mode threads, and / or a combination of user mode and kernel mode threads. In some embodiments, the PSM may be implemented as part of a virtualization manager. A packet metadata analyzer 1367 of the PSM 1320 may inspect the metadata 1341 (and / or other metadata included in the protocol P1 packet 1342) to select a particular protocol stack instance from among one or more protocol stack instances running on the ATOD 1310 that needs to further process the contents of the encapsulated packet 1340 and perform the necessary corresponding auxiliary tasks. The PSM may implement a set of socket level APIs 1370 for communicating with protocol stack instance(s), such as protocol P1 stack instance 1352, protocol P2 stack instance 1353, etc.

[0099] Protocol P1 stack instance 1352 may include a set of user mode or user space threads and associated data structures that collectively emulate multiple layers of a protocol stack, such as the transport layer and the application layer in the illustrated embodiment. The threads of a given protocol stack instance may collectively interpret the contents of messages formatted according to an auxiliary protocol such as BGP, examine state information generated as a result of previous messages of the auxiliary protocol (in the case of a stateful protocol), determine what, if any, actions need to be taken based on the received messages (e.g., modify state information such as BGP attributes used to select an optimal next hop, store an indication of current membership of a multicast group, etc.), and implement such actions. Thus, a given protocol stack instance may, in various embodiments, be described as including a protocol processing engine for the auxiliary protocol. In at least some embodiments, copying the contents of encapsulated packet 1340 from DMA buffer 1335 may not be required. For example, packet metadata analyzer 1367 may simply inspect the DMA buffer (arrow 1373) and pass a pointer to the DMA buffer to the protocol P1 stack instance (arrow 1374). Such "zero copy" techniques can be much more efficient when processing received network messages than techniques that copy the contents of a message from one set of memory locations to another set of memory locations.

[0100] The results of the processing of the encapsulated packet in the selected protocol stack instance 1352 (e.g., new routes, multicast group membership information, etc.) may be sent back to the PSM 1320 via the socket level API 1370 and sent to the virtual router via the network interface card 1330 in some embodiments. In at least one embodiment, the PSM may be responsible for encapsulating the results according to the encapsulation protocol being used to communicate with the virtual router. In other embodiments, the protocol stack instance may encapsulate the results. In some embodiments, a given encapsulated packet received at the offload device may be processed by multiple protocol stack instances, and the results of the processing may be combined in a multiplexer before being sent back to the virtual router in a single packet or message.

[0101] In at least some embodiments, a protocol stack multiplexer of the kind introduced above may be agnostic with respect to the programming language and / or runtime environment used for the protocol stack instances. FIG. 14 illustrates an example scenario in which protocol stack instances developed in several different programming languages ​​may be executed within respective software containers of an auxiliary task offload device, according to at least some embodiments. An auxiliary task offload device 1410 includes a protocol stack multiplexer 1420 that implements a set of PSM APIs for interacting with the protocol stack instances. A protocol P1 stack instance 1452 implemented in Java may be executed within a software container 1480 in the illustrated embodiment. A protocol P2 stack instance 1453 implemented in Scala executes within a second software container 1481, while a protocol P3 stack instance 1454 is implemented in C and executes within a third software container 1482. Such flexibility with respect to programming languages ​​and associated runtime environments may facilitate the packet processing service, in various embodiments, to collect protocol stack instances from a wide variety of development groups. Each development group or individual developer may package the code for a protocol stack instance within a software container whose contents cannot be easily modified, and multiple such containers may run on the same ATOD without interfering with each other. In some embodiments, a client of the packet processing service may provide a software container that includes custom processing code used for auxiliary tasks performed on traffic between the client's isolated networks, and such a container may be deployed on the ATOD.

[0102] 15 illustrates an example scenario in which multiple independent instances of a given protocol stack may run simultaneously on an auxiliary task offload device, according to at least some embodiments. In the illustrated embodiment, the ATOD 1510 includes a protocol stack multiplexer (PSM) 1520 and at least three protocol stack instances.

[0103] The protocol P1 stack instance 1552 configured in multitenant mode is used for auxiliary tasks related to traffic processed by virtual router VR1 configured for client C1 of the packet processing service, as well as for auxiliary tasks related to traffic processed by virtual router VR4 of another client C4. The protocol P2 stack instance 1553A is configured in the ATOD 1510 to implement auxiliary tasks related to traffic processed by virtual router VR2 configured for client C1. Messages formatted according to protocol P2 can also be processed by another stack instance of the ATOD 1510 in the illustrated embodiment, namely protocol P2 stack instance 1553B, established to implement auxiliary tasks related to traffic processed by virtual router VR3 configured for client C2. The two protocol stack instances operate independently of each other, so that overlapping address ranges between packets processed by stack instances 1553A and 1553B can be easily managed. For example, one encapsulated packet received at ATOD 1510 with a destination IP address D1 (e.g., the address of BGP engine BE1 participating in a BGP session with BGP engine BE2 outside the provider network) may be processed by stack instance 1553A, while another packet received at ATOD 1510 with the same destination IP address D1 may be processed by stack instance 1553B.

[0104] In the embodiment shown in FIG. 15, multi-tenancy may be implemented at several levels. First, the ATOD 1510 as a whole may be considered in a multi-tenant mode in that auxiliary tasks for several different clients (C1, C2, and C3) of the packet processing service are implemented using the ATOD. Second, a single protocol stack instance, such as protocol P1 stack instance 1552, may operate in a multi-tenant mode in that it performs auxiliary tasks for clients C1 and C3. In at least some embodiments, as discussed above in the context of FIG. 9, a client may indicate a preference regarding the tenancy mode to be used for resources used for auxiliary tasks, and the packet processing service may configure the ATOD and protocol stack instance accordingly. For example, if a client requests single-tenancy, in one embodiment, an ATOD may be configured or assigned exclusively for the auxiliary tasks of that client. In some embodiments, a client may specify tenancy requirements at the device (ATOD) level or at the protocol stack instance level.

[0105] Exemplary Auxiliary Task State Information Management Depending on the type of auxiliary task being performed, in some embodiments, it may be necessary to maintain state information that applies to multiple messages exchanged between the virtual router and the auxiliary task offload device. For example, some protocol processing stacks may require the preservation of Transmission Control Protocol (TCP) connection state information. Figure 16 illustrates an example of an alternative approach for preserving protocol state information related to the auxiliary tasks of a virtual router, according to at least some embodiments.

[0106] In protocol stack state management approach A, a protocol stack instance 1652 runs within a process 1653 (Java™ Virtual Machine or JVM), which uses a garbage collected heap for memory management in an execution environment (EE) 1610 (e.g., a computing instance or non-virtualized server). To ensure that state information associated with auxiliary tasks processed using the protocol stack instance is not lost permanently if the process 1653 crashes or terminates unexpectedly, in the illustrated embodiment, an off-heap data structure 1660, such as a hash table that does not utilize a heap, may be used to persistently store the state information. A new protocol stack processing instance process may be started as a replacement in case of termination of the process 1653, and the new process may access the off-heap data structure. It should be noted that state information may be lost in approach A if the execution environment 1610 crashes or terminates unexpectedly.

[0107] In protocol stack state management approach B, a separate persistent state management process (PSMP) 1623 (as opposed to simply an off-heap data structure) may be allocated in some embodiments to manage state information for auxiliary tasks processed by the protocol stack instance process 1622. The PSMP 1623 may be longer-lived than the stack instance process 1622. The process 1622 that performs computations for the auxiliary tasks may run in the same EE 1611 as the PSMP 1623 in the illustrated embodiment. As such, premature termination or failure of the EE may still result in loss of state information.

[0108] Protocol stack state management approach C may use distributed techniques for state information management in the illustrated embodiment. A persistent state management cluster 1640 may be configured to include several different PSMPs, such as 1624A and 1624B, each running in an EE 1613 (e.g., 1613A or 1613B) separate from the EE 1612 in which the protocol stack instance process 1632 runs. Any given PSMP of the cluster 1640 may be capable of taking over the role of a failed PSM. Additionally, as state information for an auxiliary task changes, it may be propagated to one or more service resources available for persistent storage in the provider network, such as database service 1601 in the illustrated embodiment. If desired by a client on whose behalf an auxiliary task is to be executed, in some embodiments the state information may be provided to or made accessible to the client via notification server 1602 or message queue service 1603.

[0109] How to offload auxiliary tasks using protocol stack multiplexing FIG. 17 is a flow diagram illustrating aspects of operations that may be performed to offload some types of tasks from a virtual router using a protocol stack multiplexer and independent instances of protocol stacks, according to at least some embodiments. As shown in element 1701, execution environments (EEs), such as computing instances of a virtualized computing service or non-virtualized servers, in the illustrated embodiment, may be identified by the control plane of the packet processing service to perform offloaded auxiliary protocol tasks on offload devices of one or more virtual routers. In one embodiment, one or more such EEs may be configured when the virtual router is established. In other embodiments, the EEs may be configured later during the lifetime of the virtual router, e.g., in response to a program request indicating one or more categories of auxiliary tasks to be performed with respect to traffic routed / forwarded through the virtual router(s).

[0110] A protocol stack multiplexer (PSM) (e.g., a process or thread that has access to a DMA buffer in which network packets received by a network interface card of an offload device used for an EE are placed) may be launched in the EE, in various embodiments, for example, by the control plane of a packet processing service (element 1704). In addition, one or more protocol stack instances (PSIs) including threads that run in user space or user mode (as opposed to running in privileged or kernel mode) may be launched in the execution environment. The PSIs may implement or emulate the functionality of one or more Open Systems Interconnection network stack layers (e.g., network layer, transport layer, or application layer) required to perform one or more types of auxiliary tasks related to network traffic sent through the virtual router and may perform additional logic required to process messages associated with the auxiliary tasks. An individual PSI of a PSI may implement a processing engine for one or more protocols (e.g., BGP, IGMP, TWAMP, etc.) used for the auxiliary tasks in various embodiments. In at least some embodiments, a cell-based architecture similar to the architecture discussed in the context of Figures 5 and 6 may be used for the virtual router and / or resources configured for the auxiliary protocol tasks. In some embodiments, at least some of the PSI may include one or more kernel mode or privileged threads.

[0111] A network connection may be established between the virtual router(s) and the EE, in various embodiments, for example, by configuring one or more virtual network interfaces and / or encapsulation protocol tunnels (element 1707). Techniques similar to those shown in Figure 7 may be used in some embodiments to enable the communication, including, for example, establishing a GENEVE tunnel, storing metadata in an exception path node of the virtual router indicating an address of a virtual network interface of a cell of an autonomous task processing resource as a destination for packets indicating an auxiliary task to be performed, and storing metadata in the EE indicating one or more addresses of the virtual network interface of the virtual router as a destination for results of the auxiliary task.

[0112] After a connection is established between the EE and the VR(s), in various embodiments, at some point a message may be received at the EE from the established VR indicating an auxiliary task to be performed to forward packets between the isolated networks IN1 and IN2 (element 1710). This message may, in some embodiments, include metadata (e.g., including the contents of the header of the encapsulating protocol, such as GENEVE) about an encapsulated packet (e.g., a packet sent by a BGP processing engine at a facility outside the provider network) embedded within the message. The metadata may include, for example, (a) an identifier of the networking protocol used for the encapsulated packet in the message, (b) a virtual router identifier of the VR from which the message is received, (c) an identifier of the virtual network interface, (d) an identifier of a client of the provider network, or (e) an identifier of the isolated network whose traffic is being routed through the virtual router.

[0113] The PSM may inspect the metadata and determine which particular PSI (e.g., PSI-1) running in the EE should further process the message, element 1713. In at least some embodiments, the message contents may not need to be copied from the DMA buffer for analysis by the PSM or for processing of the message contents by the selected PSI.

[0114] The selected PSI, PSI-1, in various embodiments, may analyze the contents of the message, perform auxiliary tasks necessitated by the contents of the message, and send the results of the auxiliary tasks to the PSM (element 1716). In at least some embodiments, the PSM may implement a set of socket-level or socket-layer programmatic interfaces, and PSI-1 may send the results to the PSM via such interfaces. In some embodiments, PSI-1 may store state information about its auxiliary tasks (e.g., TCP connection state, protocol-specific sequence number information, etc.) in storage external to the EE. In some embodiments, the PSI and / or the EE in which the PSI executes may be configured in a single-tenant mode, e.g., at the request of a client for which a VR has been established. In other embodiments, a given EE and / or a given PSI may process auxiliary tasks for several different clients and / or several different VRs in a multi-tenant mode.

[0115] The PSM, in the illustrated embodiment, may send the results to the VR from which it received the message (element 1719). The VR, in the illustrated embodiment, may use the results of the auxiliary task to send at least some packets between IN1 and IN2 (element 1722).

[0116] Example system environment with dynamic routing enabled for peered virtual routers In some embodiments, a path including multiple virtual routers may be required to forward traffic between isolated networks, such as in scenarios where traffic needs to be sent across continental, national, state, or regional boundaries. For such traffic, a pair of virtual routers may be programmatically attached to one another. Such an attachment between VRs may be referred to as a "peering attachment," and the attached VRs may be said to be peered with one another. FIG. 18 illustrates an exemplary system environment in which dynamic routing, including the exchange of routing information using a Border Gateway Protocol (BGP) processing engine, according to at least some embodiments, may be enabled for a peering pair of virtual routers upon request of a client of a packet processing service. In the system 1800, a pair of virtual routers (VRs) 1810A and 1810B may be configured or established in response to a programmatic request of one or more clients of a packet processing service (PPS) similar to the packet processing service previously described, for example, received via a programmatic interface 1870 of a PPS control plane 1890. VR1810A may, for example, be established in a geographic region GR1 (e.g., using a computing device in one or more provider network data centers located in country C1 or state S1), and VR1810B may be established in another geographic region GR2 (e.g., using a computing device in one or more provider network data centers located in country C2 or state S2).

[0117] VRs 1810A and 1810B can be programmatically attached to each other and to one or more isolated networks in response to programmatic attachment requests sent to the client on whose behalf the VRs and isolated networks are configured in the illustrated embodiment. A given attachment involving a VR can belong to one of several categories in the illustrated embodiment, such as an IVN attachment (associating an isolated virtual network (IVN) of a virtualized computing service (VCS) with a VR), a DX attachment (associating an isolated network of a client premises connected via a dedicated physical link to a provider network with a VR), a VPN attachment (associating an isolated network of a client premises connected to a provider network via one or more VPN tunnels with a VR), a peering attachment (associating two VRs), an SD-WAN attachment (associating a software-defined wide area network device of a client with a VR), etc. In the scenario shown in FIG. 18, both isolated network 1840A (including an IVN) and isolated network 1840B (including another IVN) may be programmatically attached to VR 1810A via IVN attachments IA-1 and IA-2, respectively. In addition, isolated network 1840A (including an IVN) may be attached to VR 1810B via IVN attachment IA-3, and isolated network 1840D (including resources at a client premises outside the provider network's data center) may be attached to VR 1810B via DX attachment DA-1. A peering attachment PA-1 may be established between VR 1810A and VR 1810B. Each of these five attachments (IA-1, IA-2, IA-3, DA-1, and PA-1) may be established in response to one or more programmatic requests from a packet processing service (PPS) client 1895 in the illustrated embodiment.

[0118] In the illustrated embodiment, the transfer of dynamic routing information according to a version or variant of BGP may be enabled between VRs 1810A and 1810B based at least in part on input received via program interface 1870, for example, as part of a peering attachment request to PA-1 or following peering of the two VRs 1810A and 1810B. In scenarios where multiple paths are available for transmitting application data packets between isolated networks, the routing information may enable the virtual router to dynamically select a more optimal path for the application data packets. This type of routing may be referred to as dynamic routing in various embodiments. In at least some embodiments, any of several different factors may be considered by the virtual router when selecting a next hop or path for an application data packet when dynamic routing is enabled, such as bandwidth availability, delay, past congestion patterns, and / or agreements with intermediary or transit network providers.

[0119] In addition to enabling the transfer of dynamic routing information, in at least some embodiments, the client may use the program interface 1870 to provide a group of one or more dynamic routing protocol configuration settings to be used for the transfer. Such settings may indicate various preferences of the client regarding aspects of routing information transfer. One such setting may include, for example, a filter rule used to determine whether a route to a particular destination is transferred from one VR to another. Another setting may indicate respective priorities assigned to individual attributes of a plurality of routing-related attributes for selecting a next hop to a destination, for example, (a) a local preference attribute, (b) a local prefix origin attribute, (c) an autonomous system (AS) path length attribute, (d) a multi-exit discriminator attribute, or (e) a router identifier attribute. The local preferences may indicate respective preferences used by the VR for various available paths, for example, using numerical values ​​propagated in route updates from the VR's neighbors' BGP in the same autonomous region. The client may use the local preferences to influence which exit point is preferred among multiple exit points of an autonomous system. In one embodiment, the route with the highest local preference value (among the available alternate routes) may be selected by the VR for the packet. In some embodiments, the local prefix origin attribute may be used in the VR to prefer paths in IVNs directly attached to the VR if alternate paths involving other VRs are also available. The VR may select the path with the shortest AS path length (among the available alternate paths) in embodiments where the AS path length attribute is used. In some embodiments, a multi-egress discriminator (MED) may be obtained at the VR from neighboring BGPs in different ASes, and if alternate paths with different NEDs are available, the VR may select the path with the lowest MED. In some embodiments, a numeric router identifier may be assigned to each VR, as well as to client-owned hardware routers, SD-WAN equipment, etc.In one embodiment, among the alternative paths with forwarding to the respective routers, the path of the router with the lowest router identifier may be selected if none of the other attributes considered lead to a preference. In some embodiments, the client-specified configuration may indicate a particular variant and / or version of BGP to be used, such as iBGP, eBGP, MP-BGP, etc., and / or a CIDR (Classless Inter-Domain Routing) block from which addresses are assigned to the BGP processing engine associated with the VR 1810. Other parameters governing the forwarding of routing information may be specified by the client via the interface 1870 in some embodiments.

[0120] According to the requirements for enabling dynamic routing information transfer, in various embodiments, respective BGP processing engines 1814 may be established or instantiated for the two VRs in the illustrated embodiment. For example, BGP processing engine 1814A may be configured for VR 1810A, and BGP processing engine 1814B may be configured for VR 1810B. One or more BGP sessions may be initiated between the two processing engines to exchange dynamic routing information. This information enables each VR to forward network packets to the isolated network via the other VR according to the configuration settings specified by the client in the illustrated embodiment. The transfer of routing information from one BGP processing engine to another BGP processing engine for different sets of destination endpoints may be referred to as "advertising" the routing information.

[0121] In the illustrated embodiment, each of the virtual routers may maintain at least one route table associated with peering attachment PA-1. Thus, route table 1871 is maintained by VR 1810A, and route table 1872 is maintained by VR 1810B. Entries in a given route table may indicate next hops for various groups of destination endpoints, referred to as destination prefixes, specified in the CIDR format of FIG.

[0122] Isolated network 1840A, in the depicted exemplary scenario, includes a set of network endpoints having IP version 4 addresses in the range ABCD / 16 (represented in CIDR notation). Isolated network 1840B includes a set of network endpoints having IP version 4 addresses in the range AFCD / 16. Isolated network 1840C includes a set of network endpoints having IP version 4 addresses in the range AGCD / 16, while isolated network 1840D includes a set of network endpoints having IP version 4 addresses in the range KLMN / 16. To allow traffic to flow through peering attachment PA-1, in the depicted embodiment, BGP processing engine 1814A sends advertisements for ADCD / 16 and AFCD / 16 to BGP processing engine 1814B, while processing engine 1814B sends advertisements for AGCD / 16 and KLMN / 16 to BGP processing engine 1814A. As a result, route table 1871 is populated with one entry pointing to peering attachment PA-1 as the next hop for destinations or destination prefixes (Dst prefix) in the AGCD / 16 range, and another entry pointing to peering attachment PA-1 as the next hop for destinations in the KLMN / 16 range. Route table 1872 is populated with entries pointing to PA-1 as the next hop based on the ABCD / 16 and AFCD / 16 advertisements received by BGP processing engine 1814B from BGP processing engine 1814A in the illustrated scenario.

[0123] Route tables 1871 and 1872, in the illustrated embodiment, may also include entries for next hops of isolated networks directly attached to the corresponding VR. For example, route table 1871 includes an entry indicating IA-1 as the next hop for ABCD / 16 and another entry indicating IA-2 as the next hop for AFCD / 16. Similarly, route table 1872 includes an entry indicating IA-3 as the next hop for AGCD / 16 and another entry indicating DA-1 as the next hop for KLMN / 16. In some embodiments, some or all of isolated networks 1840 may include their own BGP processing engines. For example, advertisements for KLMN / 16 may be sent to BGP processing engine 1814B from another BGP processing engine configured in isolated network 1840D.

[0124] In the illustrated embodiment, dynamic routing information (e.g., BGP advertisements) forwarded between VRs according to client configuration settings may be used to forward network packets from one isolated network to another. For example, if a packet originating at isolated network 1840A is destined for an address in the KLMN / 16 range, then the KLMN / 16 entry in route table 1871 may be utilized in VR 1810A to send the packet via PA-1 to VR 1810B, from where it may be forwarded to isolated network 1840D based on the KLMN / 16 entry in route table 1872. Although attachment identifiers (IA-1, IA-2, IA-3, and DA-1) are used in FIG. 18 to indicate the next hop, such attachment identifiers may be translated into corresponding virtual network interface (VNI) identifiers or addresses (each VNI is configured with one of the attachments) for packet forwarding in at least some implementations. Note that in the illustrated embodiment, static routes may not need to be provided by the client to be able to send network packets between any of the isolated networks, since routing information is dynamically exchanged between the BGP processing engines of the virtual routers. In some embodiments, static routes may not be required, although clients may still specify static routes if desired.

[0125] In some embodiments, the BGP processing engine 1814 may be instantiated with offload resources, such as the auxiliary task offloader described above. In other embodiments, such offload techniques may not be necessary, and the BGP processing engine may be launched with the same resources used for one of the VR nodes. In some embodiments, a protocol other than BGP or a variant thereof may be used to transfer at least some of the routing information between the virtual routers. For example, a custom protocol developed in the provider network may be used.

[0126] 19 illustrates an example scenario in which dynamic routing information exchange may be enabled for several different types of programmatic attachments of a virtual router, according to at least some embodiments. As noted above, a virtual router may attach to other sources of routing information via any of several different types of attachments, e.g., in response to programmatic requests from clients of the PPS. In some embodiments, the different types of attachments may differ from one another in the type of metadata that may be stored for them in the PPS control plane (e.g., including the type of protocol processing engine used for the routing information associated with the attachment, the route tables associated with the attachment, respective limitations on the amount or rate of traffic that can be forwarded, how and how often to update the routing information associated with the attachment, the attachment's virtual network configuration information, etc.).

[0127] In the embodiment shown in FIG. 19, the VR1910A is attached to four other entities. An IVN 1940 including a client-configured SD-WAN (Software-Defined Wide Area Network) device 1990 is attached to the VR1910A via an IVN attachment IA-1. A VPN-connected client premises isolated network 1941 (i.e., an isolated network including network endpoints and resources located in a facility outside the provider network in which the VR1910A is established) including a client premises router 1991 is attached to the VR1910A via a VPN attachment VA-1. A direct physical link connected client premises isolated network 1942 (i.e., an isolated network including network endpoints and resources located in a facility outside the provider network in which the VR1910A is established) including a client premises router 1992 is attached to the VR1910A via a DX attachment DA-1. In addition, another VR1910B is attached to the VR1910A via a peering attachment PA-1.

[0128] Each of the entities to which VR1910A is attached may, in the illustrated embodiment, include a respective protocol processing engine for a dynamic routing information exchange protocol (such as a BGP variant, or a custom protocol). Thus, dynamic routing information exchange may be enabled between each pair of attached entities, as indicated by the double-dashed arrows labeled Dynamic Routing Information Exchange (DRIE) 1922, DRI 1923, DRE 1924, and DRIE 1925. In some embodiments, different protocols may be used for dynamic routing information exchange between different pairs of entities. For example, protocol P1 (and associated protocol processing engine PE1) may be used to exchange routing information between VR1910A and 1910B, while protocol P2 (and associated engine protocol processing engine PE2) may be used to exchange routing information between IVN 1940 and VR1910A.

[0129] An example of using a custom protocol while maintaining BGP compatibility FIG. 20 illustrates an exemplary scenario in which a custom protocol for routing information transfer is used by a virtual router to exchange information originally sent to the virtual router using BGP, according to at least some embodiments. In the embodiment illustrated in FIG. 20, the PPS client 2095 may send a routing configuration request 2078 (e.g., including the types of settings discussed above that control aspects of the transfer of routing information) using BGP terminology and attributes via the program interface 2070. Internally, the PPS control plane 2088 may utilize a custom routing information transfer protocol (CRITP) 2044 to transfer routing information between VRs while still maintaining compatibility with BGP from the client's perspective. A custom protocol may be preferred for internal use for a variety of reasons in different embodiments, such as avoiding the implementation of some of the lesser used features of BGP and removing some of the constraints imposed by BGP (e.g., limitations on the amount of routing information that can be transferred within a particular BGP session). In the illustrated embodiment, the configuration settings transformer 2055 may convert the BGP-based routing configuration request 2078 into the format used for the CRITP 2044.

[0130] Dynamic routing information exchange (DRIE) messages between the client premises and the VR 2010 may continue to be formatted according to BGP in the illustrated exemplary scenario, as indicated by labels 2023 and 2024. For example, the BGP processing engine 2091A of the router 2090A of the client premises CP1 may establish a BGP session with the BGP-compliant processing engine 2066A of the VR 2010A, and the BGP processing engine 2091B of the router 2090B of the client premises CP2 may establish a BGP session with the BGP-compliant processing engine 2066B of the VR 2010A. When routing information obtained via BGP messages from the router 2090 is transferred from one VR to another VR via the peering attachment PA-1, in the illustrated embodiment, the information may be expressed according to CRITP. That is, the VRs 2010A and 2010B may exchange dynamic routing information using CRITP messages instead of BGP messages, as indicated by label 2025. In fact, the BGP-compliant processing engine 2066 may convert the same underlying routing information from BGP to CRITP and vice versa, as necessary, and thus may be capable of processing messages of both protocols.

[0131] Example of using multiple peering attachments for network segmentation FIG. 21 illustrates an example scenario in which multiple peering attachments may be established between pairs of virtual routers, according to at least some embodiments. In the illustrated embodiment, a client may want to ensure that traffic is permitted to flow between a specified pair of isolated networks attached to VRs 2110A and 2110B, while network flows are prevented or prohibited between other pairs of isolated networks attached to the same VR. For example, a client may want to enable dynamic routing of packets (e.g., using the exchange of advertisements of the type described above) between isolated networks (INs) 2140A and 2140B, and also between isolated networks 2140C and 2140D. However, a client may also want to prevent traffic flow between (a) IN 2140A and IN 2140D, (b) IN 2140A and IN 2140C, (c) IN 2140B and IN 2140C, and (d) IN 2140B and 2140C.

[0132] To achieve this type of network segmentation while still using dynamic routing information exchange using BGP or a similar protocol, in some embodiments, two different peering attachments (and associated different pairs of dynamic routing protocol engines) may be established: Peering attachment PA-1 may be configured for only traffic (and associated dynamic routing information transfer) between IN2140A and 2140B, while peering attachment PA-2 may be configured for only traffic (and associated dynamic routing information transfer) between IN2140C and 2140D.

[0133] Example programmatic interaction of dynamic routing through peered virtual routers FIG. 22 illustrates an exemplary set of programmatic interactions related to configuration of dynamic routing of peered virtual routers, according to at least some embodiments. Packet processing service (PPS) 2212, which is functionally similar to the packet processing service described above in the context of FIG. 1, may implement a set of programmatic interfaces 2277 in the illustrated embodiment. Programmatic interfaces 2277 may include, for example, a set of APIs, command line tools, a web-based console, a graphical user interface, etc. Using interface 2277, a client may send messages related to virtual router configuration similar to those described in the context of FIG. 8 and FIG. 9, as well as additional messages illustrated in FIG. 22, and receive corresponding responses.

[0134] After previously establishing a number of virtual routers for managing traffic between a set of isolated networks (e.g., using the CreateVirtualRouter request 814 shown in FIG. 8), the client 810 may, in the embodiment shown in FIG. 22, send a CreateVRPeeringAttachment request 2214 to request that a peering attachment be created between a specified pair of VRs. Metadata indicating which particular VRs are attached may be stored in the PPS 2212, and in some embodiments a PeeringAttachmentCreated message 2215 may be sent to the client.

[0135] In at least some embodiments, peering attachments may be created between VRs established on behalf of different clients or different client accounts. For example, a virtual router VR-1 may be created for client C1 in a provider network, and a virtual router VR-2 may be created for client C2, and the two clients may want to enable the transfer of application data packets between various isolated networks owned by the clients via a peering attachment established between VR-1 and VR-2. In such a scenario, when one of the clients requests a peering attachment, the PPS may need to ensure that the owner of the other VR agrees to the attachment. In some embodiments, when such a "cross-account" attachment is requested, the PPS 2212 may send an ApproveCrossAccountVRPeering request 2217 to the client for which permission or approval is required. Thus, in the above example where client C1 owns VR-1 and requests peering with VR-2, an ApproveCrossAccountVRPeering request 2217 may be sent to C2. If C2 approves, C2 may respond with a CrossAccountVRPeeringApproved message 2219 and, in the illustrated embodiment, the peering attachment requested by C1 may be established.

[0136] In various embodiments, a client may request that dynamic routing (e.g., including the transfer of routing information between peered VRs and the use of the routing information to dynamically select optimal next hops in the VRs for various application data packet flows) be enabled for the peering attachment, for example, by sending an EnableDynamicRoutingForVRPA request 2221. In response, in at least some embodiments, a respective routing information exchange protocol processing engine may be configured for each of the peered VRs (e.g., using an offload device as discussed above or using the same device as used for the VR's forwarding plane node) and a protocol session may be initiated between the protocol processing engines. A DynamicRoutingEnabled message 2223 may be sent to the client to confirm that dynamic routing is enabled. In at least one embodiment, dynamic routing may be enabled by default when a peering attachment is created, so a separate EnableDynamicRoutingForVRPA may not be required.

[0137] One or more RoutingInfoTransferConfigSettings messages 2225 may be sent by the client 2210 to indicate various configuration settings regarding the transfer of dynamic routing information between peered VRs in the illustrated embodiment. Any of a variety of configuration settings may be specified, including the particular protocol to be used (e.g., any of the various flavors of BGP, such as eBGP, iBGP, MP-BGP, etc.), the filtering of outbound advertised routes, the filtering of inbound advertisements, the relative priorities assigned to various BGP attributes for selecting the next hop to the destination, the CIDR block to be used for the IP address of the protocol processing engine, the autonomous system identifier assigned to the protocol processing engine, etc. The set of BGP attributes for which the relative priorities of each are indicated by the client in one embodiment may include, for example, one or more of: (a) a local preference attribute, (b) a local prefix origin attribute, (c) an autonomous system (AS) path length attribute, (d) a multi-egress discriminator (MED) attribute, or (e) a router identifier attribute. The configuration settings, which in at least some embodiments may also be referred to as dynamic routing protocol control settings, may be indicated as parameters of the EnableDynamicRoutingForVRPA request in some embodiments. In some embodiments, the client may use the programmatic interface 2277 to indicate various factors to be used when making dynamic routing decisions in the VR, such as measured latency, bandwidth availability, and the relative priorities assigned to the factors, for example, as part of the peering attachment configuration settings. After the client-specified settings are obtained at the PPS 2212, they may be stored in a database and applied in various embodiments at the protocol processing engine configured for the peered VR. In at least some embodiments, a SettingsApplied message 2227 may be sent to the client 2210.

[0138] According to some embodiments, various metrics related to the transfer and use of dynamic routing information, such as the number of route advertisements sent in either direction between pairs of routing information exchange protocol processing engines used, health status information of the protocol processing engines (e.g., responsiveness, uptime, etc.), the rate at which advertisements are sent over time, the number of times a particular attribute is used to change the next hop setting, etc. may be collected by the PPS 2212. A client may send a ShowDynamicRoutingMetrics request 2229 to request such metrics, and in the illustrated embodiment, the requested metrics may be presented to the client via one or more MetricsSet response messages 2231.

[0139] In one embodiment, a client may send a ShowLearnedRoutes request 2233 requesting information about a set of dynamically learned routes of a peered virtual router or a particular route table of a peered virtual router. In response, in the illustrated embodiment, the next hop addresses learned in the VR may be presented to the client via one or more LearnedRoutesSet responses 2235. In some embodiments, a protocol processing engine of a given VR may receive BGP messages from multiple processing engines, with different engines each providing information about alternative paths to the same destination. In one such embodiment, the learned route information provided to the client via the LearnedRouteSet message may include several different next hop alternatives for a given destination address or prefix, each obtained from a different protocol processing engine. For example, the contents of a table similar to the following, containing routing information obtained from at least two different BGP engines, may be presented to the client in a LearnedRoutesSet message: { “Network”: “ABCD / 32” “NextHop”: “EFGH”, “EFGK” “MED”: “0”, “0” “Local Preference”: “100”, “300” “ASN-Path”: “777 911 711i”, “777 911 711 715i” }

[0140] In the example table above, two different next hops (EFGH and EFGK) have been learned for a destination with an address of ABCD / 32. Values ​​for various attributes used to select a preferred next hop when multiple next hop alternatives are available, such as a multi-exit discriminator (MED), local preference attribute, and autonomous system number path (ASN-Path), may also be provided in the LearnedRoutesSet message in at least some embodiments for different next hop options.

[0141] How to configure and use dynamic routing for peered virtual routers 23 is a flow diagram illustrating aspects of operations that may be performed to enable and utilize dynamic routing for peered virtual routers, according to at least some embodiments. As shown in element 2301, a set of virtual routers (VRs) including VR1 and VR2 may be created or established in a packet processing service (PPS) in response to programmatic requests from one or more clients of the PPS, for example using a cell-based approach of the type described above. The VRs may be created to transmit network packets between a set of isolated networks including IN1 and IN2.

[0142] In the illustrated embodiment, the INs and VRs may be programmatically attached to one another, e.g., IN1 may be attached to VR1 and IN2 may be attached to VR2, and a peering attachment PA may be created between VR1 and VR2 based on a request received from a client on behalf of which the IN and VR are established (element 2304).

[0143] A determination may be made that dynamic routing should be enabled for the inter-VR peering attachment (element 2307). That is, a determination that dynamic routing information, such as updated attribute values, performance metrics, etc., should be transferred between the VRs and that such routing information should be used to route application data packets between the attached IVNs. In at least some embodiments, the client may specify configuration settings (rules for filtering inbound or outbound route advertisements, respective priorities assigned to attributes / factors used to select next hops for various destinations, etc.) for the transfer of routing information between the peered VRs according to a selected protocol, such as a variant of BGP. The routing information exchanged may indicate, for example, routes or next hops to groups of destination addresses (e.g., expressed as CIDR blocks) in various IVNs, values ​​of BGP or other attributes (such as MED values) associated with the groups of destination addresses, delay measurements associated with various paths available to the destination addresses, measurements of available bandwidth along the various paths, metrics of errors / failures encountered along the paths, etc.

[0144] In the illustrated embodiment, respective protocol processing engines E1 (associated with VR1) and E2 (associated with VR2) may be instantiated and / or connected to each other to initiate a dynamic routing information exchange (DRIE) session (e.g., a BGP session) (element 2310). In some embodiments, an offload device of the type described above may be used for one or more of the protocol processing engines E1 and E2.

[0145] Routing information regarding IN1 may be obtained at VR2 via a DRIE session, and routing information regarding IN2 may be obtained at VR1 via a DRIE session (element 2313). The obtained routing information may be utilized to send at least some network packets originating at one of the INs to the other IN without having to configure static routes for such packets (element 2316).

[0146] An example wide area networking service using VR with dynamic routing enabled Many organizations have offices and computing resources distributed across geographic regions, and a given organization's facilities may span continents. Managing connections between such remote facilities can be complex, as they may involve different entities, different hardware devices, and related software from different vendors. Figure 24 illustrates an example environment in which a wide area network linking an organization's geographically separated facilities may be managed by the organization using leased fiber lines and equipment from different vendors, according to at least some embodiments. Organization A may have an headquarters site (OAHQ) 2420 in country / region 2410A, and may also have facilities in country / region 2410B and country / region 2410C. Country / region 2410B may include, for example, one or more of organization A's data center (OADC) 2412B, branch offices (OABO) 2415B, and point of sale site (OAPOS) 2418B, while country / region 2410C may include OADC 2412C, OABO 2415C, and OAPOS 2418C. Country / region 2410A may also include OADC 2412A, OABO 2415A, and OAPOS 2418A. Within a given country / region 2410, organization A may rely on a local Internet Service Provider (ISP) for connectivity between different facilities, for example. In some scenarios where large amounts of data need to be transferred between organization A's facilities from one country / region to another, the organization may acquire leased fiber lines such as 2444A, 2444B, 2444C. Wide area networking management equipment 2457 (e.g., routers for packets flowing over leased fiber lines), such as WAN equipment 2457A from hardware vendor A in country / region 2410A, WAN equipment 2457B from hardware vendor B, and WAN equipment 2457C from hardware vendor C, may also need to be purchased and managed by organization A.Additionally, if organization A also utilizes resources 2421 (e.g., computing instances for a virtualized computing service, database systems for a database service, etc.) in a provider network or cloud computing environment, in some cases, there may be no easy way to interconnect the leased fiber lines 2444 with the provider network, so it may be necessary to configure one or more custom hubs 2491 to route traffic between remote regions and the provider network. If demand for traffic between regions increases, it may take more time than organization A would like to expand the leased lines it has acquired. Managing an organization's global WAN can be cumbersome, as administrators may need to use different tools to handle each part of the network.

[0147] The provider network resources may be distributed across various regions / countries, and the provider network may use a high-bandwidth private fiber backbone network to connect to its own data centers spread across the world. In some embodiments, a service may be implemented that allows an organization to configure its own WAN using the provider network's backbone network and a collection of virtual routers of the types described above. FIG. 25 illustrates an exemplary system environment in which traffic between remote facilities of clients of the provider network is transmitted using the provider network's wide area network (WAN) service, according to at least some embodiments. The service employs an internal fiber backbone network and a collection of virtual routers that enable dynamic routing. As shown, the system 2500 includes resources and products of a provider network WAN service 2502 that are used to enable connectivity between the facilities of organization A, which has a headquarters OAHQ 2520, and various other facilities distributed across country / region 2510A, country / region 2510B, and country / region 2510C. The WAN service 2502 includes a set of control plane servers 2544, a set of client WAN metadata 2546, a WAN scalability manager 2548, and client-side WAN management interfaces / tools 2550. Clients of the WAN service may utilize the provider network's high-performance (e.g., low latency, high bandwidth) private fiber backbone links 2570 to transmit packets between client facilities located in different countries or regions, and may in fact configure private WANs using the provider's network resources and easy-to-use configuration management tools. The high-performance fiber backbone links 2570 may be described as private because they are used exclusively by the provider network service (on behalf of clients of the service and / or for internal management purposes) and, in at least some embodiments, may not include links in the public Internet.The WAN service may manage the scalability and availability of the private fiber backbone links, adding resources / links as needed, and the client may not even need to be aware of the link details (e.g., exactly which backbone links link to which data centers, the bandwidth supported by the various links, etc.).

[0148] The control plane server 2544 may be responsible for management tasks of the WAN services, such as provisioning computing instances of the virtualized computing services of the provider network, for example, for running a scalability manager 2548, and for responding to input obtained via a client-side WAN management interface / tools 2550, which in various embodiments may include a set of programmatic interfaces, such as a web-based console, a command line tool, a graphical user interface, and / or an API. Using such interfaces, in some embodiments, a potential client of the WAN services 2502 (such as an administrator or manager of organization A) may be provided with a view of multiple client premises between which network traffic is routed over the private fiber backbone of the provider network. For example, the client may provide information such as the physical locations of facilities in different geographic regions (including OAHQ 2520, OADC 2512A, 2512B, and 2512C, OABO 2515A, 2515B, and 2515C, and OAPOS 2518A, 2518B, and 2518C), the expected rates of traffic between facilities across the regions, the desired range of packet delays, etc. In at least some embodiments, the client may indicate or specify a particular protocol (e.g., a version or variant of BGP) to be used to obtain dynamic routing information regarding the different facilities. The information provided by the client may be stored as part of the client WAN metadata 2546 in the illustrated embodiment.

[0149] According to some embodiments, the WAN service may analyze the provided information about the client's premises and provide the client, via a programmatic interface, with a recommendation to establish a number of virtual routers (VRs) of the type described above for the client's private WAN. In at least one embodiment, a mapping between the VRs and the premises, preferably having local networks to which the VRs are programmatically attached, may also be provided to the client via the programmatic interface. Such a mapping, in at least some embodiments, may be based on the physical location of the provider network data centers relative to the location of the client premises. For example, if the provider network data centers are distributed in regions defined in the provider network (e.g., US Region A, US Region B, Europe Region A, etc.), the mapping may, in some embodiments, indicate a recommended provider network region in which one or more virtual routers should be established for one or more nearby client premises. Based on the provided recommendation, the client, in some embodiments, may send a programmatic request (e.g., directly to a packet processing service of the type described above or via a WAN service) to establish a set of virtual routers. In other embodiments, instead of requesting the client to configure the VRs, the WAN service itself may configure the set of virtual routers on behalf of the client. One or more provider networks VR2572A may be configured in country / region 2510A, one or more provider networks VR2572B may be configured in country / region 2510A, and one or more provider networks VR2572C may be configured in country / region 2510C.

[0150] In at least some embodiments, each VR may be configured as part of the client's private WAN using a set of provider network resources (e.g., computing instances of the fast path nodes, exception path nodes, and / or auxiliary task offloader virtualized computing services described above) that meet a proximity criterion with respect to one or more of the client premises. In some embodiments, validating that a VR meets a proximity criterion with respect to the client premises may include ensuring that the VR is in a region defined in the same provider network where the computing instance would be established by default if a computing instance launch request was sent from the client premises. In other embodiments, validating that a VR meets a proximity criterion may include ensuring that a dedicated direct physical link (direct connect link) can be set up between the client premises and the provider network data center, if the client desires, or that a VPN tunnel with an average packet forwarding delay time of T milliseconds or less can be set up between the client premises and the provider network. Other types of proximity criteria may be used in different embodiments.

[0151] In various embodiments, connections may be established or enabled between some or all of the VRs themselves, as well as between the VRs and client premises networks, for example, using different types of attachments as shown in FIG. 19. For example, peering attachments with dynamic routing enabled may be set up between pairs of VRs 2572A, VRs 2572B, or VRs 2572C. Depending on client preferences, for example, as indicated in a programmatic attachment request, VPN attachments using one or more VPN tunnels may be created between one or more VRs 2572 and some client premises networks, while direct physical link-based attachments (DX attachments of the kind described above) may be set up between one or more VRs 2572 and other client premises networks. In at least some embodiments, such attachments may be used to establish network connections between VRs and a dynamic routing information source (DRIS) 2577, such as a client-owned router, client-managed SD-WAN equipment, etc., at a given client's premises. In various embodiments, network configuration information, such as IP addresses of the DRIS, may be provided by the client to the WAN service via a programmatic interface to enable the VR to communicate with the DRIS. In the illustrated embodiment, the OADC2512C may include one or more DRIS2577A, the OABO2515C may include a DRIS2577B, while the OAPOS2518C may include a DRIS2577, and may establish connections between at least some of these DRIS and the VR2572, such that dynamic routing information regarding endpoints in local or isolated networks at the client's premises can be obtained by the VR and used to direct traffic between regions. Note that in some embodiments, not all client premises include a DRIS.In some implementations, a respective protocol processing engine of a routing information exchange protocol (such as a version of BGP) indicated by a client may be set up for each VR (e.g., using an auxiliary task offloader of the type described above), and a protocol session may be initiated between the protocol processing engine of the VR and the DRIS to transfer routing information for the various client premises. The contents of a network packet originating at a given client premises (such as OAHQ, OADC, OABO, or OAPOS) may be transmitted to another client premises via several VRs 2572 and private fiber backbone links 2570, e.g., along a route identified using a set of dynamic routing information obtained at the VR from the DRIS in the illustrated embodiment.

[0152] Organization A, whose traffic is instead transmitted between client facilities shown in FIG. 25, may also be able to easily connect provider network resources 2521 to its private WAN built using the provider network's backbone links in the illustrated embodiment. For example, a manager administrator of organization A may use client side WAN management interface / tools 2550 to request a connection between one or more client facilities and organization A's isolated virtual network established with the provider network's virtualization computing services. In response to such a request, configuration settings may be changed in one of the VRs configured for the client, or a new VR may be configured, and in various embodiments, the requested connection may be enabled using the modified or new VR. The provider network may have data centers (and backbone links connected to such data centers) in additional countries, such as country / region 2510D in the illustrated embodiment, where organization A may not currently have any facilities or equipment. If organization A expands into such a country / region, only a few programmatic interactions may be required to extend the private WAN established using the provider network's backbone network. When adding additional facilities to an existing private WAN configured using the WAN service (either in a region where other client facilities are already connected to the WAN service, or in another region), the client need only provide the same type of information about the new facility as was previously provided for the other facilities via the programmatic interface. Connections between the VRs at the additional facility and the specified DRIS may then be established, and in various embodiments, routing information related to the additional facility may be propagated among some or all of the VRs already used for the client's VAN without requiring manual configuration of static routes.

[0153] In various embodiments, a client of the WAN service may obtain various metrics (e.g., total bytes transferred per unit time, bandwidth usage trends, measured latency, packet drop rates, etc.) of network traffic flowing between client premises in different geographic regions over the private fiber backbone, e.g., via a client-side WAN management interface 2550. In some embodiments, the client may select a preferred granularity at which the metrics are presented from a set of granularities including, for example, (a) a region-level granularity (metrics for all traffic flowing between client premises in a region pair are aggregated), (b) a client premises-level granularity (metrics are presented separately for different pairs of client premises), or (c) an isolated network-level granularity (metrics are presented separately for each IVN pair, as well as for each combination of IVN and client premises network). In at least some embodiments, a unified interface may be used to present not only intra-region traffic metrics, but also inter-region traffic metrics.

[0154] A client of the WAN service, in some embodiments, may utilize the service to manage various types of exception events for its applications, e.g., fail over some application workloads from one region to another in the event of an outage or other network problem. The WAN service may obtain from the client, e.g., via a programmatic interface, an indication of one or more diversion criteria (e.g., detection of failures, network slowdowns, etc.) for traffic directed to a first set of network endpoints at the client's premises in a given geographic region. The WAN service, in various embodiments, may monitor network performance data related to traffic to and from different client premises utilizing a backbone network, and reroute or divert traffic based on the client's expressed criteria. For example, in response to a determination that a diversion criterion has been met, some network packets whose original or first destination is an endpoint in a first region, such as 2510A, may be delivered instead to a failover or backup set of endpoints in another region, such as 2510B or 2510C, using an appropriate peered VR. In various embodiments, the diversion or failover criteria of different clients may be stored as part of the client's WAN metadata.

[0155] According to some embodiments, a client may request custom processing or actions for at least some of the packets transmitted through the WAN service 2502. For example, regulatory or organizational policies may require that an audit record be generated and stored when a packet is transmitted from a set of endpoints in one country or region to another country or region. The client may use a programmatic interface to indicate the custom actions to be performed and the conditions under which the actions are to be performed, and the WAN service may ensure that the actions are performed according to the illustrated embodiment. In some embodiments, for example, an offload device similar to those described above may be used for such custom actions.

[0156] In some embodiments, using a programmatic interface, a client of the WAN service may provide an indication of a target bandwidth limit for network traffic flowing over the private fiber backbone of the provider network between a first set of one or more client facilities in a first geographic region and a second set of one or more facilities in a second geographic region. The WAN service may ensure that such limits are enforced, e.g., by causing network packets to be dropped at an appropriate VR if the limit is reached. In some embodiments, the client may dynamically request an increase in the bandwidth limit via the programmatic interface. In response to such an increase request, the WAN control plane may provide an indication via the programmatic interface that confirms that there are sufficient resources in the backbone to support or sustain the increase (e.g., sufficient unused bandwidth on various links used for the client's inter-regional traffic) to confirm that the new target bandwidth limit is acceptable. In at least some embodiments, the WAN service may periodically and proactively provision additional backbone fiber links to clients in anticipation of potential requests for additional bandwidth from the clients, so that the clients do not have to wait long periods of time when higher bandwidth is required.

[0157] In various embodiments, multiple paths may be available through the private backbone of the provider network for traffic between a given pair of client facilities. For example, for availability and performance reasons with respect to the provider network's other services (e.g., virtualized computing services, various database services, etc.), multiple sets of fiber links may be provisioned by the provider network between its own data centers in different countries or regions, and such fiber links may provide alternative options for routing the WAN service client's traffic. In at least some embodiments in which the WAN service client specifies performance goals (e.g., latency goals) for traffic between the pair of client facilities, the VR used for the pair of client facilities may dynamically select a particular set of backbone links that can meet the client's performance goals using current or recent performance metrics obtained from several different alternative sets of backbone links available for traffic between the client's facilities. At least some packets may then be transmitted between the client facilities using the selected set of links. In some embodiments, the client may provide rules for forwarding dynamic routing information for a specified facility or region, e.g., using configuration settings similar to those described above in the context of Figures 18 and 20. It should be noted that the various features and functions of the virtual routers described above in the context of Figures 1 through 23 may be utilized by or for the WAN services of a provider network in some embodiments.

[0158] Example Graphical Interface for WAN Services FIG. 26 illustrates an exemplary web-based interface that may be used to provide WAN service quality metrics for traffic between client-specified locations, according to at least some embodiments. As illustrated, web-based interface 2602, implemented by a WAN service functionally similar to service 2502 of FIG. 25, may include an introductory message region 2604 in which the potential client is requested to provide a list of cities / regions in which the potential client's facilities are located. This interaction may be initiated, in some embodiments, before the potential client agrees to use the WAN service so that the WAN service can determine whether it can provide a backbone connection between the client's locations and / or so that the potential client can ascertain the WAN service's quality metrics for traffic between the regions. In the illustrated embodiment, the potential client indicates that the facility is located in City A, City B, and City C, which are located within State A of Country A, State B of Country B, and State C of Country C, respectively. After the client facility locations are entered into table 2606, the client may use submit button 2608 to submit the information to the WAN service.

[0159] In response to the submission of the client facility location information, the WAN service may present a set of service quality metrics for traffic transmitted over the provider network backbone network between the regions in which the client facilities are located in the illustrated embodiment. Several metrics for each direction of traffic flow between pairs of client facility locations, such as metric-1, metric-2, metric-3, metric-4, metric-5, and metric-6, may be presented to the client via the web-based interface 2602. Such metrics may include, for example, packet transmission latency between locations, bytes transferred / second or bytes transferred / hour, packet drop rates, etc. In some embodiments, nominal or expected values ​​for some metrics may be provided along with actual measurements taken over a recent time interval. In one embodiment, instead of first asking the client for the facility location, a table 2610 showing such metrics for various country / region combinations may be presented by the WAN service as a way of informing the client about locations available to the WAN service.

[0160] In the embodiment shown in FIG. 26, the WAN service may present a recommendation message 2612 regarding a virtual router that should be configured if a potential client wishes to use the WAN service to route traffic over a provider network backbone link between client facilities. Table 2614 shows a list of provider network defined regions (e.g., Region-Country A-1, Region-Country B-1, Region-Country C-2) for which establishment of VRs is recommended, and a mapping between the provider network defined regions and the client facilities. Note that in at least some embodiments, at least some of the provider network defined regions may not correspond exactly to individual countries or states defined by government recognized boundaries. For example, regions may be defined by the provider network for internal administrative purposes based on the location of data centers, and a given provider network defined region may include a portion of a state / country rather than a complete state / country. When requesting resources (e.g., VRs) from a provider network, in at least some embodiments, the client may need to specify the provider network defined region in which the resource is to be established or created in the illustrated embodiment. For example, the provider network defined region may be indicated by a parameter in an API or command for requesting the resource. In the illustrated embodiment, instructions regarding next steps, such as how the VR should be configured, may be provided to the client via web-based interface 2602. For example, the client may be informed of configuration information regarding dynamic routing information sources for networks set up within the client's premises, the protocols used to exchange dynamic routing information, and / or any custom or supplemental actions on the client's traffic (such as the creation of audit log records) that the client must provide if the client wishes to utilize WAN services.

[0161] FIG. 27 illustrates an exemplary web-based interface that may be used to present status information for traffic flowing between client-specified locations, according to at least some embodiments. In the web-based interface 2702, message 2704 indicates how the client may change the granularity at which status information (including health or availability information, and traffic rates measured in either direction) for various portions of the client's wide area network is presented. In some embodiments, the client may select from information aggregated at a regional level, at the level of individual facilities (as shown in the example of FIG. 27), or even at the level of individual isolated networks within facilities and within the provider network, among other granularity options. In graph 2710, health statistics ("Status: OK") and the latest traffic rates in both directions of traffic are shown for provider backbone-based connections between facility P1 (City A) and facility P2 (City B), facility P1 and facility P3 (City C), and facility P2 and facility P3, with the three facilities configured with virtual routers VR-1, VR-2, and VR-3, respectively. In some embodiments, the granularity may be changed using a zoom in / out control element 2711. For example, if the client zooms out so that several different regions (each containing one or more client facilities) are visible, the granularity of the displayed information may automatically change to a region-level granularity. The client may change the granularity by clicking on connectors shown between VRs or facilities in the graph 2710, in the illustrated embodiment.

[0162] In addition to viewing their WAN status information using the web-based interface 2702, clients may view and / or change target data transfer rates between client facilities using table 2714 of the web-based interface 2702 in the illustrated embodiment. Current requested data transfer rates in either direction between various facilities (e.g., P1 to P2, P2 to P1, etc.) or various regions may be shown in the "Current Limit" column of table 2714. The "New Limit" column may be used to change the target data transfer rate supported by the WAN service for any pair of facilities in the illustrated embodiment. In some embodiments, a client may wish to increase the limit based on an expected increase in application traffic demand. A client may wish to lower the target limit in embodiments in which clients are charged by the WAN service based on requested bandwidth limits. In at least some embodiments, other types of information may be provided to WAN service clients other than the information shown in Figures 26 and 27.

[0163] Example of custom processing using WAN services In some embodiments, as described above, a client may request that a specified custom processing action be performed on at least a portion of traffic sent through the WAN service on the client's behalf. Figure 28 illustrates an example scenario in which mandatory intermediation for traffic flowing between specified locations may be configured on behalf of a client of the WAN service, according to at least some embodiments. In the illustrated example, VRs 2825A and 2825B are configured to route traffic between a client premises P1 in region A and a client premises P2 in region B.

[0164] In response to a client request, a mandatory intermediary 2835 including an audit engine 2871 may be configured for the client's inter-region traffic (i.e., for packets sent between region A and region B). In the illustrated embodiment, based on a custom action specification provided by the client, the audit engine 2871 may inspect some or all packets sent between regions and generate and store corresponding records in an audit log. In some embodiments, a mandatory intermediary may be established in an offload device of the type described above such that custom actions do not need to be performed by a routing plane node in the VR. In other embodiments, a separate computing device not utilized to offload workloads from the VR may be used. In at least one embodiment, the client may provide executable code used to perform custom actions on the client's traffic, and the executable code may be deployed to one or more devices by the WAN service. In some embodiments, custom actions for multiple client requests may be performed on inter-region traffic, for example, using respective processing engines (such as the audit engine 2871) or using a single processing engine configured to perform all actions.

[0165] In some embodiments, the provider network WAN service may be configured as a primary path for traffic between several client facilities, while a secondary path for traffic may be configured using resources outside the provider network. Thus, two types of WAN links may be used: a provider network private backbone link (used as a primary WAN link), and an external WAN link (e.g., a leased fiber line similar to that shown in FIG. 24). In some embodiments, a client's inter-region traffic may be distributed between the two types of WAN links based on splitting conditions specified by the client, for example. For example, a client may specify that 60% of the traffic should flow on the provider network's backbone link, and the remaining 40% should be sent via the leased fiber line. In one such embodiment, the client's traffic splitting preferences may be provided to the WAN service along with information regarding how the VR should direct the portion of the traffic that is not sent via the backbone link. The VR may be configured to send the requested portion of the traffic to the external WAN link (e.g., 60% of the packet flow is sent via the backbone, and 40% is sent from the VR to the WAN equipment specified by the client and sent via the leased fiber line). In another approach, the provider network backbone links may be used by default, and the client's traffic may be switched to the external fiber line in the event of a failure reported by the WAN service or in response to performance metrics reaching thresholds specified by the WAN service. In some embodiments, the served client may provide the WAN service with configuration information that can be used to access performance metrics of the leased fiber WAN, and an integration tool or interface (similar to the interface shown in FIG. 27) may be used to provide the client with performance metrics and health status for both types of WANs.

[0166] Example programmatic interactions with WAN services FIG. 29 illustrates an example set of programmatic interactions related to the use of private provider network backbone network links for traffic between client premises, according to at least some embodiments. In the illustrated embodiment, a WAN service 2912, functionally similar to WAN service 2502 of FIG. 5, may implement a set of programmatic interfaces 2977 that may be used by clients 2910 to send requests and messages regarding desired WAN configurations and receive corresponding responses. Programmatic interfaces 2977 may include, for example, web-based consoles, command line tools, APIs, and / or graphical user interfaces similar to those illustrated in FIGS. 26 and 27. In some embodiments, the WAN service may be implemented as a subcomponent of a more general packet processing service (PPS) of the type previously described, and the PPS programmatic interfaces may be used for the interactions illustrated in FIG. 29.

[0167] A client 2910 (or a potential client that has not yet decided whether to start using the WAN service 2912) may, in the illustrated embodiment, send information in one or more ClientPremisesInfo messages 2914 about a set of client facilities between which a network connection over the private fiber backbone of the provider network is desired. In some embodiments, the facility information may include only location information (e.g., city, state, country) of the individual facilities. In other embodiments, more detailed information may be included in the ClientPremisesInfo message, such as, for example, if the client has already decided to use the WAN service, the desired bandwidth and / or latency, the protocol(s) used to exchange dynamic routing information about the facility, the IP address of a dynamic routing information source such as a client managed router or SD-WAN equipment for the facility, etc. In response, in some embodiments, the WAN service may send a WANServiceInfoForPremises message 2915. The WANServiceInfoForPremises message may provide, for example, performance information (e.g., nominal available bandwidth, data transfer rates measured in recent time intervals, latency, packet error rates, etc.) of private backbone traffic between the provider network closest to the data center for the facility. In some embodiments, the WANServiceInfoForPremises message may include recommendations regarding the number of virtual routers that may be required and the mapping between client premises and virtual routers (e.g., specific client premises that should programmatically attach their networks to each of the recommended virtual routers).

[0168] In the embodiment shown in FIG. 29, the client 2910 may send one or more EstablishVirtualRouters requests 2921 via the program interface 2977, for example, to create a set of virtual routers recommended by the WAN service. In response, the virtual routers may be established, for example, by using a computing instance of a virtualization computing service of the provider network, as described above, with metadata related to the virtual routers being stored / managed in a packet processing service (PPS) of the provider network. A VRsEstablished message 2923 may be sent back to the client in some embodiments after the VRS has been established and programmatically attached to the network at the client premises. In at least some embodiments, in addition to allocating / assigning the fast path and exception path nodes of the VR, auxiliary task offloaders including a protocol processing engine (e.g., for a particular protocol indicated by the client) may also be configured for the VR by the PPS. A dynamic routing information exchange session may then be initiated between the VR protocol processing engine and the dynamic routing information source at the client premises. In addition to attachments between VRs and client premises networks, in at least some embodiments, peering attachments with dynamic routing enabled (similar to the attachments described in the context of FIG. 18) may be established between various pairs of VRs. In one embodiment, a separate program request may be sent by the client 2910 to create a VR and then attach the VR to the client premises network as well as other VRs. In another embodiment, the client may not need to create a VR or request an attachment. Instead, the WAN service may automatically set up and configure the VR based on information provided by the client regarding the client premises network. In some embodiments, a VPN tunnel may be set up between the VR and one or more client premises networks based on client preferences indicated, for example, via program interface 2977.In other embodiments, a dedicated private physical link (direct connect link) may be set up for communication between one or more client premises and their corresponding VRs.

[0169] In some embodiments, the client 2910 may provide rules for controlling the type of routing information sent between VRs and / or between VRs and dynamic routing information sources at the client premises. Such rules may be indicated via a DynamicRoutingConfigSettings message 2925, which may include information similar to that included in the RoutingInfoTransferConfigSettings message 2225 of FIG. 22. Any of a variety of configuration settings may be specified, including the particular version or variant of the protocol to be used (e.g., any of the various flavors of BGP, such as eBGP, iBGP, MP-BGP, etc.), filtering of outbound advertised routes, filtering of inbound advertisements, relative priorities assigned to various BGP attributes for selecting the next hop to a destination, CIDR blocks to be used for the IP addresses of the protocol processing engines, autonomous system identifiers assigned to the protocol processing engines, etc. The specified settings and / or rules may be stored in the WAN service's client WAN metadata store and applied to the client's WAN configuration before the ConfigSettingsApplied message 2927 is sent to the client.

[0170] The client may provide WAN failover-related configuration using, for example, in the illustrated embodiment, one or more TrafficDiversionConfigInfo messages 2931 to the WAN service. The TrafficDiversionConfigInfo message may indicate one or more diversion criteria for traffic normally directed to a set of network endpoints in one or more client premises. The diversion criteria may include, for example, detection that one or more links to the set of network endpoints have failed or that the latency of delivering a packet to the set of endpoints exceeds a threshold. The TrafficDiversionConfigInfo may indicate an alternative endpoint, such as a customer premises in another region, to which traffic should be diverted if the criteria are met. In some embodiments, the traffic diversion configuration information may be stored and a DiversionInfoSaved message 2934 may be sent to the client. According to the provided information, in various embodiments, the VR configured for the client's WAN may divert packets from one set of network endpoints in one region (the packet's original destination) to another set of network endpoints in another region when the diversion criteria are met.

[0171] In some embodiments, a client may request that custom actions be performed on at least some packets sent on the client's behalf over the provider network's private fiber backbone, e.g., by an audit engine or other intermediary similar to that described in the context of FIG. 28. Rules for selecting the set of packets on which such actions are to be performed, and a descriptor of the desired actions themselves (e.g., in source code or executable code format), may be provided by the client 2910 in the illustrated embodiment in one or more ConfigureCustomActionsForInterregionTraffic messages 2937. The configuration operations required to perform the custom actions may be completed by the WAN service (such as instantiating an audit engine or other intermediary engine on an offload device configured for VR), and a CustomActionsEnabled message 2940 may be sent to the client.

[0172] The client 2910 may set and / or modify WAN performance objectives using, for example, a SetWANTrafficLimits message 2943. If the client desires to increase the traffic rate, the WAN service, in some embodiments, may verify that sufficient capacity is available in the private fiber backbone network to support the increase before sending a WANTrafficLimitsSet response 2947 to the client.

[0173] A client 2910, in the illustrated embodiment, may request performance metrics, availability metrics, and / or health status updates for a backbone-based private WAN by sending a ShowWANMetrics request 2951. In response, the requested set of metrics may be presented to the client via one or more MetricsSet responses 2954. The metrics may include, for example, data transfer rates, packet delays, packet drop / loss rates, uptime, etc., provided at any of several granularities selected by the client (e.g., geographic granularity, facility granularity, per isolated network granularity, etc.).

[0174] According to at least one embodiment, clients 2910 may wish to enable connectivity between their client premises networks and one or more isolated virtual networks (IVNs) set up on the client's behalf in the provider network's virtualized computing service. In such an embodiment, an EnableConnectivityWithIVN request 2957 may be sent to the WAN service. In response, one or more configuration changes may be made in the appropriate VRs (e.g., IVN attachments may be created) of the client's WAN and / or the specified IVNs so that traffic may be routed between the IVNs and the client's premises. An IVNConnectivityEnabled message 2960 may then be sent to the client. Once a client provides information regarding additional client premises networks (indicated in a ClientPremisesInfo message) or additional IVNs (indicated in an EnableConnectivityWithIVN message) to add to the WAN, routing information for the additional networks may be automatically propagated between VRs set up in the client's private WAN without the client needing to provide static routes or perform additional configuration operations. In some embodiments, programmatic interactions other than those shown in FIG. 29 may be supported by the WAN service 2912.

[0175] How to implement WAN services FIG. 30 is a flow diagram illustrating aspects of operations that may be performed in a wide area networking service of a provider network that transmits traffic between client premises over a private fiber backbone, according to at least some embodiments. As shown in element 3001, information regarding the various premises of clients to which the inter-premises traffic is to be transmitted using the private fiber backbone network of the provider network to enable dynamic routing may be determined or obtained through a programmatic interface of a WAN service, for example, functionally similar to WAN service 2502 of FIG. 25. In addition to the WAN service itself, the provider network may implement various services accessible on the network, such as virtualized computing services (VCS) and packet processing services (PPS) of the type previously described. The information may include, for example, in the illustrated embodiment, the location of the premises, the protocol used to exchange dynamic routing information that can be used to transmit packets to and from the premises' network, the address of a dynamic routing information source in the client premises network, such as a client-owned router, SD-WAN equipment, etc.

[0176] Optionally, in some embodiments, the client may be provided with at least some WAN service quality information related to the location of the client facility, such as expected or measured backbone-based data transfer rates (based on aggregated metrics for the provider network's internal traffic and / or for other WAN service clients' traffic) between provider network datacenters located near the client's facility, packet transfer latencies between such datacenters, packet loss rates, etc. (element 3004). In some embodiments, the provider network may organize its datacenters into regional groups, obtain metrics for traffic flowing between selected datacenters in different regions, and use the metrics to generate and provide WAN service quality metrics. Approval may be obtained from the client to enable backbone-based connectivity between the facilities (also referred to as the client's private WAN). In some embodiments, approval may be provided by the client in the same message / request in which the client provides information about the facilities to which it will be connected using the private fiber backbone.

[0177] In the illustrated embodiment, using provider network resources (e.g., using the computing instances of the VCS and / or the control plane components of the PPS), a set of virtual routers (VRs) may be configured for the client's private WAN (element 3007). A given VR may be configured, in the illustrated embodiment, using provider network resources that satisfy a proximity criterion with respect to one or more client facilities. For example, if a client facility is in a state S1 of a country, the VR may be configured in a region R1 defined by a provider network that includes at least some data centers in S1 or a neighboring state in preference to a region R2 defined by another provider network whose data centers are more distant. Connections may be established, in various embodiments, using programmatic attachments of the type described above, for example, at least between pairs of VRs, as well as between individual VRs and neighboring client facility networks.

[0178] In some embodiments, a respective protocol processing engine (e.g., a BGP processing engine) may be instantiated or configured for each of the VRs for received dynamic routing information regarding the client premises network (element 3010). In embodiments where some or all of the client premises include dynamic routing information sources, such as client-owned or client-managed hardware routers and / or SD-WAN equipment, a connection may be established between at least one VR configured for the client and each of the dynamic routing information sources. For example, a BGP session may be established between a protocol processing engine configured for the VR and the dynamic routing information source of the client premises. In some embodiments, an auxiliary task offloader may be used for the protocol processing engine of the VR.

[0179] Routing information for a given client premises network may be obtained at a protocol processing engine of the VR (element 3013). The information may be propagated to other VRs configured on behalf of the client, for example, in some embodiments according to protocol forwarding configuration settings indicated by the client. In various embodiments, routing information for a given client premises network may be propagated through one or more VRs to client premises routers or SD-WAN equipment in another client premises network.

[0180] The routing information obtained and propagated between the VRs of the client's private WAN may, in various embodiments, be used to forward at least some network packets between the client facilities over the private fiber backbone of the provider network (element 3016).

[0181] It should be noted that in various embodiments, some of the operations shown in Figures 10, 17, 23, and / or 30 may be performed in a different order than that shown in the figures, or may be performed in parallel rather than sequentially. Additionally, some of the operations shown in Figures 10, 17, 23, and / or 30 may not be required in one or more implementations.

[0182] Exemplary Computer System In at least some embodiments, a server implementing the types of technologies described herein (e.g., packet processing services, auxiliary task offloading, wide area networking services, or various functions of other services of a provider network) may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media. FIG. 31 illustrates such a general-purpose computing device 9000. In the illustrated embodiment, the computing device 9000 includes one or more processors 9010 coupled to a system memory 9020 (which may include both non-volatile and volatile memory modules) via an input / output (I / O) interface 9030. The computing device 9000 further includes a network interface 9040 coupled to the I / O interface 9030.

[0183] In various embodiments, the computing device 9000 may be a uniprocessor system including one processor 9010, or a multiprocessor system including several processors 9010 (e.g., two, four, eight, or another suitable number). The processor 9010 may be any suitable processor capable of executing instructions. For example, in various embodiments, the processor 9010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, ARM, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of the processors 9010 may typically, but not necessarily, implement the same ISA. In some implementations, a graphics processing unit (GPU) and / or a field programmable gate array (FPGA) may be used in place of or in addition to a conventional processor.

[0184] The system memory 9020 may be configured to store instructions and data accessible by the processor(s) 9010. In at least some embodiments, the system memory 9020 may include both volatile and non-volatile portions, while in other embodiments, only volatile memory may be used. In various embodiments, the volatile portion of the system memory 9020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM, or any other type of memory. For the non-volatile portion of the system memory (which may include, for example, one or more NVDIMMs), in some embodiments, flash-based memory devices, including NAND flash devices, may be used. In at least some embodiments, the non-volatile portion of the system memory may include a power source, such as a supercapacitor or other power storage device (e.g., a battery). In various embodiments, any of memristor-based resistive random access memory (ReRAM), 3D NAND technology, ferroelectric RAM, magnetoresistive RAM (MRAM), or various types of phase change memory (PCM) may be used for at least the non-volatile portion of the system memory. In the illustrated embodiment, program instructions and data implementing one or more desired functions, such as those methods, techniques, and data described above, are stored in system memory 9020 as code 9025 and data 9026.

[0185] In one embodiment, the I / O interface 9030 may be configured to coordinate I / O traffic between the processor 9010, the system memory 9020, and any peripheral devices within the device, including the network interface 9040, or other peripheral interfaces such as various types of persistent and / or volatile storage devices. In some embodiments, the I / O interface 9030 may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., the system memory 9020) into a format suitable for use by another component (e.g., the processor 9010). In some embodiments, the I / O interface 9030 may include support for devices attached through various types of peripheral buses, such as, for example, the Peripheral Component Interconnect (PCI) bus standard, or variants of the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 9030 may be split into two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of the I / O interface 9030, such as an interface to the system memory 9020, may be incorporated directly into the processor 9010.

[0186] The network interface 9040 may be configured to allow data to be exchanged between the computing device 9000 and other devices 9060 attached to the network(s) 9050, such as, for example, other computer systems or devices as illustrated in FIGS. 1-30. In various embodiments, the network interface 9040 may support communication over any suitable wired or wireless general data network, such as, for example, a type of Ethernet network. Additionally, the network interface 9040 may support communication over a telecommunications / telephony network, such as an analog voice network or a digital fiber communications network, over a storage area network, such as a Fibre Channel SAN, or over any other suitable type of network and / or protocol.

[0187] In some embodiments, the system memory 9020 may represent one embodiment of a computer-accessible medium configured to store at least a subset of the program instructions and data used to implement the methods and apparatus discussed in the context of FIGS. 1-30. However, in other embodiments, the program instructions and / or data may be received, transmitted, or stored in different types of computer-accessible media. Generally speaking, the computer-accessible medium may include non-transitory storage or memory media, such as magnetic or optical media, e.g., disks or DVDs / CDs, coupled to the computing device 9000 via the I / O interface 9030. The non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may be included in some embodiments of the computing device 9000 as the system memory 9020 or another type of memory. In some embodiments, multiple non-transitory computer-readable storage media may collectively store program instructions that, when executed on one or more processors or across one or more processors, implement at least a subset of the methods and techniques described above. The computer-accessible medium may further include a transmission medium or a signal, such as an electrical, electromagnetic, or digital signal, transmitted over a communication medium, such as a network and / or wireless link, as may be implemented via the network interface 9040. Some or all of the computing devices as shown in FIG. 31 may be used in various embodiments to implement the described functionality, e.g., software components executing on various different devices and servers may cooperate to provide the functionality. In some embodiments, some of the described functionality may be implemented using storage devices, network devices, or special purpose computer systems in addition to or instead of being implemented using a general purpose computer system.The term "computing device," as used herein, refers to at least all of these types of devices, but is not limited to these types of devices.

[0188] Embodiments of the present disclosure can be described in light of the following provisions. Clause 1. A system comprising: one or more computing devices; The one or more computing devices, when executed on or across the one or more computing devices, cause the one or more computing devices to: obtaining, via one or more programmatic interfaces of a wide area networking service of a provider network including a virtualized computing service, an indication of (a) a plurality of client facilities between which network traffic is routed via a private fiber backbone of the provider network, the plurality of client facilities including a first facility in a first geographic region and a second facility in a second geographic region, and (b) a particular protocol used to obtain dynamic routing information regarding at least the first facility and the second facility; configuring a first virtual router using at least a first set of resources of the virtualized computing service at a first provider network data center that satisfies a proximity criterion to the first facility, and a second virtual router using at least a second set of resources at a second provider network data center that satisfies the proximity criterion to the second facility; (a) enabling connectivity between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source located at the first facility, and (c) the second virtual router and a second dynamic routing information source located at the second facility; causing contents of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a set of routing information. Includes instructions, at least a portion of the set of routing information is obtained from the second dynamic routing information source by a protocol processing engine associated with the second virtual router, the protocol processing engine being configured to process messages of the particular protocol. The system.

[0189] Clause 2. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices are causing one or more metrics of network traffic flowing between the first geographic region and the second geographic region over the private fiber backbone to be presented via the one or more programmatic interfaces. 2. A system as described in clause 1, including further instructions.

[0190] Clause 3. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices are causing, via the one or more programmatic interfaces, to obtain an indication of a particular granularity at which the one or more metrics are presented, selected from a set of granularities including: (a) a geographical area level granularity; (b) a facility level granularity; or (c) an isolated network level granularity, wherein the one or more metrics are presented at the particular granularity; 2. A system as described in clause 2.

[0191] Clause 4. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices are obtaining, via the one or more programmatic interfaces, an indication of diversion criteria for traffic directed to a first set of network endpoints in the second geographic region; determining that the diversion criteria are met; In response to determining that the diversion criteria is satisfied, causing a particular network packet originating in the first geographic region to be delivered via the first virtual router to a network endpoint in a third geographic region. Further instructions include: an initial destination address of the particular network packet is an address of a network endpoint of the first set; A system according to any one of clauses 1 to 2.

[0192] Clause 5. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices are obtaining, via the one or more programmatic interfaces, a representation of an additional facility, wherein network traffic between the additional facility and one or more of the plurality of client facilities is transmitted over the private fiber backbone; Propagating routing information related to the additional facility to one or more virtual routers, including the first virtual router and the second virtual router, without requiring specification of a static route to the additional facility. 5. A system as described in any one of clauses 1-2 or 4, including further instructions.

[0193] Clause 6. Obtaining, via one or more programmatic interfaces, a representation of a plurality of facilities between which network traffic is routed over a private fiber backbone of the provider network, including a first facility in a first geographic region and a second facility in a second geographic region; establishing a first virtual router using a first set of resources that meets a proximity criterion to the first facility and establishing a second virtual router using a second set of resources that meets the proximity criterion to the second facility; (a) enabling connectivity between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source located at the first facility, and (c) the second virtual router and a second dynamic routing information source located at the second facility; causing content of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a first set of routing information; Including, At least a portion of the first set of routing information is obtained by the second virtual router from the second dynamic routing information source. Computer-implemented method.

[0194] Clause 7. Obtaining, via the one or more programmatic interfaces, an indication of a custom action to be performed with respect to at least some network packets transmitted between the first geographic region and a second geographic region; executing said custom action on a particular network packet; 7. The computer-implemented method of claim 6, further comprising:

[0195] Clause 8. The computer-implemented method of clause 7, wherein the custom action includes storing an audit log record.

[0196] Clause 9. Obtaining a request via a program interface to enable a connection between the first facility and an isolated virtual network of a virtualized computing service of a provider network; modifying a configuration of the first virtual router to enable the connection between the first facility and the isolated virtual network; 8. The computer-implemented method of any one of clauses 6-7, further comprising:

[0197] Clause 10. The computer-implemented method of any one of clauses 6-7 or 9, wherein the portion of the first set of routing information is obtained in a session of a version of a Border Gateway Protocol (BGP).

[0198] Clause 11. The computer-implemented method of any one of clauses 6-7 or 9-10, wherein establishing a connection between the first virtual router and the first dynamic routing information source includes establishing one or more virtual private network (VPN) tunnels.

[0199] Clause 12. The computer-implemented method of any one of clauses 6-7 or 9-11, wherein a connection between the first virtual router and the first dynamic routing information source is established using a dedicated physical link.

[0200] Clause 13. The second dynamic routing information source includes a software-defined wide area networking (SD-WAN) device, and the computer-implemented method further comprises: and obtaining, via the one or more programmatic interfaces, network configuration information for the SD-WAN equipment; enabling a connection between the second virtual router and the second dynamic routing information source includes utilizing the network configuration information. 13. The computer-implemented method of any one of clauses 6-7 or 9-12.

[0201] Clause 14. obtaining, via said one or more programmatic interfaces, an indication of a target bandwidth limit for network traffic flowing over said private fiber backbone between a first set of facilities in said first geographic region and a second set of facilities in said second geographic region; providing, via the one or more programmatic interfaces, an indication that the private fiber backbone has sufficient resources to allow traffic to flow at the target bandwidth limit; 14. The computer-implemented method of any one of clauses 6-7 or 9-13, further comprising:

[0202] Clause 15. causing the display, via one or more programmatic interfaces, of (a) a recommended number of virtual routers for routing traffic among said plurality of facilities, and (b) a recommended mapping between each facility of said plurality of facilities and the virtual routers to which connections from said each facility will be established. 15. The computer-implemented method of any one of clauses 6-7 or 9-14, further comprising:

[0203] Clause 16. When executed on or across one or more processors, causes said one or more processors to: obtaining, via one or more programmatic interfaces, a representation of a plurality of facilities between which network traffic is routed over a private fiber backbone of the provider network, the plurality of facilities including a first facility in a first geographic region and a second facility in a second geographic region; establishing a first virtual router using a first set of resources that meets a proximity criterion to the first facility and establishing a second virtual router using a second set of resources that meets the proximity criterion to the second facility; (a) enabling connectivity between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source located at the first facility, and (c) the second virtual router and a second dynamic routing information source located at the second facility; causing contents of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a first set of routing information. It stores program instructions, At least a portion of the first set of routing information is obtained by the second virtual router from the second dynamic routing information source. One or more non-transitory computer-accessible storage media.

[0204] Clause 17. The one or more non-transitory computer-accessible storage media of clause 16, wherein the first virtual router includes a fast path node and an exception path node, the fast path node configured to (a) obtain one or more routing actions from the exception path node and (b) implement the routing actions, the one or more non-transitory computer-accessible storage media, when executed on or across one or more processors, further cause the one or more processors to: and further storing program instructions for configuring an offload device linked to one or more nodes of the first virtual router with a routing information exchange protocol processing engine; the portion of the first set of routing information is processed by the routing information exchange protocol processing engine; said one or more non-transitory computer-accessible storage media.

[0205] Clause 18. The one or more non-transitory computer-accessible storage media of any one of clauses 16-17, wherein the portion of the first routing information is obtained from the second dynamic routing information source via a Border Gateway Protocol (BGP) session.

[0206] Clause 19. When executed on or across said one or more processors, causes said one or more processors to: obtaining, via the one or more programmatic interfaces, a performance objective for traffic between the first facility and the second facility; obtaining performance metrics for each of a plurality of links of the private fiber backbone that can be used to transmit traffic between the first facility and the second facility; selecting a particular link of the plurality of links for transmitting one or more network packets originating at the first facility to the second facility based at least in part on the respective performance metrics and the performance objectives. One or more non-transitory computer-accessible storage media according to any one of clauses 16 to 18 storing further program instructions.

[0207] Clause 20. When executed on or across said one or more processors, causes said one or more processors to: obtaining, via said one or more program interfaces, one or more rules for forwarding routing information relating to said second facility. storing further program instructions; the first set of routing information is provided to the first virtual router in accordance with the one or more rules. One or more non-transitory computer-accessible storage media according to any one of clauses 16 to 19.

[0208] conclusion Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing description of computer-accessible media. In general, computer-accessible media may include storage or memory media, such as magnetic or optical media, e.g., disks or DVDs / CD-ROMs, volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, and transmission media or signals, such as electrical, electromagnetic, or digital signals conveyed over a communication medium, such as a network and / or wireless link.

[0209] The various methods illustrated in the figures and described herein represent example embodiments of the methods. The methods may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed, and various components may be added, reordered, omitted, modified, etc.

[0210] Various modifications and variations may be made, as would be apparent to one skilled in the art having the benefit of this disclosure. All such modifications and variations are intended to be included, and therefore the above description should be considered in an illustrative rather than a limiting sense.

Claims

1. 1. A system comprising: one or more computing devices; The one or more computing devices, when executed on or across the one or more computing devices, cause the one or more computing devices to: obtaining, via one or more programmatic interfaces, a representation of a plurality of client premises between which network traffic is routed via a private fiber backbone of the provider network, the plurality of client premises including a first premises in a first geographic region and a second premises in a second geographic region; establishing a first virtual router using a first set of resources that meets a proximity criterion to the first facility and establishing a second virtual router using a second set of resources that meets the proximity criterion to the second facility; (a) enabling a connection between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source at the first facility, and (c) the second virtual router and a second dynamic routing information source at the second facility; causing contents of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a first set of routing information. Includes instructions, at least a portion of the first set of routing information is obtained by the second virtual router from the second dynamic routing information source; The system.

2. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices may causing one or more metrics of network traffic flowing between the first geographic region and the second geographic region over the private fiber backbone to be presented via the one or more programmatic interfaces. The system of claim 1 further comprising instructions.

3. When the one or more computing devices are executed on or across the one or more computing devices, the one or more computing devices may obtaining, via the one or more programmatic interfaces, a representation of an additional facility, wherein network traffic between the additional facility and one or more facilities of the plurality of facilities is transmitted over the private fiber backbone; Propagating routing information related to the additional facility to one or more virtual routers, including the first virtual router and the second virtual router, without requiring specification of static routes to the additional facility. A system according to any preceding claim, further comprising further instructions.

4. obtaining, via one or more programmatic interfaces, a representation of a plurality of facilities between which network traffic is routed over a private fiber backbone of the provider network, the plurality of facilities including a first facility in a first geographic region and a second facility in a second geographic region; establishing a first virtual router using a first set of resources that meets a proximity criterion to the first facility and establishing a second virtual router using a second set of resources that meets the proximity criterion to the second facility; (a) enabling connectivity between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source at the first facility, and (c) the second virtual router and a second dynamic routing information source at the second facility; causing content of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a first set of routing information; Including, at least a portion of the first set of routing information is obtained by the second virtual router from the second dynamic routing information source; Computer-implemented method.

5. obtaining, via the one or more programmatic interfaces, an indication of a custom action to be performed with respect to at least some network packets transmitted between the first geographic region and a second geographic region; executing said custom action on a particular network packet; The computer-implemented method of claim 4 , further comprising:

6. obtaining a request via a programmatic interface to enable a connection between the first facility and an isolated virtual network of a virtualized computing service of a provider network; modifying a configuration of the first virtual router to enable the connection between the first facility and the isolated virtual network; The computer-implemented method of any one of claims 4 to 5, further comprising:

7. The computer-implemented method of any one of claims 4 to 6, wherein the portion of the first set of routing information is obtained in a Border Gateway Protocol (BGP) version session.

8. 8. The computer-implemented method of claim 4, wherein establishing a connection between the first virtual router and the first dynamic routing information source comprises establishing one or more virtual private network (VPN) tunnels.

9. The second dynamic routing information source includes a software-defined wide area networking (SD-WAN) device, and the computer-implemented method further comprises: obtaining network configuration information for the SD-WAN device via the one or more program interfaces; enabling a connection between the second virtual router and the second dynamic routing information source includes utilizing the network configuration information. A computer implemented method according to any one of claims 4 to 8.

10. obtaining, via the one or more programmatic interfaces, an indication of a target bandwidth limit for network traffic flowing over the private fiber backbone between a first set of premises in the first geographic region and a second set of premises in the second geographic region; providing, via the one or more programmatic interfaces, an indication that the private fiber backbone has sufficient resources to allow traffic to flow at the target bandwidth limit; The computer-implemented method of any one of claims 4 to 9, further comprising:

11. causing a display, via one or more programmatic interfaces, of (a) a recommended number of virtual routers for routing traffic between said plurality of facilities, and (b) a recommended mapping between each facility of said plurality of facilities and the virtual routers to which connections from said each facility will be established. The computer-implemented method of any one of claims 4 to 10, further comprising:

12. When executed on one or more processors or across one or more processors, the one or more processors are caused to: obtaining, via one or more programmatic interfaces, a representation of a plurality of client premises between which network traffic is routed via a private fiber backbone of the provider network, the plurality of client premises including a first premises in a first geographic region and a second premises in a second geographic region; establishing a first virtual router using a first set of resources that meets a proximity criterion to the first facility and establishing a second virtual router using a second set of resources that meets the proximity criterion to the second facility; (a) enabling a connection between the first virtual router and the second virtual router, (b) the first virtual router and a first dynamic routing information source at the first facility, and (c) the second virtual router and a second dynamic routing information source at the second facility; causing contents of at least one network packet originating at the first facility to be forwarded over the private fiber backbone to the second facility using a first set of routing information. It stores program instructions, at least a portion of the first set of routing information is obtained by the second virtual router from the second dynamic routing information source; One or more non-transitory computer-accessible storage media.

13. 13. The one or more non-transitory computer-accessible storage media of claim 12, wherein the first virtual router includes a fast path node and an exception path node, the fast path node configured to (a) obtain one or more routing actions from the exception path node and (b) implement the routing actions, the one or more non-transitory computer-accessible storage media, when executed on or across one or more processors, further causing the one or more processors to: and further storing program instructions for configuring an offload device linked to one or more nodes of the first virtual router with a routing information exchange protocol processing engine; the portion of the first set of routing information is processed by the routing information exchange protocol processing engine. said one or more non-transitory computer-accessible storage media.

14. When executed on or across the one or more processors, the one or more processors are caused to: obtaining, via the one or more programmatic interfaces, a performance objective for traffic between the first facility and the second facility; obtaining performance metrics for each of a plurality of links of the private fiber backbone that can be used to transmit traffic between the first facility and the second facility; selecting a particular link of the plurality of links for transmitting one or more network packets originating at the first facility to the second facility based at least in part on the respective performance metrics and the performance objectives. One or more non-transitory computer-accessible storage media according to any one of claims 12 to 13, storing further program instructions.

15. When executed on or across the one or more processors, the one or more processors are caused to: obtaining, via said one or more program interfaces, one or more rules for forwarding routing information relating to said second facility; storing further program instructions; the first set of routing information is provided to the first virtual router in accordance with the one or more rules. One or more non-transitory computer-accessible storage media according to any one of claims 12 to 14.

Images