Virtual network interface for managed layer 2 connectivity in computing service extension locations

Abstract

A networking manager of an expansion server of the virtualized computing service detects that a data link layer frame has been obtained at the expansion server. In response to determining that a destination media access control (MAC) address of the frame matches a MAC address of a local premises access virtual network interface attached to the compute instance, the networking manager delivers at least a portion of the contents of the frame to a compute instance running on the expansion server. The local premises access virtual network interface is not assigned an Internet Protocol (IP) address from a range of IP addresses managed by the virtualized computing service.

Description

[Background technology]

[0001] Many businesses and other organizations operate computer networks that interconnect numerous computing systems to support their operations, such as with computing systems that are co-located (e.g., as part of a local network) or located in multiple separate geographic locations (e.g., connected via one or more private or public intermediate networks). Data centers that house significant numbers of interconnected computing systems have become commonplace, including private data centers operated by and on behalf of a single organization and public data centers operated as businesses by entities to provide computing resources to customers. Some public data center operators provide network access, power, and secure installation facilities for various customer-owned hardware, while others offer "full-service" facilities that also include hardware resources made available for use by customers.

[0002] The emergence of virtualization technology for commodity hardware has provided benefits for managing large-scale computing resources for many customers with diverse needs, enabling various computing resources to be shared more efficiently and securely among multiple customers. For example, virtualization technology may enable a single physical virtualization host to be shared among multiple users by providing each user with one or more “guest” virtual machines hosted by the single virtualization host. Each such virtual machine may represent a software simulation that functions as a separate logical computing system, giving the user the illusion that they are the sole operator of a given hardware computing resource, while also providing application isolation and security among the various virtual machines. Instantiating several different virtual machines on the same host may help increase overall hardware utilization levels in a data center, resulting in a higher return on investment. In some cases, the benefits of virtualization management technology and its associated resource manageability may be extended to premises outside of a cloud provider's data center. [Brief explanation of the drawings]

[0003] [Figure 1] 1 illustrates an example system environment in which a local premises access virtual network interface used for Layer 2 communication in a managed open systems interconnection model can be programmatically attached to a compute instance launched on an extended server of a virtualized computing service, according to at least some embodiments. [Figure 2] 1 illustrates an example use of mapping services and encapsulation protocols in a virtualized computing service where a logical network is configured on top of a substrate network, according to at least some embodiments. [Figure 3]1 illustrates an example use of an enhanced traffic mediation device, according to at least some embodiments, that allows control plane traffic to flow securely from a data center of a provider network to an external premises where compute instances managed by a virtualized computing service are launched. [Figure 4] 1 illustrates an example network configuration for compute instances on an expansion server, according to at least some embodiments. [Figure 5] Illustrates an example configuration in which compute instances configured within separate isolated virtual networks of a virtualized computing service on an extended server, according to at least some embodiments, may share network cables to communicate with devices in a VCS extended premises. [Figure 6] 1 illustrates an example configuration in which several local premises access virtual network interfaces, each with its own security and performance-related properties, may be configured in an extension server of a virtualized computing service, according to at least some embodiments. [Figure 7] 1 illustrates example programmatic interactions associated with configuring and using a local premises access virtual network interface, according to at least some embodiments. [Figure 8] 1 is a flow diagram illustrating aspects of operations that may be performed to configure and utilize a local premises access virtual network interface, according to at least some embodiments. [Figure 9] 1 illustrates example components of an enhanced server, according to at least some embodiments. [Figure 10] FIG. 1 is a block diagram illustrating an example of a computing device that may be used in at least some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0004] Although embodiments are described herein by way of example in several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. The drawings and detailed description thereof are not intended to limit the embodiments to the particular disclosed forms; on the contrary, the intent is to cover all modifications, equivalents, and alternatives falling within the spirit and scope defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or claims. As used throughout this application, the word "may" is used in an permissive sense (i.e., meaning having the potential to), rather than a mandatory sense (i.e., meaning necessary). Similarly, the terms "include," "including," and "includes" mean including, but not limited to, something. When used in the claims, the term "or" is used as an inclusive or, not an exclusive or. For example, the phrase "at least one of x, y, or z" means any one of x, y, and z, and any combination thereof. Unless otherwise indicated, articles such as "a" or "an" should generally be construed throughout this application to include one or more listed items. Thus, phrases such as "a device configured to" are intended to include one or more listed devices. Such one or more listed devices may be collectively configured to perform the listed enumeration. For example, "a processor configured to perform enumerations A, B, and C" may include a first processor configured to perform enumeration A working in conjunction with a second processor configured to perform enumerations B and C.

[0005] The present disclosure relates to a method and apparatus for setting up premises external to a provider network's data center and local devices at the external premises to enable managed Open Systems Interconnection Model Layer 2 (OSI L2) communication between compute instances of a virtualized computing service (VCS) in a provider network or cloud computing environment. Such external premises may include customer data centers, manufacturing facilities, restaurants, etc., where the owners of the external premises wish to run applications within VCS compute instances on local servers and also wish to implement applications and protocols that require OSI L2 connectivity between non-VCS devices (devices not managed or owned by VCS) located on the premises and the compute instances. Because various types of features and functions supported by VCS in the provider network's data center are extended to the external premises, the external premises are referred to as VCS-extended premises. The local servers on which VCS compute instances run at the external premises are referred to as VCS-extended servers or "outpost" servers.

[0006] Compute instances can be set up using commands sent over secure network paths established between the VCS control plane server (located in the provider network's data center) and the extension server. Multiple types of virtual network interfaces, or VNIs (logical devices managed by VCS and mapped to physical network interfaces by VCS), including cloud access VNIs (CVNIs) and local premises access VNIs (LVNIs), can be used to enable network connectivity for VCS compute instances. VNIs (also known as "elastic network interfaces," or ENIs) enable various network-related attributes, such as IP (Internet Protocol) addresses and / or security settings governing inbound and outbound messages, to be easily transferred between compute instances without necessarily reconfiguring physical network cards. Such attribute transfer can be achieved by programmatically detaching a VNI from one compute instance and programmatically attaching it to another, independent of the specific hardware network interface card (NIC) of the host on which the compute instances run. For example, when a VNI is created or modified at the request of a VCS client or customer, metadata indicating the VNI's properties (such as IP address, Media Access Control (MAC) address, security rules, etc.) is stored or updated by the VCS's management component.

[0007] By programmatically attaching a cloud access virtual network interface (CVNI) to a compute instance on an extended server, the compute instance can be configured with an IP (Internet Protocol) address within an isolated virtual network (IVN) established within VCS on behalf of the client. As a result of the use of the CVNI, the compute instance running on the extended server has connectivity to other provider network resources (e.g., other compute instances implemented on virtualized servers in the provider network datacenter, resources within other provider network services such as storage services and database services) equivalent to the connectivity provided if the compute instance were launched in the provider network datacenter. A second type of virtual network interface, called a local premises access virtual network interface (LVNI), can also be programmatically attached to the compute instance on an extended server to provide OSI L2 connectivity managed using the VCS control plane. A MAC address is assigned to the LVNI by the VCS control plane, and this address can be used for bidirectional OSI L2 communication (including broadcast, if necessary) between the compute instance and devices in the customer-setup on-premises network without the need for an intermediary networking device such as a router or gateway. By using a combination of LVNI and CVNI, compute instances set up on a VCS extended server can therefore access local on-premise resources (via OSI L2) and cloud-based resources (via IP) with equal ease, without the administrator of the on-premise resources having to go to the trouble of configuring or managing intermediary networking devices for local traffic.

[0008] As one skilled in the art will appreciate in light of this disclosure, certain embodiments may be capable of achieving various advantages, including some or all of: (a) enabling programs running on VCS compute instances on extended servers to participate in applications and protocols running at OSI Layer 2 in the extended premises without having to configure intermediary devices, such as gateways or routers, in the extended premises, while still maintaining full connectivity to data center resources in the provider network; and (b) improving the user experience for local network administrators of the extended premises, for example, by enabling configuration and management of both cloud access virtual network interfaces and local premises access virtual network interfaces using a common toolset.

[0009] According to some embodiments, a system may include one or more control plane servers (CPSs) of a provider network's virtualized computing service (VCS) and a networking manager running on an extension server (ES) of the VCS. The CPSs may be located within one or more data centers of the provider network, and the ESs may be located on premises outside the provider network's data centers, referred to as VCS extension premises. The CPS, in various embodiments, may programmatically attach at least one pair of virtual network interfaces (VNIs) to compute instances (e.g., virtual machines) of the VCS running on the ESs. The programmatic attachment process, in at least some embodiments, may involve sending a command or request from the CPS to a local management component of the ES. One of the VNIs, referred to as the cloud access VNI or CVNI, may be assigned an IP address from an IP address range of an isolated virtual network (IVN) configured in the VCS on behalf of VCS clients whose requests the extension server configures, and may be primarily used to communicate with resources in the provider network's data centers, such as compute instances, storage servers for a storage service, and database servers for a database service. An IVN (also known as a virtual private cloud) may include a collection of networked resources (e.g., including compute instances) allocated to a given VCS client that are logically isolated from (and, by default, inaccessible to) resources allocated to other clients in other isolated virtual networks. The client on whose behalf the IVN is established may be granted significant flexibility regarding the network configuration for the IVN's resources.For example, a private IP address for a compute instance may be selected by a client without having to consider whether other resources in other IVNs may be assigned the same IP address, whether a subnet of the client's choice may be established within the IVN, whether security rules may be set up by the client for incoming and outgoing traffic on the IVN, etc. In at least some embodiments, the CVNI may be assigned an address from within a private IP address range selected by the client. In contrast to the CVNI, a second VNI programmatically attached to the compute instance, called a local premises access VNI (LVNI) or simply local VNI, may not be assigned an IP address in the IVN. The CPS may assign a media access control (MAC) address to the LVNI, which, in various embodiments, may be used as a source or destination address for OSI data link layer (Layer 2) communications with non-VCS devices in the extended premises.

[0010] The networking manager of the expansion server (which may be implemented, for example, as part of the virtualization management software and / or in a virtualization management offload hardware card of the expansion server) may, in various embodiments, be responsible for determining how to respond to a data link layer frame or message received at the expansion server. In some embodiments, the expansion server may be assigned an IP address from within the VCS substrate network or underlying physical network, similar to how a VCS virtualization server in a provider network datacenter is assigned a substrate address. In response to determining that a data link layer frame received at the expansion server includes a first IP packet having a substrate address as a destination address, the networking manager may utilize the VCS encapsulation protocol to extract a second IP packet from the first IP packet. If the destination address of the extracted IP packet matches an IP address assigned to a CVNI of a compute instance launched on the expansion server (note that there may be multiple such compute instances, and more than one of them may have a CVNI attached), the extracted IP packet may be delivered to that compute instance by the networking manager in various embodiments.

[0011] In contrast, if a received data link layer frame does not include an IP packet with a substrate destination address, the networking manager may attempt to determine whether the destination MAC address of the received frame matches a MAC address assigned to an LVNI of any of the compute instances running on the expansion server, and an encapsulation protocol may not be used. If an LVNI with a matching MAC address is found, the networking manager, in various embodiments, may deliver at least a portion of the contents of the received frame to the compute instance to which the LVNI is attached. The received frame may also be a broadcast frame (with a special destination MAC address), in which case, in at least some embodiments, the networking manager may deliver the contents of the received frame to each of zero or more compute instances (a) running on the expansion server, (b) to which an LVNI is attached, and (c) that are configured to accept L2 broadcast frames. (Note that in some embodiments, a configuration setting may be used to prevent delivery of broadcast frames to one or more compute instances or their attached LVNIs.)

[0012] In some embodiments, LVNIs may be attached to compute instances of an expansion server as part of a workflow executed in response to a launch request for the compute instance. In other embodiments, a separate programmatic request may be submitted by a VCS client to create LVNIs and / or programmatically attach LVNIs to specified compute instances (e.g., prior to launching the compute instances to which they are ultimately programmatically attached). Similarly, in some embodiments, CVNIs may be automatically attached to compute instances as part of a launch workflow, but in at least some embodiments, programmatic requests may also be submitted to create CVNIs and / or attach them.

[0013] In at least one embodiment, the VCS maintains a pool of MAC addresses from which individual MAC addresses are assigned to LVNIs, to which the MAC addresses of decommissioned LVNIs are returned for reuse. In some embodiments, by default, an LVNI may be deleted or decommissioned when the compute instance to which it was attached ceases execution, and an indication that the MAC address of an LVNI is available for reuse may be stored in the VCS control plane when a request to terminate the compute instance is received (or when the compute instance terminates for other reasons, such as an error or failure). Such a MAC address may later be reassigned to another LVNI by the VCS control plane, for example, before programmatically attaching the LVNI to another compute instance. In one embodiment, a VCS client may request that a MAC address that satisfies specified properties (e.g., matches a specified substring indicated by the client) be used for the LVNI to be attached to the client's compute instance. If the VCS can identify a MAC address that satisfies the client's request from among the set of MAC addresses that the VCS is permitted to use, that MAC address may be used. In this way, a VCS client may, at least in some cases, be able to influence the MAC address utilized for the client's LVNI.

[0014] According to at least some embodiments, the VCS may collect a respective set of metrics for each VNI attached to a given compute instance and provide LVNI metrics and CVNI metrics separately (if desired) via a programmatic interface to the client on whose behalf the VNI is being used. Such metrics, in different embodiments, may include, for example, the total number of messages received / sent using each VNI, the total number of bytes received / sent, the distribution of inbound / outbound traffic among different IP addresses and / or MAC addresses, etc. In at least some embodiments, a client may obtain the MAC addresses assigned to the client's LVNI using a programmatic request.

[0015] In various embodiments, data link layer traffic associated with an LVNI may be transmitted over a physical cable (e.g., an Ethernet cable) linking the extended server to an on-premises network switch (a portion of the physical network set up at the extended premises and managed by the client, not the VCS). Other devices configured in the on-premises network may communicate with the compute instance to which the LVNI is attached using OSI L2 frames, and may include, for example, in different embodiments, Internet of Things (IoT) devices, industrial automation devices (e.g., components that implement an automated pipeline of tasks in a manufacturing facility), servers owned / managed by the VCS client on whose behalf the extended server is configured, etc. The VCS client, in various embodiments, may specify security settings for traffic associated with the LVNI (e.g., restrictions on the sources from which frames may be received or the destinations to which frames may be sent, expressed using VLAN (Virtual Local Area Network) identifiers or tags, which may enable VLAN-based segmentation of local premises OSI Layer 2 traffic), and the networking manager may apply such security settings when determining the disposition of data link layer frames received at or transmitted from the extended server.

[0016] In various embodiments, an LVNI (and / or CVNI) can be programmatically detached from a compute instance and reattached to a different compute instance while retaining properties such as a MAC address and / or IP address. The compute instance Cl2 to which an LVNI is attached after being detached from the previously attached compute instance Cl1 can run on the same expansion server or a different expansion server. In at least some embodiments, Cl2 can be configured (e.g., via its CVNI) as part of a different IVN than Cl1. In some embodiments, multiple LVNIs (and / or CVNIs) can be attached to a given compute instance. For example, multiple LVNIs can be attached to a given compute instance to facilitate segmentation of local premises OSI Layer 2 traffic by a network manager of the expansion server on which the compute instance is running; e.g., messages associated with different VLAN identifiers / tags can be handled using each LVNI. After an LVNI with a given MAC address is detached from a compute instance, the networking manager, in various embodiments, may no longer be able to deliver frames with that MAC address designated as a destination to that compute instance. OSI L2 messages sent / received using the MAC address of the LVNI may, in some embodiments, include, among others, messages of Address Resolution Protocol (ARP), Domain Host Configuration Protocol (DHCP), or custom data link layer applications implemented within the extended premises' local network.

[0017] In at least some embodiments, a VCS may be implemented as one of a set of services in a provider network or cloud computing environment. A cloud provider network (sometimes simply referred to as a "cloud") refers to a pool of network-accessible computing resources (such as compute, storage, and networking resources, applications, and services), which may be virtualized or bare metal. A cloud may provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to fluctuating loads. Thus, cloud computing can be viewed as both applications delivered as services over publicly accessible networks (e.g., the Internet or cellular communications networks) and the hardware and software in cloud provider data centers that provide those services.

[0018] A cloud provider network may be formed into several regions, where a region is a distinct geographic area where a cloud provider clusters its data centers. Such regions may also be referred to as provider-network-defined regions because their boundaries may not necessarily coincide with those of countries, states, or other countries. Each region may include two or more availability zones interconnected via a private high-speed network, e.g., a fiber optic connection. An availability zone (also called an availability domain or simply a "zone") refers to an isolated failure domain containing one or more data center facilities with separate power sources, separate networks, and separate cooling from other availability zones. A data center refers to a physical building or enclosure that houses and provides power and cooling to the servers of a cloud provider network. Availability zones within a region are preferably located far enough apart from each other so that the same natural disaster does not take more than one availability zone offline at the same time. Customers may connect to availability zones in a cloud provider network through a publicly accessible network (e.g., the Internet, a cellular communication network) via a transit center (TC). TCs can be considered key backbone locations linking customers to the cloud provider network and may be located on other network provider facilities (e.g., internet service providers, telecommunications providers) and securely connected to availability zones (e.g., via VPN or direct connection). Each region may operate two or more TCs for redundancy. Regions are connected to a global network that connects each region to at least one other region. The cloud provider network may deliver content from points of presence outside of, but networked with, these regions via edge locations and regional edge cache servers (points of presence, or PoPs).This partitioning and geographic distribution of computing hardware enables cloud provider networks to offer customers global, low-latency resource access with a high degree of fault tolerance and stability.

[0019] A cloud provider network may implement various computing resources or services, which may include virtualized compute services (VCS), data processing service(s) (e.g., map-reduce, dataflow, and / or other large-scale data processing techniques), data storage services (e.g., object storage services, block-based storage services, or data warehouse storage services), network function virtualization services or packet processing services, and / or any other type of network-based service (which may include various other types of storage, processing, analytics, communication, event handling, visualization, and security services). Resources (e.g., computational and storage resources) required to support the operation of such services may be requested by users of the cloud provider network and provisioned in accounts associated with the cloud provider, as opposed to resources that may be provisioned in user accounts.

[0020] Various network-accessible services may be implemented in one or more data centers of a provider network in different embodiments. The network-accessible computing services may include elastic compute cloud services (referred to in various implementations as elastic compute services, virtual machine services, computing cloud services, compute engines, or cloud compute services). The services may provide compute instances (also referred to as guest virtual machines, or simply "instances") with varying computational and / or memory resources, which are managed by a compute virtualization service (referred to in various implementations as elastic compute services, virtual machine services, computing cloud services, compute engines, or cloud compute services). In one embodiment, each of the virtual compute instances may correspond to one of several instance types or families. An instance type may be characterized by its hardware type, computational resources (e.g., the number, type, and configuration of virtualized central processing units (VCPUs or VCPU cores)), memory resources (e.g., the capacity, type, and configuration of local memory), storage resources (e.g., the capacity, type, and configuration of locally accessible storage), network resources (e.g., the characteristics of its network interfaces and / or network capabilities), hardware accelerator resources, and / or other suitable descriptive characteristics (such as a “burstable” instance type that has a baseline performance guarantee and the ability to periodically burst above that baseline, or a non-burstable or dedicated instance type that is allocated and guaranteed a fixed amount of resources). Each instance type may have specific ratios of processing, local storage, memory, and network resources, and different instance families may have different types of these resources as well. Within a given instance type, multiple sizes of these resource configurations may be available.The instance type selection function may be used to select an instance type for a customer based (at least in part) on input from the customer, for example. For example, a customer may select an instance type from a set of predefined instance types. As another example, a customer may specify the desired resources of the instance type and / or the requirements of the workload on which the instance will run, and the instance type selection function may select an instance type based on such specifications. A suitable host for the requested instance type may be selected based at least in part on factors such as collected network performance metrics, resource utilization levels on different available hosts, etc. In some embodiments, in response to a programmatic request from a client, instances of several different instance types may be launched (e.g., with LVNIs attached) on the extended premises.

[0021] The provider network's computing services may also include container orchestration and management services (referred to in various embodiments as container services, cloud container services, container engines, or container cloud services). A container represents a logical packaging of a software application that abstracts the application from the computing environment in which it runs. For example, a containerized version of a software application includes the software code and dependencies used by that code, allowing the application to run consistently on any infrastructure that hosts a suitable container engine (e.g., a Docker® or Kubernetes® container engine). Compared to a virtual machine (VM), which emulates an entire computer system, a container virtualizes at the operating system level and thus typically represents a more lightweight package for running an application on a host computing system. An existing software application may be "containerized" by packaging the software application in an appropriate manner and generating other artifacts (e.g., a container image, container file, or other configuration) used to make the application executable on a container engine. In some embodiments, the container engine may run on a virtual machine instance, which is selected based at least in part on the described network performance metrics. In some embodiments, other types of network-accessible services, such as packet processing services, database services, wide area networking (WAN) services, etc., may also be implemented in the cloud provider network.

[0022] Cloud provider network traffic and operations, in various embodiments, can be broadly subdivided into two categories: control plane operations, which occur on a logical control plane, and data plane operations, which occur on a logical data plane. The data plane represents the movement of user data through a distributed computing system, and the control plane represents the movement of control signals through the distributed computing system. The control plane generally includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic generally includes management operations such as system configuration and management (e.g., resource deployment, hardware capacity management, diagnostic monitoring, or system state information). The data plane includes customer resources (e.g., computing instances, containers, block storage volumes, databases, or file storage) implemented on the cloud provider network. Data plane traffic generally includes non-management operations, such as the transfer of customer data to and from customer resources. Some control plane components (e.g., tier 1 control plane components such as a control plane for virtualized computing services) are typically implemented on a set of servers separate from the data plane servers, while other control plane components (e.g., tier 2 control plane components such as analytics services) may share virtualization servers with the data plane, and control plane traffic and data plane traffic may be transmitted over separate / distinct networks.

[0023] FIG. 1 illustrates an example system environment in which local premises access virtual network interfaces used for managed Open Systems Interconnection model Layer 2 communications can be programmatically attached to compute instances launched on an extension server of a virtualized computing service, according to at least some embodiments. As shown, system 100 includes resources and artifacts of a virtualized computing service (VCS) 110 in a provider network 101. VCS 110 can implement one or more programmatic interfaces 188 that can be used by clients to submit programmatic requests from client devices 185 (such as laptops, desktops, mobile computing devices, etc.), including, for example, requests to launch, terminate, or reconfigure compute instances 130. Programmatic interfaces 188 can include, for example, one or more web-based consoles, command line tools, graphical user interfaces, APIs, etc. Requests submitted by VCS clients are directed to a set of VCS control plane servers 111, which can cause corresponding internal requests or commands to be implemented in various other components of the VCS.

[0024] In the embodiment shown in FIG. 1 , various hardware servers may be used to set up compute instances 130 on behalf of VCS clients. An operator of VCS 110 may set up racks containing various categories of standardized servers in one or more data centers of provider network 101, including, for example, virtualization servers (VSs) 125A, 125B, and 125C. Various categories of compute instances (CIs), such as CIs 130A, 130B, 130C, 130D, 130E, or 130C, may be launched on servers in the data center in response to client requests submitted to the control plane. Just as the capabilities of the underlying virtualization servers may differ from one another, the compute instances launched on different sets of servers may also differ in capabilities. For example, CI 130A may be optimized for compute-intensive applications, while CIs 130B and 130C may be optimized for storage-intensive tasks.

[0025] In addition to enabling clients to obtain and use compute instances at provider network data centers, VCS 110, in various embodiments, may also enable clients to request compute instances at premises outside the provider network. For example, in the illustrated embodiment, a logical extension of the VCS may be set up at client premises 121 in response to a request from a client (C2) that owns / manages premises 121. Thus, the client premises may be referred to as a VCS-extended premises in some embodiments. A group of VCS-managed resources, referred to as a VCS-extended resource group (ERG) 122, may be configured at client premises 121. ERG 122 may include one or more VCS extension servers (ESs), such as ES 127 in the illustrated embodiment. A given ES, in the illustrated embodiment, may include zero or more compute instances (e.g., CI 130K) as well as various management components, such as networking manager 155 and ES configuration manager 154, which receives and coordinates responses to commands from the VCS control plane.

[0026] Just as compute instances may be launched on VS 125 in response to client requests received on the VCS control plane, compute instances may, in the illustrated embodiment, be launched on a VCS expansion server via a secure communication channel established between the control plane and the expansion server. Additionally, a VCS control plane server may attach zero or more virtual network interfaces (VNIs) to a compute instance, such as CI 130K. A VNI, in various embodiments, may be represented as a set of metadata entries stored in the VCS control plane that indicate a set of network-related properties (e.g., IP addresses, MAC addresses, security rules for inbound / outbound traffic, etc.) that can be associated / disassociated with a compute instance without modifying the configuration of the physical networking devices (e.g., network interface cards, or NICs) of the server on which the compute instance is running.

[0027] VCS 110 may include physical network 115, also referred to as a substrate network, to which, in the illustrated embodiment, virtualization servers may be connected. Links 111A, 111B, and 111C indicate physical connections between VSs and substrate network 115. In the illustrated embodiment, tunneling-based techniques may be used to associate substrate network addresses with extension servers, such as ES 127, and this association is represented by line 117. Thus, VCS extension servers may effectively be integrated into substrate network 115.

[0028] In the embodiment shown in FIG. 1 , a set of logical networks, such as logical network 116A or 116B, may be configured using substrate network 115 as the physical infrastructure. Examples of logical networks, in at least some embodiments, may include isolated virtual networks (IVNs) established on behalf of VCS clients. For example, in the scenario shown in FIG. 1 , logical network 116A may include the IVN of client C1, while logical network 116B may include the IVN of client C2. In the illustrated embodiment, compute instance 130 may be provided with a network address within a logical network by the VCS control plane server by programmatically attaching a cloud access VNI (CVNI) to which the network address is assigned. Thus, for example, CI 130A of VS 125A may be configured within logical network 116A via CVNI 160A, as indicated by element 112A. CI 130B of VS 125B may be configured within logical network 116B using CVNI 160B, as indicated by line 112D. 1. CI 130C of VS 125B may be configured within logical network 116A using CVNI 160C, as shown by line 112C. CI 130D of VS 125C may be configured within logical network 116B using CVNI 160D, as shown by line 112E. CI 130E may be configured within logical network 116A using CVNI 160E, as shown by line 112B. In addition, the VCS control plane server may also configure CI 130K running on expansion server 127 as part of logical network 116B using CVNI 160F (which is assigned an IP address within logical network 116B), as shown by line 112F in the scenario shown in FIG. 1.To manage and route network traffic directed to and from logical network entities (such as compute instances 130) using the underlying substrate network to which the virtualization servers and extension servers are connected, mapping services and / or encapsulation protocols may be employed in VCS 110, as described in further detail below with respect to FIG. 2.

[0029] Client premises 121, in the illustrated embodiment, may include a local network 192 having a switch 144 or similar networking intermediary device. Various types of client-owned or client-managed devices, such as IoT devices including sensors or smart appliances, servers running client management applications, industrial automation devices, etc., may be configured as part of client premises local network 192. Some such client premises devices 177 may be physically linked to switch 144 via cables such as 118B.

[0030] In at least some embodiments, client C2, whose request involves VCS configuration of expansion server 127, may desire to establish communications at OSI Layer 2 between a number of client premises devices 177 and expansion server CI 130K. To facilitate management of such communications, local premises access virtual network interface (LVNI) 162 may be programmatically attached to CI 130K by the VCS control plane in the illustrated embodiment. A MAC address may be assigned to LVNI 162 by the VCS control plane server, although an IP address within logical network 116B may not be assigned to the LVNI. The MAC address may be used by networking manager 155 to direct inbound traffic received from local network 192, via physical cable 118A in the illustrated embodiment, to CI 130K. The logical network address assigned to CVNI 160F, in contrast, may be used, at least in some embodiments, to communicate with resources within a provider network data center. Note that ERG 122 may include one or more other compute instances (not shown in FIG. 1 ) running on ES 127 or another ES server. Network communications between such extended-premises compute instances may be transmitted, in some cases, using OSI Layer 2 (e.g., utilizing a hardware loopback interface on a networking card attached to the ES) or using IP and other higher-level protocols, depending on factors such as the application for which the communication is generated and whether the source and destination compute instances are running on the same ES. In some cases, the VCS encapsulation protocol may be employed for IP packets transmitted between at least some pairs of compute instances in the ERG.

[0031] When a data link layer frame is received at ES 127, networking manager 155 may be responsible for its disposition in the illustrated embodiment. The networking manager may inspect the frame's contents to determine whether the frame includes an IP packet with the substrate address of ES 127 as its destination address. If the frame includes such an IP packet, the networking manager may use the VCS encapsulation protocol to extract a second IP packet from the first IP packet. If the destination address of the extracted IP packet matches the logical network address (CVNI address) of any CI (e.g., CI 130K) running in the ES, the extracted IP packet may be delivered by the networking manager to that CI (otherwise, the extracted IP packet may be discarded, at least in some embodiments). If the frame did not include an IP packet with a substrate destination IP address, the networking manager may inspect the frame's destination MAC address and, in the illustrated embodiment, deliver it to a CI, such as 130K, whose attached LVNI has the same MAC address. Broadcast data frames (which are special addresses whose destination MAC address does not match the MAC address of an LVNI) may be delivered by the networking manager to one or more CIs of an ES in the illustrated embodiment. Frames that are not broadcast frames, do not have a matching MAC address, and do not contain an IP packet with a substrate destination IP address may be dropped in the illustrated embodiment.

[0032] Figure 2 illustrates an example use of mapping services and encapsulation protocols in a virtualized computing service where a logical network is configured on a substrate network, according to at least some embodiments. The illustrated scenario is presented to illustrate the types of networking-related operations that may need to be performed to enable traffic flow between compute instances set up within such a logical network (either in the VCS's data center or external premises). While IP (Internet Protocol) version 4 addresses are used as examples in Figure 1, addresses formatted according to other protocols, such as IPv6, may also be used in at least some embodiments. Note that the specific IP addresses shown in Figure 2 are chosen as examples, and the techniques described herein are not limited to any particular addresses or address ranges.

[0033] In the illustrated embodiment, two virtualization servers (VSs) and expansion servers (ESs) are shown in VCS datacenter 210. VSs 225A and 225B may each be physically connected to the VCS's substrate network, for example, via one or more Ethernet or similar cables linked to top-of-rack switches configured in the substrate network. ES 226 may be connected to the VCS datacenter via at least some network links not managed / owned by the provider network (e.g., public Internet links and / or private links owned by other network service providers), but may still be assigned a network address in the substrate network using techniques described in detail below. Substrate address 192.168.0.3 is assigned to VS 225A, substrate address 192.168.1.3 is assigned to VS 225C, and substrate address 192.168.1.4 is assigned to ES 226.

[0034] In the illustrated embodiment, compute instances launched on the virtualization server may be assigned network addresses within the isolated virtual network. For example, CIs 230A (of VS 225A), 230D (of VS 225B), and 230G (of ES 226) may all be configured within the same IVN 233A and assigned IVN private addresses 10.0.0.2, 10.0.0.4, and 10.0.0.3, respectively. Similarly, CIs 230B and 230C may be assigned IVN private addresses 10.0.0.2 and 10.0.0.3 within IVN 233B. Note that, as previously indicated, address ranges used within an IVN for private addresses assigned to CIs may overlap with each other. Thus, CIs 230A and 230B have the same private address 10.0.0.2 within separate IVNs (233A and 233B). In some embodiments, intra-IVN addresses may be considered private in that they are not, at least by default, published or made accessible outside the IVN. In at least some embodiments, as described above, private addresses may be assigned to each cloud access virtual network interface (CVNI), which may be programmatically attached or associated with a compute instance. In at least some embodiments, in contrast, at least some of the substrate addresses may be assigned to physical network interface cards, i.e., NICs (or NIC emulators), for example, in a virtualized server.

[0035] To send a network packet originating on one CI to another, in the illustrated embodiment, three types of network information may need to be considered: the source and destination IVN private addresses, the IVN to which the source and destination belong, and the substrate address of the underlying virtualization server. For example, a packet originating on CI 230A and destined for CI 230G may show its source (private) address as 10.0.0.2 and destination address as 10.0.0.3. However, the packet may actually have to be forwarded from substrate network address 192.168.0.3 to substrate network address 192.168.1.4 to reach its intended destination. An encapsulation protocol 244 (used to envelope or encapsulate packets associated with logical network sources / destinations within larger "extended" packets associated with substrate network sources / destinations) and the associated mapping service 245 of the VCS may be used to accomplish this type of forwarding in the illustrated embodiment. The networking virtualization management components of the VCS (including the networking manager running on the expansion servers and the networking manager running on the virtualization management hardware / software stack of the VS225) perform protocol encapsulation and decapsulation operations and may utilize the mapping service 245 to determine the specific substrate addresses to which packets involved in such transfers should be sent.

[0036] In the above example where a packet is sent from CI 230A to CI 230G, mapping service 245 may indicate to the networking manager associated with VS 225A that, for IVN 233A, destination private address 10.0.0.3 corresponds to substrate address 192.168.1.4. The networking manager associated with VS 225A may generate an encapsulated packet containing the original packet, having a substrate source address of 192.168.0.3, a substrate destination address of 192.168.1.4, and identifying IVN 233A as the IVN within which the packet is being forwarded. On the receiving side of external premises 220, the networking manager running in ES 226 may extract (decapsulate) the original packet from the encapsulated packet and provide it to destination CI 230G. In some embodiments, to ensure that a packet is from a trusted / valid source, the networking manager may query a mapping service to perform reverse mapping (e.g., identify the packet's origin) before extracting the original packet. Thus, the mapping service 245 may provide security by preventing the opening of packets that have not been validated. For packets sent in the reverse direction, in at least some embodiments, the networking manager may query a mapping service to obtain the correct substrate address of the destination and perform the necessary encapsulation operations.

[0037] 3 illustrates an example use of an enhanced traffic mediation device to enable control plane traffic to flow securely from a provider network data center to external premises where compute instances managed by a virtualized computing service are launched, according to at least some embodiments. Data plane resources 345 of a VCS 310 located in a provider network data center 301 can be organized into a set of isolated virtual networks (IVNs) similar to those described above. For example, IVN 315A can be set up on behalf of client C1, IVN 315B can be established on behalf of client C2, and so on, with clients given flexibility regarding network configuration settings within each IVN. IVN 315A can include compute instances running on virtualization servers 317A and 317B, and IVN 315B can include compute instances (CIs) (e.g., CI 325A) launched on virtualization servers 317J and 317K. VCS control plane resources 341 may include a set of control plane servers (CPS) 302, such as 302A and 302B in the illustrated embodiment, that are responsible for management tasks such as receiving and responding to requests to set up VCS extended resource groups (ERGs), instance configuration requests on data center virtualization servers or extended servers 360, provisioning additional virtualization servers, and monitoring the health of data plane resources.

[0038] In the embodiment shown in FIG. 3, each expansion server 360 may be set up as part of an expansion resource group that extends VCS data plane functionality at some premises outside of data center 301. For example, expansion server (ES) 360A may be set up as part of ERG 335A at customer premises 332C of client C2, ES 360B may be set up as part of ERG 335B at customer premises 332A on behalf of client C1, and ES 360C may be set up in ERG 335C at a second customer premises 332B of client C2. Each ES may include a respective configuration manager (ECM) (such as ECM 361 of ES 360A) that is responsible for receiving management commands from control plane resources 341 and executing them on the ES. In addition to ES 360, at least some of external premises 332A, 332B, and 332C may also include non-VCS devices (e.g., servers not utilized to host VCS compute instances). For example, in the illustrated embodiment, premises 332A may include non-VCS device 323B, premises 332B may include non-VCS device 323C, and premises 332C may include non-VCS device 323A. The non-VCS devices may communicate with compute instances (such as CIs 325P, 325Q, or 325R) launched on the ES using local premises access VNIs (LVNIs) and OSI Layer 2 messages sent over links in local premises networks 368A, 368B, or 368C. Thus, an expansion server may be configured in multiple networks, including a local on-premises network and one or more VCS networks (e.g., using addresses assigned to the expansion server from the range used for the VCS substrate network). Note that the various capabilities of ES 360 may differ from one another depending on the needs of the clients on whose behalf ES 360 is configured by the VCS control plane.For example, ES360A may have a different or more CPUs than ES360B, ES360C may have more memory or a different CPU architecture than ES360A, etc. Also, note that while only a single ES is shown in each external premises in Figure 3, in at least some embodiments, multiple such ESs may be deployed and used to host VCS compute instances within a single external premises.

[0039] From the perspective of a VCS client, in various embodiments, an ERG may represent a local extension of the VCS capabilities and may be set up in any desired physical location with Internet access and accommodate extension servers acceptable to the VCS and its clients. From the perspective of the VCS itself, an ERG may be considered to be physically located within a customer-selected premises, while virtually located in the same provider network data center as the core VCS infrastructure. In various embodiments, once an ERG is set up in a customer-selected location, the ERG's resources may be managed by control plane components of the VCS located in the provider network data center. Thus, in at least some embodiments, setting up and using an ERG at a given premises may not require local replication of the VCS's control plane capabilities. Instead, a secure network connection may be set up to send control plane commands from the provider network data center to the ERG's PVMD, and the ERG's resources may be devoted primarily to data plane operations.

[0040] In some embodiments, one or more of the compute instances (CIs) launched on the ESs of an ERG may be configured within the IVN 315 (e.g., the CIs may be programmatically attached to a cloud access virtual network interface (CVNI) having an address within the IVN's address range, the CIs may be included in the IVN's database of compute instances, information about the CIs' membership in the IVN may be provided to the VCS's mapping service, etc.). In some cases, CIs running in an ERG may be included in an IVN that also includes CIs in a provider network datacenter, and such an ERG may be said to extend the IVN. For example, ERG 335A extends IVN 315B for client C2 in FIG. 3 , and ERG 335B extends IVN 315A configured for client C1. Some ERGs may include CIs configured in an IVN that does not include any compute instances in a VCS datacenter, or may not even be configured in an IVN.

[0041] Because external premises 332 do not have direct access to the VCS substrate network, additional work may be required from the VCS to configure compute instances at the external premises than is required for compute instances in data center 301. In the embodiment shown in FIG. 3, ES configuration manager (ECM) 361 may perform several functions. For example, the ECM may establish connections with one or more enhanced traffic intermediation devices (ETIs) 377 of the VCS over one or more secure channels (such as virtual private network (VPN) tunnels) 366 (e.g., channels 366A, 366B, or 366C). Secure channels 366 may be required because traffic between the ECM and the VCS control plane (and data plane) must flow over physical network paths that are not controlled or managed by the VCS and, therefore, such physical network paths may be untrusted from the VCS's perspective. The ETI 377 may be implemented on a computing device within the provider network's data center 301 (e.g., using one or more compute instances and / or IVNs set up on behalf of a VCS control plane), and the ETI may be configured to perform one or more security operations regarding traffic between the external premises and the provider network's data center. Control plane commands may be allowed to flow only in one direction, from the provider network's data center to the external premises, in the illustrated embodiment, using the secure channel established with the ETI. In contrast, data plane traffic may flow in either direction between the external premises and the provider network in various embodiments. Additional details regarding how control plane and data plane traffic flows between the provider network's data center and the external premises in various embodiments are provided below.

[0042] The ECM of the ES 360 may also, in at least some embodiments, assign the ES a network address in the VCS substrate network (i.e., a network address selected from the range of addresses assigned to virtualization servers in the VCS substrate network). The substrate address assigned to the ES may also, in at least some embodiments, be assigned to one of the ETIs 377, allowing the ETI to act as a proxy or representative for the ES within the provider network data center.

[0043] Additionally, the ECM, in at least some embodiments, may act as an intermediary for configuration commands related to CIs initiated at the ES 360 in the ERG 335. For example, a client of a VCS may submit a request for a configuration operation for a CI in an ERG to the VCS control plane (using a path other than the secure channel set up with the ETI), and an indication of the request may be provided from the ETI to the ECM of that ERG. In response to this indication, one or more configuration operations for the compute instances may be executed in the CSS. In at least some embodiments, the ECM may be instantiated in response to the detection of one or more trigger conditions in the ES (e.g., detection of power and / or Internet connectivity). The ECM may then automatically initiate (or at least participate in) the automatic establishment of secure network connections with one or more VCS components (e.g., ETI 377) of one or more provider network data centers, without requiring additional configuration guidance from, for example, the VCS client on whose behalf the ES is configured. After the connection is established, in various embodiments, the client may issue commands to instantiate the compute instance in the ES (and / or perform other operations related to the compute instance, such as attaching an LVNI), in a manner similar to how such commands are issued for compute instances that use only provider network resources. From the VCS client's perspective, VCS functionality can now be seamlessly utilized using local resources (and resources located in the provider network datacenter, as needed). Compute instances set up in the ERG, in various embodiments, may communicate with non-VCS devices 323 and other CIs set up in the provider network datacenter or external premises, as needed.At least some CIs set up in an ERG, and associated upstream services that use such CIs as building blocks, may, in some embodiments, continue to function even during periods when connectivity to the provider network datacenter is temporarily interrupted. In particular, for VCS customers who want low-latency access and processing of large amounts of application data stored in their datacenters (e.g., for regulatory compliance, security, or other reasons), the ability to set up VCS CIs that are co-located with the application data may be highly beneficial in various embodiments.

[0044] In the provider network data center 301 in which the VCS 310 is implemented, the control plane resources 341, in the illustrated embodiment, may include one or more extended connection managers (EXCMs) 378 (implemented using some combination of hardware and software in one or more servers of the VCS). The EXCMs may determine that a connection over a secure channel should be established with an ECM, such as ECM 361, for example, in response to a programmatic request received in the control plane. In some embodiments, the ECMs may send a request for a secure connection to the VCS control plane (e.g., over a public Internet path) once the ECMs are online and connected to the Internet through the external premises' local premises network. To enable the secure connection, a set of extended traffic intermediaries (ETIs) 377 may be set up by the EXCMs. The ETIs 377 may include, for example, extended server proxies (ESPs) and tunnel endpoints (referred to as provider network-side tunnel endpoints) in various embodiments. In some embodiments, the ESPs and tunnel endpoints may include respective compute instances of the VCS. In at least one embodiment, the ESP may be set up within one IVN established on behalf of the VCS control plane (different from the IVN set up on behalf of the VCS client), while the tunnel endpoint may be set up within a separate IVN established for the VCS control plane.

[0045] In some embodiments, a virtual network interface (VNI) may be programmatically attached to an ESP by the EXCM. In at least some embodiments, a VNI attached to an ESP may be assigned the same substrate network address assigned to the ES 360 for which the ESP is configured. In at least some embodiments, the ECM may receive an indication of the substrate network address from the EXCM and assign that address to the ES 360. The ECM may function as the external endpoint of a secure (e.g., encrypted) network tunnel established with the provider network-side tunnel endpoint. Any of a variety of tunneling protocols (e.g., standard VPN protocols, customized protocols utilized within VCS, etc.) may be used to transmit packets between the provider network and external premises over a potentially untrusted network, in various embodiments.

[0046] Because the ESP and ES are assigned the same substrate network address, the VCS control plane, in the illustrated embodiment, may use the ESP as the destination for control plane commands (e.g., configuration commands generated in response to client-issued requests) that are actually intended for the ES. Thus, from the perspective of the VCS control plane, the ES may appear as just another virtualization server that can be accessed via the VCS's substrate network. When a control plane command is generated, it is received at the ESP, and a version of the command may be transmitted over channel 366. In some embodiments, the ESP and / or the provider network-side tunnel endpoint may apply one or more security-related transformation operations to the command before transmitting it to the ECM. For example, the version of the command obtained at the proxy from the VCS control plane may, in some embodiments, include one or more security tokens (e.g., tokens that can be used to verify the identity of the requestor of an action performed as a result of the command, and that the requestor has the necessary permissions to request that action), and the ESP may strip or exclude the tokens from the version of the command forwarded to the ECM in the ERG. In at least some embodiments, the original security token may be transformed, and the transformed version of the security token may be included in the forwarded version of the command. In at least one embodiment, a respective message authentication code (e.g., including a hash-based message authentication code, or HMAC) may be generated for each outbound control plane command sent from the ESP to the ES. In various embodiments, the ESP may log all outbound communication messages sent to a given ECM, and the logged messages may be inspected, if necessary, by a client for which the ES is set up.In some embodiments, at least two virtual network interfaces may be associated with a given ESP, one used to receive commands from the VCS control plane and one used to communicate with the tunnel endpoints.

[0047] In some embodiments, data plane traffic may also flow between the VCS and external premises, e.g., via a secure channel or tunnel separate from the channel used for control plane traffic. Data plane traffic may flow in either direction over such a tunnel and may not involve the use of a proxy. In one embodiment, a VNI with the same substrate address as the address assigned to the ES may also be set up in the VCS for data plane traffic to / from the ES.

[0048] 4 illustrates example network configurations for compute instances on an expansion server, according to at least some embodiments. Depending on the particular application that a VCS client wants to run on a given compute instance, in the illustrated embodiment, any of at least three network configurations can be set up for a given CI by the VCS control plane.

[0049] CI 410 represents a first example network configuration for VCS extension server 401, in which a set of applications running on the CI communicate with resources located in a provider network data center and also communicate with resources on other devices in the local premises network. CI 410 has an attached CVNI 415A and an attached LVNI 417A. OSI Layer 2 messages 423A to / from devices in the local premises network use MAC addresses assigned to LVNI 417A as source / destination MAC addresses. Data plane messages 422A to / from the provider network data center use IP addresses assigned to attached CVNI 415A as source / destination IP addresses. Management commands 421A from the VCS control plane (e.g., including commands to start / stop the CI or to change various configuration settings of the CI) can also be delivered to CI 410 using an intermediary, such as a secure channel and an ECM of the type described above.

[0050] In a second type of network configuration, represented by CI 412, a CI running in an ES may not have an attached LVNI but may have an attached CVNI 415B. Management commands 421B from the VCS control plane may still be received at CI 412, and data plane messages 422D using the CVNI's IP address may be exchanged with resources in the provider network's data center, but OSI Layer 2 messages using the CVNI's MAC address cannot be exchanged. In some embodiments, data plane messages between a CI having the configuration of CI 412 and a CI having the configuration of CI 410 may be forwarded through the provider network's data center or through a local networking intermediary device set up at the extended premises. In at least one embodiment, as an optimization, data plane messages between CI 412 and CI 410 may be sent using a hardware loopback mechanism implemented by the ES's networking manager, as indicated by arrow 477.

[0051] CI414 in FIG. 4 is an example of a third type of network configuration. CI415B has CVNI415B and LVNI417B attached to it, but the security parameters of CVNI415B (and / or the IVN to which CVNI415B is assigned) are configured to prohibit data plane traffic to / from the provider network data center. OSI Layer 2 messages 423B may be exchanged with devices in the local premises network using the MAC address assigned to LVNI417B, and management commands 421C from the VCS control plane may be received by CI414. The type of configuration shown for CI414 may be utilized in embodiments where, for example, (a) a CVNI must be attached to a CI based on VCS configuration rules, e.g., to receive control plane commands, (b) a client desires to use the CI for applications requiring OSI L2 connectivity to non-VCS devices on the premises, or (c) a client desires to isolate the CI from VCS data plane traffic. In one embodiment, if a client wants to enable communication at OSI Layer 2 between CI 414 and CI 410 (both CIs have LVNIs attached), a hardware loopback mechanism implemented by the networking manager of the ES may be used, as indicated by arrow 478. In some embodiments, other types of network configurations than those shown in Figure 4 may be employed for the CIs of the expansion server.

[0052] In some embodiments, CIs belonging to separate IVNs may be launched on a given VCS extension server. Figure 5 illustrates an example configuration in which compute instances configured within separate isolated virtual networks of a virtualized computing service on an extension server, according to at least some embodiments, may share a network cable to communicate with devices on the VCS extension premises. VCS extension server 501 includes CIs 510A and 510B and a networking manager 555. CI 510A is used by a VCS client to run application set 561A, and CI 510B is used to run a different application set 561B. A VCS client for which extension server 501 is set up may want to isolate the two application sets while still enabling OSI L2 communication between both application sets and a set of (potentially separate) on-premises devices.

[0053] Thus, in the illustrated embodiment, CI 510A may be configured into IVN 544A by assigning IP address 525A of IVN 544A to CVNI 515A attached to CI 510A, and CI 510B may be configured into a different IVN 544B by assigning IP address 525B of IVN 544B to CVNI 515B attached to CI 510B. To enable OSI L2 connectivity, a shared network cable 570 (e.g., an Ethernet cable) may be used to link the physical network interface of VCS expansion server 501 to the on-premises local network switch 544, and each LVNI 517 (e.g., 517A or 517B) may be attached to two CIs. OSI L2 frames 507A may be exchanged between CI 510A and a local network device via network cable 570, and OSI L2 frames 507B may also be exchanged between CI 510B and a local network device using the same network cable 570, with the networking manager acting as an intermediary (logically similar to a switch) for both sets of frames.

[0054] In the illustrated embodiment, various non-VCS devices may be attached to local network switch 544 via respective network cables, such as 573A, 573B, and 573C, and some or all of these devices may communicate at OSI Layer 2 with one or more CIs running in ES 501. Such devices may include, among others, on-premises DHCP server 575, non-VCS general-purpose server 571, and / or one or more IoT devices 572. Note that in some embodiments, multiple network cables may be attached between the extension servers and devices in the local network (such as network switches). For example, one such cable may be used to communicate with provider network data center resources and the VCS control plane via tunneling over the public Internet, and another cable may be used for OSI Layer 2 traffic with the local network.

[0055] In some embodiments, a VCS client may decide to apply custom traffic management policies to local premises traffic of LVNIs attached to different compute instances on a given expansion server. Figure 6 illustrates an example configuration in which several local premises access virtual network interfaces, each with its own security and performance-related properties, may be configured on an expansion server of a virtualized computing service, according to at least some embodiments. In the example scenario illustrated in Figure 6, a VCS expansion server 601 includes a pair of compute instances CI610A and CI610B. CI610A is used for application set 661A and has CVNI615A and LVNI617A attached to it. CI610B is used for application set 661B and has CVNI615B and two LVNIs, 617B and 617C, attached to it. An IP address 625A within the IVN of the VCS is assigned to CVNI 615A, and an IP address 625B within the IVN (which may be the same IVN as the address assigned to CVNI 615A or a different IVN) is also assigned to CVNI 615B.

[0056] The VCS clients on which CI 610 and LVNI 617 are configured are configured, in the illustrated embodiment, to apply different sets of security and performance-related rules to their respective OSI Layer 2 traffic using the MAC addresses of the LVNIs. Respective security settings 644A, 644B, and 644C may be specified by the client via a VCS control plane programmatic interface for LVNIs 617A, 617B, and 617C, respectively. Additionally, respective traffic rate limits 645A, 645B, and 645C may be specified by the client via a VCS control plane programmatic interface for LVNIs 617A, 617B, and 617C, respectively. The security settings may specify properties such as, for example, whether broadcast frames received by the networking manager should be sent to the compute instance to which the LVNI is attached, whether frames with a specified VLAN tag or identifier should be delivered to the compute instance, etc. In the illustrated embodiment, traffic rate limits may be used to throttle outbound OSI Layer 2 traffic from a given CI (or a given LVNI of a given CI). Recall that in some embodiments, a shared network cable (e.g., to the switch shown in FIG. 5) may be used to connect the expansion server with the rest of the local premises network, and such rate limits may be useful to ensure that the available bandwidth on such a cable is divided according to the VCS client's intent. Security settings and rate limits obtained via a programmatic interface from a client at the VCS control plane server may be sent to an expansion server's expansion configuration manager (ECM) and / or the expansion server's networking manager, and in at least some embodiments, the settings may be applied or enforced by the networking manager. Other LVNI parameters other than those shown in FIG. 6 may also, or instead, be customized by the VCS client in some embodiments.

[0057] FIG. 7 illustrates example programmatic interactions associated with the configuration and use of a local premises access virtual network interface, according to at least some embodiments. Similar in features and functionality to VCS 110 of FIG. 1, VCS 712, in the embodiment illustrated in FIG. 7, may implement a set of programmatic interfaces 777, such as one or more web-based consoles, command line tools, graphical user interfaces, application programming interfaces (APIs), etc. A VCS client 710 may submit a ConfigureExtensionServer request 714 to the VCS control plane via the programmatic interfaces to request the establishment of at least one extension server of the type described above at a specified premises outside the provider network's data center. After the requested server is provisioned and online, an ExtensionServerOnline response message 716 may be sent to the client in the illustrated embodiment. Note that in some cases, it may take some time (e.g., days or weeks) to ship or transmit the necessary server components from the provider network to the external premises, then configure the servers (e.g., in standard server racks), attach the servers to the Internet, and power on the servers, and the ExtensionServerOnline message may, in at least some embodiments, be sent only after all these tasks are completed. As described above, the extension server configuration manager at the external premises may, in at least some embodiments, establish a connection over a secure channel with the extension connection manager in the VCS control plane, located in the provider network's data center, as part of the initial configuration of the extension server.

[0058] In the illustrated embodiment, client 710 may request the creation of one or more LVNIs via CreateLVNIs request 718 and may request the creation of one or more CVNIs via CreateCVNIs request 723. In some embodiments, such LVNIs and / or CVNIs may be created prior to the launch of the compute instances to which they are programmatically attached. In response, metadata indicating properties of the requested LVNI / CVNI (e.g., name, description, IP address if specified / requested by the client, MAC address if specified / requested by the client, security rules, traffic rate limits if applicable, etc.) may be stored in the VCS control plane. LVNI-IDs message 720, in some embodiments, may be sent to the client indicating successful creation of the requested LVNI and providing an identifier for the created LVNI that may be used in subsequent interactions as needed. CVNI-IDs message 725 may be sent to the client after the requested CVNI has been created to indicate that the created LVNI has been created and provide an identifier for the created LVNI that may be used in subsequent interactions as needed.

[0059] In the illustrated embodiment, a compute instance may be launched on an extension server in response to a LaunchCIAtExtensionServer request 727. Parameters of the launch request may govern how VNIs (e.g., one or more CVNIs and / or LVNIs) are programmatically attached to the CI. In some cases, the client may request that the VCS control plane create and attach at least one CVNI within a specified IVN at instance launch time and / or that the VCS create and attach at least one LVNI with a VCS-selected MAC address at instance launch time. In other cases, the client may postpone attaching one or more VNIs until the instance is launched and a CILaunched message 728 is received at the client.

[0060] In the illustrated embodiment, a client may request programmatic attachment of one or more VNIs (e.g., LVNIs or CVNIs, or both) to a CI in an extension server by submitting an AttachVNIToExtensionServerCI request 730 to the VCS. Optionally, the client may specify an identifier for the VNI (e.g., CVNI or LVNI) to be attached, or may request that the VCS control plane create and attach a new VNI. In the illustrated embodiment, after metadata indicating the association of the VNI and the CI is stored in the VCS control plane and propagated to the extension server, a VNIAttached message 732 may be sent to the client.

[0061] In the illustrated embodiment, an LVNI or CVNI may be programmatically detached from a CI by the VCS control plane in response to a DetachVNIFromExtensionServerCI request 734. Such detachment may include removing an indication of the association between the VNI and the CI to which it is attached, while retaining at least some of the VNI's other properties, such as the MAC address and / or IP address. A VNIDetached message 736 may be sent to the client to indicate that the VNI has been disassociated from the CI.

[0062] In the illustrated embodiment, a DescribeVNIs request 741 may be submitted by a client 710 to view the properties of a specified set of VNIs (e.g., LVNIs, CVNs, or both types of VNIs) and / or all VNIs attached to a CI. The properties of the VNI(s) may be presented to the client via one or more VNInfo messages 743. For example, the client may use the DescribeVNI request to obtain the MAC address of an LVNI (before or after attaching the LVNI to a CI) and may use that information to make configuration changes in the on-premises network so that OSI L2 traffic from the on-premises network is exchanged with the CI to which that LVNI is attached.

[0063] As noted above, for example, in the context of Figure 6, a client may wish to customize various properties of an LVNI (and / or CVNI), such as security settings or traffic rate limits. In some embodiments, a ChangeVNISettings request 747 indicating the desired settings may be submitted by the client using a programmatic interface 777. The metadata stored for the VNI may be modified accordingly, and a VNISettingsChanged response message 749 may be sent to the client.

[0064] In different embodiments, a VCS may collect various types of metrics associated with particular CVNIs and LVNIs, such as the rate at which messages are received / sent using addresses assigned to the VNIs, the distribution of message sizes, trends in traffic rates, etc. A client may request metrics for an LVNI and / or CVNI, e.g., per VNI type or per VNI, in the illustrated embodiment, by submitting a GetVNIMetrics request 751. The requested metrics may be provided in one or more MetricsSet response messages 753.

[0065] In some embodiments, an LVNI may be treated (by the VCS control plane) as a special case of a more general VNI construct supported by VCS. In such embodiments, if a client 710 requests the creation or attachment of a VNI without specifying that the VNI be used as an LVNI, the VCS may default to assuming that the VNI be used as a CVNI (e.g., that the VNI be assigned an IVN address). In embodiments in which such a general VNI construct is supported, the VCS client 710 may use an IVN subnet configuration setting to indicate that an IP address (i.e., a CVNI IP address) is configured for a CI within the IP address range of that subnet. The VCS client may submit a SetSubnetAttributesForLVNIs request 755 to indicate that, for a compute instance configured within a specified subnet of a specified IVN, any VNI at a specified integer interface index in the array or list of VNIs attached to that CI is to be used or treated as an LVNI. In response, the VCS may store the LVNI index as an attribute of the subnet and treat any subsequently attached VNIs at that index (e.g., in response to an attach request specifying the index) as an LVNI. A SubnetAttributesSet message 757 may be sent from the VCS to the client. The subnet attribute method for configuring LVNIs may simplify LVNI management for VCS clients because clients may simply use the same types of programmatic requests on LVNIs as they use for other VNIs without using LVNI-specific parameters (as long as the correct interface index is used for the LVNI). Note that programmatic interactions related to the use of LVNIs, CVNIs, and extension servers other than those shown in FIG. 7 may be supported by the VCS in at least some embodiments. Furthermore, FIG. 7 is not intended to depict a required sequence in which programmatic interactions must occur.For example, although the DescribeVNI request is shown below the LaunchCIAtExtensionServer request in FIG. 7, the DescribeVNI request may be submitted before the LaunchCIAtExtensionServer request.

[0066] FIG. 8 is a flow diagram illustrating aspects of operations that may be performed to configure and utilize a local premises access virtual network interface, according to at least some embodiments. As shown in element 801, a VCS extension server (ES), similar in features and functionality to VCS 110 of provider network 101 of FIG. 1, may be configured on a premises outside the provider network's data center in the illustrated embodiment. Configuring the ES may include establishing a connection over a secure channel from the VCS control plane server (CPS) to an ES configuration manager (ECM) running on the ES. The ECM, in various embodiments, may be responsible for launching compute instances (CIs) on the ES in response to requests / commands from the VCS control plane, among other tasks. The ES may also, in various embodiments, include a networking manager responsible for managing the flow of network traffic between CIs launched on the ES and communication endpoints external to the ES, and between CIs on the ES.

[0067] In response to a request from a VCS client for which an ES is being set up, the VCS control plane may activate a CI, CI1, in the ES. In the illustrated embodiment, a local premises access virtual network interface (LVNI) may be programmatically attached to CI1 by the VCS control plane, and a cloud access virtual network interface (CVNI) may also be programmatically attached to the CI (element 804). In at least some embodiments, a control plane server in the VCS may receive a request to attach an LVNI or CVNI to a CI from a client via a programmatic interface, and the control plane server may then send an internal request / command to a configuration management component or networking manager in the ES, which completes the attach operation. The CVNI may be assigned an IP address, IPAddr1, from the address range of an isolated virtual network (IVN) configured in the VCS. The LVNI may be assigned a MAC address, M1, by the VCS control plane, but not from within that IVN or any range of IP addresses managed by the VCS. In at least some embodiments, the networking manager may be provided with configuration information (including, for example, IP addresses and MAC addresses) associated with attached CVNIs and attached LVNIs from the VCS control plane.

[0068] At the ES, the networking manager may utilize the VCS's encapsulation protocol to translate outbound IP packets from CI1 with destination addresses within the VCS (or within other services in the provider network) (element 807). Outbound data link layer frames with destination MAC addresses within the premises' local network (or with destination addresses used for broadcast) may be directed by the networking manager to a network switch or other similar networking intermediary device at the premises via a cable connecting the ES to the intermediary device.

[0069] An inbound data link layer frame DLLF may be obtained (element 810) at the physical networking interface (e.g., a network interface card or equivalent) of the ES and processed by the networking manager. In some embodiments, the DLLF may have associated metadata (e.g., an indication of the source queue into which it is placed by lower driver software or a hardware networking component) that indicates whether the source from which the DLLF was received was a provider network resource or whether it was received from a device local to the premises where the ES is located. In such embodiments, such metadata may first be checked by the networking manager to determine the type of source from which the DLLF was sent, and then the information about the source may be used, along with the contents of the DLLF, to determine how the DLLF should be processed. For example, the VCS encapsulation protocol may be used, as described below, only if the DLLF was received from a provider network resource. If the DLLF includes an IP packet IPP1 having a destination address SDA that is a VCS substrate network address previously assigned to the ES (e.g., during initial configuration of the ES), as determined by operations corresponding to element 813, the networking manager may extract a second IP packet IPP2 encapsulated within IPP1 using the encapsulation protocol of the VCS (element 819). If the destination IP address of IPP2 matches an IP address assigned to the CVNI of any CI running in the ES, such as IP address IPAddr1 assigned to the CVNI of CI1, the networking manager may deliver the contents of the second IP packet IPP2 to that CI. In the unlikely event that no IP address matches the CVNI of a CI in the ES, in the illustrated embodiment, IPP2 may be dropped. In some implementations, metadata of the type described above may not be used.

[0070] In at least some embodiments, if the DLLF does not include an IP packet with SDA as the destination address, the networking manager may attempt to use the DLLF's destination MAC address to identify one or more CIs to which the DLLF content should be delivered. If the DLLF destination MAC address matches a MAC address assigned to any CI running on the ES (such as MAC address M1 assigned to the LVNI of CI1) (as determined in operations corresponding to element 816), the networking manager, in the illustrated embodiment, may deliver at least a subset of the DLLF content to that CI (element 859).

[0071] If an exact match with the LVNI MAC address is not found, the networking manager may determine whether the DLLF is a broadcast frame (element 822). If the DLLF is a broadcast frame, its contents may be distributed to all CIs running on ESs with LVNIs configured to accept broadcast frames (e.g., based on the LVNI's security settings or other parameters), in the illustrated embodiment (element 825). If not, the networking manager could not find a suitable destination CI for the DLLF using the CVNI IVN IP address match or the LVNI MAC address match, and the DLLF may be dropped in various embodiments because the DLLF was not a broadcast frame (element 828). Operations corresponding to elements 810 et seq. may be repeated for at least some inbound data link layer frames by the network manager in the illustrated embodiment.

[0072] It should be noted that in various embodiments, some of the operations shown in the flowchart of Figure 8 may be performed in an order different from that shown in the figure, or may be performed in parallel rather than sequentially. For example, in some embodiments, a MAC address-based check (elements 816 and 859) may be performed before an IP address-based check (elements 813 and 819) to identify a destination CI for an inbound message. Furthermore, some of the operations shown in Figure 8 may not be required in one or more implementations.

[0073] In some embodiments, a networking manager responsible for directing incoming messages to compute instances of the expansion server may be implemented at least in part on an offload card of the expansion server. FIG. 9 illustrates example components of an expansion server, according to at least some embodiments. As illustrated, a VCS expansion server (ES) 902 may include a primary physical processor set 904, main memory (e.g., one or more modules of random access memory, or RAM) 908, a partially offloaded virtualization manager (PVM) 970, and CIs 950, such as CIs 950A, 950B, and 950C, to which LVNIs and / or CVNIs of the type described above may be programmatically attached. In some embodiments, each CI of CI 950 may include a respective virtual machine, referred to as a guest virtual machine (GVM). ES 902 may also include several other components, such as various persistent storage devices, which are not shown in FIG. 9 to avoid clutter. Primary physical processor set 904 may include several physical CPUs (pCPUs), including pCPUs 905A-905C in the illustrated embodiment. A virtualized version of a pCPU, called a vCPU or virtual CPU, may be assigned to an individual CI by PVM 970 during the CI's lifetime. Each CI 950 may include a respective instance of an operating system (e.g., operating systems 952A-952B) and a set of applications (e.g., 954A-954C) running on behalf of clients of the virtualized computing service. One or more of the applications 954 and / or associated operating systems 952 may exchange data link layer messages with devices in the local premises network using the LVNI, as described above.

[0074] In the illustrated embodiment, the PVM 970 may include an opportunistic stripped-down hypervisor 920 (running on a pCPU) and one or more offloaded virtualization manager components (OVMCs) 972 that do not use the pCPUs. The OVMC 972 may include, for example, the virtualization controller 915 and at least a portion of the networking manager 916 in the offload card 910. The networking manager may inspect data link layer frames received at the ES via a cable connected to the network cable port 991 of the offload card 910 and determine the CI to which the frame's contents are to be delivered using the methodology described above (e.g., whether the IVN address of the CVNI is used to select the destination CI or whether the MAC address of the LVNI is used). The other end of the cable may be connected to a local network switch, as described in the context of FIG. 4. In some embodiments, each of the OVMCs may be implemented using a respective system-on-chip design. Although in the illustrated embodiment, both OVMCs 972 are shown as being incorporated within a single offload card 910 (e.g., a Peripheral Component Interconnect Express, or PCIe, card), other approaches to OVMC placement and organization may be employed in different embodiments. For example, in one embodiment, a single system-on-chip implementation may be used to perform the functions of the virtualization controller and networking manager, thereby eliminating the need for two different OVMCs. In another embodiment, respective offload cards may be used for the virtualization controller 915 and networking manager 916. The virtualization controller, as suggested by its name, may be responsible for orchestrating or coordinating many of the virtualization management activities performed in the ES 902 in the illustrated embodiment. For example, it may be the first of the PVM's components to boot and may trigger the startup of other components of the PVM, communicate with the VCS control plane, make memory allocation decisions for the CIs, etc.In the illustrated embodiment, the networking manager 916 may implement one or more networking protocols (including, for example, the encapsulation protocol used within VCS) and may serve as an intermediary between the external networking endpoints of the CI and the ES.

[0075] In the illustrated embodiment, the hypervisor 920 may be described as stripped down because much of the work performed by at least some conventional hypervisors may instead be handled by the OVMC 972, thereby reducing the complexity and size of the hypervisor 920. Additionally, the hypervisor 920 may be shown as opportunistic because in most situations, it may wait until the CI voluntarily relinquishes control of the pCPU 905 before the hypervisor uses CPU cycles. Thus, for example, if a particular CI 950 issues an I / O request (where I / O is expected to take approximately time T1 to complete) and relinquishes the pCPU until a response to the I / O request is received, the hypervisor can take advantage of this opportunity to perform one or more virtualization management tasks (which may typically take time T2, where T2 << T1) using the pCPU while the CI does not anticipate using the pCPU. In this way, in the illustrated embodiment, the impact on the performance of the hypervisor 920's application 954 can be minimal.

[0076] The hypervisor 920 may itself include several subcomponents, including, in the illustrated embodiment, a set of operating system kernel-level components 922, a hypervisor coordinator 925, one or more GVM managers 928, an isolation / security component 929, and / or a messaging manager 931. The hypervisor coordinator 925, each of the GVM managers 928, the isolation / security component 929, and / or the messaging manager 931 may, in at least some embodiments, be implemented as respective user-mode processes. In various embodiments, at least some of these components may be implemented as instances of respective statically linked programs that communicate with each other via pipes using a simple, specialized protocol. The hypervisor subcomponents, in the illustrated embodiment, may remain passive or dormant by default, reacting and activating only in response to events (e.g., messages from other subcomponents, context switches initiated by the CI, etc.). In some implementations, for example, some of the hypervisor subcomponents may typically remain blocked in a polling system call (e.g., epoll() or its equivalent) most of the time.

[0077] Kernel-level components 922 may provide support for various lower-level operations, such as the initial response to a VM exit instruction issued by a CI (e.g., when a CI relinquishes a pCPU). Hypervisor coordinator 925, as its name suggests, may be responsible for coordinating the operations of the other subcomponents. Hypervisor coordinator 925 may, for example, implement an API that may be used to communicate between OVMC 972 and the hypervisor, initiate the launch and termination of CIs (e.g., at the request of OVMC receiving a request from the VCS control plane to launch or terminate a CI), expose metrics collected by the GVM manager, provide debugging capabilities, etc.

[0078] Each GVM manager 928 may be responsible for launching or instantiating each CI based on specifications provided by the coordinator 925, monitoring GVM metrics and logs, and so on. In some embodiments, the GVM manager 928 may also assist with I / O operations requested by a CI to a device, for example, by trapping the CI's I / O requests and converting them into memory-mapped I / O operations that are completed with the help of the OVMC. In at least some embodiments, in accordance with the security-related principle of least privilege, the GVM manager 928 may drop many of its own privileges as early as possible during CI instantiation. For example, after one or more vPCU (virtual CPU) threads are created for a CI and the CI's memory is mapped, the GVM manager may revoke some of its privileges to reduce the opportunity for security breaches. In some embodiments, there may be a one-to-one mapping between GVM managers and CIs, while in other embodiments, a single GVM manager may be responsible for multiple GVMs.

[0079] The messaging manager 931 may act as an intermediary between the virtualization controller 915 and the hypervisor, for example, by translating commands issued by the virtualization controller using a queue-based protocol into pipe messages within the hypervisor. The security and isolation component 929 may be responsible for scrubbing or cleaning up the memory of a CI when the CI terminates, for example, to prevent unintended sharing of data between CIs. In some embodiments, at least a portion of the networking manager functionality may be implemented in the hypervisor. Note that the PVM may include additional components (not shown in FIG. 9) in at least some embodiments, and that in at least one embodiment, one or more of the PVM components shown in FIG. 9 may not be required.

[0080] In at least some embodiments, a server (e.g., including a VCS control plane function and / or an external server function) implementing techniques of the type described herein may include a general-purpose computer system including or configured to access one or more computer-accessible media. FIG. 10 illustrates such a general-purpose computing device 9000. In the illustrated embodiment, the computing device 9000 includes one or more processors 9010 coupled to system memory 9020 (which may include both non-volatile and volatile memory modules) via an input / output (I / O) interface 9030. The computing device 9000 further includes a network interface 9040 coupled to the I / O interface 9030.

[0081] In various embodiments, the computing device 9000 may be a uniprocessor system including one processor 9010, or a multiprocessor system including several processors 9010 (e.g., two, four, eight, or another suitable number). The processor 9010 may be any suitable processor capable of executing instructions. For example, in various embodiments, the processor 9010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as the x86, PowerPC, SPARC, ARM, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each processor 9010 may typically, but not necessarily, implement the same ISA. In some implementations, a graphics processing unit (GPU) and / or a field programmable gate array (FPGA) may be used in place of or in addition to a conventional processor.

[0082] The system memory 9020 may be configured to store instructions and data accessible by the processor(s) 9010. In at least some embodiments, the system memory 9020 may include both volatile and non-volatile portions, while in other embodiments, only volatile memory may be used. In various embodiments, the volatile portion of the system memory 9020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM, or any other type of memory. For the non-volatile portion of the system memory (which may include, for example, one or more NVDIMMs), in some embodiments, flash-based memory devices, including NAND flash devices, may be used. In at least some embodiments, the non-volatile portion of the system memory may include a power source, such as a supercapacitor or other power storage device (e.g., a battery). In various embodiments, any of memristor-based resistive random access memory (ReRAM), 3D NAND technology, ferroelectric RAM, magnetoresistive RAM (MRAM), or various types of phase change memory (PCM) may be used for at least the non-volatile portion of the system memory. In the illustrated embodiment, program instructions and data that implement one or more desired functions, such as those methods, techniques, and data described above, are shown stored in system memory 9020 as code 9025 and data 9026.

[0083] In one embodiment, the I / O interface 9030 may be configured to coordinate I / O traffic between the processor 9010, the system memory 9020, and any peripheral devices within the device, including the network interface 9040 or other peripheral interfaces such as various types of persistent and / or volatile storage devices. In some embodiments, the I / O interface 9030 may perform any necessary protocol, timing, or other data conversions to convert data signals from one component (e.g., the system memory 9020) into a format suitable for use by another component (e.g., the processor 9010). In some embodiments, the I / O interface 9030 may include support for devices attached through various types of peripheral buses (including various types of hardware accelerators and / or offloaders), such as, for example, variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 9030 may be split into two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of the I / O interface 9030, such as an interface to the system memory 9020, may be incorporated directly into the processor 9010.

[0084] The network interface 9040 may be configured to allow data to be exchanged between the computing device 9000 and other devices 9060 attached to one or more networks 9050, such as, for example, other computer systems or devices as illustrated in FIGS. 1-9. In various embodiments, the network interface 9040 may support communication over any suitable wired or wireless general data network, such as, for example, an Ethernet network type. Additionally, the network interface 9040 may support communication over a telecommunications / telephony network, such as an analog voice network or a digital fiber communications network, over a storage area network, such as a Fibre Channel SAN, or over any other suitable type of network and / or protocol.

[0085] In some embodiments, system memory 9020 may represent one embodiment of a computer-accessible medium configured to store at least a subset of the program instructions and data used to implement the methods and apparatus discussed in the context of FIGS. 1 through 9. However, in other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible medium. Generally speaking, computer-accessible medium may include non-transitory storage or memory media, such as magnetic or optical media, e.g., disks or DVDs / CDs, coupled to computing device 9000 via I / O interface 9030. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., that may be included in some embodiments of computing device 9000 as system memory 9020 or another type of memory. In some embodiments, multiple non-transitory computer-readable storage media may collectively store program instructions that, when executed on or across one or more processors, implement at least a subset of the methods and techniques described above. Computer-accessible media may also include transmission media or signals, such as electrical, electromagnetic, or digital signals, transmitted over a communications medium, such as a network and / or wireless link, as may be implemented via network interface 9040. Some or all of multiple computing devices, such as those shown in FIG. 10, may be used to perform the described functionality in various embodiments; for example, software components executing on various different devices and servers may cooperate to provide functionality. In some embodiments, some of the described functionality may be implemented using storage devices, network devices, or special-purpose computer systems in addition to, or instead of, being implemented using a general-purpose computer system.As used herein, the term "computing device" refers to at least all of these types of devices, but is not limited to these types of devices.

[0086] Embodiments of the present disclosure may be described in light of the following clauses.

[0087] Article 1. 1. A system, comprising: a control plane server for a virtualized computing service of a provider network, the control plane server being located in a data center of the provider network; a networking manager of an extension server of the virtualized computing service, the extension server being located on a premises outside the provider network; The control plane server: attaching a cloud access virtual network interface to a compute instance of the virtualized computing service, the compute instance running on the expansion server, and assigning a first Internet Protocol (IP) address to the cloud access virtual network interface from a range of IP addresses of an isolated virtual network configured in the virtualized computing service; attaching a local premises access virtual network interface to the compute instance, the local premises access virtual network interface being assigned a specific Media Access Control (MAC) address by the control plane server, and configured such that the local premises access virtual network interface is not assigned an IP address from the network address range of the isolated virtual network; The networking manager: in response to at least determining that a first data link layer frame received at the enhanced server includes a first IP packet having a destination address within a set of virtual host network addresses of a substrate network of the virtualized computing service; extracting a second IP packet from the first IP packet using an encapsulation protocol of the virtualized computing service; In response to determining that the destination address of the second IP packet matches the first IP address, deliver the second IP packet to the compute instance; In response to determining that a destination MAC address of a second data link layer frame received at the enhanced server matches the particular MAC address, the second data link layer frame does not include an IP packet having the destination address of the first IP packet, and the system is configured to deliver at least a portion of the contents of the second data link layer frame to the compute instance without utilizing the encapsulation protocol.

[0088] Article 2. The control plane server: 10. The system of claim 1, further configured to receive a request to launch the compute instance via a programmatic interface, wherein the local premises access virtual network interface is created and programmatically attached to the compute instance in response to the request to launch the compute instance.

[0089] Article 3. The control plane server: In response to receiving a request to terminate the compute instance via a programmatic interface, store an indication that the particular MAC address is available for reuse; assigning the particular MAC address to a separate local premises access virtual network interface; 10. The system of claim 1, further configured to programmatically attach the other local premises access virtual network interface to another compute instance.

[0090] Article 4. The control plane server: 10. The system of claim 1, further configured to determine at least a portion of the particular MAC address based on input received via a programmatic interface from a client of the virtualized computing service on whose behalf the compute instance is launched.

[0091] Article 5. The control plane server: 10. The system of claim 1, further configured to provide, via a programmatic interface, (a) a first set of metrics related to network traffic associated with the cloud access virtual network interface, and (b) a second set of metrics related to network traffic associated with the local premises access virtual network interface.

[0092] Article 6. 1. A computer-implemented method comprising: Attaching a first local premises access virtual network interface to a first compute instance of the virtualized computing service by a control plane server of the virtualized computing service, wherein the control plane server is located in a data center of a provider network, the first compute instance is running on a specific server located in a premises outside the data center, and the first local premises access virtual network interface is assigned a first media access control (MAC) address by the control plane server; in response to determining, by a networking manager of the particular server, that a first data link layer frame received at the particular server includes a first IP packet having a particular destination address assigned to the particular server; extracting a second IP packet from the first IP packet using an encapsulation protocol of the virtualized computing service; delivering the second IP packet to the first compute instance; and in response to the networking manager determining that a destination MAC address of a second data link layer frame received at the particular server matches the first MAC address, delivering at least a portion of the contents of the second data link layer frame to the first compute instance.

[0093] Article 7. 10. The computer-implemented method of claim 6, further comprising creating, by the control plane server, the first local premises access virtual network interface in response to a programmatic request and prior to launching the first compute instance on the particular server.

[0094] Article 8. The computer-implemented method of any one of clauses 6-7, wherein the second data link layer frame is received at the particular server from one of: (a) an Internet of Things (IoT) device on the premises; or (b) an industrial automation device on the premises.

[0095] Article 9. 9. The computer-implemented method of any one of clauses 6 to 8, wherein the second data link layer frame is received at the particular server via a cable connected to a network switch in the premises' local network.

[0096] Article 10. detaching the first local premises access virtual network interface from the first compute instance and attaching the first local premises access virtual network interface to a second compute instance in response to one or more requests received via a programmatic interface; 10. The computer-implemented method of claim 6, further comprising: after the first local premises access virtual network interface is programmatically attached to the second compute instance, in response to determining that a destination MAC address of a third data link layer frame received at a server on which the second compute instance executes matches the first MAC address, delivering at least a portion of the contents of the third data link layer frame to the second compute instance.

[0097] Article 11. 11. The computer-implemented method of claim 10, wherein the first compute instance is configured within a first isolated virtual network of the virtualized computing service and the second compute instance is configured within a second isolated network of the virtualized computing service.

[0098] Article 12. attaching, by the control plane server, a second local premises access virtual network interface to a second compute instance launched on the particular server of the virtualized computing service, wherein the second local premises access virtual network interface is assigned a second MAC address and the second local premises access virtual network interface is not assigned an Internet Protocol (IP) address by the control plane server; 10. The computer-implemented method of any one of clauses 6-9, further comprising: in response to the networking manager determining that a destination MAC address of a third data link layer frame received at the particular server matches the second MAC address, delivering at least a portion of the contents of the third data link layer frame to the second compute instance.

[0099] Article 13. The computer-implemented method of any one of clauses 6-9, further comprising providing the first MAC address in response to a request received via a programmatic interface.

[0100] Article 14. attaching, by the control plane server, a second local premises access virtual network interface to the first compute instance, wherein the second local premises access virtual network interface is assigned a second MAC address and the second local premises access virtual network interface is not assigned an Internet Protocol (IP) address by the control plane server; 10. The computer-implemented method of any one of clauses 6-9, further comprising: in response to the networking manager determining that a third data link layer frame having the second MAC address as its destination has been received at the particular server, delivering at least a portion of the contents of the third data link layer frame to the first compute instance.

[0101] Article 15. obtaining, via a programmatic interface at the control plane server, an indication of security settings to be applied with respect to network traffic associated with the first local premises access virtual network interface, the security settings including restrictions on sources from which data link layer frames are accepted; 10. The computer-implemented method of any one of clauses 6-9, further comprising: enforcing, by the networking manager, the security settings, including verifying that a source from which the second data link layer frame is received does not violate the restrictions before delivering a portion of the content of the second data link layer frame to the first compute instance.

[0102] Article 16. A non-transitory computer-accessible storage medium that, when executed on a processor, Detecting, by a networking manager in an expansion server of a virtualized computing service, that a first data link layer frame has been acquired at the expansion server; the non-transitory computer-accessible storage medium storing program instructions for, in response to the networking manager determining that a destination media access control (MAC) address of the first data link layer frame matches a specific MAC address of a first local premises access virtual network interface attached to the first compute instance, delivering at least a portion of the contents of the first data link layer frame to a first compute instance running on the expansion server, and not assigning an Internet Protocol (IP) address to the first local premises access virtual network interface from a range of IP addresses managed by the virtualization computing service.

[0103] Article 17. When executed on the processor, 17. The non-transitory computer-accessible storage medium of clause 16, further storing program instructions for throttling a rate at which data link layer frames are transmitted from the first compute instance based at least in part on a rate limit indicated via a programmatic interface.

[0104] Article 18. When executed on the processor, 18. The non-transitory computer-accessible storage medium of any one of clauses 16-17, further storing program instructions for, in response to detecting that a broadcast data link layer frame has been received at the expansion server, the networking manager to distribute at least a portion of the contents of the broadcast data link layer frame to a plurality of compute instances of the expansion server, including the first compute instance and a second compute instance to which a second local premises access virtual network interface is programmatically attached.

[0105] Article 19. When executed on the processor, 19. The non-transitory computer-accessible storage medium of any one of clauses 16 to 18, further storing program instructions for determining, by the networking manager, following detaching the first local premises access virtual network interface from the first compute instance, that contents of a second data link layer frame received at the expansion server should not be delivered to the first compute instance, wherein a destination MAC address of the second data link layer frame matches the specified MAC address.

[0106] Article 20. 20. The non-transitory computer-accessible storage medium of any one of clauses 16-19, wherein the first data link layer frame includes a message formatted in accordance with one of Address Resolution Protocol (ARP), Domain Host Configuration Protocol (DHCP), or a custom data link layer application implemented within a local network at the premises.

[0107] conclusion Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing description of a computer-accessible medium. Generally speaking, a computer-accessible medium may include storage or memory media such as magnetic or optical media, e.g., disks or DVD / CD-ROMs, volatile or non-volatile media such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc., and transmission media or signals such as electrical, electromagnetic, or digital signals transmitted over a communications medium such as a network and / or wireless link.

[0108] The various methods shown in the figures and described herein represent exemplary embodiments of the methods. The methods may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

[0109] Various modifications and changes may be made, as will be apparent to those skilled in the art having the benefit of this disclosure. All such modifications and changes are intended to be included and the above description is therefore to be considered in an illustrative rather than a limiting sense.

Claims

1. 1. A system, comprising: a control plane server for a virtualized computing service of a provider network, the control plane server being located in a data center of the provider network; a networking manager of an extension server of the virtualized computing service, the extension server being located on a premises outside the provider network; The control plane server: attaching a cloud access virtual network interface to a compute instance of the virtualized computing service, the compute instance running on the expansion server, and assigning a first Internet Protocol (IP) address to the cloud access virtual network interface from a range of IP addresses of an isolated virtual network configured in the virtualized computing service; attaching a local premises access virtual network interface to the compute instance, the local premises access virtual network interface being assigned a specific media access control (MAC) address by the control plane server, and configured such that the local premises access virtual network interface is not assigned an IP address from the range of IP addresses of the isolated virtual network; The networking manager: In response to at least determining that a first data link layer frame received at the enhanced server includes a first IP packet having a destination address within a set of virtual host network addresses of a substrate network of the virtualized computing service, extracting a second IP packet from the first IP packet using an encapsulation protocol of the virtualized computing service; In response to determining that the destination address of the second IP packet matches the first IP address, deliver the second IP packet to the compute instance; In response to determining that a destination MAC address of a second data link layer frame received at the enhancement server matches the particular MAC address, the second data link layer frame does not include an IP packet having the destination address of the first IP packet, and the system is configured to deliver at least a portion of the contents of the second data link layer frame to the compute instance without utilizing the encapsulation protocol.

2. The control plane server:

10. The system of claim 1, further configured to receive a request to launch the compute instance via a programmatic interface, wherein the local premises access virtual network interface is created and programmatically attached to the compute instance in response to the request to launch the compute instance.

3. The control plane server: In response to receiving a request to terminate the compute instance via a programmatic interface, store an indication that the particular MAC address is available for reuse; assigning the particular MAC address to another local premises access virtual network interface; The system of claim 1 , further configured to programmatically attach the other local premises access virtual network interface to another compute instance.

4. The control plane server:

10. The system of claim 1, further configured to assign the particular MAC address based on input received via a programmatic interface from a client of the virtualized computing service on whose behalf the compute instance is launched.

5. The control plane server:

10. The system of claim 1, further configured to cause, via a programmatic interface, to provide: (a) a first set of metrics related to network traffic associated with the cloud access virtual network interface; and (b) a second set of metrics related to network traffic associated with the local premises access virtual network interface.

6. 1. A computer-implemented method comprising: Attaching a first local premises access virtual network interface to a first compute instance of the virtualized computing service by a control plane server of the virtualized computing service, wherein the control plane server is located in a data center of a provider network, the first compute instance is running on a specific server located in a premises outside the data center, and the first local premises access virtual network interface is assigned a first media access control (MAC) address by the control plane server; in response to determining, by a networking manager of the particular server, that a first data link layer frame received at the particular server includes a first IP packet having a particular destination address assigned to the particular server; extracting a second IP packet from the first IP packet using an encapsulation protocol of the virtualized computing service; If a destination address of the extracted second IP packet matches a logical network address of the first compute instance running on the particular server, delivering the second IP packet to the first compute instance; and in response to the networking manager determining that a destination MAC address of a second data link layer frame received at the particular server matches the first MAC address, delivering at least a portion of the contents of the second data link layer frame to the first compute instance.

7. 10. The computer-implemented method of claim 6, further comprising creating, by the control plane server, the first local premises access virtual network interface in response to a programmatic request and prior to launching the first compute instance on the particular server.

8. obtaining, via a programmatic interface at the control plane server, an indication of security settings to be applied with respect to network traffic associated with the first local premises access virtual network interface, the security settings including restrictions on sources from which data link layer frames are accepted; 8. The computer-implemented method of claim 6, further comprising: enforcing, by the networking manager, the security settings, the enforcing including verifying that a source from which the second data link layer frame is received does not violate the restrictions before delivering the portion of the content of the second data link layer frame to the first compute instance.

9. detaching the first local premises access virtual network interface from the first compute instance and attaching the first local premises access virtual network interface to a second compute instance in response to one or more requests received via a programmatic interface; 8. The computer-implemented method of claim 6, further comprising: after the first local premises access virtual network interface is programmatically attached to the second compute instance, in response to determining that a destination MAC address of a third data link layer frame received at a server on which the second compute instance executes matches the first MAC address, delivering at least a portion of the contents of the third data link layer frame to the second compute instance.

10. attaching, by the control plane server, a second local premises access virtual network interface to the first compute instance, wherein the second local premises access virtual network interface is assigned a second MAC address and the second local premises access virtual network interface is not assigned an Internet Protocol (IP) address by the control plane server; 8. The computer-implemented method of claim 6, further comprising: in response to determining, by the networking manager, that a third data link layer frame having the second MAC address as its destination has been received at the particular server, delivering at least a portion of the contents of the third data link layer frame to the first compute instance.

11. A non-transitory computer-accessible storage medium that, when executed on a processor, detecting, by a networking manager in an extension server of a virtualized computing service of a provider network, that a first data link layer frame has been acquired at the extension server, the extension server being located on a premises outside the provider network, the virtualized computing service comprising a control plane server located in a data center of the provider network, a first compute instance of the virtualized computing service having a cloud access virtual network interface and a first local premises access virtual network interface attached thereto, the cloud access virtual network interface being assigned a first Internet Protocol (IP) address from a range of IP addresses managed by the virtualized computing service; the non-transitory computer-accessible storage medium storing program instructions for, in response to the networking manager determining that a destination media access control (MAC) address of the first data link layer frame matches a particular MAC address of the first local premises access virtual network interface attached to a first compute instance running on the expansion server, delivering at least a portion of the contents of the first data link layer frame to the first compute instance, and not assigning an Internet Protocol (IP) address to the first local premises access virtual network interface from a range of IP addresses managed by the virtualization computing service.

12. When executed on the processor, 12. The non-transitory computer-accessible storage medium of claim 11, further storing program instructions for throttling a rate at which data link layer frames are transmitted from the first compute instance based at least in part on a rate limit indicated via a programmatic interface.

13. When executed on the processor, 13. The non-transitory computer-accessible storage medium of claim 11, further storing program instructions for, in response to detecting that a broadcast data link layer frame has been received at the expansion server, distributing, by the networking manager, at least a portion of the contents of the broadcast data link layer frame to a plurality of compute instances of the expansion server, including the first compute instance and a second compute instance to which a second local premises access virtual network interface is programmatically attached.

14. When executed on the processor, 13. The non-transitory computer-accessible storage medium of claim 11, further storing program instructions for determining, by the networking manager, following detaching the first local premises access virtual network interface from the first compute instance, that content of a second data link layer frame received at the expansion server should not be delivered to the first compute instance, wherein a destination MAC address of the second data link layer frame matches the specified MAC address.

15. 13. The non-transitory computer-accessible storage medium of claim 11, wherein the first data link layer frame comprises a message formatted according to one of Address Resolution Protocol (ARP), Domain Host Configuration Protocol (DHCP), or a custom data link layer application implemented within a local network at the premises where the external server is located.

Images