Peripheral device for configuring a compute instance on a client-selected server

Abstract

The present disclosure relates to a peripheral device for configuring compute instances on a server selected by a client. The peripheral device includes one or more processors and memory storing program instructions that, when executed, implement a virtualization offload component of a virtualization computing service that includes a storage manager. The offload component establishes a network connection with a control plane of the service. Based on detecting that a hardware server in a separate chassis has linked to the peripheral device, the hardware server is presented as a virtualization host of the service. The offload component initiates compute instance configuration operations at the server in response to commands issued to the control plane, including at least one configuration operation initiated by the storage manager to allow access to a logical storage device from a compute instance.

Description

[0001] This application is a divisional application of invention patent application No. 202080061704.3, filed on September 10, 2020, entitled "Peripheral device for configuring computing instances on a server selected by a client". Technical Field

[0002] This disclosure relates to peripheral devices for configuring compute instances on a server selected by the client. Background Technology

[0003] Many companies and other organizations operate computer networks that interconnect numerous computing systems to support their operations, such as computing systems being co-located (e.g., as part of a local network) or alternatively located in multiple different geographical locations (e.g., connected via one or more private or public intermediary networks). For example, data centers housing large numbers of interconnected computing systems have become commonplace, such as dedicated data centers operated by and on behalf of a single organization, and public data centers operated by entities as businesses to provide computing resources to customers. Some public data center operators provide network access, power, and security installations for a variety of customer-owned hardware, while others provide "full-service" facilities that also include hardware resources available to the operator's customers.

[0004] The emergence of virtualization technology for commodity hardware has already offered benefits in managing large-scale computing resources for many customers with varying needs, allowing multiple customers to efficiently and securely share diverse computing resources. For example, virtualization technology can allow a single physical virtualization host to be shared among multiple users by providing each user with one or more compute instances (e.g., "customer" virtual machines) hosted by a single virtualization host. Each such compute instance can represent a software emulation acting as a different logical computing system, giving users the illusion that they are the sole operator of the given hardware computing resources, while also providing application isolation and security between the various virtual machines. Realizing several different compute instances on the same host can also help increase the overall hardware utilization level at the data center, resulting in a higher return on investment.

[0005] In response to client requests, various compute instance types suitable for different types of applications (such as compute-intensive applications, storage-intensive applications, etc.) can be deployed at data centers within some cloud computing provider networks. The provider network operator can select the underlying virtualization servers used for these compute instances. Furthermore, higher-level services such as database services that utilize virtualized computing services to materialize database instances can be provided to provider network clients, relying on such provider networks' virtualized computing services. Summary of the Invention

[0006] A method for managing virtual machines on remote computing devices is provided, comprising: causing a virtualization management component to be implemented on at least one of one or more computing devices located outside a cloud service provider's underlying network; responding to receiving a request from a client at a client interface of a virtualization computing service on the cloud service provider's underlying network, performing one or more configuration operations to instantiate a virtual machine on at least one of the one or more computing devices located outside the cloud service provider's underlying network; and causing one or more operations to be performed to manage the virtual machine instantiated on the computing device located outside the cloud service provider's underlying network. Attached Figure Description

[0007] Figure 1 The illustration depicts an example system environment according to at least some embodiments, in which peripheral devices connected to the underlying physical network of the virtualized computing service can be used to enable computing instances to be configured on a server selected by the client.

[0008] Figure 2 The illustration shows an example of using a mapping service and encapsulation protocol at a virtualized computing service, according to at least some embodiments, where a logical network is configured on the underlying network.

[0009] Figure 3 The illustration shows example elements of a peripheral virtualization management device, according to at least some embodiments, that can be used to enable computing instances to be set up at a server selected by the client.

[0010] Figure 4 The illustration shows an example virtualization management software component that can be executed at a peripheral virtualization management device according to at least some embodiments.

[0011] Figure 5 The illustration shows example differences between a virtualization server using a host-based hypervisor according to at least some embodiments and a hardware server selected by a client that can configure a computing instance using a peripheral virtualization management device.

[0012] Figure 6The illustration shows an example use of a peripheral virtualization management device at a provider network data center and a cooperative positioning facility, according to at least some embodiments.

[0013] Figure 7 The illustration shows example program interactions related to the configuration of a computing instance using a peripheral virtualization management device, according to at least some embodiments.

[0014] Figure 8 This is a flowchart illustrating aspects of operations that can be performed at a provider network, according to at least some embodiments, to configure a compute instance using a server selected by a client connected to a peripheral virtualization management device.

[0015] Figure 9 An exemplary system environment according to at least some embodiments is shown, in which the virtualization computing services of a provider network can be extended at a location outside the provider network using peripheral devices.

[0016] Figure 10 An example extended traffic intermediary device according to at least some embodiments is shown, which can be configured to enable control plane traffic to flow securely from the provider network’s data center to an external location where a peripheral virtualization management device is deployed.

[0017] Figure 11 The illustration shows an example communication channel established between extension managers running on various peripheral virtualization management devices, according to at least some embodiments.

[0018] Figure 12 The illustration depicts an example geographically distributed logical network that can be configured using multiple peripheral virtualization management devices located at appropriate locations outside the provider network, according to at least some embodiments.

[0019] Figure 13 An example of using additional provider network services at an extended resource group of a virtualized computing service, according to at least some embodiments, is shown.

[0020] Figure 14 Exemplary programming interactions involving the configuration of extended resource groups for virtualized computing services using peripheral devices are illustrated according to at least some embodiments.

[0021] Figure 15 This is a flowchart illustrating aspects of operations that can be performed according to at least some embodiments to establish an extended resource group using a peripheral virtualization management device.

[0022] Figure 16 This is a flowchart illustrating aspects of operations that, according to at least some embodiments, can be performed to enable the establishment of a multi-site logical network in an environment where conflicting network addresses may have already been assigned at multiple sites.

[0023] Figure 17 This is a block diagram illustrating an example computing device that can be used in at least some embodiments.

[0024] Although embodiments have been described herein by way of example with respect to several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed description thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the scope defined by the appended claims. The headings used herein are for organizational purposes only and are not intended to limit the scope of this specification or claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning “possibly”) rather than a mandatory sense (i.e., meaning “must”). Similarly, the words “include, including, and includes” mean including but not limited to. When used in the claims, the term “or” is used inclusively rather than exclusively. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, and any combination thereof. Detailed Implementation

[0025] This disclosure relates to methods and apparatus for configuring compute instances of a virtualized computing service (e.g., a cloud provider network) on a client-selected hardware server using external peripheral devices connected to the server (as opposed to the default practice of setting up compute instances on hardware servers selected by the VCS operator). The peripheral devices may reside in a separate chassis or enclosure from the client-selected hardware server and may be ordered and shipped independently of the client-selected hardware server. Clients of a virtualized computing service (VCS) may wish to configure compute instances on such servers (rather than only on servers selected by the VCS operator) for various reasons, such as because the server contains components tailored to the client's specific applications (e.g., high-end processors / cores, large storage devices, or custom network components), because software licenses are associated with the client-selected server and cannot be easily transferred to servers owned by the VCS operator, etc. However, such clients typically must choose from hardware maintained by the VCS operator. For ease of maintenance, VCS operators often include a large number of servers of the same type in their fleets, and therefore clients requiring dedicated hardware may not be able to use VCS compute instances or other services on their desired hardware. A VCS can be one of multiple network-accessible services (e.g., including storage services, database services, etc.) implemented within a cloud provider's network or cloud computing environment. The client-selected hardware server (also referred to as the client-selected hardware server or target hardware server) can be located in a data center within such a provider network, in a co-location facility (e.g., a location such as a building or room where computing-related resources of more than one organization can be hosted), and / or in a client-owned location. Therefore, the disclosed technology advantageously enables VCS services and resources to be used in a wider range of hardware types (including dedicated hardware that would otherwise not be incorporated into a VCS fleet) and / or in non-traditional locations.

[0026] In at least some embodiments, peripheral devices, which can be provided by virtualized computing service operators, may each include various hardware components (e.g., processors / cores, memory, storage devices, circuitry for power management, security management, etc.) and software that collectively implements network and storage virtualization management. They provide access to storage volumes via block-device interfaces and consolidate compute instances into isolated virtual networks (IVNs) or other logical networks set up for clients on the VCS. Peripheral devices can be encapsulated in small chassis that can be mounted on top of an industry-standard server rack or in an empty slot, and can be connected to the hardware server and network using industry-standard components and connectors. In fact, as long as the hardware server chosen by the client conforms to widely used peripheral connectivity industry standards (e.g., PCIe or Universal Serial Bus (USB)), the functionality included in the peripheral device allows compute instances launched on the hardware server to enjoy all the advantages (e.g., manageability, security, connectivity to other network-accessible services, etc.) that can be provided to compute instances set up in a server cluster chosen by the VCS operator. The compute instances set up on the operator-selected servers of the VCS can be called default cluster (DF) compute instances, while the compute instances set up on the client-selected servers of the VCS can be called client cluster (CF) compute instances.

[0027] In various embodiments, a VCS may include a physical network referred to as the underlying network, to which hardware servers for DF compute instances and various other devices (e.g., network middleware, including routers, switches, gateways, etc.) can be connected. Utilizing the underlying network as the underlying infrastructure, logical networks can be configured in these embodiments to represent various VCS clients. For example, a set of compute instances (including virtual machines, bare-metal instances allowing non-virtualized access to at least some hardware components of the underlying servers, etc.) may be configured for client C1 within a logical network called an isolated virtual network IVN1, while another set of compute instances may be configured for a different client C2 within another isolated virtual network IVN2. An IVN may include a set of networking resources (containing compute instances) assigned or allocated to a given client, which are logically isolated from (and inaccessible by default from) resources allocated to other clients in other isolated virtual networks. Clients establishing an IVN on their behalf can be granted substantial flexibility in network configuration of resources used within the IVN—for example, a dedicated IP address for a compute instance can be chosen by the client without considering the possibility that other resources within the IVN may have been assigned the same IP address; client-selected subnets can be established within the IVN; security rules can be set by the client for incoming and outgoing traffic relative to the IVN; and so on. Furthermore, in at least some embodiments, custom network endpoints can be configured within the IVN to enable compute instances within the IVN to communicate with network-accessible services of the provider network (e.g., storage services, database services, machine learning services, etc.), using dedicated network paths of the provider network without traversing or using links or devices on the public internet. In various embodiments, the network address assigned to a compute instance within the IVN may differ from the underlying network address assigned to the hardware server running the compute instance. In various embodiments, encapsulation protocols and associated mapping services can be used to transport the flow of network traffic within and across the IVN via links and servers of the underlying network (e.g., from one compute instance to another, between client devices outside the VCS and compute instances, or between compute instances and other compute network services). In various embodiments, the VCS may also include a set of management or data plane components responsible for tasks such as supplying hardware, monitoring other resources, and receiving and processing instance configuration commands from clients.

[0028] In at least some embodiments, the peripheral device used to configure compute instances at a server selected by the client may be referred to as a Peripheral Virtualization Management Device (PVMD). In various embodiments, the PVMD may be configured to execute different combinations of software / firmware, thereby supporting different combinations of virtualization-related functions depending on the specific needs of the client, and the compute instance will be set up on behalf of that client on various client-selected hardware servers. For example, in some cases, a client may want to launch their compute instances at a location (e.g., a data center implementing a VCS provider network, or certain types of cooperative location facilities) via a direct physical connection to the underlying network of the VCS. In such locations, the PVMD can be physically connected to the underlying network via one or more cables (e.g., Ethernet cables), without requiring traffic between the PVMD and the underlying network to flow through any network links not managed / controlled by the VCS operator. When used in these types of locations, the PVMD may be said to operate in “Direct Underlying Connection” (DSC) mode, and a first set of virtualization-related software can be executed at the PVMD. In various embodiments, configuration operations related to the compute instance launched at the target hardware server connected to the PVMD in DSC mode can be performed by the PVMD, for example, in response to commands submitted by the client to the VCS control plane. In such an embodiment, the software running at the PVMD can implement the VCS encapsulation protocol, which is used to transmit traffic to and from the compute instance over the underlying network.

[0029] In other scenarios, clients may wish to leverage PVMD to set up compute instances on client-selected hardware servers located in locations where direct physical access to the underlying VCS network is not possible, such as client-owned data centers, client-organized office facilities, etc. While some colocation facilities can provide direct access to the underlying network, others may not in at least some embodiments, and the latter type of colocation facility can also be used on client-selected hardware servers running PVMD-enabled compute instances. In practice, in this case, the client may wish to logically extend the VCS beyond the provider network's data center. When used for this purpose, PVMD may operate in "Extended Venue" (EP) mode, and in at least some embodiments, software including a VCS Extension Manager (EM) may run. Note that when used in EP mode, PVMD can also run all (or at least most) of the software that runs in DSC mode. Using the described techniques, in at least some embodiments, clients of virtualized compute services may be able to leverage selected hardware devices located outside the provider network to host compute instances with the same characteristics and capabilities as at least some compute instances that could be set up in the provider network's data center.

[0030] As those skilled in the art will understand from this disclosure, certain embodiments may enable various advantages, including some or all of the following: (a) enabling various virtualized computing applications to be implemented in a hardware-agnostic and location-independent manner, for example, using client-selected hardware servers, while still retaining the operational advantages that may be implemented through the provider network (e.g., manageability / monitoring, connectivity with other provider network services, etc.); (b) reducing the amount of application data and results that must be transmitted over long distances, such as links between customer data centers and provider network data centers; (c) improving the overall latency and responsiveness of applications by moving applications closer to the data source / destination to use potentially large amounts of application data as input or as output; and / or (d) enhancing the security of sensitive application data. In various embodiments, the guiding principles and objectives for the design and implementation of a PVMD for enabling compute instances on a hardware server of client choice may include, among others: (a) using industry-standard connectivity technologies and protocols; (b) utilizing PVMD enclosures with small form factors, minimizing space, cooling, and other physical requirements; (c) ensuring customer data security by limiting and thoroughly logging network traffic and providing physical mechanisms (e.g., physical keys) to ensure that data stored on the PVMD's storage devices is not stolen; (d) providing first-level support for various provider network services (equivalent to support that may utilize resources located in the provider network's data center); (e) protecting the provider network's own data center from potential hostile actors (e.g., operating at the location where the PVMD is installed); and / or (f) supporting continuous service fulfillment on servers connected to the PVMD, even in cases of imperfect network connectivity.

[0031] According to some embodiments, the system may include one or more computing devices in the control plane of the provider network's VCS, a target hardware server (THS) indicated by a client of the provider network, and peripheral devices (e.g., PVMDs) located in a physical chassis outside the target hardware server. The term "target" can be applied to the hardware server selected by the client, as it is the resource targeted by the operation initiated by the PVMD. The PVMD may include one or more processors and memory, etc., storing program instructions. In some embodiments, the processor and / or memory may be incorporated into one or more SOCs (System-on-a-Chip). When executed at one or more processors of the PVMD or across one or more processors of the PVMD, the program instructions of the PVMD may implement one or more virtualization management offloading components, which in some embodiments include one or more storage managers and one or more networking managers. In some embodiments, the PVMD may establish a connection to the underlying network of the VCS via a first cable, or detect that such a connection has been established; therefore, in such embodiments, the PVMD may operate in DSC (Direct Underlying Connection) mode. In at least some embodiments, encapsulation protocols implemented at multiple devices attached to the underlying network may be used to implement connections between one or more logical networks of the VCS (e.g., similar to those IVNs described previously). Upon detecting that (a) the THS peripheral card has been connected to the external peripheral port of the peripheral device via a second cable, and (b) the THS system management service processor has been connected to the peripheral device via a third cable, in at least some embodiments, the peripheral device can enable the VCS control plane to present the THS to the client as a client-selectable virtualization server to run one or more compute instances. Therefore, in such embodiments, the THS can be linked to the PVMD via a peripheral card (e.g., a PCIe card) suitable for the THS slot and via the THS system management service processor (e.g., a baseboard management controller or BMC). In various embodiments, the THS can be linked to the underlying network solely via the peripheral device; in practice, the peripheral device, which can be designed and supplied by the VCS operator, can act as a gatekeeper or security intermediary between the THS (selected by the client) and the underlying network. As part of the operations performed to enable the THS to be presented as a virtualization server, in at least some embodiments, a network address of the VCS underlying network (i.e., a network address selected from the address range allocated to virtualization hosts linked to the underlying network) can be assigned to the THS. The terms “virtualized server” and “virtualized host” are used synonymously in this document to refer to a hardware server on which one or more computing instances of a VCS can be executed or run.

[0032] In at least some embodiments, the VCS can provide its clients with the ability to configure compute instances in either a multi-tenant mode (where a given hardware server can be used to launch and run compute instances for several different clients) or a single-tenant mode (where a given hardware server can be used to launch and run compute instances for no more than one client). One of the single-tenant options may include assigning a virtualization server designated as a "dedicated virtualization host" to a client, enabling the client to specify / identify the virtualization server as a resource for one or more compute instances without sharing the host with other clients, and allowing the client to use server-based software licenses (e.g., per hardware slot, per kernel, or per virtual machine license). In some embodiments, when the PVMD is connected to a client-selected THS as described above, the PVMD can cause the VCS control plane to present the THS as such a dedicated virtualization host. For example, information about the newly connected THS can be transferred from the PVMD to the VCS control plane (e.g., via the VCS underlying network), and the VCS control plane can add the THS to its database of dedicated virtualization hosts, assign an underlying network address to the THS, and include the THS in a list of dedicated virtualization hosts provided to clients using the PVMD on its behalf.

[0033] In various embodiments, the PVMD can initiate one or more configuration operations on a compute instance at the THS on behalf of a client, including, for example, starting the compute instance, changing network or other configuration settings, terminating the instance, etc. In at least one embodiment, a bare-metal compute instance can be instantiated at the THS on behalf of a client via the PVMD, thereby enabling non-virtualized access to at least some of the hardware devices / components of the THS. In some embodiments, multiple compute instances can be configured at the THS via the PVMD. In one embodiment, a single PVMD can be used to configure corresponding compute instances on servers selected by more than one client connected to the PVMD. In various embodiments, compute instances of the THS can be configured within an isolated virtual network of the provider network, at least in part, based on operations performed using one or more network managers running at the PVMD. For example, such network managers can store indications of network addresses assigned to compute instances configured at the THS (within the range of private network addresses of the isolated virtual network established at the virtualized compute service), and / or can assign such addresses to virtual network interfaces programmatically attached to such compute instances. In one embodiment, the compute instance of the THS provides access to the root volume (and / or other logical storage devices, file systems, etc.) based at least in part on operations performed by one or more storage managers running at the PVMD. For example, in some embodiments, the storage manager may use block storage services and / or other logical storage devices, file systems, etc., from a provider network to set up, modify, or otherwise configure the root volume. In some embodiments, the PVMD may include one or more persistent storage devices (e.g., devices accessible via an NVME (Non-Volatile Memory Fast) interface) on which the contents of the root volume and / or other storage objects accessed from the compute instance of the THS may be stored.

[0034] In some embodiments, the PVMD can be connected to the THS via one or more PCIe connectors. In other embodiments, connectors configured to multiplex PCIe signals with other types of signals (e.g., connectors multiplexing PCIe signals with DisplayPort signals) can be used, and / or a USB (Universal Serial Bus) connector can be used. In one embodiment, a signal repeater card or signal retiming card (e.g., a PCIe retiming card or repeater card) can be inserted into a slot in the THS and linked to the PVMD via a cable. In other embodiments, such a signal repeater card or retiming card may not be required.

[0035] According to at least one embodiment, the PVMD's networking manager may include a network interface card (NIC) emulator and / or an IVN connection manager. In some embodiments, encapsulation / decapsulation operations of the VCS's encapsulation protocol may be implemented at the networking manager, for example, for packets directed from a client cluster (CF) compute instance within a specific IVN to a default cluster (DF) compute instance running on the same or a different IVN. In at least one embodiment, the PVMD's networking manager may be configured to log various types of network traffic to and / or from the CF compute instance, including, for example, domain name service traffic to internal or external DNS servers of the provider network, and to provide such logs to clients configuring the CF compute instance on its behalf via a programming interface. The VCS may implement a variety of programming interfaces (e.g., a web-based console, command-line tools, a graphical user interface, an application programming interface (API), etc.) to enable clients to submit requests related to PVMD-enabled compute instances in various embodiments and receive corresponding responses. For example, a client may submit a programming request to enable the THS to host one or more compute instances via the PVMD, and the PVMD may connect to the substrate and the THS (e.g., within a provider network data center or colocation facility) to respond to such requests.

[0036] In one embodiment, the PVMD may include a persistent storage device and an associated removable cryptographic storage security device (e.g., a physical key, accessible from the front of the PVMD housing), which can be physically disconnected / detached from the PVMD. In such an embodiment, removing the cryptographic storage security device may render the contents of the persistent storage device unreadable / unwritable; that is, the security device may need to be physically present to allow reading or writing of the contents of the persistent storage device.

[0037] According to at least one embodiment, the PVMD may include one or more small form factor pluggable (SFP) ports. Such ports can be used to establish connections to the underlying VCS network and / or other networks. In one embodiment, the PVMD may be mounted in a 1-U (one-unit) slot / tray of a server rack (e.g., a server rack that also houses the THS).

[0038] In some embodiments, the PVMD can be used in Extended Premises (EP) mode, i.e., in a customer-owned premises or other third-party premises where direct connection between the PVMD and the VCS substrate is not possible (e.g., via one or more cables), such as in a provider network data center or some co-location facility where the PVMD can operate in DSC (Direct Underlying Connection) mode. According to one such embodiment, the system may include one or more computing devices of a provider network's VCS (located within one or more data centers of the provider network), a target hardware server (THS) located at a premises outside the one or more data centers, and a PVMD located in a physical enclosure separate from the THS. The PVMD may include program instructions that implement an Extended Manager (EM) of the VCS when executed at one or more processors of the PVMD or across one or more processors of the PVMD. In some embodiments, the EM may be configured to establish connections with one or more Extended Traffic Intermediaries (ETIs) of the VCS via one or more secure channels. The ETI may be implemented at one or more computing devices within the provider network data center. In various embodiments, the ETI may be configured to perform secure operations on traffic between the external premises where the PVMD is located and one or more data centers of the provider network.

[0039] After determining that the THS's peripheral card is linked to the PVMD (e.g., via the PVMD's external peripheral port), in some embodiments, the EM can assign a specific private network address of the VCS's underlying network to the THS, causing the THS to appear as a virtualization host of the VCS. In at least some embodiments, the specific private network address assigned to the THS can also be assigned to one of the ETIs, thereby enabling the ETI to act as a substitute for the THS within the VCS's underlying network (from a networking perspective). In this configuration of assigning public addresses to the THS and ETI, messages to be sent from within the substrate to the THS (e.g., originating from the default cluster compute instance or from the VCS control plane) may initially be sent to the ETI. The message can then be transmitted securely to the EM and from the EM to the THS (e.g., to the compute instance started at the THS, or to the control plane agent running at the THS). In response to an indication received at the EM from the ETI that a client has submitted a programming request to the VCS, the EM can cause one or more configuration operations of the compute instance to be performed at the THS (e.g., including starting / stopping the instance, setting networking attributes, etc.). In at least one embodiment, programming requests from the client may have been submitted using a path that does not include one or more secure channels established between PVMD and ETI. In such embodiments, the secure channels may be configured as unidirectional paths for commands from the VCS control plane to the PVMD / THS combination and may not be used to submit commands to the VCS control plane. However, in at least some embodiments, data plane message traffic may be allowed in both directions between PVMD and the VCS data center.

[0040] From the perspective of the VCS client that configures and uses PVMD, THS can support the same type of compute instance-related functionality regardless of whether the PVMD and THS combination operates in a location remote from the VCS data center or, in at least some embodiments, in a provider network data center. In fact, when PVMD is configured to run EM, it can extend the VCS data plane to external locations. For example, compute instances launched under THS can be configured within the VCS's IVN; that is, network addresses from the client's IVN address range can be assigned to the compute instances. PVMD can also act as a bridge between the VCS and other networks (e.g., networks established by the client in external locations), performing encapsulation, address translation, and other operations to enable packets to flow into and out of the VCS's compute instances and devices configured in other networks.

[0041] In some embodiments, individual compute instances running on several THSs at a location outside the provider network can be configured as part of the same logical network—for example, as part of the same IVN, or even as part of the same subnet within the IVN. In this case, secure communication channels or tunnels can be established between the individual PVMDs connected to the THS located at the location, so that network packets can flow securely between instances using the physical network links and devices available at the location.

[0042] According to some embodiments, as described above, the provider network of the VCS can implement one or more other services, such as database services or object storage services, which can be accessed from at least some compute instances of the VCS using credentials assigned to compute instances via the VCS's Instance Metadata Service (IMDS). Such an IMDS can also provide additional metadata elements for the compute instance, including a unique identifier assigned to the compute instance by the VCS, an identifier for the machine image of the compute instance (if any), block device mapping information for the instance, and so on. In some embodiments, the metadata can be accessed from the compute instance via a local link HTTP (Hypertext Transfer Protocol) address that can only be accessed from within the instance itself. In at least one embodiment, an IMDS proxy can run on the PVMD, and this metadata (including credentials that can be used to access other provider network services from the compute instance) can be provided by the proxy.

[0043] In some embodiments, as previously described, a dedicated service endpoint (PSE) can be set up within the IVN, for example, to enable network traffic to flow between compute instances within the IVN and other publicly accessible provider network services without using the public internet. In at least one such embodiment, the client can define various types of policies and associate them with such a PSE—for example, a policy could instruct that only instances CI1, CI2, and CI3 of the IVN are allowed to use endpoint PSE1 to access a specific storage object SO1 in storage service SS1. In at least some embodiments, compute instances set up at a client-owned facility's THS can also utilize such a PSE and associated policies. For example, different policies can be defined and implemented for granting access permissions to compute instances running at a client-owned facility, relative to granting access permissions to compute instances established within a provider network representing the same client.

[0044] In at least one embodiment, multiple THSs with PVMDs can be set up in their respective external locations, and logical networks (e.g., IVNs) can be configured across different locations. Therefore, in such an embodiment, compute instances CI1 (running at THS1 connected to the PVMD at external location EP1), CI2 (running at THS2 connected to the PVMD at a second external location EP2), and CI3 (running at THS2 connected to the PVMD at a second external location EP2) can all be configured within the same IVN. If desired, in some embodiments, CI1, CI2, and CI3 can be located within corresponding subnets of the IVN.

[0045] Note that, from a hardware perspective, in at least some embodiments, the PVMD used in an external location (where direct connection to the VCS underlying network is not feasible) may be the same as the PVMD used in a provider network data center or certain types of cooperative location facilities (where the PVMD is physically connected to the underlying network). From the perspective of the characteristics and functionality of compute instance configuration and use, an Extended Location (EP) mode PVMD (i.e., with an extension manager) can provide at least some additional functionality beyond that supported by Direct Underlying Connection (DSC) mode. In at least some embodiments, such additional functionality may include the ability to communicate between a compute instance (set up with the help of the PVMD) and a non-VCS device within the network set up in an external location, with the help of an extension manager. In at least some embodiments, if a particular category of VCS compute instances (e.g., bare metal compute instances) can be set up and managed using a PVMD within a provider network data center, it is also possible to configure the same category of compute instances with the help of a PVMD outside the data center.

[0046] Example system environment with direct underlying connections

[0047] Figure 1The illustration depicts an example system environment according to at least some embodiments, where peripheral devices connected to the underlying physical network of a virtualized computing service can be employed to enable computing instances to be configured on servers of client choice. As shown, system 100 includes resources and artifacts of a Virtualized Computing Service (VCS) 110 of a provider network 101. In some embodiments, a network established by an entity such as a company or public sector organization to provide one or more network-accessible services (such as various types of cloud-based computing, storage, or analytics services) accessible to a distributed set of clients via the Internet and / or other networks may be referred to as a provider network (or cloud provider network). A provider network may sometimes be referred to as a “public cloud” environment (sometimes simply “cloud”), which refers to a large pool of network-accessible computing resources (such as compute, storage, and network resources, applications, and services), which may be virtualized or bare metal. The cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically configured and published according to client commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads.

[0048] In some cases, a provider network's resources are distributed across multiple data centers, which in turn may be distributed across numerous geographical regions (e.g., each region corresponds to one or more cities, states, or countries). For example, a cloud provider network can be structured as multiple regions, where a region is a geographical area of ​​a cluster of cloud provider data centers. Each region may contain two or more availability zones interconnected via a dedicated high-speed network (e.g., fiber optic communication connection). An availability zone is an isolated fault domain containing one or more data center facilities that have separate power supplies, separate networks, and separate cooling compared to facilities in another availability zone. Preferably, availability zones within a region are far enough apart that the same natural disaster should not simultaneously take down more than one availability zone. Customers can connect to the availability zones of the cloud provider network via a publicly accessible network (e.g., the Internet, cellular communication networks).

[0049] VCS 110 can implement one or more programming interfaces 188, which clients can use to submit programming requests from client devices 185 (e.g., laptops, desktops, mobile computing devices, etc.), including requests to start, terminate, or reconfigure compute instances 130, such as virtual machines, bare metal instances, etc. Bare metal instances can access at least some non-virtualized hardware components of the virtualization server running the bare metal instance. Programmable interfaces 188 can include, for example, one or more network-based consoles, command-line tools, graphical user interfaces, APIs, etc. Requests submitted by VCS clients can be directed to a set of control plane devices 111 of the VCS, which in turn can cause corresponding internal requests or commands to be implemented at various other components of the VCS. The VCS control plane can primarily handle management or configuration tasks, while the data plane (which may include compute instances 130 running on virtualization servers 125 selected by the VCS and hardware servers 175 or 176 selected by the client) can primarily be used for client applications and data.

[0050] exist Figure 1 In the illustrated embodiment, various hardware servers can be used to set up compute instances 130 on behalf of VCS clients. For example, the operator of VCS 110 can set up racks containing various types of standardized servers in one or more data centers, such as Type A server group 120 and Type B server group 121 selected by the VCS. The virtualization servers selected by the VCS can differ from each other in terms of computing power, memory, or storage capacity—therefore, for example, Type A virtualization server 125A (e.g., VS 125A or 125B) may have more or faster CPUs / cores than Type B virtualization server 135 (e.g., 135A or 135B), while Type B virtualization server 135 may have more local storage than Type B virtualization server. In response to client requests submitted to the control plane, various types of compute instances (CIs), such as CI 130A, 130B, 130C, 140A, 140B, or 140C, can be launched on the servers selected by the VCS. Just as the capabilities of the underlying virtualization servers may differ from group 120 to group 121 selected by the VCS, the capabilities of compute instances launched on servers in different groups may also differ—for example, CI 130 may be optimized for compute-intensive applications, while CI 140 may be optimized for storage-intensive tasks. In some embodiments, the VCS may default to using only a single type of virtualization server, without using type A servers, type B servers, etc.

[0051] VCS 110 may include a physical network 115 to which the virtualization server can connect in the depicted embodiments. Using the underlying network 115 as the physical infrastructure, a set of logical networks, such as 116A or 116B, can be configured. Instances of logical networks may include isolated virtual networks (IVNs) established on behalf of VCS clients in at least some embodiments—for example, in… Figure 1 In the illustrated scenario, logical network 116A may include the IVN of client C1, while logical network 116B may include the IVN of client C2. In at least some embodiments, compute instances 130 and 140 may be assigned network addresses of the logical networks. Therefore, for example, CI 130A of virtualization server (VS) 125A may be configured within logical network 116A, while CI 140B of VS 135B may be configured within logical network 116B. To manage and route network traffic to or from entities (e.g., compute instances 130 or 140) within the logical networks to which the virtualization hosts are connected, mapping services and / or encapsulation protocols may be employed at VCS 110, as described below. Figure 2 To be discussed in more detail.

[0052] Some clients of VCS 110 may wish to set up and use compute instances on hardware servers instead of those configured by default at the VCS in the depicted embodiments. VCS 110 can allow at least some client-selected hardware servers (e.g., hardware server 175 selected by client C1 or hardware server 176 selected by client C2) to be used as virtualization hosts, provided that the hardware servers selected by the client are connected to the VCS underlying network 115 via a VCS provider peripheral virtualization management appliance (PVMD) 160 (e.g., PVMD 160A or 160B in the illustrated embodiments). PVMD 160 can be implemented in a different chassis or enclosure than the client-selected hardware servers physically connected in the depicted embodiments. In at least some embodiments, PVMD 160 can be provided by the VCS in response to a program request submitted via programming interface 188. In some embodiments, the PVMD may incorporate hardware, firmware, and / or software components that perform the networking and storage virtualization tasks required to establish compute instances (e.g., instances 185 or 195) at a client-selected hardware server (e.g., server 175 or 176), while also acting as a secure intermediary between the server and the underlying network 115. The client-selected hardware server 175 may differ from the server selected by the VCS in various dimensions—for example, some client-selected hardware servers may have larger compute or storage capacities than the servers selected by the VCS, including specialized networking or storage hardware not available on the servers selected by the VCS, or simply have server-specific software licenses required to run client applications that cannot be transferred to the servers selected by the VCS.

[0053] According to at least some embodiments, the PVMD may include program instructions that, when executed at the PVMD's processor, implement one or more network managers and one or more storage managers. In one embodiment, a given PVMD 160 may, for example, establish connections with: (a) the VCS underlying network 115, (b) a peripheral card of a client-selected target hardware server (e.g., 175 or 176) which will host at least one VCS compute instance, and (c) a system management service processor (e.g., a baseboard management controller or BMC) of the client-selected server. In one example implementation, a connection to the underlying network may be established using an Ethernet cable (e.g., inserted into a small form factor (SFP) port of the PVMD), a connection to the retiming card of the target hardware server may be established using PCIe or USB, and a connection to the system management service processor may be established using an RJ45 connector. Other types of ports, cables, and / or connectors may be used in other embodiments. After PVMD detects that these three types of connections have been successfully established, in at least some embodiments, PVMD can cause the VCS control plane device 111 to present the client-selected hardware server 175 or 176 as a virtualized host selectable by the client to run compute instances. In some embodiments, the client-selected hardware server can be presented as a dedicated virtualized host on which only a single client's compute instance can be set up. In one embodiment, a given VCS client C1 can allow other VCS clients to set up compute instances on the server selected by C1 that is connected to PVMD.

[0054] After the hardware server selected by the client is specified and presented as a virtualization host, various types of compute instance configuration operations (e.g., starting or terminating a compute instance or changing its configuration settings) can be performed on the server in the depicted embodiments, for example, in response to requests submitted by the client through interface 188. For example, compute instances can be configured within a logical network (e.g., an IVN) using operations performed by a network manager running on PVMD. Figure 1In the illustrated scenario, compute instance 185 (e.g., a bare-metal instance) can be launched at hardware server 175 selected by the client and assigned a network address within logical network 116A, just as compute instance 130A of virtualization server 125A selected by the VCS is assigned a network address within logical network 116A. Similarly, compute instance 195 can be launched at hardware server 176 selected by the client and assigned a network address within logical network 116B, just as compute instances 130B of virtualization server 125B and 140B of virtualization server 135B selected by the VCS are assigned corresponding network addresses within logical network 116B. A storage manager running at the PVMD can configure and provide access to the root volume and / or other logical storage devices, file systems, etc., of compute instances 185 or 195. In at least one embodiment, such root volumes and / or other logical storage devices can be configured using persistent storage devices (e.g., NVME devices) incorporated within the PVMD enclosure. In one embodiment, the PVMD's networking manager may include, for example, a NIC emulator, an IVN connection manager, etc. In some embodiments, the PVMD may also include a physical security device, such as a physical key, which renders the PVMD's storage device unusable if the physical key is removed from the PVMD.

[0055] In some embodiments, the hardware server selected by the client and the corresponding PVMD 160 can be located within the data center of the provider network 110, for example, within the same standardized server rack, such as a 19-inch wide rack. In one embodiment, the PVMD 160 can occupy a 1-U slot within such a rack, while the hardware server can occupy one or more other slots or locations within the rack. In one embodiment, the hardware server selected by the client and its PVMD can be located within a cooperative positioning facility with direct access to the underlying VCS network 115.

[0056] In at least some embodiments, various types of functionality supporting compute instances 130 or 140, such as the ability to log network traffic flows (including DNS requests and responses), perform intrusion detection, and other types of security-related operations, may be transparently supported for compute instances 185 or 195. In some embodiments, instance metadata may be generated at PVMD 160, including, for example, credentials that enable compute instances 185 or 195 to issue authorized service requests to other services of provider network 110, and provided to the compute instances. Metrics (e.g., including resource utilization levels, network packet counts / drops, etc.) may be collected for compute instances launched on a server selected by the client and provided to the client, just as they are for CIS 130 and 140, etc.

[0057] Example mapping and encapsulation of network traffic in logical networks

[0058] As previously mentioned, in various embodiments, the VCS may include a logical network configured using an underlying physical or underlying network, and the PVMD may connect to such an underlying network to enable network communication of compute instances set up at a server selected by the client. Figure 2 The illustration depicts an example of using mapping services and encapsulation protocols at a virtualized compute service, according to at least some embodiments, where a logical network is configured on the underlying network. The described scenario is intended to illustrate various network-related operations that may need to be performed to enable traffic flow between compute instances configured in such a logical network; such operations are not limited to environments using PVMD. IP (Internet Protocol) version 4 addresses are... Figure 1 The examples used herein are for illustrative purposes only, although in at least some embodiments addresses formatted according to other protocols such as IPv6 may be used. Note that Figure 2 The specific IP address shown is selected as an example, and the PVMD-based techniques described in this article are not limited to any specific address or address range.

[0059] In the depicted embodiment, three server racks 220A, 220B, and 220C are shown, each rack including one or more virtualization servers used by VCS 210. Rack 220A includes virtualization servers (VS) 225A and VS 225B, rack 220B includes VS 225C, and rack 220C includes a client-selected virtualization server 226 connected to PVMD 260. PVMD 260 can provide a virtualization server similar to... Figure 1 The PVMD 160's functionality and features are discussed in the context of [previous discussion]. The VS255A, 255B, and 255C can each connect to the underlying network of the VCS, for example, via one or more Ethernet or similar cables connected to a top-of-rack switch configured within the underlying network. The PVMD 260 can also connect directly to the underlying network, for example, using Ethernet or similar cables. The customer-selectable VS 226 can connect to the PVMD, for example, via a PCIe or USB connector, but may not connect directly to the underlying network itself; that is, the PVMD 260 can act as an intermediary between the underlying network and the customer-selectable VS 226.

[0060] In the depicted embodiments, each of the virtualization servers 225A, 225B, 225C, and 226 can be assigned an underlying network address, such as 192.168.0.3 for VS 225A, 192.168.0.4 for VS 225B, 192.168.1.3 for VS 225C, and 192.168.1.4 for the virtualization server 226 selected by the client. In some embodiments, the baseboard address of server 226 can be obtained by PVMD 260, for example, from the VCS control plane.

[0061] In the depicted embodiments, network addresses within isolated virtual networks can be assigned to compute instances launched at the virtualization server. For example, CI 230A (in VS 225A), 230F (in VS 225C), and 230G (in VS 226) can all be configured within the same IVN 233A and assigned the corresponding IVN private addresses 10.0.0.2, 10.0.0.4, and 10.0.0.3, respectively. Similarly, in the illustrated example, CI 230B and 230E can be assigned IVN private addresses 10.0.0.2 and 10.0.0.3 within IVN 233B, and CI 230C and 230D can be assigned IVN private addresses 10.0.0.4 and 10.0.0.6 within IVN 233C. Note that, as previously mentioned, the address ranges used for assigning proprietary addresses to CIs within an IVN may overlap—therefore, CIs 230A and 230B have the same proprietary address 10.0.0.2 within different IVNs (233A and 233B). In some embodiments, addresses within an IVN may be considered proprietary, at least by default, because they are not publicly advertised or accessed outside the IVN. In at least some embodiments, proprietary addresses may be assigned to virtual network interfaces, and virtual network interfaces may be programmatically attached to or associated with compute instances. Conversely, in at least some embodiments, at least some underlying addresses may be assigned to physical network interface cards or NICs (or NIC emulators), such as at a virtualization server or PVMD 260.

[0062] To transmit network packets originating from one CI to another, three types of network information may need to be considered in the illustrated embodiment: the source and destination IVN proprietary addresses, the IVNs to which the source and destination belong, and the underlying virtualization server's underlying address. For example, a packet originating from CI 230A and destined for CI 230G may indicate its source (proprietary) address as 10.0.0.2 and its destination address as 10.0.0.3. However, the packet may actually need to be transmitted from the underlying network address 192.168.0.3 to the underlying network address 192.168.1.4 to reach its intended destination. In the illustrated embodiment, encapsulation protocol 244 (for encapsulating or encapsulating packets associated with the logical network source / destination within a larger "enhanced" packet associated with the underlying network source / destination) and VCS's related mapping service 245 can be used to implement this type of transmission. The VCS networking virtualization management components (including the networking manager running on PVMD 260 and the networking manager running in the virtualization management hardware / software stack of VS 225) can perform protocol encapsulation and decapsulation operations, and use mapping service 245 to determine the specific underlying address to which packets contained in such transmissions should be sent.

[0063] In the example described above, where data packets are sent from CI 230A to CI 230G, mapping service 245 may instruct the network manager associated with VS 225A that for IVN 233A, the destination proprietary address 10.0.0.3 corresponds to the underlying address 192.168.1.4. The network manager associated with VS 225A may generate a packet containing the original data packet, with the underlying source address 192.168.0.3, the underlying destination address 192.168.1.4, and identify IVN 233A as the IVN in which the data packet is being transmitted. At the receiving end, the network manager operating at PVMD 260 may extract (decapsulate) the original data packet from the packet and provide it to the destination CI 230G. In some embodiments, to ensure that the data packet originates from a trusted / valid source, PVMD may consult the mapping service to perform a reverse mapping (e.g., to identify the origin of the data packet) before extracting the original data packet. Mapping service 245 can thus provide security by preventing the opening of unauthenticated data packets. For packets transmitted in reverse, PVMD's network manager can consult the mapping service to obtain the correct base address of the destination and perform the necessary encapsulation operations.

[0064] Example components of peripheral virtualization management devices

[0065] Figure 3The illustration shows example elements of a peripheral virtualization management device, according to at least some embodiments, that can be used to enable computing instances to be set up at a server selected by a client. As shown, in the illustrated embodiment, it is similar in function and features to... Figure 1 The peripheral virtualization management device (PVMD) 360 of the illustrated PVMD 160 may include one or more virtualization offloading cards 320, a local persistent storage device 304 and an internal PCIe card 366, a security key device 365 linked to a physical key 395, a power distribution board 302, and several ports including one or more external PCIe ports 355, one or more small form factor ports (SFP) 344, and an RJ45 (registered jack 45) port 345. In the depicted embodiment, the PVMD 360 may be encapsulated in a separate physical chassis 333 or mounted outside a target hardware server 380 on which one or more compute instances will be configured. Note that Figure 3 The aim is to provide a conceptual overview of PVMD components; Figure 3 The relative positions and sizes of the components shown may not correspond to at least some implementations.

[0066] Virtualization offloading card 320 may include one or more processors / cores 370 and one or more memories 328. The term "virtualization offloading" can be used to describe card 320 because, in the depicted embodiments, much of the work required to configure and manage compute instances running on target hardware server 380 (e.g., a server selected by the VCS client) can be offloaded to PVMD 360, allowing a larger portion of the server's compute and other resources to be used for the compute instances and the client applications running on them. Figure 3 In the illustrated embodiments, the code and data 322 of multiple virtualization management component programs (e.g., software and / or firmware) can be stored in memory 328 and run using processor / core 370. In at least some embodiments, each virtualization management component can be executed using a corresponding subset of available kernels / processors—for example, one kernel could be used for an embedded operating system, another for a network interface card emulator, and so on. At least a portion of the code residing in memory 328 can be used to manage various aspects of networking and storage of the compute instances launched at the target hardware server 380, and can therefore be referred to as a combination of a networking manager and a storage manager. Note that, in at least some embodiments, at least a portion of the code and / or data 322 can be dynamically updated or modified, for example, after one or more compute instances are launched at the target hardware server using the code and data.

[0067] In at least some embodiments, the PVMD can be physically linked to the target hardware server in at least two ways using cables 355A and 355B. Cable 355A can link a PCIe retimer card 385 installed in the target hardware server 380 to an external PCIe port 355, which in turn enables connectivity between the target hardware server and other components of the PVMD 360, including local persistent storage device 304 and virtualization offload card 320, via an internal PCIe card 366. The retimer card 385 can be used to improve PCIe signal integrity, thereby enhancing system performance and reliability across the potentially long cable 355A. In other embodiments, a repeater card can be used instead of or attached to the retimer card. In one embodiment, neither a repeater card nor a retimer card is required. Cable 355B can be used to connect the RJ45 circuitry 332 of the PVMD 360 to the system management service processor 386 (e.g., a baseboard management controller or BMC) of the target hardware server 380 via an RJ45 port 345. In some embodiments, a system management service processor 386, which can be connected to the motherboard of the target hardware server, can be responsible for tasks such as monitoring the physical status of the target hardware server using sensors, providing the results of such monitoring, and restarting / restarting the target hardware server when necessary. Cable 355C can be used to link the PVMD 360 to one or more networks 323 (e.g., the VCS underlying network in a scenario where the PVMD is located in a provider network data center providing direct access to the underlying network, and / or an external network set up at the client location). In at least some embodiments, small form factor circuitry 335 linked to one or more SFP ports 344 can be used to access network 323.

[0068] In some embodiments, one or more types of local persistent storage devices 304 may be incorporated into the PVMD 360, such as NVME (Non-Volatile Memory Fast) devices 306, other (non-NVME) solid-state drives (SSDs) 308 accessible from the SATA (Serial Advanced Technology Attached) circuitry of the virtualization offloading card 320, and so on. In at least some embodiments, storage manager code running at the virtualization offloading card 320 may use the local persistent storage device 304 to configure root volumes and / or other logical storage devices for compute instances instantiated at the target hardware server 380. In some embodiments, the storage manager code may locally implement a block-level device interface (effectively implementing a subset of the functionality of block storage services). In other embodiments, the storage manager may access block storage services (and / or other network-accessible storage services) of a provider network to configure at least some storage devices.

[0069] exist Figure 3In the illustrated embodiment, PVMD 360 includes a physical key 355 (e.g., accessible from the front of the PVMD housing and physically detachable). A physical key 395, which can be attached to a cryptographic security key device 365 located within PVMD 360, can be used to provide physical security for data stored at the PVMD, including data stored within local persistent storage device 304. In at least some embodiments, removing or detaching the physical key 395 from the PVMD can render the contents of the persistent storage device unreadable and / or unwritable. Power distribution board 302 can be connected to a standard power supply (e.g., a server rack power supply) in the facility where the PVMD and the target hardware server reside. Note that in some embodiments, a power supply can be used... Figure 3 Alternatives to specific components shown are provided—for example, USB could be used instead of PCIe, or attached to PCIe; multiplexed connectors / cables could be used between the target hardware server and the PVMD, etc. In some embodiments, the PVMD may include components not in... Figure 3 Additional components, such as batteries and additional security modules, are shown. In at least one embodiment, the housing 333 may occupy a 1-U slot within a standard server rack.

[0070] Figure 4 The illustration depicts example virtualization management software components that can execute at a peripheral virtualization management device according to at least some embodiments. In the depicted embodiments, a set of virtualization management software components 420 running on the kernel / processor of the PVMD may include an embedded operating system 425 (which can coordinate the operation of various hardware components of the PVMD itself), one or more network interface card (NIC) emulators 429, and one or more emulators 429 for legacy devices.

[0071] The block-device storage manager 437 running at the PVMD can, for example, use the PVMD's local persistent storage to configure a root volume for a compute instance running on a hardware server selected by the client. In some embodiments, the NVME device emulator 439 can be used to manage access to NVME-based persistent storage. The IVN data plane connection manager 441 can, for example, implement encapsulation protocol operations (e.g., encapsulating outbound packets or decapsulating inbound packets) for traffic between the compute instance on the target hardware server and other endpoints. Such other endpoints may include, for example, other compute instances within the provider's network data center, other compute instances on hardware servers selected by different clients, services other than the VCS, etc. Calls similar to... Figure 2The VPC mapping service of mapping service 245 can be initiated from IVN data plane connectivity manager 441. In some embodiments, IVN data plane connectivity manager can initiate or implement configuration operations to assign network addresses within IVN to one or more virtual network interfaces that are programmatically attached to a compute instance running on a target hardware server, thus including the compute instance in the network address range specified by the client for IVN.

[0072] In some embodiments, the instance metadata service agent 449 may provide various elements of metadata in response to queries issued from a compute instance launched at a target hardware server. For example, such metadata may include credentials for authorizing / verifying requests sent from the compute instance to other provider network services, block device mapping information of the compute instance, identifiers of the compute instance, and so on. In some embodiments, a local link HTTP address accessible only from within the instance itself may be used to obtain metadata at the compute instance.

[0073] In at least some embodiments, one or more agents 453 of the VCS control plane may run at the PVMD. For example, such agents may be responsible for receiving commands generated at the VCS control plane and initiating operations (e.g., configuration change operations) at the PVMD and / or target hardware server in response to such commands.

[0074] In some embodiments where the PVMD is installed in a facility that cannot directly access the underlying VCS network, such as a client-owned location, the PVMD may include a VCS Extension Manager (EM) 457. In at least one embodiment, in a scenario where the PVMD can directly access the underlying VCS network, the EM may not need to be installed, instantiated, or activated at the PVMD. The Extension Manager 457 may establish a secure communication channel (e.g., a Virtual Private Network or VPN tunnel) with an extension traffic intermediary located in the provider network data center, allowing VCS commands and data to securely flow through a non-underlying network not managed or controlled by the VCS. Based on commands received through such a channel, compute instances can be set up and configured at the target hardware server, effectively extending the VCS to a location outside the provider network, as described in further detail below. In some embodiments, the Extension Manager may include a control plane agent in an environment where direct access to the underlying network is not possible; that is, if the Extension Manager runs on the PVMD, a separate VCS control plane agent may not be required in such embodiments.

[0075] In at least one embodiment, one or more network security agents 461 may run at the PVMD. Such network security agents may be responsible for a variety of operations, such as causing the generation of logs for various traffic (including DNS requests and responses, and standard IP packets) directed to or originating from computing instances connected to the target hardware server at the PVMD, initiating or performing intrusion detection or penetration detection operations, etc. Note that in some embodiments, different combinations of software and / or firmware components may run at the PVMD, rather than through... Figure 4 The example shown in the text.

[0076] In some virtualized computing services, the virtualization management software stack, including the hypervisor, may run on the virtualization server itself. Therefore, a significant portion of server resources may sometimes have to be dedicated to virtualization management tasks rather than client workloads. Furthermore, in such cases, the hardware server running the compute instance may be selected by the VCS, as the virtualization management software stack may have to be customized for the specific hardware used by the server; consequently, VCS clients may not have much flexibility regarding the specific hardware used for their compute instances.

[0077] Figure 5 The illustration shows example differences between a virtualization server using a host-based hypervisor according to at least some embodiments and a hardware server selected by a client that can configure compute instances using peripheral virtualization management devices. The host-based hypervisor virtualization server 510 may include server hardware 530, such as a CPU, memory, persistent storage devices, a NIC, etc. A virtualization management software stack 525, including a hypervisor, a VCS networking and storage virtualization manager, a management instance of an operating system, etc., may act as an intermediary between the hardware 530 and one or more compute instances 515, such as guest virtual machines on the server 510.

[0078] In contrast, in scenarios where a PVMD 565 is used as an intermediary to configure compute instances, all logic for virtualization management and connectivity to the VCS (including logic for containing compute instances in an isolated virtual network, logic for setting up storage devices such as the root volume of a compute instance, etc.) can be executed within the PVMD itself, as indicated by arrow 590. Standard peripheral connectors or cables (e.g., PCIe or USB cables) can be used to physically link the PVMD to its hardware 531, which is selected by the client, to the virtualization server 550. The client can choose any server hardware of their choice, as long as the PVMD is connectable to the server in such an embodiment. For example, the client can choose a CPU that implements the desired processor architecture, optimizes for the client's application, or the client can decide to use a server that requires a software license tied to the server. In some cases, the client may want to use a server that can be configured with custom networking configurations that may not be supported on the default VCS-selected server, such as multicast configurations. Note that while the PVMD 560 can act as an intermediary between the client-selected virtualization server and the underlying VCS network (and the VCS logical network built on top of the underlying network), this does not preclude the client-selected virtualization server 550 from being configured in other networks selected or set by the client in at least one embodiment. The PVMD 565 can therefore provide VCS clients with greater flexibility in server selection than could be offered using a server with a conventional virtualization management software stack running on the server's main CPU.

[0079] Example use of PVMD in provider network data centers and co-location facilities

[0080] Figure 6 The illustration depicts an example use of a peripheral virtualization management device at a provider network data center and a cooperative positioning facility, according to at least some embodiments. In a given data center 602 of the provider network, a VCS is implemented within this network-accessible services suite, and multiple server racks can be configured in the depicted embodiments. Some default server racks, such as 612A or 612B, can be used for virtualized servers that do not use PVMD. Instead, for example, each individual rack in these racks may include a top-of-rack switch 610 (e.g., switch 610A or 610B) configured within or linked to the VCS underlying network 615. Thus, individual default servers located within the default server racks can be connected to the underlying network via their respective top-of-rack switches.

[0081] In response to one or more programming requests from a VCS client, in some embodiments, a client-requested server rack 622A can be established within data center 602. Such a client-requested server rack 622A can house a client-selected hardware server 667A connected to a PVMD 665A, which is directly physically linked (i.e., without traversing network links not controlled by the VCS operator) to the underlying VCS network 615. Using virtualization management hardware and software integrated within the PVMD, one or more compute instances can be configured at the client-selected hardware server 667A. In some embodiments, a single PVMD can be used to configure compute instances on multiple client-selected servers; in other embodiments, a one-to-one relationship may exist between client-selected servers and PVMDs. In one embodiment, multiple client-selected servers, such as 667A and / or multiple PVMDs 665A, can be housed within a single rack. In some embodiments, a separate client-requested rack 622A is not required to configure compute instances using the PVMD; instead, for example, the PVMD and client-selected servers can be housed within a server rack already used for other virtualized servers in the provider network data center.

[0082] In some embodiments, a cooperative location facility 652 that can access the VCS underlying network can house a PVMD for compute instances and a client-selected server. In the depicted embodiments, the cooperative location facility 652 may include a collection of provider network operator-managed devices 640 and client-managed or third-party-managed devices 642. Provider network operator-managed devices 640 may include, for example, one or more networking devices 660, such as routers, gateways, switches, etc., that provide access to the VCS underlying network and / or are configured within it. In response to one or more programming requests from a client, a client-selected server 667B may be located at the cooperative location facility and linked to a PVMD 665B capable of accessing the underlying network via networking devices 660. In some embodiments, a server rack 622B requested by the client may be used for both server 667B and PVMD 665B, although such a dedicated rack may not necessarily be required or used in other embodiments. In some embodiments, multiple client-selected servers may be deployed at the cooperative location facility, each linked to a corresponding PVMD (or sharing a PVMD). In some embodiments, the server selected by the client for the computing instance that enables PVMD can also be connected to a third-party or client-owned device 677 located at the colocation facility 652.

[0083] In some embodiments, the PVMD that directly accesses the underlying VCS network and the associated client-selected server can be located in a location other than the provider network data center 602 or the cooperative location facility 652. For example, in one embodiment, the provider network operator may set up a temporary facility (e.g., in a shipping container located near the client's work location) and may deploy the PVMD to enable compute instances to be configured on client-selected hardware placed in such a temporary facility.

[0084] Example of DSC mode operation: PVMD related programming interaction

[0085] Figure 7 The illustrations depict sample program interactions related to the configuration of a computing instance using a peripheral virtualization management device, according to at least some embodiments. Although Figure 7 The programmatic interactions described herein are supported in environments where the PVMD can directly connect to the underlying VCS network (i.e., when the PVMD is used in direct underlying connection or DSC mode), but at least some of the interactions shown are also supported in environments where the PVMD's connection to the underlying network requires traversing network paths that are not necessarily managed / controlled by the VCS operator.

[0086] Its functions and features are similar to Figure 1 VCS 712 of VCS 110 can implement the programming interface 777 in the illustrated embodiment. Such an interface may include, for example, a set of application programming interfaces (APIs), command-line tools, graphical user interfaces, network-based consoles, etc. VCS client 710 may submit a PVMDConnectedTargetServerSetupRequest 714 via programming interface 777 to initiate the process of enabling a compute instance at a hardware server selected by the client via PVMD. In at least some embodiments, request 714 may indicate various attributes of the requested PVMD connection configuration, such as detailed information about the specifications of the client-selected server to be used (peripheral connection types available on the server, CPU type, memory, storage, physical footprint, power requirements, etc.), the type of compute instance to be set up on the client-selected server (e.g., bare metal instance, microvirtual machine, or virtual machine belonging to another category defined in the VCS), the IVN in which the compute instance will be configured, physical location preferences (e.g., provider network data center, colocation facility, etc. in a specified location), and so on.

[0087] In response, after recording the details of request 714, VCS 712 may send a SetupInitiated message 715 to the client in the depicted embodiment, indicating that the tasks required to set up the desired configuration have been scheduled. In some cases, the actual physical setup of the PVMD and server combination may take some time (e.g., if a new rack must be set up, or if hardware servers must be shipped to the required facility, etc.). In some embodiments, a PVMDConnectedServerOnline message 716 may be sent to the client when the server selected by the client is physically connected to the PVMD, the PVMD is connected to the underlying VCS network, and sufficient information about the client-selected server has been provided from the PMVD to the VCS control plane to present the client-selected server as a virtualization host. In at least one embodiment, message 716 may include a host identifier that the client can use to refer to the client-selected server and / or the corresponding PVMD in subsequent programming requests pointing to the client-selected server.

[0088] After the server selected by the client is designated as a virtualization host (e.g., as a dedicated host allocated to the client in single-tenant mode), the client can begin requesting configuration operations for one or more compute instances on the client-selected server. For example, the client can issue a LaunchCIAtPVMDConnectedServer request 717 to establish a compute instance of a specified class on the client-selected server. In response, such a compute instance can be launched, for example, with the assistance of PVMD's virtualization management components such as the Network Manager and Storage Manager, and a LaunchInitiated message 721 can be sent to the client 710. The LaunchInitiated response may include, for example, an identifier assigned by the VCS to the launched compute instance, and / or one or more IVN network addresses assigned to the compute instance.

[0089] From the client's perspective, the compute instance launched at the server selected by the client can be considered the same as any other compute instance set up for the client in the depicted embodiments. For example, the client can request network traffic logs to / from the compute instance, e.g., via GetCITrafficLogs request 723. In some embodiments, various types of traffic can be logged, including, for example, Domain Name Service (DNS) requests and responses, traffic to storage or database services within the provider network, traffic to and from other compute instances within the same IVN, traffic to and from endpoints outside the compute instance's IVN, and so on. In at least some embodiments, GetCITrafficLogs request 723 may include one or more filtering parameters that specify the specific types of traffic for which logs should be provided—e.g., whether only DNS request / response logs are required, whether traffic logs to a specific IVN or service are required, the time range for providing log entries, and so on. In some embodiments, one or more components running at the PVMD may be responsible for initiating logging, and / or at least a subset of the logs may be stored at the PVMD. In other embodiments, the logs may be stored in other resources of the VCS or in a storage service within the provider network. In some embodiments, the requested log content (if available) can be provided to the client via a LogRecords response 725.

[0090] In at least some embodiments, the client may submit a GetInstanceMetadata request 731, for example, via an HTTP request initiated from a compute instance launched from a server selected by the client. In the depicted embodiments, a proxy of the instance metadata service running at PVMD may provide the requested metadata in a MetadataEntry response 733. In other embodiments, a component of the instance metadata service not running on PVMD may provide the requested metadata. Such metadata may include, for example, an identifier of the compute instance, credentials that can be used to submit requests from the compute instance to other provider network services, etc. In at least one embodiment, when a request is directed from the compute instance to another service, such credentials may be automatically provided by / from the PVMD / s metadata service proxy, and a separate request to obtain credentials may not be required.

[0091] If and when a client wishes to terminate a compute instance running on a server connected to PVMD, a TerminateCI request 741 can be sent to the VCS via programming interface 777. In response, the compute instance can be shut down, and a CITerminated message 743 can be sent to the client. If and when a client wishes to stop using a client-selected server for a compute instance, a DecommissionPVMDConnectedServer request 747 can be submitted. In response, in some embodiments, PVMD can disconnect from the client-selected server, and a ServerDecommissioned response 749 can be provided to the client 710. Note that in some embodiments, other types of programming interactions associated with using PVMD in an environment that may have a direct underlying network connection from PVMD can be supported. Figure 7 (Not shown in the image). Examples of programmatic interactions for managing the use of PVMD in locations outside the provider network of the VCS are discussed in more detail below, for example, in places where the underlying network cannot be directly accessed.

[0092] Methods for configuring compute instances using PVMD in DSC mode

[0093] Figure 8 This is a flowchart illustrating aspects of operations that can be performed at a provider network, according to at least some embodiments, to configure compute instances using servers selected by a client connected to a peripheral virtualization management device. As shown in block 801, instructions can be obtained at the VCS (e.g., through programmatic interaction with one or more VCS clients): a target hardware server (THS) selected by the client will be used to host one or more compute instances at a location or facility with direct access to the underlying network of the VCS. The underlying network may include physical network equipment of the provider network, such as cabling, switches, etc., linking at least some virtualization servers selected by the VCS operator as the default servers for the compute instances. Using the underlying network as infrastructure, various logical networks can be configured at the VCS, including isolated virtual networks (IVNs) of the type discussed earlier. Traffic between entities (e.g., various compute instances) assigned network addresses in logical networks can be transmitted via links in the underlying network, for example, using... Figure 2 The encapsulation protocol and / or VCS mapping service are discussed in the context of this discussion. In at least some embodiments, the client can suggest a specific location where the THS will be located, such as a provider network data center or cooperative positioning facility. In some embodiments, suggestions or recommendations for such locations can be programmatically provided to the client from the VCS, and the client can select or approve such a location.

[0094] In various embodiments, a peripheral virtualization management device (PVMD) for setting up compute instances at the THS can be identified and, if necessary, transmitted to the facility where the THS will reside. In some embodiments, VCS operators may have multiple versions or categories of such PVMDs available, which may differ from one another in attributes such as the specific type of peripheral connector / protocol used, the total compute capacity on the device, the type of local persistent storage device included in the PVMD, the capacity of the local persistent storage device, the physical footprint of the PVMD, and so on. In such embodiments, a THS-compatible PVMD can be identified and used—for example, a PVMD that can be connected via a peripheral protocol supported by the THS can be selected, or a PVMD configured to enable a specific type of compute instance preferred by the client can be discovered together with the THS. In the depicted embodiments, the PVMD (and the VCS, based on communication from the PVMD) can detect that a connection has been established between the PVMD and the underlying network, and between the PVMD and the THS (block 804). In some embodiments, the PVMD can transmit various details about the THS to the VCS control plane, for example, via the underlying network. As previously discussed, in various embodiments, encapsulation protocols implemented at multiple devices attached to the underlying network can be used to transmit traffic between one or more logical networks of the VCS. In at least one embodiment, multiple physical connections can be established between the PVMD and the THS—for example, one connection may utilize a first cable connected to an external PCIe port of the PVMD, while another connection may utilize an RJ45 connector to enable the transmission of commands and data between the PVMD and the system management service processor of the THS.

[0095] In some embodiments, PVMD can enable the THS to be presented, for example, as a dedicated virtualization host via a client-accessible VCS control plane programming interface (block 807). In some embodiments, such a dedicated virtualization host can be designated by a client as a resource for launching one or more compute instances in single-tenant mode—that is, as the term "dedicated" implies, such a virtualization host can be used on behalf of only one client. In other embodiments, the THS can be presented as a different type of virtualization host. In some embodiments, a network address of the underlying network can be assigned to the THS, for example, with the assistance of PVMD and the VCS control plane, as part of performing prerequisite operations to make the THS present as a virtualization host.

[0096] In response to a programming request from the client, in various embodiments, a compute instance can be launched at the THS after the THS has been designated as a virtualization host available to the client (box 810). For such a compute instance, storage and network virtualization tasks can be performed at the PVMD, including, for example, configuring the root volume using the PVMD's persistent storage, including the compute instance in the client's IVN, etc. In the depicted embodiments, the hypervisor or other components of the software virtualization management stack may not necessarily run on the THS, thereby enabling the client to use all (or at least the vast majority) of the THS's compute, memory, and storage capacity for the client's applications.

[0097] In at least some embodiments, after a compute instance is launched on the THS, it can be treated (from the client's perspective) just like any other compute instance on the client and can access all VCS features available for compute instances launched without the assistance of PVMD. In response to additional requests received from the client in the VCS control plane, in some embodiments, PVMD can initiate appropriate instance configuration operations for the compute instance launched on the THS (block 813). Such operations may include, for example, changing the network address, stopping / restarting the instance, changing security settings, etc. In the depicted embodiments, network traffic logs (including DNS requests and responses) for the compute instance can be generated and provided to the requesting client. In some embodiments, network security analysis, such as for intrusion detection, penetration detection, etc., can be performed with the help of components of PVMD and / or VCS. Access to other services implemented at the provider network (e.g., database services, storage services, load balancing services, etc.) can be provided to programs running on the compute instance on the THS. For example, a service request to another service (e.g., a database service) on the provider network, originating from a process running within the THS compute instance, can be transmitted / forwarded to that service by PVMD. An agent for the instance metadata service can run at PVMD to provide instance metadata for compute instances, such as credentials required to access some other services.

[0098] It should be noted that in each embodiment, Figure 8 Some of the operations shown may be performed in a different order than that shown in the figures, or may be performed in parallel rather than sequentially. Additionally, in one or more embodiments, it may not be necessary to... Figure 8 Some of the operations shown.

[0099] Example system environment for deploying PVMD in an external location

[0100] As previously described, in various embodiments, PVMD can be deployed in at least two operating modes: Direct Substrate Connectivity (DSC) mode and External Venue (EP) mode. In EP mode, PVMD can be used to configure and manage compute instances using a hardware server selected by the client, located in a venue outside the provider's network that may not have direct access to the VCS underlying layer from that venue. Figure 9 An exemplary system environment according to at least some embodiments is illustrated, in which the virtualization computing services of a provider network can be extended at locations outside the provider network using peripheral devices. As shown, system 900 includes resources located within a data center of provider network 901, and resources located at locations outside the provider network that do not have access to the underlying VCS network, such as customer locations 932A and 932B and cooperative positioning facility 930.

[0101] The data plane resources 945 of the VCS 910 of system 900 can be organized into a set of isolated virtual networks (IVNs) similar to those discussed previously. For example, IVN 915A can be established on behalf of client C1, IVN 915B can be established on behalf of client C2, and so on, providing clients with flexibility regarding network configuration settings within their respective IVNs. IVN 915A may include compute instances running on virtualization servers 917A and 917B selected by the VCS, while IVN 915B may include compute instances launched on virtualization servers 917J and 917K (e.g., CI 925A) selected by the VCS. The VCS control plane resources 941 may include a set of servers 902, such as 902A, 902B, and 902C in the illustrated embodiment, which are responsible for management tasks such as receiving and responding to instance configuration requests from clients, provisioning additional virtualization servers selected by the VCS, monitoring the health of the data plane resources, etc.

[0102] exist Figure 9In the illustrated embodiment, each PVMD 960 (e.g., 960A, 960B, or 960C) running an Extension Manager (EM) can be used to enable compute instances to be launched and configured at client-selected servers at external locations 932A, 932B, and 930. For example, PVMD 960A can be used to enable a client-selected server (CSS) 918A at a colocation facility 930 to host compute instances, PVMD 960B can be used to enable compute instances to be established at CSS 918J at client location 932A for VCS client C1, and PVMD 960C can connect to CSS 918K at client location 932B for client C3. Note that in the depicted embodiment, in addition to the EM, the PVMD 960 may also include a networking and storage virtualization manager. In the depicted embodiments, in addition to the server 918 selected by the client, at least some external locations 932A, 932B, and 930 may also include non-VCS devices (i.e., servers not intended to host VCS compute instances) – for example, location 932A may include non-VCS device 923B, location 932B may include non-VCS device 923C, and colocation facility 930 may include non-VCS device 923A. At least some non-VCS devices may also be linked to the CSS via a local network 968 (e.g., 968A, 968B, or 968C) – therefore, the CSS can be configured across multiple networks, including the local network 968 and one or more VCS networks (e.g., using addresses from a range used for the underlying VCS network). Note that CSS 918 may differ from each other – for example, CSS918J may have a different or more CPU than CSS 918K, CSS 918A may have more memory or a different CPU architecture than CSS 918K, and so on. Also note that, although in Figure 9 Each external location in the document displays only one CSS, but in at least some embodiments, multiple such CSSs can be located and used to host VCS compute instances within a single external location.

[0103] A set of client-selected servers and PVMDs in a given external location, along with computing instances running using the PVMDs, associated metadata, and configuration information, can be collectively referred to as the Extended Resource Group (ERG) 935 of the VCS in various embodiments. For example, in the depicted embodiments, ERG 935B (including PVMD 960B, CI 925K, and CSS 918J) can be configured at client location 932A, ERG 935C (including PVMD 960C, CI 925P, and CSS 918K) can be configured at client location 932B, and ERG 932C (including PVMD 960A, CI 925D, and CSS 918A) can be configured within a cooperative location facility 930. In practice, from the perspective of the VCS client, in various embodiments, the ERG can represent a local extension of the VCS's capabilities, which can be set up at any desired physical location that has internet access and can (e.g., relative to physical space, power, etc.) accommodate a set of PVMDs and client-selected hardware devices. From the perspective of the VCS itself, the ERG can be considered to reside virtually in the same provider network data center as the core VCS infrastructure, but physically in a location of customer choice. In various embodiments, when the ERG is set up at a customer-selected location, its resources can be managed by the control plane components of the VCS located in the provider network data center. Therefore, in at least some embodiments, setting up and using the ERG at a given location may not require locally replicating the VCS's control plane capabilities; instead, a secure network connection can be configured to transmit control plane commands from the provider network data center to the PVMD at the ERG, and the ERG's resources can be primarily used for data plane operations.

[0104] In some embodiments, one or more CIs of an ERG can be configured within an IVN 915 (e.g., CIs can be programmatically attached to virtual network interfaces having addresses within the address range of the IVN; CIs can be included in a database of compute instances within the IVN; information about the membership of CIs in the IVN can be provided to a mapping service of the VCS, etc.). In some cases, CIs running on an ERG with the aid of a PVMD can be included within an IVN that also includes CIs within the provider network data center, and it can be said that such an ERG extends the IVN. For example, the ERG 935B extends... Figure 9 The IVN 915A is for client C1, while the ERG 935A extends the IVN 915B configured for client C2. Some ERGs may include CIs configured within an IVN that are not located within a VCS data center, or may not even be configured within an IVN.

[0105] Because external locations 930 and 932 cannot directly access the VCS underlying layer, configuring compute instances using PVMD may require additional work compared to using PVMD in DSC mode. Figure 9 In the illustrated embodiments, the PVMD's extension manager can perform several functions. For example, the extension manager can establish a connection with one or more Extended Traffic Intermediaries (ETIs) 977 of the VCS via one or more secure channels (e.g., Virtual Private Network (VPN) tunnels) 966. Secure channels 977 may be necessary because traffic between the PVMD and the VCS control plane (and data plane) must flow through physical network paths not controlled or managed by the VCS, and therefore such physical network paths may not be trustworthy from the VCS's perspective. ETI 977 can be implemented at computing devices within the provider network 901's data center (e.g., using one or more compute instances and / or an IVN set up on behalf of the VCS control plane), and the ETI can be configured to perform one or more security operations related to traffic between the external site and the provider network's data center. In the depicted embodiments, using a secure channel established with the ETI can allow control plane commands to flow only in one direction: from the provider network's data center to the external site. Conversely, in various embodiments, data plane traffic can flow in either direction between the external site and the provider network. The following provides additional details about how control plane and data plane traffic flows between the provider network data center and external locations in various embodiments.

[0106] In at least some embodiments, the PVMD 960's extension manager can also assign a network address of the VCS underlying network (i.e., a network address selected from the address range allocated to virtualization servers within the VCS underlying network) to the CSS 918 to which the PVMD is attached. For example, in one embodiment, the underlying network address can be assigned after it has been determined that the CSS 918's perimeter card is already linked to the PVMD (e.g., linked to the PVMD's external perimeter port). In at least some embodiments, the underlying address assigned to the CSS 918 can also be assigned to one of the ETI 977s, enabling the ETI to act as a proxy or representative of the CSS 918 within the provider network data center.

[0107] Furthermore, in at least some embodiments, the extension manager can act as an intermediary for configuration commands related to the CI initiated at CSS 918 within ERG 935. For example, a VCS client can submit a request to the VCS control plane for configuration operations on the CI at ERG (using a path other than the secure channel established with ETI), and the instruction for this request can be provided to the extension manager at ERG from ETI. In response to this instruction, one or more configuration operations for the compute instance can be performed at CSS.

[0108] In various embodiments, the extension manager can also act as a bridge between the VCS and the local site network at an external location. For example, to enable traffic to flow between the non-VCS device 923 and the VCS compute instance, the extension manager can perform packet encapsulation / decapsulation according to encapsulation protocols and / or network address translation operations.

[0109] In various embodiments, the CI located at CSS 918 in ERG 935 can perform the same functions as a CI located in the data center of provider network 901 and be granted the same capabilities and rights. As previously described, the ERG CI can be configured within the IVN; for example, an IVN address is assigned to a virtual network interface programmatically attached to the CI, and IVN security rules are enforced on the CI. PVMD may include an instance metadata service agent that provides credentials to the CI, can be used to access other provider network services as previously described, and other types of instance metadata, such as CI identifiers, network configuration details, etc. In some embodiments, PVMDs located in several different external locations can be used to configure compute instances within the same IVN—that is, the IVN may include compute instances set up at hardware servers of various client selections in multiple locations outside the provider network. In some embodiments, for example to simplify network management, CI groups in each different external location may be configured within a corresponding subnet of the IVN. Network traffic logs of the CI running at the ERG may be captured, for example, at the PVMD or with the assistance of the PVMD, and provided to VCS clients in various embodiments. In at least one embodiment, traffic from the ERG CI to a DNS server outside the provider network (e.g., a DNS server set up for the client network) can be captured and logged, and provided to the VCS client upon request. Such a dedicated endpoint can be configured to enable the ERG CI to access publicly accessible services of the provider network without using a path or link to the public internet, if required by the VCS client configuring the CI on its behalf, and if client-specified policy controls can enforce the determination of whether a given packet from the CI should be forwarded to a service based on the traffic flow through the dedicated endpoint. In various embodiments, the PVMD960 used in external locations may include... Figure 3Some or all of the components shown can be connected to the CSS 918 and the local network using any suitable connector or cable.

[0110] In at least some embodiments, the Extension Manager (EM) can be instantiated in response to the detection of one or more triggering conditions (e.g., detection of power and / or internet connectivity) at the PVMD. The EM can then initiate (or at least participate in) the automatic establishment of a secure network connection to one or more VCS components (e.g., ETI 977) at one or more provider network data centers, without requiring, for example, additional configuration guidance from the VCS client configuring the PVMD on its behalf. Once the connection has been established, in various embodiments, the client can issue commands to instantiate a compute instance (and / or perform other operations using the compute instance) on a server of client-selected connection to the PVMD, similar to how such commands would be issued for a compute instance using only provider network resources. From the VCS client's perspective, the functionality of the VCS can now be seamlessly utilized using local resources (and, if needed, resources located in the provider network data center). In various embodiments, the compute instance set up at the ERG can (e.g., with the assistance of the PVMD capable of performing address translation and / or other encapsulation protocol-related processing) communicate with non-VCS devices 923, and, as needed, with other CIs set up in the provider network data center or at external locations. In some embodiments, even during periods of temporary interruption of connectivity to the provider's network data center, at least some CIs set up at the ERG, as well as associated higher-level services that use such CIs as building blocks, can continue to operate. In various embodiments, the ability to set up VCS CIs located in the same location as the application data can be particularly beneficial for VCS customers who wish to access and process large amounts of application data stored in their customer's data center with low latency (e.g., for legal compliance, security, or other reasons).

[0111] Example of an extended traffic intermediary device

[0112] Figure 10 An example extended traffic intermediary device according to at least some embodiments is illustrated, which can be configured to enable control plane traffic to securely flow from a provider network's data center to an external location where a peripheral virtualization management device is deployed. In the depicted embodiment, an extended resource group (ERG) 1035 including a client-selected server 1018 may be located at an external location 1032. Its features and functions are similar to... Figure 9 The PVMD 960 and PVMD 1060 can be attached to the server 1018, for example, via one or more cables or connectors. The PVMD 1060 may include an extension manager (EM) 1061.

[0113] In the depicted embodiment, at the data center 1001 of the provider network implementing VCS 1010, control plane resource 1072 may include one or more ERG connection managers 1074 (implemented using some combination of hardware and software at one or more computing devices of the VCS). The ERG connection managers may determine that a connection via a secure channel will be established with EM 1061, for example, in response to a programming request received at control plane resource 1072. In some embodiments, when the EM is online and connected to the Internet via a local network of external location 1032, EM 1061 may send a secure connection request to the VCS control plane (e.g., via a path over the public Internet). To enable secure connections, a set of Extended Traffic Intermediaries (ETIs) 1021 may be set up by the ERG connection managers.

[0114] In the depicted embodiments, ETI 1021 may include, for example, a CSS proxy 1014 and a tunnel endpoint 1016 (referred to as a provider network-side tunnel endpoint). In some embodiments, the CSS proxy 1014 and tunnel endpoint 1016 may include corresponding compute instances of the VCS. In at least one embodiment, the CSS proxy may be set within an ENI established on behalf of the VCS control plane (different from the ENI set on behalf of the VCS client), while the tunnel endpoint 1016 may be set within another ENI established for the VCS control plane.

[0115] Virtual Network Interface (VNI) 1012 can be programmatically connected to CSS Agent 1014 via Connection Manager 1074. In various embodiments, VNIs (also referred to as “elastic network interfaces”) can be configured at the VCS and / or at VCS extended resource groups, enabling some networking-related attributes such as IP (Internet Protocol) to be transferred relatively easily between compute instances without reconfiguring physical network interface cards (NICs). For example, such attribute transfer can be accomplished by programmatically detaching a VNI from one compute instance and programmatically attaching it to another. In the depicted embodiments, one or more VNIs, such as VNI 1012, can be programmatically attached to (or detached from) a given compute instance, independent of, for example, the specific hardware network interface card (NIC) of the host running the compute instance. A given VNI may have numerous attributes, including an IVN identifier indicating the IVN that configures the VNI, one or more private IP addresses (addresses that are not visible or public outside the IVN, at least by default), zero or more public IP addresses, one or more subnet identifiers (e.g., represented in classless inter-domain routing or CIDR format), one or more security rules (similar to firewall rules), and so on. In at least some embodiments, the same underlying network address may be assigned to VNI 1012, which is also assigned to CSS 1018 for which proxy 1014 is configured. In at least some embodiments, EM 1061 may receive an indication of the underlying network address from the ERG connection manager and assign that address to CSS 1018. EM 1061 may be used as an external endpoint for a secure (e.g., encrypted) network tunnel established with provider network-side tunnel endpoint 1016. In various embodiments, any of various tunneling protocols 1091 (e.g., standard VPN protocols, custom protocols used within the VCS, etc.) may be used to transmit data packets between the provider network and an external location over a potentially untrusted network.

[0116] In the described embodiments, because the same underlying network address is assigned to CSS proxy 1014 (via its attached VNI 1012) and CSS 1018, the VCS control plane can use CSS proxy 1014 as the destination for control plane commands actually used by CSS 1018 (e.g., configuration commands generated in response to requests from clients). Therefore, from the perspective of the VCS control plane, CSS may simply be another virtualization server accessible through the underlying network of the VCS. When a control plane command is generated, it can be received at the CSS proxy, and a version of the command can be sent via outbound control plane traffic path 1045. In some embodiments, CSS proxy 1014 and / or provider network-side tunnel endpoint 1016 can apply one or more security-related transformations to the command before sending it to ERG 1035. For example, in some embodiments, the version of the command obtained from the VCS control plane at the proxy may include one or more security tokens, and the CSS proxy may remove or exclude tokens (e.g., tokens that can be used to verify the identity of the requester of the operation performed as a result of the command, and tokens that the requester has the necessary permissions to request the operation) from the version of the command forwarded to the ERG. In at least some embodiments, the original security token can be transformed, and the transformed version of the security token can be included in the forwarded version of the command. In at least one embodiment, a corresponding message authentication code (including, for example, a hash-based message authentication code or HMAC) can be generated for outbound control plane commands sent from the agent to the ERG. In various embodiments, the CSS agent can log all outbound communication messages sent to a given ERG, and the logged messages can be checked by the ERG as a client set up on behalf of the client if needed. In some embodiments, at least two virtual network interfaces can be associated with a given CSS agent—one virtual network interface for obtaining commands from the VCS control plane, and another virtual network interface for communicating with tunnel endpoint 1016.

[0117] In some embodiments, data plane traffic can also flow between the VCS and external locations via secure channels or tunnels, for example, separate from channels used for controlling plane traffic. Data plane traffic can flow in either direction over such tunnels. Figure 10 (Not explicitly shown in the text), and may not involve the use of a CSS proxy. In one embodiment, a VNI with the same underlying address assigned to CSS 1018 may also be established within the VCS for data plane traffic to and from the CSS.

[0118] Example site internal PVMD communication channel

[0119] Figure 11The illustration depicts an example communication channel established between extension managers running on various peripheral virtualization management devices, according to at least some embodiments. In the depicted embodiments, multiple compute instances 1124 can be established using a client-selected hardware server within a given external location 1132. For example, compute instance (CI) 1124A can be set up on a client-selected server 1122A with the assistance of PVMD 1102A, CI 1124B can be set up on a client-selected server 1122B with the assistance of PVMD 1102B, and CI 1124C can be set up on a client-selected server 1122C with the assistance of PVMD 1102C.

[0120] Each client-selected server (CSS) 1122 can connect to an external location network 1191, for example, a network configured by or on behalf of the VCS client within the external location, to initiate CI 1124 upon request from the VCS client. In addition to providing a secure connection between the VCS and CI at external location 1132, PVMD 1102 can also be responsible for configuring a secure channel 1174 (e.g., 1174A, 1174B, or 1174C) between CIs 1124 in the depicted embodiments. In at least some embodiments, the secure channel can be configured with extension managers (EMs) 1112, such as 1112A, 1112B, and 1112C, and used to transmit data packets between CIs according to the VCS's encapsulation protocol. For example, CIs 1124 can be assigned network addresses within the proprietary address range of their respective IVNs (e.g., as a result of PVMD 1102 programmatically attaching their respective virtual network interfaces to CIs 1124), while CSSs can be assigned their respective underlying network addresses.

[0121] In some embodiments, such as Figure 11 As shown, each PVMD used in a given external location may include a corresponding Extension Manager (EM) 1112. In other embodiments, only a single "master" EM may be needed in a given external location, for example, for an entire extension resource group that may include multiple CSSs and multiple CIs. Such a master EM can establish a secure channel mediating all traffic between the VCS's data center and the extension resource group with the extension traffic. Within the ERG itself, such a master EM, running on a specific PVMD connected to a specific CSS, can communicate with PVMDs set up for other CSSs via a secure communication channel. In some embodiments, all CIs 1124 set up at the ERG may be configured within a single IVN. In other embodiments, some CIs of the ERG may be configured within different IVNs.

[0122] In at least some embodiments, not all CIs of the ERG must be configured using customer-selected hardware servers; instead, some hardware servers may be pre-configured by the VCS operator and shipped to an external location. The PVMD 1102, including the EM 1112, can also connect to such VCS-selected or VCS-configured servers and be used to manage compute instances launched on VCS-configured servers, just as a PVMD can be used to manage compute instances on a client-selected server. In some embodiments, the PVMD can even be used to configure and manage compute instances on VCS-selected hardware servers within the provider network data center.

[0123] Example of a geographically distributed logical network with PVMD enabled

[0124] In some embodiments, a given client of a VCS may wish to use a server of client selection for compute instances at several different locations and include several such instances in the same logical network such as an IVN. Figure 12 The illustration depicts an example geographically distributed logical network that can be configured using multiple peripheral virtualization management devices located at respective locations outside the provider network, according to at least some embodiments. In the depicted embodiments, VCS clients may have resources distributed across multiple locations outside the provider network data center, such as external locations 1202A, 1202B, or 1202N. In one example scenario, these external locations may each represent a corresponding branch of a bank (e.g., B1, B2, ..., Bk).

[0125] A client may wish to run applications using compute instances launched at a client-selected server (CSS) 1222 at each external location 1202—for example, using CSS 1222A at location 1202A, CSS 1222B at location 1202B, and CSS 1222N at location 1202N. The client may also wish to include all compute instances launched at different locations (e.g., CI 1224A, 1224B, and 1224N) and a set of additional CI 1224Z launched at data center 1255 in the provider network within a single logical network 1277. In the depicted embodiment, such a geographically distributed logical network can be configured using an IVN connection manager and / or EM set up at each PVMD in each location. For example, the corresponding EMs 121A, 1212B, and 1212C can be executed within PVMDs 1206A, 1206B, and 1206D, and such EMs can establish connections between the corresponding Extended Resource Groups (ERGs) at location 1202 and the various extended intermediaries described above. Extended group instances 1224A, 1224B, and 1224C can be assigned network addresses within a logical network or IVN 1277, for example, using the same address range assigned to the additional CI 1224Z. Components of the PVMD can perform the required encapsulation / decapsulation operations of the encapsulation protocol used at the VCS to enable packets to flow between different CIs 1224 (and between CI 1224 and other endpoints). In some embodiments, separate subnets (identified by corresponding CIDR address ranges) can be used for each CI at each external location, for example, to simplify network management, configure location-specific security, and / or for other reasons.

[0126] In the depicted embodiments, the corresponding VCS underlying network address (also assigned to the extended traffic intermediary at the provider network data center as described above) can be assigned to CSS 1222. Each CSS may also have an address assigned to it within an external location local network 1223 (e.g., networks 1223A, 1223B, or 1223N established by the client's IT personnel at different locations). In at least one embodiment, it is possible that the network address ranges of devices (including CSSs) assigned to different external location local networks 1223 at different locations 1202 may overlap—for example, the IPv4 address 10.0.0.1 may be assigned to CSS 1222A within local network 1223A, or to CSS 1222B within local network 1223B. In some cases, in order to be able to handle traffic flows between CSSs that were initially assigned the same address in their local networks, it may be necessary to change the overlapping addresses assigned to at least some CSSs. In some embodiments, an alternative approach to the overlapping IP address problem may be adopted, wherein bidirectional Network Address Translation (NAT) can be implemented with the help of PVMD 1206.

[0127] Example use of provider network services via PVMD

[0128] In some embodiments, the provider network implementing virtualized computing services may also provide access to other higher-level network-accessible services that utilize VCS compute instances as building blocks. For example, database instances may be implemented using VCS compute instances and provided to clients of network-accessible database services. In various embodiments, such higher-level services may also be available at Extended Resource Groups (ERGs), for example, by implementing service features using local compute instances set up within an ERG. Furthermore, in some embodiments, other services of the provider network that do not directly depend on VCS compute instances may also be accessed from VMs set up at an ERG, similar to how such services may be accessed from compute instances set up within the provider network's data center. Figure 13 An example of using additional provider network services at an extended resource group of a virtualized computing service, according to at least some embodiments, is shown.

[0129] In the depicted embodiment, the provider network 1301 includes at least a Virtualized Computing Service (VCS) (functionally and in terms of features similar to...) Figure 1(Similar to VCS 110 in function and features), storage service 1330 and database service 1342. A portion of VCS 1310 can be implemented using resources located at the provider network data center, and an extension of VCS can be set up at a location outside the provider network, such as within a customer data center 1332, which also includes one or more non-VCS devices 1323 (e.g., servers that will not have VCS compute instances set up) in ERG 1335.

[0130] exist Figure 13In the exemplary scenario depicted, an isolated virtual network (IVN) 1315 of the type previously discussed has been established for the VCS client. IVN 1315 includes multiple compute instances (such as CIs 1317A and 1317B) within the provider network data center portion 1310 of the VCS, and multiple PVMD-enabled CIs (such as 1318J and 1318K) located in ERG 1335 of the customer data center 1332. The term "PVMD-enabled CI" refers to a compute instance launched on a hardware server of client selection with the aid of the PMVD type discussed earlier, for example, to perform virtualization management operations on the CI on the PVMD. In the depicted embodiment, programs running at any CI can utilize resources from other provider network services. For example, storage instance 1322 of storage service 1330 can be accessed from CI 1017B in the provider network and CI 1318K in ERG 1335; similarly, database instance 1342 of database service 1340 can be accessed from CI 1317A in the provider network and CI 1318J in the ERG. In the depicted embodiments, access to services of other provider networks can be provided to CIs in ERG 1335, which is logically equivalent to providing access to CIs instantiated within the provider network. Within ERG 1335 itself, in the depicted embodiments, the configuration and use of services built on top of the VCS compute instance (i.e., services using CIs as building blocks) can be performed locally without accessing resources outside the ERG. For example, in one embodiment where the database instance of DB service 1340 includes CIs of the VCS, in response to a request for a database instance from another ERG CI 1318K, a new database instance can be created locally within the ERG using ERG CI 1318J. In the described embodiments, therefore, for a VCS customer represented by ERG 1335, even if the VCS control plane components are instantiated within the provider network data center and not replicated at ERG, ERG-based CIs such as 1318J and 1318K may be functionally equivalent to CIs such as 1317A and 1317B launched within the provider network. In at least some embodiments, in the unlikely event of a connection interruption between the ERG and the provider network data center, the CIs already instantiated at ERG and resource instances of other services set up at ERG (such as the aforementioned database service) may continue to operate for at least a period of time; therefore, clients may not need a persistent connection to the provider network data center when using ERG.

[0131] According to at least some embodiments, one or more additional provider network services can typically be accessed by submitting a request to an address accessible from the public Internet; therefore, such requests may have to be made via a link to the public Internet. To avoid having to use such a public network path, and also for performance reasons, in some embodiments, the VCS may set up a dedicated service endpoint (e.g., endpoint 1399). Instead of sending a request for service 1330 or 1340 from CI 1318 via the public Internet, the request can instead be directed to the dedicated service endpoint 1399, and in such embodiments, the dedicated network path of the provider network is used to deliver the request to the target service. In at least some embodiments, the response to the request can also be transmitted back to the requester via a dedicated path configured for or associated with endpoint 1399. In some embodiments, such a dedicated service endpoint can also be set up for services implemented within the IVN of a VCS client. For example, a VCS client C1 can implement a publicly accessible service S1 in C1's IVN, allowing other VCS clients C2, C3, ... to access S1 from their respective IVNs via a dedicated service endpoint such as 1399. In at least some embodiments, PVMD-enabled CIs such as 1318J and 1318K can also utilize such service endpoints to avoid using public network paths for service requests and responses. In the depicted embodiments, a client-specified policy or rule 1388 can be associated with the service endpoint, and such a policy or rule can be applied to service-related traffic for any CIs 1317 and 1318. For example, such a policy could instruct that only instances 1318J and 1317A of IVN 1335 are allowed to use endpoint 1399 to access a specific storage object SO1 at storage service 1330. In at least some embodiments, the decision regarding whether to forward packets containing service requests generated at a PVMD-enabled CI such as 1318J or 1318K can be made based on such a policy (e.g., at the PVMD used for the CI).

[0132] Example procedural interactions related to PVMDs deployed in external locations

[0133] In some embodiments, the VCS client can use a number of programming interfaces to submit and set data. Figure 9 Using ERG (similar to) with the help of PVMD introduced in the context of Figure 9 The request associated with ERG 935 introduced in [the document]. Figure 14Exemplary programming interactions involving the configuration of extended resource groups for virtualized computing services using peripheral devices are illustrated according to at least some embodiments. In the depicted embodiments, VCS 1412 may implement one or more programming interfaces 1477, such as a set of application programming interfaces (APIs), a web-based console, a graphical user interface, command-line tools, etc., any combination of which can be used by VCS client 1410 to interact with VCS.

[0134] Client 1410 may submit a SetupERGUsingPMVD message 1414, for example, via programming interface 1477, requesting the configuration of an extended resource group with one or more PMVDs at a specified location outside the VCS's own data center. In some embodiments, message 1414 may include attributes or details of the target hardware server selected by the client for compute instances at the requested ERG, such as information about... Figure 7 Some details are discussed in PVMDConnectedTargetServerRequest 714. In some embodiments, in response to SetupERGUsingPMVD message 1414, a workflow including various preliminary tasks to be performed at the provider network can be initiated, and response messages such as ERGSetupInitiated 1415 can be transmitted to the client. In at least one embodiment, with Figure 10 The configuration of one or more Extended Traffic Intermediaries (ETIs) similar to ETI 1021 can be used as part of an initial task workflow to initiate the requested ERG. For example, in some embodiments, the compute instance set up as a CSS proxy can actually actively wait to be contacted by an Extension Manager (EM) at a PMVD located at a target external location. In at least one embodiment, the EM may not attempt to communicate with the CSS proxy until the PMVD is delivered or obtained at the external location, powered on, connected to the Internet, etc., which may only occur some time after the SetupERGUsingPMVD request is processed at VCS 1412. In some embodiments, the VCS's ERG connection manager (similar to...) Figure 10 The ERG connection manager (1074) can be responsible for instantiating and configuring ETI. In at least one embodiment, a given CSS proxy instance can be used, for example, for several different CSSs of one or more VCS clients in a multi-tenant operation mode.

[0135] In the depicted embodiments, VCS 1412 may send an ERGONline message 1416 to client 1410 after the ERG, including at least one client-selected server and the PVMD connected to the client-selected server, has been configured at the desired location, indicating that the client can begin using the ERG. In some embodiments, implicit communication may be used to notify the client that the requested ERG is ready for use, rather than an explicit message; for example, the entry for the requested ERG in the console may be displayed via a visual icon indicating that the ERG has become available. In at least some embodiments, one or more preconditions may have to be met before the equivalent of the ERGONline message is transmitted to the client: for example, a secure communication channel that can be used by the CSS proxy instance to transmit management commands to the ERG may have to be established, at least one CSS may have to be successfully launched via the PVMD, and so on.

[0136] In the depicted embodiments, VCS client 1410 may use programming interface 1477 to submit request 1417 to launch one or more compute instances at a specified client-selected server in the ERG. In at least some embodiments, request 1417 may be transmitted to the VCS control plane via a path different from the unidirectional path used to transmit one or more corresponding commands to the ERG. In some embodiments, after the command for launching the CI has been transmitted from ETI to PVMD in the ERG, a message 1421 indicating that launch has been initiated may be transmitted back to the client. Note that, at least in some embodiments, LaunchInitiated message 1421 may simply indicate that the command for launching the VM has been sent to the ERG; in such embodiments, the process at PVMD and / or at the client-selected server that actually performs the work of launching the CI may not necessarily transmit an outbound management message confirming whether the launch was successful.

[0137] In some embodiments, a DescribeERG request 1423 may be submitted to obtain information about the compute instance of the ERG, the server selected by the client, and / or PVMD, and the requested information may be provided in the form of an ERGInfo message 1425. In some embodiments, this information may include, for example, a list of entities authorized to use the ERG (e.g., starting or terminating a CI at the ERG), the set of servers and / or CIs selected by the client of the ERG, the setup date of the ERG, data related to ERG billing, etc.

[0138] In at least some embodiments, client 1410 may request changes to the ERG, for example, by submitting a ModifyERG request 1428 indicating the desired changes. Such changes may include, for example, increasing or decreasing the number of servers and associated PVMDs selected by the client for the ERG, modifying the set of users / groups / entities permitted to use the ERG, and so on. In the depicted embodiments, if the requested modifications can be accommodated based on applicable rules and policies of the VCS, a corresponding workflow for implementing the changes can be initiated, and a ModifyInitiated message 1433 indicating that a change workflow is underway can be transmitted to the client. In one embodiment, client 1410 may submit an UnconfigureERG request 1441 to indicate that the ERG is no longer needed. In response, the VCS may initiate a workflow to terminate accessibility to the ERG, shut down, dismantle, and transport equipment configured for the ERG, and in some embodiments, an ERGUnconfigured message 1443 may be sent to the client to indicate that a workflow has been initiated and the programming request no longer allows access to the ERG.

[0139] Apart from Figure 14 In addition to the procedural interactions related to ERG shown, client 1410 can also submit various requests related to IVN instances established at ERG, and / or, in some embodiments, various requests related to individual compute instances established at ERG, similar to those in... Figure 7 The requests discussed in the context of [the relevant context]. For example, a client might submit a request to obtain network traffic logs for a PVMD-enabled compute instance set up in the ERG, to configure a private endpoint, etc. In at least one embodiment, a client might submit a programming request to establish a single logical network 1277, such as an IVN spanning multiple external locations, similar to [the previous context]. Figure 12 The logical network discussed in the context of [the previous context], or adding a CI to an existing logical network specifying an external location. For ERG-related operations, in some embodiments, [the following can be supported:] more than [previous context]. Figure 14 or Figure 7 The additional types of procedural interactions shown.

[0140] Methods for configuring and using PVMD in external locations

[0141] Figure 15This is a flowchart illustrating aspects of operations that can be performed to establish an extended resource group using a peripheral virtualization management device, according to at least some embodiments. As shown in block 1501, instructions can be obtained at the VCS (e.g., through one or more programmatic interactions with a VCS client): a target hardware server (THS) selected by the client will be used to host one or more compute instances in a location or facility (e.g., the client's own data center or some type of colocation facility) that does not have direct access to the underlying network of the VCS. The underlying network may include physical network equipment of the provider network, such as cabling, switches, etc., linking at least some virtualization servers selected by the VCS operator as default servers for the compute instances and located in the data center of the provider network implementing the VCS. At the underlying layer, various logical networks can be configured at the VCS, including isolated virtual networks (IVNs) of, for example, the type discussed above. Traffic between entities (e.g., various compute instances) assigned network addresses in logical networks can be transmitted through links in the underlying network, for example, using... Figure 2 The encapsulation protocols and / or VCS mapping services discussed in the context of this discussion. In various embodiments, traffic between the THS and the VCS data center, where the VCS control plane components and most of the VCS data plane components reside, may have to traverse network paths of the public internet and / or other networks not controlled or managed by the provider network. Therefore, from the VCS's perspective, its level of trust is lower than that of the provider network's internal network.

[0142] In various embodiments, a peripheral virtualization management device (PVMD) for setting up compute instances at the THS can be identified, which has characteristics and functions similar to those in... Figure 9 The PVMD 960 discussed in the context of [previous context] will be transmitted to the facility where THS will be located, if necessary. In some embodiments, as discussed earlier in the context of using PVMDs in direct underlying connection mode, VCS operators may have multiple versions or categories of such PVMDs available, which may differ from one another in attributes such as the specific type of peripheral connector / protocol used, the total computing power on the device, the type of local persistent storage device included in the PVMD, the capacity of the local persistent storage device, the physical footprint of the PVMD, and so on. In such embodiments, THS-compatible PVMDs can be identified and used—for example, PVMDs that can be connected via peripheral protocols supported by THS can be selected, or PVMDs configured with THS to enable specific types of compute instances preferred by the client can be discovered.

[0143] In various embodiments, in addition to the baseline PVMD components used when the underlying layer is directly accessible, the PVMD used with the THS in an external location may also include Figure 9 The type of extension manager (EM) discussed in the context of this document. In some embodiments, such an EM may be started based on the detection that the PVMD is already connected to power and the Internet (e.g., using an Ethernet or similar cable linked to a local network set up at an external location, which in turn is linked to the Internet) (box 1504). In other embodiments, other triggering conditions or commands may cause the EM to be started at the PVMD.

[0144] One or more secure network tunnels (e.g., corresponding tunnels for control plane traffic and data plane traffic) can be established between the EM and a set of Extended Traffic Intermediaries (ETIs) similar to the previously discussed ETIs (box 1507). ETIs can be configured at the data center for the VCS in the depicted embodiments, for example, at one or more compute instances within an isolated virtual network established on behalf of the VCS, to manage connectivity to external locations. In some embodiments, an External Resource Group (ERG) Connection Manager within the VCS control plane can establish an ETI. At least some ETIs can be configured to perform secure operations (e.g., encrypt / decrypt packets, strip or replace previously discussed credentials, etc.) on at least a portion of the traffic between the EM and the data center.

[0145] In the illustrated embodiment, the PVMD (and the VCS control plane, based on communication from the PVMD) can detect that a connection has been established between the PVMD and the THS. In at least one embodiment, multiple physical connections can be established between the PVMD and the THS—for example, one connection can utilize a first cable connected to an external PCIe port of the PVMD, while another connection can utilize an RJ45 connector to allow commands and data to be transferred between the PVMD and the system management service processor of the THS. After the THS is connected to the PVMD, the EM can assign a private network address PNA1 of the underlying network (i.e., an address selected from the address range allocated to VCS virtualization servers within the provider network data center) to the THS (block 1510). In some embodiments, PNA1 can also be assigned to one or more ETIs, for example, via a programmably connected virtual network interface as described above, enabling such ETIs to act as proxies for the THS within the underlying network. Note that the THS may also be assigned one or more other network addresses, for example, within a local network at an external location in at least some embodiments; in these embodiments, the underlying network address PNA1 may be used for traffic between the VCS data center and the THS, while the local network address may be used for traffic between the THS and other devices at the external location.

[0146] In some embodiments, PVMD can cause the THS to be presented, for example, as a dedicated virtualization host via a client-accessible control plane programming interface of the VCS. In some embodiments, such a dedicated virtualization host can be designated by the client as a resource for launching one or more compute instances in single-tenant mode—that is, as the term "dedicated" implies, such a virtualization host can be used on behalf of only one client. In other embodiments, the THS can be presented as a different type of virtualization host.

[0147] After the underlying address has been assigned to the THS and it has been presented as a virtualized host, the client can use the THS for various configuration operations related to compute instances, just as other virtualized hosts located within the provider network data center could be used in the depicted embodiment. In response to a programming request from the client, which is received at the EM via one or more secure channels, the compute instance can be launched at the THS, for example (box 1513). In addition to a secure connection to the VCS, baseline storage and network virtualization tasks can be performed at the PVMD, including, for example, configuring the root volume using the PVMD's persistent storage, including the compute instance in the client's IVN, etc. In the depicted embodiment, the hypervisor or other components of the software virtualization management stack may not need to run on the THS, thereby enabling the client to use all (or at least the vast majority) of the THS's compute, memory, and storage capacity for the client's applications.

[0148] In at least some embodiments, once a compute instance is launched on a THS at an external location, it can be treated (from the client's perspective) just like any other compute instance on the client and can access all VCS features available for compute instances launched without the assistance of PVMD. In response to additional requests received from the client in the VCS control plane and transmitted to the EM via a secure channel, in some embodiments, PVMD can initiate appropriate instance configuration operations for the compute instance launched on the THS (block 1516). Such operations may include, for example, changing the network address, stopping / restarting the instance, changing security settings, etc. In the depicted embodiments, network traffic logs (including DNS requests and responses) for the compute instance can be generated and provided to the requesting client. In some embodiments, network security analysis, such as for intrusion detection, penetration detection, etc., can be performed with the help of components of PVMD and / or VCS. Programs running on the compute instance on the THS can be provided with access to other services implemented at the provider network (e.g., database services, storage services, load balancing services, etc.). An instance metadata service proxy can run at PVMD to provide instance metadata for the compute instance, such as credentials required to access some other services. In some embodiments, compute instances using several different THS configurations at one or more external locations can be configured as part of the same Isolated Virtual Network (IVN). In some cases, an IVN may include one or more compute instances configured within a provider network data center, and one or more compute instances configured using PVMD and THS at external locations.

[0149] As mentioned earlier, in some embodiments, network addresses for local networks used in multiple external locations may overlap; if left unaddressed, this could lead to problems with configuring multi-location logical networks using PVMD, as VCS may expect all virtualized hosts in the underlying network to have unique network addresses. Figure 16 This is a flowchart illustrating aspects of operations that, according to at least some embodiments, can be performed to enable the establishment of a multi-site logical network in an environment where conflicting network addresses may have already been assigned at multiple sites. As shown in block 1601, a programming request can be received, for example, at the VCS control plane to configure a logical network (e.g., an IVN) spanning multiple external sites. For example, the request could instruct the use of a corresponding PVMD at each site to consolidate client-selected servers into virtualized hosts for the VCS.

[0150] In some embodiments, as a prerequisite for establishing such a logical network, the current networking configuration of various locations (including the range of network addresses assigned to various servers, including those selected by the client, and / or those assigned to servers selected by the client if they are not already configured) can be obtained programmatically in the VCS. For example, this configuration information can be checked through the control plane component of the VCS (block 1604).

[0151] If it is determined that addresses assigned (or later assigned) to various servers conflict in different locations, as detected in the operation corresponding to block 1607, in some embodiments, the VCS can notify the client via a programming interface about one or more methods for handling or resolving address conflicts (block 1610). Such methods may include, for example, the reallocation of network addresses (e.g., assigning addresses from various non-overlapping IP address ranges at different locations), and / or the implementation of a bidirectional Network Address Translation (NAT) algorithm. In bidirectional NAT, in various embodiments, both the source and destination addresses of a given packet can be translated at a network intermediary such as a virtual or physical router or traffic hub. In some embodiments, the VCS may provide the client with more detailed guidance (e.g., suggesting new non-overlapping ranges of IP addresses, instructions for establishing a virtual router or hub for NAT, etc.) to make the required configuration changes (block 1613). If the client wishes to continue establishing a multi-location logical network, configuration changes can be made at one or more locations, and the operations corresponding to blocks 1604 and later can be performed again.

[0152] If no address conflict is detected (also determined in the operation corresponding to block 1607), a logical network spanning multiple locations can be configured (block 1616), for example, using PVMD and VCS control plane resources at external locations. For example, an extension manager (EM) similar to those discussed earlier can be used to implement secure connections between servers at external locations and VCS data centers, and compute instances set up on target hardware servers selected and linked to by clients to PVMDs can, in some embodiments, be assigned network addresses within a single IVN.

[0153] It should be noted that in various embodiments, Figure 15 and / or Figure 16 Some of the operations shown can be implemented in a different order than that shown in the accompanying drawings, or they can be performed in parallel rather than sequentially. Furthermore, in one or more implementations, it may not be necessary to... Figure 15 and / or Figure 16 Some of the operations shown in the image.

[0154] Use Cases

[0155] The aforementioned technologies that enable compute instances of a provider network's virtualized computing services to be launched on a client-selected server using small-footprint external peripherals, and that allow the data plane logic of the virtualized computing service to be logically extended to the client's premises using such peripherals, can be very useful in a variety of situations. Many organizations that leverage provider network resources (such as virtual machines of various capability levels) to achieve scalability, availability, reliability, and security may want to use similar characteristics on their chosen hardware servers—for example, on hardware familiar to the organization's IT staff, or optimized for certain types of applications. The described peripherals can be attached to such servers with minimal effort, for example, in the provider network's data center where most of the virtualized computing resources reside, at a co-location facility, and / or at the client's premises. In some cases, the peripherals can simply be placed in the same rack as the client-selected hardware server and connected to the server using standard cables to allow compute instances to be launched on the server. Such compute instances can automatically and transparently inherit all the characteristics of compute instances launched using servers selected by the virtualized computing service provider, such as secure access to other services, automatic metrics and log collection, network intrusion detection, etc. In some cases, such as due to access to large amounts of application data that can be stored at such locations, clients may prefer to launch instances using selected hardware located at their own premises. If possible, provider network clients may prefer to avoid the costs and / or latency associated with transferring such large amounts of data over the internet to the provider network's own data center and may want to ensure that data stays as close to the client's premises as possible. Clients may sometimes need to use compute instances and / or use compute instances as building blocks for services in remote locations where internet connectivity may be unreliable or expensive (e.g., near oil rigs, cellular phone towers, scientific data collection sensor arrays, etc.). Some organizations may have a large number of engineers / designers, for example, in offices or other physical locations, and may need to use compute instances within very low latency distance of the engineers / designers to perform rendering or other compute-intensive operations. Using local equipment in any desired location with internet connectivity to support the same capabilities offered at the provider network's data center can significantly extend the range of applications that can run efficiently and securely by provider network clients.

[0156] Explanatory computer system

[0157] In at least some embodiments, the server implementing one or more of the control plane and data plane components for supporting the technology type described herein (e.g., establishing a connection to PVMD, setting up and using an extended resource group for virtualized computing services at a selected location outside the provider network data center, etc.) and / or the server selected by the client for hosting computing instances may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media. Figure 17 A general-purpose computing device 9000 of this type is illustrated. In the illustrated embodiment, the computing device 9000 includes one or more processors 9010 coupled to system memory 9020 (which may include both non-volatile memory modules and volatile memory modules) via an input / output (I / O) interface 9030. The computing device 9000 further includes a network interface 9040 coupled to the I / O interface 9030.

[0158] In various embodiments, computing device 9000 may be a single-processor system including one processor 9010, or a multiprocessor system including several processors 9010 (e.g., two, four, eight, or another suitable number). Processor 9010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 9010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPSISA, or any other suitable ISA. In a multiprocessor system, each processor among processors 9010 may, but is not required to, implement the same ISA. In some implementations, a graphics processing unit (GPU) may be used in place of a conventional processor or as a supplement to a conventional processor.

[0159] System memory 9020 can be configured to store instructions and data accessible by processor 9010. In at least some embodiments, system memory 9020 may include both volatile and non-volatile portions; in other embodiments, only volatile memory may be used. In various embodiments, the volatile portion of system memory 9020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM, or any other type of memory. For the non-volatile portion of the system memory (e.g., which may include one or more NVDIMMs), in some embodiments, flash-based memory devices, including NAND flash devices, may be used. In at least some embodiments, the non-volatile portion of the system memory may include a power source, such as a supercapacitor or other power storage device (e.g., a battery). In various embodiments, memristor-based resistive random access memory (ReRAM), 3D NAND technology, ferroelectric RAM, magnetoresistive RAM (MRAM), or any type of phase-change memory (PCM) may be used at least for the non-volatile portion of the system memory. In the illustrated embodiment, program instructions and data that implement one or more of the desired functions, such as the methods, techniques, and data described above, are shown stored in system memory 9020 as code 9025 and data 9026.

[0160] In one embodiment, I / O interface 9030 may be configured to coordinate I / O traffic between processor 9010, system memory 9020, and any peripheral devices within the device, including network interface 9040 or other peripheral interfaces such as various types of persistent and / or volatile storage devices. In some embodiments, I / O interface 9030 may perform any necessary protocols, timing, or other data transformations to convert data signals from one component (e.g., system memory 9020) into a format suitable for use by another component (e.g., processor 9010). In some embodiments, for example, I / O interface 9030 may include support for devices attached via various types of peripheral buses (e.g., peripheral Component Interconnect (PCI) bus standards or variants of the Universal Serial Bus (USB) standard). In some embodiments, for example, the functionality of I / O interface 9030 may be split into two or more independent components, such as a northbridge and a southbridge. Moreover, in some embodiments, some or all of the functionality of I / O interface 9030, such as the interface for system memory 9020, may be directly incorporated into processor 9010.

[0161] Network interface 9040 can be configured to allow data exchange between computing device 9000 and other devices 9060 attached to one or more networks 9050, such as... Figures 1 to 16Other computer systems or devices illustrated herein. For example, in various embodiments, network interface 9040 may support communication over any suitable wired or wireless general data network (such as Ethernet type). Additionally, network interface 9040 may support communication via telecommunications / telephone networks such as analog voice networks or digital fiber optic communication networks, via storage area networks such as Fibre Channel SAN, or via any other suitable type of network and / or protocol.

[0162] In some embodiments, system memory 9020 may represent an embodiment of a computer-accessible medium configured to store information for implementing... Figures 1 to 16 The methods and apparatus discussed in the context of this document include at least a subset of program instructions and data. However, in other embodiments, the program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may comprise non-transitory storage media or storage media such as magnetic or optical media, for example, a disk or DVD / CD coupled to computing device 9000 via I / O interface 9030. Non-transitory computer-accessible storage media may also comprise any volatile or non-volatile media, such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., which may be included as system memory 9020 or another type of memory in some embodiments of computing device 9000. In some embodiments, multiple non-transitory computer-readable storage media may jointly store program instructions that, when executed on or across one or more processors, implement at least a subset of the methods and techniques described above. Computer-accessible media may also include transmission media or signals transmitted via communication media such as networks and / or wireless links, such as electrical signals, electromagnetic signals, or digital signals, such as those implemented via network interface 9040. Figure 17 Some or all of the computing devices illustrated may be used to implement the functions described in the various embodiments; for example, software components running on various different devices and servers may cooperate to provide the functions. In some embodiments, in addition to or instead of implementing using a general-purpose computer system, storage devices, network devices, or dedicated computer systems may be used to implement parts of the described functions. As used herein, the term "computing device" refers to at least all of these types of devices, but is not limited to these types of devices.

[0163] Embodiments of this disclosure may be described in light of the following terms:

[0164] 1. A system comprising:

[0165] One or more computing devices that provide virtualized computing services within a cloud provider's network;

[0166] The target hardware server indicated by the client of the cloud provider network; and

[0167] A peripheral device located within a physical chassis outside the target hardware server, comprising program instructions that, when executed at or across one or more processors of the peripheral device, implement (a) one or more storage managers and (b) one or more networking managers;

[0168] The peripheral device is configured as follows:

[0169] A connection is established with the underlying network of the virtualization computing service via a first cable, wherein an encapsulation protocol implemented at multiple devices attached to the underlying network is used to transmit traffic between one or more logical networks of the virtualization computing service;

[0170] After determining that (a) the peripheral card of the target hardware server is connected to the external peripheral port of the peripheral device via a second cable and (b) the system management processor of the target hardware server is connected to the peripheral device via a third cable, the one or more computing devices of the virtualization computing service present the target hardware server as a dedicated virtualization host selectable by the client to run one or more computing instances, wherein the target hardware server is connected to the underlying network only through the peripheral device; and

[0171] One or more configuration operations are initiated on behalf of the client to launch a compute instance on the target hardware server, wherein at least part of the configuration of the compute instance is based on operations performed using the one or more network managers, and wherein at least part of the access to the root volume is provided to the compute instance based on operations performed by the one or more storage managers.

[0172] 2. The system according to Clause 1, wherein the computing instance is a bare metal instance, and from the bare metal instance, one or more hardware resources of the target hardware server can be accessed without virtualization.

[0173] 3. The system according to any one of Clauses 1-2, wherein the one or more computing devices are configured to present the target hardware server as the dedicated virtualization host in response to a programming request received from a client, wherein the programming request indicates one or more attributes of the target hardware server.

[0174] 4. The system according to any one of Clauses 1-3, wherein the target hardware server is located within a data center of one or more data centers of the cloud provider network implementing the virtualized computing service.

[0175] 5. The system according to any one of Clauses 1-4, wherein the peripheral card is linked to the peripheral device using one or more of the following: (a) a PCIe (Peripheral Component Interconnect Fast) connector, (b) a connector configured to multiplex PCIe signals with one or more other types of signals, or (c) a USB (Universal Serial Port) connector.

[0176] 6. A method comprising:

[0177] A connection to the underlying network of the virtualization computing service is determined at a peripheral device including one or more virtualization offloading components containing a storage manager, wherein an encapsulation protocol implemented at multiple devices attached to the underlying network is used to transmit traffic between one or more logical networks of the virtualization computing service.

[0178] After detecting that the target hardware server is linked to the peripheral device, the target hardware server is presented to the client of the virtualization computing service as a virtual host selectable by the client to run one or more computing instances, wherein the peripheral device is located in a chassis outside the target hardware server; and

[0179] One or more configuration operations are initiated by the one or more virtualization offloading components to launch a computing instance on behalf of the client at the target hardware server.

[0180] 7. The method according to Clause 6, wherein the one or more configuration operations include configuring the root volume of the compute instance by the storage manager.

[0181] 8. The method according to any one of Clauses 6-7, wherein the peripheral device comprises one or more persistent storage devices, the persistent storage devices including at least one persistent storage device accessible from the computing instance via an NVME (Non-Volatile Memory Fast) interface.

[0182] 9. The method according to any one of Clauses 6-8, wherein the virtualized computing service is implemented at the provider network, the method further comprising:

[0183] A service request is transmitted via the peripheral device to another service of the provider network, wherein the service request originates from the computing instance.

[0184] 10. The method according to any one of Clauses 6-9, wherein the target hardware server is linked to the peripheral device using one or more of the following: (a) a PCIe (Peripheral Component Interconnect Fast) connector, (b) a connector configured to multiplex PCIe signals with one or more other types of signals, or (c) a USB (Universal Serial Bus) connector.

[0185] 11. The method according to any one of Clauses 6-10, wherein the target hardware server is linked to the peripheral device using one or more of the following: (a) a signal retimer card or (b) a signal repeater card.

[0186] 12. The method according to any one of clauses 6-11, wherein the peripheral device comprises one or more of the following: (a) a network interface card (NIC) emulator or (b) an isolated virtual network connection manager, wherein the first computing instance is configured within a first isolated virtual network, the method further comprising:

[0187] At the peripheral device, one or more operations of the encapsulation protocol are performed on data packets generated at the computing instance, wherein the data packets are directed to a destination configured within the isolated virtual network of the virtualized computing service.

[0188] 13. The method according to any one of Clauses 6-12, wherein the peripheral device comprises a persistent storage device and a removable security device, wherein removing the removable security device from the peripheral device renders at least a portion of the persistent storage device unreadable.

[0189] 14. The method according to any one of Clauses 6-13, wherein the control plane of the virtualized computing service comprises a plurality of computing devices located in one or more data centers, wherein the peripheral devices and the target hardware server are located outside the one or more data centers.

[0190] 15. The method according to any one of Clauses 6-14, wherein the target physical server includes a baseboard management controller, and wherein detecting that the target hardware server has been linked to a peripheral device includes detecting that the peripheral device is connected to the baseboard management controller.

[0191] 16. A peripheral device, comprising:

[0192] One or more processors and one or more memories, wherein the one or more memories include program instructions for one or more virtualization offloading components that implement virtualized computing services of a provider network when executed on or across the one or more processors, wherein the one or more virtualization offloading components include a storage manager; and

[0193] A port that can be connected to the target hardware server;

[0194] The one or more virtualization offloading components are configured as follows:

[0195] Establish a network connection with one or more computing devices in the control plane of the virtualized computing service;

[0196] At least in part based on the detection that a specific hardware server has been linked to the port, the control plane presents the specific server as a virtualization host to the clients of the virtualization computing service, wherein the peripheral devices are located in a separate chassis from the specific hardware server; and

[0197] In response to a determination that a command has been received on the control plane, one or more compute instance configuration operations are initiated on the specific hardware server, including at least one configuration operation initiated by the storage manager, to enable compute instances launched from the specific hardware server to access logical storage devices.

[0198] 17. The peripheral device as described in Clause 16, wherein the one or more virtualization offloading components include a network manager configured to store indications of network addresses assigned to the compute instances configured at the particular hardware server, wherein the network addresses are within the range of private network addresses of an isolated virtual network established at the virtualization compute service.

[0199] 18. The peripheral device according to any one of Clauses 16-17, wherein the one or more virtualization offloading components include an instance metadata broker configured to provide credentials to the compute instance launched at the particular hardware server for access to services other than the virtualized compute service.

[0200] 19. The peripheral device according to any one of Clauses 16-18 further includes a persistent storage device, wherein the storage manager is configured to enable access from the computing instance to the persistent storage device.

[0201] 20. The peripheral device according to any one of clauses 16-19, wherein the target hardware server is mounted in one or more slots of a server rack, and wherein the peripheral device is mounted in different slots of the server rack.

[0202] 21. A system comprising:

[0203] One or more computing devices for virtualized computing services of a cloud provider network, located within one or more data centers of the cloud provider network;

[0204] Target hardware servers located at locations outside the one or more data centers of the cloud provider network; and

[0205] A peripheral device, located in a physical enclosure separate from the target hardware server, includes program instructions that, when executed on or across one or more processors of the peripheral device, implement a first extension manager for the virtualized computing service.

[0206] The first extension manager is configured as follows:

[0207] A connection is established with one or more extended traffic intermediaries of the virtualized computing service through one or more secure channels, wherein the one or more extended traffic intermediaries are implemented at the one or more computing devices, and wherein the one or more extended traffic intermediaries are configured to perform one or more security operations on traffic between the venue and the one or more data centers of the cloud provider network;

[0208] After determining that the peripheral card of the target hardware server is linked to the external peripheral port of the peripheral device, a specific private network address of the underlying network of the virtualization computing service is assigned to the target hardware server, and the target hardware server is presented as a virtualization host of the virtualization computing service, wherein the specific private network address is also assigned to the extended traffic intermediaries of the one or more extended traffic intermediaries; and

[0209] In response to an indication received from the one or more extended traffic intermediaries that a client has submitted a programming request to the virtualization computing service, one or more configuration operations of the computing instance are performed at the target hardware server, wherein the programming request was submitted using a path that does not include the one or more secure channels.

[0210] 22. The system according to Clause 21, wherein the computing instance is a bare metal instance, and from the bare metal instance, one or more hardware resources of the target hardware server can be accessed without virtualization.

[0211] 23. The system according to any one of Clauses 21-22, wherein the target hardware server is presented as the virtualization host in response to a programming request received from a client, wherein the programming request indicates one or more attributes of the target hardware server.

[0212] 24. The system pursuant to any one of Clauses 21-23, wherein the one or more secure channels comprise a VPN (Virtual Private Network) tunnel.

[0213] 25. A system according to any one of Clauses 21-24, wherein the peripheral card is linked to the peripheral device using one or more of the following: (a) a PCIe (Peripheral Component Interconnect Fast) connector, (b) a connector configured to multiplex PCIe signals with one or more other types of signals, or (c) a USB (Universal Serial Bus) connector.

[0214] 26. A method comprising:

[0215] A secure communication channel is established between a first extension manager of a virtualized computing service and one or more extension traffic brokers of the virtualized computing service, wherein the one or more extension traffic brokers are configured within one or more data centers of a provider network, and wherein the first extension manager is implemented at a first peripheral device located at a first site outside the provider network;

[0216] The first extension manager allocates a first network address of the underlying network of the virtualization computing service to a first target hardware server located at the first location, wherein the first network address is also allocated to an extension traffic intermediary of the one or more extension traffic intermediaries; and

[0217] In response to a command for the virtualized computing service, the first extension manager causes one or more computing instance configuration operations to be performed at the first target hardware server, wherein the instruction for the command is obtained from the one or more extension traffic intermediaries at the first extension manager.

[0218] 27. The method described under Clause 26 further includes:

[0219] A second network address from a range of isolated virtual networks established by a client representing the virtualized computing service is assigned to a first computing instance launched at a first target hardware server, wherein another network address from the range is assigned to a second computing instance launched on a virtualized host located in one or more data centers.

[0220] 28. The method according to any one of clauses 26-27 further comprises:

[0221] Establish another secure communication channel between the first extension manager and the second extension manager, wherein the second extension manager is implemented at a second peripheral device located at the first site; and

[0222] Network data packets originating from the first target hardware server are transmitted to the second target hardware server at the first location via the other secure communication channel.

[0223] 29. The method according to any one of Clauses 26-28, wherein the provider network includes resources for a plurality of network-accessible services, the network-accessible services including the virtualized computing service and another service, the method further comprising:

[0224] Instantiate the instance metadata broker on the first peripheral device; and

[0225] The instance metadata broker provides credentials authorizing access to the other service from the first compute instance.

[0226] 30. The method according to any one of clauses 26 to 29 further comprises:

[0227] One or more compute instances are configured by the first extension manager at a first target hardware server within an isolated virtual network established on behalf of the client of the virtualized compute service; and

[0228] One or more computing instances are configured by a second extension manager at a second target hardware server within the isolated virtual network, wherein the second target hardware server is located in a second location outside the provider network.

[0229] 31. The method according to Clause 30, wherein the one or more computing instances at the first target hardware server are assigned network addresses within a first subnet of the isolated virtual network, and wherein the one or more computing instances at the second target hardware server are assigned network addresses within a second subnet.

[0230] 32. The method according to any one of clauses 26-30 further comprises:

[0231] The extension manager performs one or more operations of the encapsulation protocol for (a) a first data packet transmitted from another device located in a first external location to a computing instance instantiated at a first target hardware server, wherein the other device is assigned an address within a client network outside the provider network, and (b) a second data packet transmitted from the computing instance to the other device located in the first external location.

[0232] 33. The method according to any one of clauses 26-30 or 32 further comprises:

[0233] In response to a programming request, a stream log is provided that indicates one or more data packets transmitted to or from a compute instance initiated at the first target hardware server.

[0234] 34. The method according to any one of clauses 26-30 or 32-33 further comprises:

[0235] In response to a programming request, a representation of the security policy of the storage endpoint is provided, through which traffic originating from a compute instance launched at the first target hardware server will be routed to another network-accessible resource; and

[0236] In response to a data packet transmitted from the computing instance to a resource, the security policy is used to determine whether to forward the data packet to the other network-accessible resource.

[0237] 35. The method according to any one of clauses 26-30 or 32-34, wherein the first peripheral device is connected to the peripheral card of the first target hardware server.

[0238] 36. A peripheral device, comprising:

[0239] One or more processors and one or more memories, wherein the one or more memories include program instructions for an extension manager that implements virtualized computing services of a provider network when executed on or across the one or more processors; and

[0240] It can connect to one or more ports of a hardware server;

[0241] The extension manager is configured as follows:

[0242] Establish a secure network channel for communicating with one or more computing devices of the virtualized computing service, wherein the one or more computing devices are located in the data center of the provider network, and wherein the peripheral devices are located in a first location outside the provider network;

[0243] Assign a network address to the underlying network of the virtualization computing service to a first target hardware server located in the first location and connected to one of the ports, wherein the first network address is also assigned to an extended traffic intermediary at the one or more computing devices; and

[0244] In response to a command for the virtualized computing service, one or more computing instance configuration operations are performed at the first target hardware server.

[0245] 37. The peripheral device according to Clause 36 further includes persistent storage, wherein the one or more memories include additional program instructions that, when executed on or across the one or more processors, instruct the memory manager to:

[0246] Provides access from a first target hardware server to a storage volume of a computing instance launched at the first target hardware server, wherein at least a portion of the contents of the storage volume is stored at the persistent storage device.

[0247] 38. The peripheral device according to any one of clauses 36-37, wherein the extension manager is further configured to:

[0248] Establish another secure communication channel with another extended manager implemented at the second peripheral device located at the first site; and

[0249] Network data packets originating from the first target hardware server are transmitted to the second target hardware server at the first location via the other secure communication channel.

[0250] 39. The peripheral device according to any one of clauses 36-38, further comprising a persistent storage device and a removable security device, wherein removing the removable security device from the peripheral device renders at least a portion of the persistent storage device unreadable.

[0251] 40. The peripheral device according to any one of Clauses 36-39, wherein the first target hardware server is connected to one of the ports via one of the following: (a) a timer card or (b) a repeater card.

[0252] in conclusion

[0253] Various embodiments may also include receiving, transmitting, or storing instructions and / or data implemented in accordance with the foregoing description on a computer-accessible medium. Generally, a computer-accessible medium may include storage media or memory media (such as magnetic or optical media, e.g., magnetic disks or DVD / CD-ROMs), volatile or non-volatile media (such as RAM (e.g., SDRAM, DDR, RDRAM, SRAM, etc.), ROM, etc.), and transmission media or signals (such as electrical signals, electromagnetic signals, or digital signals transmitted via communication media (such as networks and / or wireless links).

[0254] The figures and various methods described herein represent exemplary embodiments of the methods. The methods can be implemented in software, hardware, or a combination thereof. The order of the methods can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0255] It will be apparent to those skilled in the art who benefit from this disclosure that various modifications and changes can be made. It is intended to encompass all such modifications and changes, and therefore the above description is to be considered illustrative rather than restrictive.

Claims

1. A method for managing virtual machines on a remote computing device, comprising: This enables the virtualization management component to be implemented on at least one of one or more computing devices located outside the underlying network of the cloud service provider; In response to receiving a request from a customer at the customer interface of the virtualized computing service in the underlying network of the cloud service provider, one or more configuration operations are performed to instantiate a virtual machine on at least one of the one or more computing devices located outside the underlying network of the cloud service provider. as well as This enables the execution of one or more operations to manage the virtual machines instantiated on the computing devices located outside the underlying network of the cloud service provider.

2. The method for managing virtual machines on a remote computing device according to claim 1, further comprising: Establish a virtual private network (VPN) connection between the endpoint of the cloud service provider's network and the endpoint associated with the virtualization management component; as well as One or more management tasks are sent from the control plane of the virtualization computing service to the virtualization management component via the VPN network, wherein the one or more management tasks at least partially cause the one or more configuration operations to be performed or the one or more operations for managing the virtual machine to be performed.

3. The method for managing virtual machines on a remote computing device according to claim 1, further comprising: One or more management tasks are sent from the control plane of the virtualization computing service to the virtualization management component via a direct physical connection between the cloud service provider's underlying network and the network associated with the virtualization management component, wherein the one or more management tasks at least partially cause the execution of the one or more configuration operations or the execution of the one or more operations for managing the virtual machine.

4. The method for managing virtual machines on a remote computing device according to claims 1 to 3, The embodiment wherein the virtualization management component is implemented on at least one of one or more computing devices located outside the underlying network of the cloud service provider includes: The at least one computing device is provided to the cloud service provider’s customers, wherein the at least one computing device stores program instructions for implementing the virtualization management component.

5. The method for managing virtual machines on a remote computing device according to claims 1 to 3, further comprising: Provide the virtual machine with access to block storage services.

6. The method for managing virtual machines on a remote computing device according to claims 1 to 3, further comprising: Assign IP addresses within the range of virtual network IP addresses from the underlying network of the cloud service provider to the virtual machine.

7. The method for managing virtual machines on a remote computing device according to claim 6, wherein the virtual network comprises: The virtual machine implemented on one or more computing devices at the location located outside the underlying network of the cloud service provider; as well as One or more second virtual machines implemented on one or more computing resources within the underlying network of a cloud service provider.

8. The method for managing virtual machines on remote computing devices according to claims 1 to 3, wherein the virtualization management component implemented on at least one of the one or more computing devices located at a location outside the cloud service provider's underlying network serves as a bridge between the cloud service provider's underlying network and the network associated with the virtualization management component implemented at the location outside the cloud service provider's underlying network.

9. The method for managing virtual machines on a remote computing device according to claims 1 to 3, wherein the client interface includes one or more of the following: Web-based console; Command-line tools; Graphical user interface; or Application Programming Interface (API).

10. The method for managing virtual machines on remote computing devices according to claims 1 to 3, wherein the one or more computing devices located at a location outside the underlying network of a cloud service provider operate in a multi-tenant mode.

11. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more processors, cause the one or more processors to: At the client interface of the virtualized computing service within the cloud service provider network, requests from customers to implement virtual machines on computing devices located outside the cloud service provider network are received; and This enables the execution of one or more configuration operations to instantiate the virtual machine on the computing device located at the site outside the cloud service provider's network.

12. The one or more non-transitory computer-readable storage media of claim 11, wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to: A list of virtualization hosts available for instantiating virtual machines is presented, wherein the computing devices located at the venue outside the cloud service provider network are included in the list.

13. One or more non-transitory computer-readable storage media according to claims 11 to 12, wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to: Perform one or more operations to provide the virtual machine instantiated on the computing device located outside the service provider network with access to the contents of a volume stored in the storage service of the cloud service provider network; The contents of that volume are used to boot the operating system of the virtual machine.

14. One or more non-transitory computer-readable storage media according to claims 11 to 12, wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to: The virtual machine of the virtualized computing service is connected to the computing device located at the site outside the cloud service provider's network. The virtual machine of the virtualized computing service, which connects to the computing device outside the cloud service provider network, acts as a bridge between the management component of the virtualized computing service and the virtual machine instantiated on the computing device located outside the cloud service provider network.

15. One or more non-transitory computer-readable storage media according to claims 11 to 12, wherein the program instructions, when executed on or across the one or more processors, further cause the one or more processors to: One or more IP addresses are assigned to the virtual machine such that the virtual machine instantiated on the computing device at the location outside the cloud service provider network is included in the virtual network of the cloud service provider network's customers.

Images