Provider network service expansion

Extended Resource Groups with secure management components address the challenge of extending virtualized computing services outside the data center, ensuring low-latency and secure access while maintaining network integrity.

JP7581390B2Active Publication Date: 2024-11-12AMAZON TECH INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023002875
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-11-20
Filing Date
2023-01-12
Publication Date
2024-11-12
Estimated Expiration
2039-11-19

AI Technical Summary

Technical Problem

Existing virtualized computing services face challenges in securely extending network resources to locations outside the data center, particularly for latency-sensitive and security-critical applications, without compromising the security and performance of the provider network.

Method used

The implementation of Extended Resource Groups (ERGs) that include hardware, software, and firmware preconfigured to establish virtual machines outside the provider network, utilizing an Outbound Command Communicator (OCC) and ERG Network Manager (ENM) to securely manage and control these resources, ensuring low-latency communication and enhanced security.

Benefits of technology

ERGs enable secure, low-latency access to virtualized computing services at customer locations, maintaining network security and performance, even under incomplete connectivity, by using secure networking channels and tamper-resistant storage devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007581390000001
    Figure 0007581390000001
  • Figure 0007581390000002
    Figure 0007581390000002
  • Figure 0007581390000003
    Figure 0007581390000003
Patent Text Reader

Abstract

In a provider network, a virtual computing service includes a control plane server receiving a request to establish a compute instance, a first outbound command communicator assigned a first network address, a first outbound command communicator receiving an indication that the compute instance will be established on a target host in response to the request, and a second network address being a part of a first network established on the client premises. The message includes a command to establish the compute instance on the target host. The first network address is assigned to the target host within the second network established on the client premises. Effect: Processing of the command on the target host results in the establishment of a compute instance.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Many businesses and other organizations choose to have their computing systems co-located or (e.g. as part of a local network), or alternatively in several different geographical locations. Place them in strategic locations (e.g., one or more private or public intermediate networks) interconnect many computing systems (such as through a Operating a computer network that supports a business, e.g., by a single organization and private data centers operated on behalf of a single organization, as well as A provider of computing resources operated by an entity as a business. A significant number of interconnected computing systems, such as brick data centers Data centers that accommodate the above are becoming more common. Among public data center operators, provides network access, power, and secure configuration to various customer-owned hardware. Some provide installation facilities, while others provide hardware resources made available for use by customers. Some offer "full service" facilities, including laundry.

[0002] With the advent of commodity hardware virtualization technology, Many customers benefit from managing computing resources at scale, Allows multiple customers to share various computing resources efficiently and securely For example, in virtualization technology, one or more virtual machines hosted by a single virtualization host. By providing multiple "guest" virtual machines to each user, a single physical virtualization host can be used for multiple Each such virtual machine may be capable of being shared among multiple users on a given hardware Gives users the illusion of being the sole operator of the hardware computing resources. while providing application isolation and security across various virtual machines. A software simulation that acts as a separate logical computing system, providing It can represent a multi-threaded application that instantiates several different virtual machines on the same host. This also helps increase the overall hardware utilization level of the data center. , which may result in a higher return on investment.

[0003] Different types of applications, such as compute-intensive applications, memory-intensive applications, etc. A wide variety of virtual machine types optimized for applications meet client demands. In response, several cloud computing provider networks have In addition, virtual machines for virtual computing services can be set up Some database servers use rely on virtual computing services from such provider networks, such as Higher level services that exist are also made available to provider network clients. However, it may be necessary to store it on the customer premises outside the provider network. Some types of applications, such as those that process very large amounts of data, The application uses hardware located in the provider network's data centers. Services limited to providing virtualized resources using the / or may be suboptimal for other reasons. [Brief description of the drawings]

[0004] [Figure 1] Illustrates an example system environment in which an extended resource group of a virtualized computing service of a provider network may be set up using resources located on premises outside the provider network, according to at least some embodiments. [Diagram 2] 1 illustrates an example overview of an enhanced security virtualization host that may be included in an extended resource group of a virtualized computing service, according to at least some embodiments. [Diagram 3] 1 illustrates example elements of an offloaded virtualization management component card of a virtualization host, according to at least some embodiments. [Figure 4] 1 illustrates example attributes of a virtual network interface that may be programmatically connected to an outbound command communicator associated with an extended resource group, in accordance with at least some embodiments. [Diagram 5] 1 illustrates an example path that may be used to send a command generated in a control plane of a virtualized computing service to an extended resource group, according to at least some embodiments. [Figure 6] 1 illustrates example elements of an extended resource group network manager, according to at least some embodiments. [Figure 7] 1 illustrates an example family of virtual machines and extended resource groups that may be supported in a virtualized computing service, according to at least some embodiments. [Figure 8] 1 illustrates example programmatic interactions for extended resource groups of a virtualized computing service, according to at least some embodiments. [Figure 9] 1 illustrates an example workflow associated with starting a virtual machine in an extended resource group, according to at least some embodiments. [Figure 10] 1 illustrates an example of the use of additional provider network services in an extended resource group of a virtualized computing service, according to at least some embodiments. [Figure 11] FIG. 11 is a flow diagram illustrating aspects of operations that may be performed in a provider network to establish an extended resource group and enable virtual machines to be set up in the extended resource group, according to at least some embodiments. [Figure 12] FIG. 1 is a flow diagram illustrating aspects of operations that may be performed on an extended resource group of a virtualized computing service, according to at least some embodiments. [Figure 13] FIG. 1 is a block diagram illustrating an example computing device that may be used in at least some embodiments.

[0005] The embodiments are described herein by way of example in some embodiments and illustrative drawings. However, those skilled in the art will recognize that the embodiments are not limited to the embodiments or drawings described. The drawings and detailed description are not intended to limit the embodiments to the particular forms disclosed. Rather, the intent is to provide a method and apparatus for the preparation of a medicament for use in a computer-implemented ... All modifications, equivalents, and alternatives falling within the spirit and scope of the It should be understood that the headings used herein are for organizational purposes only. , are not meant to be used to limit the scope of the specification or claims. As used throughout this application, the word "may" means The meaning of the term is not necessarily the meaning of the term, but the meaning of the term Similarly, "include" and "include ng) and the words "includes" mean but are not limited to As used in the claims, the term "or" means an inclusive or. used as a corollary, not as an exclusive or. For example, "x, y, or z" The phrase "at least one of" refers to any one of x, y, and z, as well as Any combination of these is meant. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] The operation of the provider network virtualized computing services is controlled by the provider network. Method and apparatus for securely extending a network to a location outside of a data center Various embodiments are described. The network may be the Internet and / or other networks. One or more services ( Various types of multi-tenant and / or single-tenant cloud-based computing To provide services such as transportation, storage, or other services to companies or public institutions. Any company that uses the described technology will In this situation, clients of virtualized computing services can use the customer data center, Field location (e.g., for a client who operates a telecommunications company, mobile Set up in a data center near the tower or in the provider network data center A virtual machine with at least some of the same characteristics and capabilities as the types of virtual machines that can be Use hardware devices located in colocation facilities to host virtual machines. It is possible that this is possible.

[0007] In at least some embodiments, hardware located outside the provider network Hardware devices are distributed in various categories by, for example, provider network operators. to support multiple virtual machines in the same location, and with other local devices Supports low latency communications for the cloud without compromising the security of customer data and applications without compromising the security of the provider network itself. To do so in a manner that does not compromise the security of any other customers of the network Any suitable combination of hardware, software and / or firmware elements In at least some embodiments, the "Extended Resource Groups" ( The term ERG refers to resources located on premises outside the provider network. (e.g., hardware, software, firmware, configuration metadata, etc.) can be used to refer to a virtual machine that is established and utilized on the premises. The premises where the extended resource group is established (e.g., customer data center, field A central location (e.g., a central office location) may be referred to as a client premises in various embodiments. In fact, from the perspective of a Virtual Computing Service (VCS) customer, various implementations In this mode, the ERG can represent a local extension of the capabilities of the VCS, which can be used to It can be set up at any desired physical location with access to the hardware A set of hardware devices can be accommodated (e.g., physical space, power, etc.) From the perspective of the VCS itself, the ERG is physically located at the customer's chosen premises. while sharing the same provider network data as the core VCS infrastructure. In some embodiments, the various In addition to or instead of the standardized or customizable virtual machines in the category In addition, other types of programs, such as bare metal instances and software containers, can be deployed on the same machine. The program execution environment is provided to support one or more computing services of the provider network. In one embodiment, the VCS-supported Bare metal instances are virtualized instances that run on the same hardware as the host they are used on, without the use of virtualization. Various processes of the operating system that directly control at least a portion of the hardware (although a small subset of hardware features may be virtualized in some cases) In such an embodiment, a dedicated hypervisor for virtualization management is not required. The bare metal instance itself may not be a virtual machine in some embodiments. The term "compute instance" may include functionality for hosting In this state, resources are used in the provider network data centers and / or Use ERGs to set up and administer provider network services on behalf of your clients. All different kinds of execution environments that can be managed (virtual machines, bare metal instances, software This term may be used to refer collectively to all software applications, including software containers, that use the In this paper, virtual machines are considered as the primary example of a compute instance, but in ERG, virtual machines The techniques described for setting up and using the application may be used in various embodiments for other types of applications. Note that this can be applied with equal success to computational instances of the same group.

[0008] Virtualized Computing Service (VCS) resources, in various embodiments, may include: The resources are divided into two categories: control plane resources and data plane resources. Control plane resources are primarily used for management purposes, e.g. Monitor resource usage and status to establish isolated virtual networks for different customers Identify the specific virtualization host or server on which the requested virtual machine is started in order to monitor the To provision additional hardware as needed, etc. Data plane resources are primarily used by client applications, e.g. unmanaged activities, such as performing client application In various embodiments, the ERG can be used for customer-selected location management tasks. When set up in a datacenter, the ERG's resources are managed by the datacenter of the provider network. The VCS can be managed by the control plane component located on the VCS host. Thus, in at least some embodiments, an ERG is set up at a given premises. Upgrading and using VCS replicates the control plane capabilities locally. Instead, they may not require a data center in the provider network to Set up a secure network connection for control plane commands sent to the RG. and ERG resources can be devoted primarily to data plane operations. .

[0009] A VCS that wants to start using ERG in a specific location, such as a specific customer data center CDC1 Customer C1, in at least some embodiments, participates in a workflow similar to the following: C1 can do so by first setting up a physical ERG configuration type. (e.g., a "small" ERG configuration that contains a half rack of standard size servers, a full rack of servers a "medium" ERG configuration that includes a rack of servers, and a "large" ERG configuration that includes two full racks of servers. Optionally select a specific configuration to be set up at the desired location or premises. By default, if C1 does not want to explicitly select an ERG configuration type, In some embodiments, a particular configuration type (e.g., a "small" configuration) may be configured for a client. can be selected, thereby further reducing the effort required from C1. Next, The request may be sent via a programmatic interface (e.g., a web-based console, application Application Programming Interface (API), command line tools, graphics After the request is approved, the request may be submitted to the VCS via the VCS's , ERG hardware (with appropriate software / firmware pre-installed) The vehicle can be transported to a desired location, plugged into a power source, and connected to the Internet. .

[0010] In at least some embodiments, the network manager detects one or more trigger conditions. In response to detecting a condition (such as detecting power and / or Internet connectivity), Such an ERG network may then be instantiated in one or more of the following servers: An ENM, for example, can be configured to use the At least one provider network data center, one or more VCS components ( For example, the automatic establishment of secure network connections with The ERG (CDC1) can initiate (or at least participate in) the automatic establishment of ) and VCS components (at provider network data center(s)) After a connection is established between C1 and the For virtual machines that use only provider network resources, such commands Instantiate virtual machines that use ERG resources in the same manner as they are published (and / or use the virtual machine to perform other operations). From CDC1's point of view, the functionality of the VCS is the local resources in CDC1 (and optionally Provider network resources located in data centers) Virtual machines (VMs) set up in an ERG can be used in a variety of ways. In one embodiment, non-VCS devices located in CDC1 and, if necessary, It can communicate with other VMs set up in the datacenter ( For example, performing address translation and / or other encapsulation protocol related processing. With the help of ENMs that can M, and associated higher-level projects using such VMs as building blocks. The higher level of service may, in some embodiments, be provided by a provider network data center. It can continue to function even during periods when connectivity to the network is temporarily interrupted. The company will continue to access and process large amounts of application data stored in customer data centers. For VCS customers who wish to manage their own For these reasons, you can set up a VCS VM co-located with the application data. The ability to do this can be very beneficial in various embodiments.

[0011] As one of ordinary skill in the art will appreciate in light of this disclosure, certain embodiments include the following: In a location-independent manner, for example, clients of virtualized computing services It uses resources located on the premises selected by the provider, and The scalability, security, availability, and and other operational benefits while supporting a wide variety of virtualized computing applications. (b) the amount of application data and the customer’s Long distance links such as between customer data centers and provider network data centers (c) reducing the need to transfer data between the application and the data source / data By moving closer to the destination, potentially large amounts of data are consumed as input. The overall latency and responsiveness of the application, whether consumed or produced as output and / or (d) improve the security of sensitive application data. To achieve various benefits, including some or all of the following: In various embodiments, the design and implementation of Extended Resource Groups (ERGs) may be possible. The guiding principles and goals underlying the implementation are, inter alia: (a) the availability of computational and data resources; (b) enabling reduced latency computation based on exploiting locality; Complete logging by restricting network traffic to where data is stored. (c) ensuring the security of Customer Data by using ERG resources; We provide first-class support for various provider network services (provider Support is possible using resources located in the Datnetwork data center. (d) providing a provider network with its own data centers; protection from potential hostile actors (e.g., by operating in ERGs); and / or (e) Even under conditions of incomplete network connectivity from an ERG to a provider network, Supporting the ongoing performance of services in the ERG.

[0012] Targeted access to customer data and applications, as well as to provider networks To achieve a high level of security for Several network configuration techniques, including the use of virtual network interfaces (VNIs), and other types of configuration settings) may be utilized in different embodiments. As the name suggests, in various embodiments, an IVN is a network configuration The remainder of virtualized computing services, at least for some types of configurations An IVN may include a set of resources that are logically separate or isolated from other resources. , for various purposes, in or on virtualized computing services may be established by the computing service (and in some cases, the provider network) (This extends to ERGs located outside the IVN data center.) can be set up for a specific customer by reserving a set of resources exclusively for that customer. This allows the customer a great deal of flexibility in terms of network configuration of that set of resources. Within the IVN, customers set up subnets and configure the desired private I Security management that assigns IP addresses to various resources and manages incoming and outgoing traffic. In at least some implementations, In this state, by default, the private network is set up in one IVN. A set of addresses can be accessed from another IVN (or, more generally, from outside the IVN). In at least some embodiments, the VCS may not be available to non-provider One of the security mechanisms used to extend the network premises is , outbound commands (using resources in provider network data centers) Set one or more IVNs that contain a pool of OCC instances. This may include updating the VCS control plane servers and devices. Such communications may remain logically (and / or physically) separated. In various embodiments, the data processor may communicate with the VCS controller using a secure networking channel. It can be used to send management commands from the role plane to the ERG. In this state, management messages are sent from the ERG to the controller over a secure networking channel. Configure the OCC to ensure that the aircraft cannot be returned to the troll plane. Therefore, administrative actions that may affect other customers can be In at least some embodiments, individual O A CC instance is a single instance of a virtual machine connected to it, as discussed in more detail below. The virtual machine may include a virtual machine with one or more virtual network interfaces. The Virtual Network Interface (VNI), in various embodiments, is a programmable Public and / or private IPs (interfaces) that can be connected or associated Network properties including network protocol (IP) addresses, security settings, etc. A set of OCC instances, IVNs, and VNIs can be used to Details of the manner in which virtualized computing capacity is provided through the provision of the .

[0013] According to some embodiments, the system includes a virtualization computer of a provider network. The VCS may include one or more computing devices. A computing data provider network may include one or more data centers. The device may include instructions that, when executed on the processor, result in a computing device. In the first isolated virtual network (IVN), Instantiate an Outbound Command Communicator (OCC) associated with the In some embodiments, the ERG may include one or more data sets of the provider network. The virtualization system may include at least a first virtualization host located on a first premises outside the data center. The OCC, in various embodiments, allows commands to be submitted from the ERG. It can act as an intermediary to securely forward management commands to the ERG without requiring the The configuration is based on the assumption that at least some types of network packets from the ERG are The instructions may also be prevented from being sent to the VCS control plane via Upon receiving the request, the computing device (a) receives a notification from one or more of the provider networks. (b) a portion of a second isolated virtual network that includes resources in the data center; between one or more resources located at a first premises outside the internetwork A secure network channel (e.g., a VPN or Virtual Private Network) A network tunnel or connection, or a connection where traffic flows over a dedicated physical link Such a secure networking channel may be used, for example, For example, an ERG network manager in a provider network data center and a customer A customer IVN may be established.

[0014] OCC uses the virtual machine start command (in response to a program request from the VCS client) In various embodiments, the response (generated in response to a request) may be obtained or determined. These program requests, in at least some embodiments, are sent over a secure network channel. It may be received from a client via a path that does not include a command Requests are submitted through a web-based console and delivered to provider network data centers. Management commands submitted by clients for virtual machines set up within The instructions can be sent over the same kind of paths that are used for the other commands. In an embodiment, a representation of a virtual machine startup command is stored in a secure By having the first OCC send the extended resource group over a network channel. In at least some embodiments, the processing of the representation on the first virtualization host can include: resulting, at least in part, in instantiation of the virtual machine on the first virtualization host. The configuration controls the flow of commands from the VCS control plane out to the ERGs. Commands cannot be submitted from the ERG due to the method being limited to the bounding direction only Therefore, various embodiments may enhance the security of the VCS and its customers. This may impact operations in the VCS data center or other ERGs. In addition to the startup command, other administrative commands (for example, shutting down a virtual machine in an ERG) commands to view or change the properties of virtual machines in an ERG (e.g., ISPs, etc.) must use secure network channels established by the OCC and ERGs. It may be sent from the VCS control plane on behalf of the customer.

[0015] In at least some embodiments, the OCC determines whether the player selected by the client By mistake, the ERG hardware equipment may be set up before the delivery time. For example, in one embodiment, the time when a VCS customer orders an ERG via a program request and the The time it takes for the RG hardware to be delivered to the premises and set up, and the time it takes to establish the ERG Due to the delay between receipt of the program request and the Proactive configuration or instantiation of specially designated OCCs may be triggered A unique identifier may be generated for each ERG request, and in some embodiments, the ERG The identity of the OCC may be stored in a metadata entry accessible to the OCC designated for the The identifier may also be included in one of the hardware devices shipped as part of the ERG. Therefore, when a connection is established between the VCS and the ERG, the ERG bang is In at least some embodiments, the network manager of the ERG initiates the establishment of a secure network channel (such as a VPN tunnel) with the VCS. The OCC will then request that the Inst In some embodiments, the OCC may The present invention may be implemented using virtual machines in a data center.

[0016] In at least some embodiments, the control of the VCS in a service provider environment The rule plane allocates a specific range of service network addresses SAR1 for virtualization hosts. (The virtualization host is located in the service provider's data center.) whether it is part of an ERG set up on the client premises Within the provider network data center, the control plane A managed network consists of multiple servers configured to host compute instances. A Virtual Network Interface (VNI) and / or or other techniques, in at least one embodiment, by the control plane. The address Addr1 assigned to the target virtualization host H1 in the ERG is also Assigned to a specific OCC that is used to forward control plane commands to the ERG In such an embodiment, from a control plane perspective, OCC (Service Control Cell) Can it be configured in the same network as the virtualization hosts at the service provider data center? , or may be coupled to it) can in effect represent the host H1. To launch a compute instance on the ERG, use Addr1 as the application address: The first message including the request for the control It can be sent from the plane to the OCC.

[0017] In some embodiments, a network manager (ENM) of a target ERG: At least a second network address, Addr2, may be assigned. may be working with more than one device (e.g. the hardware cards discussed above) Addr2 may be, for example, a customer data center or other client premises. An address in the first established network that provides a client address to the device. Use the address range CAR1. The first message from the control plane at Addr1 In response to receiving the message, the OCC, in various embodiments, launches a compute instance. and preparing a second message including at least a portion of the request to Send 2 as the destination address to the ENM. Receive the second message. and the device on which the ENM operates receives at least a portion of the second message (e.g., a computation The virtualization host H1 can then forward the virtualization host H2 to the target virtualization host H1. In some embodiments, at least two network address ranges are provided: The first network uses the range CAR1, and various hosts / servers such as H1, At least the subrange of SAR1 (used by the control plane to virtualization hosts) A second network (that is, an ENM) that is assigned addresses from the address range (which may be a private network set up by a In such an embodiment, the target The virtualization host H1 is assigned the same address, Addr1, as assigned to the OCC. The ENM uses Addr1 as the destination address to A message can be sent to the target virtualization host, effectively being handled by the control plane. was initially directed to destination address Addr1, but along the path to the ERG may have been sent using one or more other addresses as the destination. The command / request to launch a compute instance is again directed to the destination address within the ERG. Address Addr1, and therefore in some embodiments, at least two The ENM itself may be implemented in some embodiments. In this case, the client premises may be part of both networks, e.g. The devices that operate at least partially are in networks where CAR1 is used and in networks where H 1 (where SAR1 is used) are assigned addresses. When host H1 receives the message sent by the ENM, it An operation may be executed, for example a compute instance may be launched on H1.

[0018] In at least some embodiments, the VCS provides a representation of a set of virtual machine categories. Any number of instances can be provided at the request of the VCS client. It may be established in the provider network data center and / or in the ERG. For example, client C1 is connected to a category C data center in the provider network. Request N VMs of at1 and assign them to the same category in the ERG set up on behalf of C1. In some embodiments, the VM may require at least M VMs in Cat1. Some categories may only be supported by ERGs or by the provider network. This may only be supported in global data centers.

[0019] The OCC, in some embodiments, may also use the virtual machine to You can perform one or more transformations on a thin configuration command (for example, a startup command). For example, the version of a command retrieved from the VCS control plane in OCC. The option, in some embodiments, may include one or more security tokens (e.g., It can be used to verify the identity of the requester of the resulting operation. and the requester may include a token having the necessary permissions to request the operation. The OCC may either strip the token from the version of the command forwarded to the ERG, or In at least some embodiments, the first security token may be omitted. The Kun can be transformed (e.g., from what was originally used in the VCS control plane) different keys) and the command is directed by the VCS control plane A transformed version of the security token that can be authenticated on the target host may be included in the forwarded version of the command. Each message authentication code (e.g., a hash-based message authentication code or HMAC) is generated for outbound commands sent from the OCC to the ERG. In various embodiments, the OCC may collect all outgoing Bound communication messages can be logged, and logged messages are Can be reviewed by the client and, if necessary, by the ERG on behalf of the client. In some embodiments, at least two virtual network interfaces are set up. A face can be associated with a given OCC, one from the VCS control plane. one is used to obtain commands, and the other is used to communicate over a secure network channel. Used to communicate with the ERG via the

[0020] According to some embodiments, the system includes a virtualization computer of a provider network. An ERG may include an Extended Resource Group (ERG) for a Virtual Channel Service (VCS). In various embodiments, a tamper-resistant storage device (e.g., a device that is not physically connected to a host) (cryptographic device that can only be decrypted if the The virtualization server may include at least a first virtualization host (VH) including a platform module. The integrator network may include one or more data centers, and the first VH may include at least one In some embodiments, it may be located on premises outside the provider network. A computing device may include instructions that, when executed on a processor, a computing device configured to receive and transmit the ERG's network master in response to one or more trigger signals; The ERG Network Manager (ENM) instantiates various In an embodiment, at least a portion of a first isolated virtual network (IVN) of the VCS Start configuring a secure network channel for communication with one or more VCS resources in Such an IVN can be, for example, a data center in a provider network. The second VH instantiated on the virtual machine may include at least a first virtual machine (VM). do.

[0021] The ENM, in at least some embodiments, communicates over a secure network channel. The virtual machine start command is obtained from the OCC associated with the extended resource group. The virtual machine start command may, in various embodiments, be Responding to program requests from clients sent over routes that do not include channels From the ENM, the virtual machine At least some versions of the application startup command may, in some embodiments, be The virtual machine may be started on the first virtualization host of the group. Processing the command may, in such an embodiment, involve creating a new virtual machine on the first virtualization host. This may result in the instantiation of

[0022] Any of several types of trigger signals may be used in various embodiments to trigger an instance of an ENM. For example, if a particular server in an ERG is powered on, or a signal that the ERG has enabled a connection to the Internet. In some embodiments, instantiation, initialization, and and / or could lead to a startup.

[0023] Acting as an intermediary between ERGs and VCS resources in the provider network In addition, in at least one embodiment, the ENM is also set up within the ERG. The virtual machines that are configured to run the ERG are (a) deployed on the premises where the ERG is set up, and (b) For example, a VCS client can act as an intermediary between a VCS and resources that are not part of the is a server that stores data that is processed using the ERG virtual machine, separate from the ERG. The ENM has a set of servers and enables communication between that set of servers and the ERG virtual machines. Such servers may, at least in some embodiments, be considered non-VCS servers. To provide connectivity, the ENM may, in some embodiments, Various types of packet transformation for outbound traffic to non-VCS servers and performs reverse translation on traffic inbound to the ERG from non-VCS servers. Conversion may be performed, for example, in different embodiments, by IP address mapping, encapsulation, etc. In some embodiments, the ERG may include a hosted protocol. What is the entity responsible for managing ERG to non-VCS resources on the premises? Different entities may be responsible for managing the communication from the ERG to the VCS. That is, a single ENM may not necessarily be responsible for both types of functions. do.

[0024] In one embodiment, the ERG also initializes / boots the ERG's virtualization hosts. In some embodiments, the ENM may include a bootstrap manager that is responsible for At least some implementations can perform bootstrapping / initialization tasks. In this form, after a VM is established in the ERG host, it is The operations of the service at the provider level can be performed using those VMs. The network is built on a single database instance using VMs as building blocks. Such a database instance may include a database service that is configured with It can be configured using a VM set up in the ERG.

[0025] In at least some embodiments, the virtualization host of the ERG is a trusted platform. Such security may include a security module such as a Total Proof of Concept (TPM). The security module instantiates a VM (or other Before performing any virtualization-related operations, check the state of the ERG host (for example, installed software, It can be used to verify that a certain set of requirements (such as software state) meets acceptance criteria. In at least one embodiment, state information associated with a VM launched on an ERG virtualization host. The ERG may be stored in encrypted form on the storage device of the virtualization host. As long as it is physically present or connected to a particular virtualization host, it can be decrypted. Tamper-resistant storage devices, such as storage devices that can be In the embodiment, it is possible to employ a method for storing encrypted data on a storage device. In such an embodiment, if the device is physically removed, the encrypted state information is not erased. Unable to decrypt from a removed device and does not support virtualization features supported by ERG Sign security will be further strengthened.

[0026] In some embodiments, for a given ERG, only those users / entities Allow or allow users to set up and use virtual machines in an ERG. specifies a set of users or entities defined in an identity management service. A given VM may be started (or other VM-related operations may be performed) in the ERG. Previously, the identity of the requesting user or entity may be determined in various embodiments. The verification can be done, for example, on the ENM, the OCC, or on the virtualization host on which the VM is started. In at least one embodiment, to help speed up VM startup in the ERG, Then, a set of VM images is stored or cached on the ERG's local storage device. The ERG can then be configured to start a VM of the desired category in the network. In some embodiments, it may not be necessary to transfer large images over the Internet. The image is stored on the ERG's computing device in the location where the ERG is set up ( For example, the data may be cached prior to physical delivery to a customer data center.

[0027] Example System Environment FIG. 1 illustrates a virtualized component of a provider network, according to at least some embodiments. The extended resource group of a computing service can access a provider outside the provider network. FIG. 1 illustrates an exemplary system environment that can be set up using resources located at a given system. As shown in the figure, in the system 100, a virtualized computing service (V The core components of the CS110 are at least the control plane resources14 1, data plane resources 145, and outbound command communicators (O A set of CCs (Common Control Points) 177, all of which are managed by the data center of the provider network. The provider network may be located within the service provider 101 in various embodiments. In the illustrated embodiment, the data plane of the VCS is External Location Groups (ERGs) are used to manage external locations (selected by the VCS customer / client). For example, the ERG135A can be extended to a colocation facility. 130 (e.g., a building in which computing-related resources of more than one organization may be hosted The ERG 135B may be established in a customer data center 132 A may be set up in customer data center 132B, and ERG 132C may be established in customer data center 132C. do.

[0028] According to some embodiments, the control plane resources 141 of the VCS 110 include: ,configuration requests submitted by clients (requests to set up new virtual machines,,network requests to change network settings), provisioning of data plane components of VCS CPS102A to 102C, etc., which are responsible for responding to the monitoring and status management operations, etc. The data plane may include several control plane servers (CPS) 102. The resource is an IVN such as IVN115A that was set up on behalf of client C2. It may include several isolated virtual networks (IVNs) 115. As previously indicated, the IVNs 115, in various embodiments, for at least some types of network configuration settings. Logically separate or disconnected from the rest of the resources of the virtualized computing service For example, a given IVN may include a set of resources that are separated from the In a network, one or more subnets and / or IPs with their respective security settings are A subnet may have a set of addresses, each of which may be used by one or more virtualization hosts (VHs). Assigned to each virtual machine 125 (e.g., VM 125A) set up in 117 In the example scenario shown in FIG. 1, IVN 115A may be connected to VH 117A and and VH 117B, while IVN 115B may include virtual machines instantiated in , VH117J or 117K. In one embodiment, at least some of the virtualization hosts 117 operate in multi-tenant mode. A given VH may be set up on behalf of several different clients, so It can potentially be used for VMs hosted on several different IVNs, and VMs on one VH can potentially be hosted on several different IVNs. Note that the object is implicitly instantiated.

[0029] In at least one embodiment, the IVN is provided for internal or administrative use, and It can be set up to host virtual machines requested by clients. In some embodiments, for example, the OCC177 is used to send commands to the ERG. One or more of these may be established within the IVN. A given OCC 177 may have, for example, several In some embodiments, the system is implemented using one or more processes or threads of execution within a virtual machine. and may be programmatically associated with at least one extended resource group 135. In at least some embodiments, the OCC configuration settings are Commands originating within a network can be sent to the ERG via the OCC, but the ERG (or Any other source) sends a message to the VCS control player via the ERG. For example, as discussed below, In the embodiment, a specific virtual network instance connected to a virtual machine used as an OCC is The security settings for the VNI determine whether messages are sent to the VCS control plane. It only allows line resource 141 to transmit to the OCC, and not in the reverse direction. This is not possible.

[0030] At a high level, in various embodiments, the ERG 135 includes at least a small amount of hardware Any VCS Customer selected that can host the equipment and has an Internet connection Secure data plane functionality of VCS at the location (e.g. provider network data The same or at least very similar to what can be set up in a data center. The ability to instantiate virtual machines that are based on the As such, one or more types or categories of ERGs may, in various embodiments, be S, and the customer may use a colocation premises 130 or a customer data center 13 2, an ERG instance is set up in a given location outside the provider network. The user may select the specific categories to be populated. The particular set of attached software and firmware may, in various embodiments, be Configured by the operator of the provider network 101 (and at least in Depending on the model, the system may be pre-configured or pre-installed.

[0031] The quality of virtualized computing and other features provided using ERG (including security) (including aspects such as reliability, performance, and availability) To ensure that the work meets the standards, different embodiments use several techniques. For example, in at least some embodiments, EVH118A, Enhanced security such as 118B, 118J, 118K, 118P or 118Q A set of virtualization hosts (EVHs) 118 may be included in an ERG. Remotely generated virtualization-related management commands, for example, may be without sending messages back to the control plane resource (e.g., control plane resource 141). Some hardware is specially designed to allow it to run completely and reliably. , software, and / or firmware elements. Such elements may include, but are not limited to, the Trusted Platform, as discussed in more detail below. Offloaded, including a Trusted Platform Module (TPM) or other security module Virtualization Management Component (OVMC), a tamper-resistant storage device, Decrypts content only if the storage device is physically connected to a specific EVH tamper-resistant storage devices that can be encrypted, and low-overhead virtualization management software In at least some embodiments, the following may include a software stack: To ensure that the EVH protocol is secure, it does not make outbound calls and uses TLS (Transport Layer Security). Implement an API for inbound commands that is protected using a security session. Such an API may include a VCS control plane agent. In some embodiments, it may have strong authorization, authentication, and accounting related controls. In some embodiments, the shared secret associated with the virtualization management is stored in the EVH itself. In various embodiments, the EVH, including the VCS software and configuration, The state information to be acquired is stored in one or more TPM or similar security modules that are sealed to the TPM or similar security module. The encrypted drive can be stored on the drive using a secure and measured boot process. In at least some embodiments, similar EVH may also be used within provider network data centers, e.g., 125A At least some subset of the virtual machines are on the EVH in the provider network. Note that the virtualization hosts (EVH118 and In some embodiments, the virtualization server (including the VH 117) may be referred to as a virtualization server, or simply as a server. Sometimes it gets discovered.

[0032] In at least some embodiments, a given ERG 135 may include one or more computing The network manager may include a network manager implemented in a provider network. In some embodiments, the network may participate in or initiate the establishment of a network connection with the network. In this mode, a secure connection such as a Virtual Private Network (VPN) tunnel or VPN is used. The network channel is ERG135 (for example, ERG's Network Manager) and resources located in the provider network data center; Such a channel can be employed to send commands from the VCS to the ERG. In the embodiment depicted in FIG. 1, for example, each one-way secure network Use channel 166 (e.g., 166A, 166B, or 166C) to request (including a request to instantiate virtual machine 125, such as 125B, in an ERG) In response to the command, the control plane resource 141 first generates a command It can be sent via OCC to the ERG for final execution on H118. In an embodiment, the secure channel used for such commands is ERG135 (network One or more resources in the ERG (such as a network manager) and the ERG allocates resources according to the request. One or more resources (virtual machines 125 or VPN gateway, etc.) and the specified After the OCC177 is set up, the channel (e.g., VPN connection identifier or You may obtain display (and permission to use) the applicable Channel Identifier.

[0033] According to some embodiments, the OCC 177 accessing the secure channel 166 The VCS control plane resource 141 generates at least one management command. The first version can be obtained. The management command is in accordance with at least one embodiment In this case, the VCS control plane generates a program in response to a program request from a client. Program requests may be directed to an ERG associated with the OCC. The body, in at least some embodiments, is configured to use a pathway that does not include a secure network channel. The request may have been received by the control plane device from a client via The representation of the command generated by the VCS control plane may, in various embodiments, be C over a secure network channel to the associated ERG 135. In one exemplary scenario, a client may be running a virtual You can programmatically submit a request to start a machine (such as a VM125B), The first version of the corresponding VM start command uses the VCS control plane resources. 141 and sent to the appropriate OCC 177, which may then The network manager of the ERG may then send the modified version to the network manager of the ERG. The manager can send commands to the selected EVH 118 and 8. One or more processes on the network requesting that a command be executed to start a virtual machine. A similar workflow can be implemented in various embodiments by issuing a command to terminate a VM. This can be done for other types of commands, such as commands to modify a VM. can.

[0034] In some embodiments, commands received at the OCC from the VCS control plane The version of the code may, for example, remove / replace one or more security-related tokens. The command can be modified in the OCC by A modified version of the command may be generated by the OCC, for example. The security artifact or object may include one or more security artifacts or objects that are In various embodiments, the target host of the ERG where the command is ultimately executed / implemented. In at least one embodiment, the encryption key can be authenticated using HMAC (Hash-Based Message Authentication). Each authentication code, such as the authentication code, is generated for each command by the OCC and stored in the ERG. It can be included in forwarded messages, making it difficult to tamper with the commands. In an embodiment, an instance of one or more families of the virtual machine category may include a provider A virtualization host 115 in a network data center may be established to At least some instances may be set up with an ERG. A virtual machine that can be used to start instances of one or more VM categories. The thin image can be transmitted, for example, over the network from the provider network to the ERG. New VMs can be created very quickly without the need to transmit potentially large VM image files. It may be cached on a storage device contained within the ERG so that it can be booted. In some embodiments, an authorized entity (e.g., a user, a group, etc., The identity service or authorization service) of those entities Only users are allowed to initiate one or more types of operations (such as starting a VM) on an ERG. Before performing such an operation, the VCS In various embodiments, the requesting entity may be verified as having appropriate authorization (e.g., , VCS control plane, OCC, and / or ERG itself) In at least some embodiments, all commands issued to the ERG (or or more generally, any messages sent to and / or from the ERG. Comprehensive logs for VCS control plane, OCC, and / or The contents of such logs may be maintained by a network manager within the ERG. This allows you to install various types of programs on the VCS client where the ERG is set up. The signal may be provided via an interface.

[0035] In at least some embodiments, the virtual machine 12 running on the EVH of the ERG 135 Regarding 5, at least two types of data plane communication can be supported: C1's customer data center 132A, client C2's customer data center 132B, Or a given external premises, such as a colocation facility 130, does not implement VCS functionality. The system may include at least some servers or computing devices. The location facility 130 may include a set of non-VCS servers 123A, a data center 132A may include a non-VCS server 123B, and data center 132B may include a non-VCS server 123B, in the illustrated implementation. In some cases, the ERG 135 may include a non-VCS server 123C. One motivation for upgrading may include proximity to a given set of non-VCS servers 123, Such a server may, for example, store data that is manipulated using the ERG's virtual machines. or intended to work closely with other applications running within the ERG. can run applications that are not part of an ERG (provider network) Thus, in various embodiments, a given ERG 135 may be , a local premises low latency data channel 168 (e.g., data center 132A) Channel 168A, Channel 168B in Data Center 132B, and Colocation One or more channels that allow connection to non-VCS resources via channel 168C of facility 130 In some embodiments, for example, the virtualization host in the EVH 118 may include resources on host (for example, a private IP address from the range associated with the VCS's IVN is assigned and non-VCS resources such as servers 123 at the ERG hosting premises. Network address translation may be required for traffic between at least In another embodiment, the network manager of the ERG 135 may Such address translation and / or other protocol processing (e.g., It can act as an intermediary to perform cellification protocol processing operations. In this form, it is necessary to set up a secure channel 166 to the VCS control plane. Allows connections to non-VCS resources using the same network manager that participates in In other embodiments, different networks can be used for different types of traffic. A network manager can be used.

[0036] In at least some embodiments, a given ERG 135 may be configured to A logical extension of the existing IVN115 established using at least some of the resources of For example, in the illustrated embodiment, ERG 135B may be configured as client C. ERG135A is an extension of IVN115A on client C1, and ERG135A is an extension of IVN115A on client C2. B. Therefore, the available IP address range, subnet settings, output / Various network configuration settings of the IVN, such as input security rules, are also In such an embodiment, it may be applied to the ERG to extend the IVN. Two-way data channels such as 67A and 167B are used to communicate with resources in the IVN. Non-management packets or data plane traffic to and from ERGs that are configured as an extension of the IVN In some embodiments, some 10Gb / s devices, such as the ERG135C, can transmit packets. Note that some ERGs may not be configured as extensions of the IVN. Such ERGs are effectively implemented outside of the provider network data center. VN, for example, in the illustrated embodiment, instead of C3. The same type of network can be done within an IVN set up across Configuration selection can be made by client C3 in ERG135C. In some embodiments, the same set of physical network links and / or the same V A PN tunnel or other secure connection is a means for (a) bidirectional data traffic (e.g., bidirectional data channel 167), and (b) unidirectional management command traffic (e.g. , logical unidirectional command channel 166).

[0037] According to some embodiments, a VCS client submits a program request to the VCS. You can begin the workflow to set up and configure an ERG by In response to such an ERG establishment request, in one embodiment, an outbound command The OCC, for example, may request that the ERG’s physical equipment be delivered to the appropriate premises even before the equipment is delivered to the appropriate premises. It may be created, selected (e.g., from a pool of existing OCCs), or instantiated. In such a scenario, a secure channel 166 (e.g., a VPN tunnel) with an ERG 135 may be used. The channel is set up at some point after the OCC is established or designated for the ERG. It is possible.

[0038] In some embodiments, the ERG device establishes a connection and begins using the ERG. It is pre-configured and pre-installed in such a way that it requires very little effort from the VCS customer to For example, in one embodiment, the ERG instance may be installed on the computer A set of computing devices is set up, powered on, and connected to the Internet. As soon as a connection is made, the network manager automatically Start up and create resources (OCC177, VPN tunnel) in the VCS data center. It can initiate connections with any gateways that have been set up to enable Therefore, the discovery that electricity and / or internet connection is available In such an embodiment, a trigger for starting up the network manager The network manager may, in some embodiments, act as a signal to the ERG. It may also be responsible for initiating a bootstrap operation on one or more of the EVHs. In addition to allowing the instantiation and use of virtual machines 125, the ERG also In some embodiments, a provider network service that utilizes a virtual machine For example, in some embodiments, a provider network may be used to provide The database service of the network uses For this purpose, a set of virtual machines is available, and such a database instance can also be set up with an ERG upon client request.

[0039] Example ERG Virtualization Host FIG. 2 illustrates a system for implementing a virtualized computing service, in accordance with at least some embodiments. 1 is an exemplary schematic diagram of an enhanced security virtualization host that may be included in an extended resource group. In the illustrated embodiment, the control of an ERG similar to ERG 135 shown in FIG. A collection of computing devices 205 is selected by the VCS client. It may be delivered to a location where the data is stored (e.g., a customer data center, a colocation facility, etc.). Each device in the collection is configured with the software and The system may already have at least some set of firmware and In some embodiments, the devices may be pre-cabled together. The collection of devices is based on a single standard rack of the kind typically installed in a data center. may fit into a rack (or some other number of racks) and may be used to measure the entire rack (or multiple racks) are packaged as units and distributed by the VCS operator to the provider network. In the illustrated embodiment, the ERG device collection The section 205 includes one or more hosts 206 that are used by the ERG network manager. This may include a VCS datacenter, which effectively acts as the primary administrator for the ERG. Establish a connection to the VCS server and the non-VCS server where the ERG is installed In some embodiments, the ERG is responsible for: A collection 205 may include one or more storage servers 207, which may include, for example, ERG allows customers to A virtual machine image that can be used to instantiate the requested virtual machine. The collection can also be used as a cache of EVH messages in the illustrated embodiment. 218A and some other numbers of Enhanced Security Virtualization Hosts (EVH) 218 In at least one embodiment, the ERG may include an EVH, an ERG network, Use the same type of computer for the network manager and / or the storage server. For network managers, and / or require a special type of hardware to cache VM images. Note that this may not be necessary.

[0040] In at least some embodiments, a given EVH of the ERG, e.g., EVH218A, , a set of 255 storage devices, an offloaded virtualization manager component OVMC 269 and Operating System (OS) strip-down backup The storage device 255 may include, for example, an EVH-powered The state information of the EVH, including any virtual machine state information, is stored, for example, in encrypted form. At least one tamper-resistant device 256 may be stored in the at least one In an embodiment, the tamper-resistant storage device 256 stores encrypted The data is sent when the device is physically connected to a specific EVH (e.g., EVH 218A). For example, the data stored in the device may be configured to be decrypted only when the The Trusted Platform Module (TPM) of the EVH is used to decrypt the encrypted version of the data. Read information stored in a security module, such as a security gateway module; and / or analysis may be required. Component 269, as discussed below in the context of FIG. 3, in some embodiments may include: The OVMC may include, for example, a security module. Through peripheral interfaces such as CI (Peripheral Component Interconnect) This may include a hardware card connected to the EVH218's memory. OVMC manages virtual machines on the primary processor of EVH. (e.g., CPU), which frees up many of the resources of the EVH In some embodiments, multiple OVs are available to the virtual machine requested by the client. The MC (e.g., multiple PCI connection cards) may be included in an EVH such as 218A. In an embodiment, the OVMC handles networking related virtualization management tasks and / or other Able to perform input / output (I / O) related virtualization management tasks. In an embodiment, a peripheral card similar to the OVMC269 is used to configure the network manager for the ERG. The server may implement at least a portion of the functionality of the manager.

[0041] In the embodiment depicted in FIG. 2, the OS 265 includes a control plane agent ( The CPA 267 may include an outbourage process of the kind discussed above. Provider network data center via the OCC It is possible to receive management commands 266 sent from the VCS control plane at the In the illustrated embodiment, the command 266 is a CPA application program. The data may be formatted according to the CPA API 278. CPA API278 is a specific implementation of CPA or other components of EVH218 Regardless of the details, it will be implemented in EVH218 as required by the VCS control plane. It can represent a contract that governs the type of operation. For example, an ERG can set up The command to start the virtual machine requested by the hosted client is At least some of the CPA API calls may be sent to the A267. In some cases, it may not be necessary to send a response to the submitter. In some such embodiments, the message may not be transmitted from the EVH by the CPA. There is a match.

[0042] In response to the VM start command, the CPA 267, in some embodiments, A replaceable virtualized intermediate process (VIP) 268 such as 8-1 may be launched. First, in the illustrated embodiment, the VIP process 268-1 receives a API handler thread 2 configured to receive the API requests or commands sent 70-1. Next, CPA 267 issues the command to actually start the VM. and launches a new child thread 263-1 of VIP 268-1 to The command that causes the start of the VM thread 263-1 is For example, what bootable VM image to use, what are the resource requirements for the VM, what is the VM's VM pool, and what is the VM pool size? Specifically, the results of network configuration operations previously performed by CPA on the VM. The command to instantiate the VM may include a parameter indicating the In an embodiment, submission may be made via a second internal API called the VIP API 279. The CPA API represents the contract between the VCS control plane and the CPA. Similarly, in various embodiments, the VIP API may be used to retrieve the identity of an entity (e.g., a VIP). The ability to start and stop VMs regardless of specific implementation details or process / thread structures. and another contract that indicates the type of requests that will be processed by the entity responsible for The lifetime of a given VIP 268 may, in some embodiments, be determined by the lifetime of the corresponding VM thread. This may coincide with the lifetime of the VM 263, e.g., when the VM is terminated, the VIP may also be terminated. During the life of the VM, the CPA communicates with the VIP through the API handler thread 270-1. For example, the VIP API to request various types of VM-related operations. 279 formatted messages to query various types of VMs. In some implementations, the user can submit a request and receive a corresponding response. In an embodiment, the resource requirements of at least some of the VMs launched in the ERG (e.g., PU, memory, storage, and / or memory requirements) are very low and / or The expected lifetime of a VM may be very short, and such a VM may be considered a "micro In one embodiment, a stripped down version of OS265. The application leverages the Linux Kernel-based Virtual Machine (KVM) and other virtualization features. The virtual machine can run such lightweight micro-VMs at near-native speed. A virtual machine monitor may include a thin monitor. The virtual machine monitor runs as a user space process. It provides minimal device emulation to the guest operating system (e.g. For example, standard PC device models, including block and network devices, faster Non-essential features to achieve startup and reduced memory footprint (e.g. VGA (Video Graphics Array) and other legacy device emulation In some embodiments, the virtual machine monitoring process may not include We have released an API similar to PA API278, implemented micro-VM sandboxing, It can handle rate limiting of virtualization related requests.

[0043] In different embodiments, the selection of the particular EVH 218 to be used for a given requested VM may be performed using Several VM placement factors can be taken into account when determining which VMs are currently running on a host. The number of VMs running on a cluster can be taken into account and the oversubscription settings of EVH can be adjusted accordingly. , and / or oversubscription of clients on whose behalf VMs are launched In some embodiments, certain Before any request to start a VM is sent to the CPA267 at the EVH, some preliminary configuration is performed. The configuration operation can be performed on the EVH218 (e.g., by the ERG Network Manager). For example, one or more virtual network interfaces of the type discussed below may be It can be configured using OVMC269 for eventual use by the VMs in the VH. do.

[0044] When a new command to start the VM is received on CPA267, the VM thread The CPA workflow shown above with respect to invocation of 263-1 is, in the illustrated embodiment, It may be repeated, for example, per-VM configuration operations may be performed by the CPA before starting the VIP. A VIP may be assigned a thread and one of the VIP's threads may be used for the requested VM. As implied by IP268-10000, in the illustrated embodiment, a given EVH 218 allows you to create a large number (e.g., tens of thousands) of VMs that run simultaneously on EVH. The actual number of VMs that can be instantiated for this purpose depends on the overall available VMs in EVH. The combination of resources, the version of the software / hardware stack installed, John, effective resource oversubscription policy / multiple resource oversubscriptions Depending on subscription policies, etc., different embodiments may vary. For example, The number of VMs may be larger or smaller than the 10,000 shown in Figure 2. The parent of 63-1, VIP268-1, is interchangeable, so the actual In some embodiments, a desired level of resource oversubscription can be implemented. For example, the total memory usage of a set of VMs running on an EVH can be calculated based on the available memory. The request to terminate a VM may exceed the amount of memory available in the VCS control Once received by the application, the corresponding command may be sent to the CPA via an API call, The VIP whose thread is used by the target VM may be terminated in the illustrated embodiment. That is, in at least one embodiment, the number of running VMs and the number of running VIPs In another embodiment, there may be a one-to-one relationship between the number of process artefacts shown in FIG. Variations in the architecture can be employed, for example, a given VIP may have You can create multiple VMs as threads and set up multiple CPAs on the VH. In some embodiments, the VIP and / or VM may The distinct processes used may not be interchangeable. In this version, a different approach to managing virtual machines is used than that shown in Figure 2. For example, instead of a control plane agent of the kind shown, In this way, a hypervisor with a larger footprint can be used.

[0045] Exemplary Offloaded Virtualization Management Component Card FIG. 3 illustrates an offloaded virtualization host according to at least some embodiments. 1 illustrates exemplary elements of a network management component card. So, would you use such a card to perform at least some types of virtualization management tasks? Run it on a virtualization host (for example, a VM in an ERG or a VM in a provider network) This allows the processing and other resources of the host being used to be freed up. Fake Virtualization Management Component (OVMC) card 302 (OVMC2 in FIG. 2) 69) are offload processors 317A and 317B (O (as distinct from the primary CPU of the virtualization host to which the VMC card is attached and used) In the illustrated embodiment, the SO One of C (e.g. 315) can be used for the boot controller, and the other , which can be used primarily or specifically for network processing offload. Other distributions of responsibilities may be implemented in different embodiments. In some embodiments, the boot controller performs multi-phase boot of the virtualization host itself. It can be used for the initial phase of operation of the system. The OVMC card 302 also Security module (Trusted Platform Module (TPM) etc.) 330 It may also include, as previously mentioned, in some embodiments, during the boot procedure and / or In at least some embodiments, the BIOS may be used extensively for boot state verification or post-boot state verification. represents a secure representation of the pre-installed software / firmware on the virtualization host. In at least some embodiments, the VCS operator may include a security module. Only the specific software stack selected by the security module 33 0 may be installable on the host on which it is installed, so that the installation Any attempt to remediate is likely to fail and compromise the security of the virtualization host. It will be further strengthened.

[0046] Additionally, the OVMC card 302, in various embodiments, may include some storage, power, and connectivity related components. For example, one or more flash drives. Incorporating the device / interface (or SSD) 335 into the offload card These devices can be managed by various virtualization management components, virtual machines, To store firmware and / or software corresponding to components, etc. In various embodiments, the PCI-E interface 340 may be used to and to communicate with processes such as VIPs and / or for communication between SOCs. In other embodiments, QuickPath Interconnect (QPI) or The UltraPath Interconnect (UPI) is a variant of the UltraPath Interconnect (UPI) and is supported by other types of interconnects. The network interface can be used. In some embodiments, the interface 345 is a virtualization host and a virtualization computer. and at least one embodiment of a control plane for a network service. The host must have a running virtual machine and at least one network endpoint. The OVMC card 302 may also be used for data plane communications between the In some embodiments, for example, in the case of an extended power outage, at least some of the targets The OVMC may include a power supply 360 sufficient to keep the OVMC functioning for any number of hours or days. In some implementations, a supercapacitor-based power source may be used.

[0047] In some embodiments, for an offloaded virtualization manager component , a separate SOC on the same card may not be required. For example, in one embodiment, Capable of performing boot controller functions and network offload tasks. In another embodiment, a single SOC may be used that is used for the boot controller. You can use a card other than the one you are using to perform network processing tasks. The set of offload card components shown in Figure 3 is intended to be comprehensive. In some embodiments, however, some other components, such as timers, In various implementations, the components shown in FIG. At least some of the points may not be needed.

[0048] Example of a Virtual Network Interface with an Outbound Command Communicator Use In various embodiments, a virtual network interface (VNI) ("elastic network The VCS (similar to VCS 110 in FIG. 1) and and / or VCS Extension Resource Groups (similar to ERG 135 in Figure 1), IP (Internet Protocol) ) addresses and other network-related attributes can be transferred relatively easily between virtual machines. Such attribute transfers can be implemented, for example, by transferring a virtual network interface to a One virtual machine (used as an Outbound Command Communicator (OCC), for example) You can programmatically disconnect from one virtual machine (the virtual machine being used) and programmatically connect to another virtual machine. FIG. 4 illustrates an extended version of the method according to at least some embodiments. Program the outbound command communicator associated with the resource group. 1 illustrates exemplary attributes of a virtual network interface that may be connected in In the illustrated embodiment, one or more VNIs, such as 491A or 491B, may be, for example, A specific hardware network interface card on the host on which the virtual machine runs. (NIC) independent and connected to a given virtual machine 490 via program operations 493 (or be disconnected therefrom).

[0049] A generalized list of exemplary attributes 495 for any given VNI 491 is shown in FIG. In at least some embodiments, not all of them are used for all VNIs. In some embodiments, the attributes or properties shown in FIG. You may only implement a subset of the attributes and not fill in all of the implemented attribute fields. It is not necessary (i.e. some of the attributes can be left blank or null). Each record contains a field / entry that contains the various VNI attributes495. In some embodiments, a persistent metadata store, such as a provider network or or VCS various control plane or management components. It may be stored in a store.

[0050] For example, in response to a program request, a new VNI is created, in the illustrated embodiment: A new interface identifier 401 may be generated for it. In this embodiment, the description field 402 may be a VNI, e.g., "OCC4 of ERG E-1142 Filled in by a client who requested the creation of an interface 654 for 85 As discussed above, in some embodiments, the VCS in which the VNI is used may include Attributes 495 may include multiple isolated virtual networks (IVNs). In the example, the VNI may include an IVN identifier 403 (indicating the IVN in which the VNI is configured).

[0051] In different embodiments, several types of network addressing related fields Any of the following may be included in the set of attributes of a VNI. In some embodiments, one or more For example, a private IP address 405 of the Private IP addresses, also referred to herein as non-public addresses, may be used in various implementations. In this state, routing within the provider network and its extensions, such as ERGs, is handled internally. can be used from outside the provider network and its ERGs (or from other In some embodiments, the VNI At least some of the non-public IP addresses associated with may not be formatted according to the provider network's proprietary protocols. You can use a strict IP address or use another public domain protocol. In general, some implementations may use addresses formatted according to the In an embodiment, zero or more public IP addresses 415 may also be associated with the VNI. These IP addresses are used outside the provider network and its extensions, e.g. Various non-VCS servers on premise, public internet In some embodiments, the VNI may be used in One or more entries, such as the identifiers of the subnets set up by the client in the VN. Subnet Identifier 425 (e.g., Classless Inter-Domain R outing or expressed in CIDR format) can be included in attribute 495 In one embodiment, the entity responsible for propagating the address(es) associated with the VNI is Identification of the Domain Name Server (DNS) that is associated with the This may be included in sex 495.

[0052] In some embodiments, attributes 495 may include security-related properties 435. Some provider networks provide a set of compute instances to which VNIs may be connected. The user can then decide which types of incoming and / or outgoing traffic are allowed at the For example, it may be possible to specify rules, including firewall-related rules. Such rules are called "security groups" and are managed by security group(s). In some embodiments, such rules may be used to You can use the RADIUS server to enforce various port and protocol restrictions, and multiple rules can be applied to each VN. For example, a client can use a security group to to ensure that only HTTP and HTTP outgoing or incoming traffic is allowed. restricting the set of TCP or UDP ports for which traffic is allowed , filter incoming and outgoing traffic according to various policies, etc. In some embodiments, the security of VNIs connected to an OCC of the type discussed above may be achieved. Use security groups and / or other security settings to control VCS It allows commands issued on the plane to be sent to the OCC while still allowing commands to be sent from the OCC. This can prevent traffic from flowing to the VCS control plane. In an implementation, which users or entities can access computing resources, such as virtual machines, in the VCS? Attachment indicates whether you are allowed to request to attach a VNI to the operating platform. In some cases, a separate detacher list may be used to You can specify which entities can detach a VNI, while the attach list Identify authorized attachers and detachers using a single list, such as 447 The IP address of the VNI (e.g., the public IP address 415 and / or Users or those permitted to set or modify private IP addresses A collection of entities may be provided in the IP address setter list 449, The user or entity that owns the I (or can modify various other fields) The set of properties may, in some embodiments, be specified in the owner / modifier field 453. For example, the owner / modifier identified in field 453 may be In this state, it is not permitted to change the attacher list 447 or the IP address setter list. Thus, it is possible to connect or disconnect a VNI or to change its IP address (multiple Changes the set of entities that are allowed to be modified. The terms used in fields 447, 449, and 453 may be used interchangeably in various embodiments. Use logical data structures other than lists (e.g., arrays, hash tables, sets, etc.) Entities that are granted various security privileges, roles, and / or capabilities using A group of

[0053] In some embodiments, the virtualized computing services of the provider network A user of a virtual machine may be allowed to programmatically terminate a virtual machine (VM). For example, a client can set up a VM, attach a VNI to the VM, and run the desired computation on the VM. and requests that the instance terminate when the desired computation is complete. In such an embodiment, the "DeleteOnTerminat Use the "e" setting 451 to specify what happens to an attached VNI when a VM is terminated. You can specify the DeleteOnTerminate command for the VNI that is connected to the VM that you want to terminate. If minate is set to "true", the VNI may be deleted. (For example, a persistent record containing the VNI attribute 495 would be stored in the repository where it was stored. DeleteOnTerminate is false. , VNIs may be preserved, e.g., some VMs or another computing platform. In this form, when a VNI is connected to a VM, a connection record is created that is separate from the VNI attribute 495. The TerminateOnTerminate property can be used to represent this relationship. Instead of, or in addition to, being associated with the VNI itself, a connection record In such an embodiment, the attributes 495 of the VNI may be associated with the A reference to each connection record or records for the connections involved can contain a pointer, and each can have a different value for "DeleteOnTerminate". Can be set on the connection record.

[0054] In one embodiment, the attributes 495 are associated with the computing platform to which the VNI is connected. The source and / or destination for network packets sent to the form Include routing-related information such as an indication of whether to perform a routing check. If the source / destination check setting is "false" or "off", If set, the routing decision is based on the source and destination IP addresses of the packet. This can be done based on addresses, e.g. packets travelling from one subnet to another. In some embodiments, if the setting is "true" or "on", In other words, the computing platform may not perform routing. Therefore, the source / destination field 465 can be used in some implementations. In this case, the computing platform to which the VNI is connected determines whether it is the final destination. Performing routing or gateway functions for non-destination packets The route table can control whether such packets are forwarded or ignored. Other types of routing related information, such as entries, may also or instead be provided in other implementations. In some implementations, the billing related information may be included in attribute 495. 95, for example, to bill for network traffic associated with the VNI. Identify the entity or user desired.

[0055] The interface status field 468 indicates the current state of the VNI, e.g. It may be used to indicate whether an I is "available", "disabled", or "under repair". Similarly, the connection status field 469, in some embodiments, indicates whether the VNI is currently Currently connected, disconnected, or in the process of being connected or disconnected In one implementation, as described above, the record of the connection The VNI is created when the corresponding connection operation is performed and is the identifier of the current connection for the VNI. Multiple identifiers may be stored in the connection ID field 471. The identifier of the virtual machine being used may be stored in the connected instance field 473. In some embodiments, the user or entity that requested the connection is identified in the connection owner field. In one embodiment, the IP address of the VNI may be specified via 475. The physical network interface cards currently available for traffic destined for this A list of identifiers for a NIC or multiple NICs, e.g., MAC address(es) In some implementations, the IP address of the VNI may be maintained in the form of a 477 field. Monitoring information, such as statistics about the amount of traffic flowing to and from the address. may also be held among attributes 495. VNI multiplexing or parent-child hierarchy is supported. In at least one embodiment, a pointer 481 to a child or branch VNI of the VNI is included. Other fields not shown in Figure 4 may be included in various embodiments.

[0056] In one embodiment, some of the fields shown in FIG. 4 are references to other objects. For example, the security of the VNI can be replaced by a reference or pointer. The security information may be stored in a separate security object, and attributes 495 may be used to Similarly, a VNI can contain a reference to a computing plan. Each connection in the platform can be represented by a connection object, whose attributes 495 include In some implementations, this may include a pointer to the appropriate connection object.

[0057] FIG. 5 illustrates a block diagram of a virtualized computing service, in accordance with at least some embodiments. Used to send control plane generated commands to an extended resource group. Illustrates exemplary paths that may be used in several isolated virtual networks of the kind introduced above. The IVN, in at least some embodiments, sets up the VCS on behalf of the VCS customer. This includes those that are hosted on a provider network or set up for administrative purposes. In the illustrated embodiment, the path is traversed in response to a client request. In response, compute instances (virtual machines, bare metal instances, and / or other The command to start the application execution environment (such as One or more control plane servers 502 (e.g., 502A or 502B). In some embodiments, the control plate may be The server 502 may be established in the VCS control plane IVN 510. Use the Service Network Address Range (SNAR) 591 to address individual hosts in the VCS. IP addresses can be assigned, where in the illustrated embodiment, various types of In at least some embodiments, a computing instance can be set up. Responding to client requests by generating internal commands, resource provisioning It runs algorithms for administrative tasks such as networking, availability management, etc. At least some of the computing devices in the control plane are not used. The device may be assigned an address that is not part of the SNAR591 and the compute instance may Provider Network A network separate from the networks set up within a data center. Note that VCS can be configured as part of a network. From a network perspective, hosts that happen to be located within an Extended Resource Group (ERG) is required for at least some types of configuration decisions, such as network address allocation. In the illustrated embodiment, it is as if the host is a data center of the provider network. For example, within the VCS control plane, one or more The hosts on which the compute instances are set up depend on where the hosts happen to be physically located. Regardless of where it is located, in various embodiments, it may be assigned to one or more SNAR addresses. In some embodiments, a computation instance may be used Metadata that indicates the specific SNAR addresses associated with each address of the The data can be stored in the VCS control plane. External premises where configuration may not be directly controlled by the VCS control plane Because it is placed in the compute instance launch command (and / or VCS control Other configuration commands generated on the plain are sent to the ERG host for different implementations. In some embodiments, the data may need to be converted in various ways. Such conversion may also be implemented in various ways. In this state, the security of the VCS and the clients on whose behalf the ERG is set up is As mentioned above, client requests (e.g., A request to launch an instance in a RG) is sent to the VCS control plane (e.g. via the VCS's public application programming interface The path sent, at least in some embodiments, sends the corresponding command to the VCS controller. This may be different from the path used to transmit from the roll plane to the ERG.

[0058] In some embodiments, an internal service called the VCS Extension Service is established to allow VCS E, including secure transmission of management commands from the CS control plane to the ERG Such internal services may handle some aspects of the management of the VC S-enhanced services, which themselves, at least in some embodiments, may include one or more The outbound command communication may include an isolated virtual network (such as IVN 530). An OCC 522 (which itself may, in some embodiments, be a VCS Extended Services The virtual machine 517 may be implemented using a virtual machine set up on the virtual machine host 517. , at least two networks, i.e., in the illustrated embodiment, a VCS control Plain IVN510 (running compute instances in provider network data centers) A host / server with an address from SNAR that can be used to launch ) and configured as part of the VCS Extended Services IVN 530; or In some embodiments, the OCC 522 may be coupled to the VNI 523A of FIG. and at least two virtual network interfaces, such as 523B (see the context of Figure 4). The present invention may be programmatically connected to a VNI having features and functionality similar to the VNI described in. One of the VNIs, such as 523A, is a VNI that the VCS control plane is directed to an ERG. 523B, while allowing the transmission of other commands such as 523B. The VNI forwards at least one version of the command to the destination ERG. In at least some embodiments, the VNI5 The configuration in 23A is that commands are sent from the VCS control plane via the VNI. However, commands cannot be directed to the VCS control plane via the VNI. Such configuration can ensure that some of the other components of the path shown in Figure 5 Either of these, in various embodiments, may be used to initiate undesirable operations in the VCS control plane. In at least some embodiments, this can help ensure that Using VNIs such as 523A, the OCC522 (at least) Target ERG host 529 acting as an intermediary for control plane commands The same network address (part of the SNAR) may be assigned. Therefore, from the VCS control plane perspective, sending a message to the OCC In such an embodiment, the ERG host is notified of the destination by sending a message to the ERG host. OCC522 is equivalent to VC Upon receiving such a message containing a command from the control plane, In at least one embodiment, the address of the ENM is used as the destination, and the translated Generate a message (containing or indicating a control plane command) and send it to the E NM. When ENM finally receives the converted version, The ENM then, in various embodiments, extracts the first command and determines if the command is the first VC S The ERG can be forwarded to the host that generated it in the control plane. In this form, the network address assigned to the host 517 on which the OCC 522 is executed is The address is different from the network address assigned to the OCC itself, and is assigned to the SNAR591 ( Address used by the VCS control plane of the host on which the compute instance can be launched. It could also be part of the response range.

[0059] In at least some embodiments, the OCC 522 also controls the VCS control perform one or more additional security-related operations with respect to the command received from the For example, one or more security-related artifacts / objects in the command can be Objects (such as bearer tokens that are initially generated in the VCS control plane) are Deletion, replacement, and / or replacement in messages sent towards the destination ERG In one embodiment, for example, the control presentation received at the OCC may be modified. The version of the call command may include one or more cryptographically signed tokens, The outbound version of the command sent to the Defense Center ERG is In some embodiments, in response to a client request, the token may include a re-signed token. For a given management command that is generated in the VCS control plane in response to (b) between the CS control plane and the OCC, and (c) between the OCC and the destination host. It is possible to establish a different TLS session with one or more resources on the path to the In various embodiments, such types of security related operations are performed by the OCC. Any security-related data or metadata emitted on the VCS control plane may be Attempts to ensure that data does not reach the destination ERG, thus to reduce the possibility of misuse of such security-related data / metadata.

[0060] In some embodiments, the VCS enhanced services IVN 530 and / or individual virtual The virtualization host 517 can be used in a multi-tenant manner. For example, multiple VCs can be used. Used to send control plane commands to the S-client ERG Several different OCCs 522 may, in one embodiment, be part of the same enhanced services IVN 530. In some embodiments, multiple OCCs for different clients may be configured as A single OCC can be used to establish a single OCC for multiple clients. In some embodiments, a single O can send commands to the ERG of the other O. CC can be used to send commands to multiple ERG hosts, and in other embodiments ,One OCC can be used per ERG host (multiple compute instances can be (Note that a given ERG host can potentially be set up with at least In some embodiments, the OCC may transmit commands issued on the VCS control plane. One or more transformation or encapsulation operations can be performed on a system. In the scenario, the VCS control plane command is NAR address) to a specific customer on a specific ERG541A. C1. In some embodiments, the address ABCD may also be assigned ) is used to check whether a command is being sent to the ERG541 before it is finally sent and implemented by the host. A's ERG network manager 528 The OCC also ensures that in such an embodiment, the ENM 528 is connected to a different IP address KL MN (KLMN is used at external premises 535A where the target host is located (part of Customer Network Address Range (CNAR) 593) In some embodiments, the OCC may determine that the command It is possible to generate messages that contain a transformed / encapsulated version of the message. The destination address of the message is indicated as KLMN. , in the illustrated embodiment, one or more additional VNIs (such as VNIs 523C and 523D) , and sent via an intermediate inter-IVN gateway 540 to the IVN 515 of customer C1. The inter-IVN gateway 540 may be implemented in some embodiments as a virtual machine or a It can be implemented using a computing instance, where the packet processing software runs. In at least some embodiments, one of the messages or commands may One or more other operations / conversions may be performed in the inter-IVN gateway 540, e.g., ERG Required for 541A to send a message to its intended destination This includes address translation operations.

[0061] As shown in FIG. 5, the IVN of C1, at least in some embodiments, The virtual machine 525 may include any number of virtual machines 525 established on hosts within the internetwork. After a message enters the customer IVN 525, for example in the illustrated embodiment, the message is forwarded to one or more Sent to the ENM at destination ERG 541A using VPN tunnel 561. A VPN tunnel can be established for secure connectivity between the provider network and the ERG. Instead of using a dedicated physical link (ERG 541B to I A dedicated link 562 connecting to the VN 515, etc., may be used, and / or other types It should be noted that any number of networks may be used. In some embodiments, this may be referred to as a direct connection link. First set up at an external premises outside the network (such as premises 535A or 535B) When this occurs, the ENM of the ERG, in at least some embodiments, 561 and / or a connection via a dedicated link 562 to the ERG. Information regarding the particular secure channel to be used may be provided to the OCC 522. One or more intermediate devices not shown may be involved in establishing the VPN tunnel and / or the dedicated For example, a VPN gateway may be involved in a connection over a customer IVN 51 and / or the edge router or other edge device may be set up with a dedicated Note that the NIC may act as an intermediary for connections over multiple physical links.

[0062] In at least some embodiments, the ERG 541 device is connected to an external premises 535 A given rack can be set up in one or more racks. 527, ENM528, and VCS control plane generated commands A set of hosts 529 on which compute instances can be set up in response (e.g., 9A, 529B, ...). In at least some embodiments, ENM 528 may include , may be configured as part of at least two networks of external premises, or In some embodiments, one of the networks may belong to an ER. A private network 531B that includes at least a subset of hosts 529 in G. Within this private network 531B, individual hosts may Network addresses are assigned from within network address range (SNAR) 591. (for example, by ENM528 as part of a host initialization operation). When the NM528 receives a command generated by the VCS control plane, it R indicates that the command is to be executed on the host with address ABCD, and EN M, in at least some embodiments, uses a private network 531B. The ENM 528 also directs commands for execution to the intended host in the illustrated embodiment. In this embodiment, the customer may be a member of the customer configuration network 531A, which is a One or more non-VCS hosts 542 may be included in the client configuration network 531A. The device, in at least some embodiments, is connected to a customer's network address range CN AR593 may be assigned addresses. Thus, for example, As a 1A component, the ENM528 also receives the address KL from the CNAR593. .MN can be assigned and this address is used by the VCS control plane command on the ENM. The IEEE 802.11b protocol may be used to receive a IEEE 802.11b signal.

[0063] In summary, in the embodiment illustrated in FIG. 5, The command is composed and sent over a path that may include one or more other intermediaries, ultimately resulting in Received at NM via a network address that is part of customer configured network 531A Along the path, for example, in various embodiments, network address translation, custom One or more transformation operations, including encapsulation, decapsulation, etc., are performed on the first command. OCC with multiple VNIs connected, IVN to IVN gateway, etc. , each implementing its own independent security operations on network traffic. In various embodiments, the use of some intermediary that can The possibility that a tactician could perform undesirable operations on the VCS and / or ERG This can help reduce the number of times a message containing or representing a command is received by an ENM. After the command is received, the ENM, in the illustrated embodiment, extracts the first command and Network 531B can be used to send commands to the target host. On the target host, the command can be executed (e.g., in the case of a start command, VCS Computing (resulting in the instantiation of one or more compute instances on a single host). From the perspective of the control plane, as previously indicated, the host 529 in the illustrated embodiment , in the same way that hosts located in the provider network are treated (at least With respect to some aspects of networking, within a given ERG, e.g. In various embodiments, the network traffic is routed to a provider network. Similar to how packets flow within a host, packets can flow between hosts 529, for example, at high speed. High bandwidth host-to-host network links can be supported. ERGs are effectively a customer I / O The network assigned to the computational instance in the ERG may represent an extension of VN515. The work address, in at least some embodiments, is determined (e.g., by the customer) by the IV The range of addresses used by the N515 (e.g. a range different from CNAR or SNAR) ) can be selected.

[0064] In some embodiments, the ENM 528 is an offloaded virtualization pipeline as shown in FIG. The OVMC 269 is also connected to the host through the peripheral interface. or may be implemented at least in part using a hardware card connected to a server. In other embodiments, other approaches to ENM implementation may be taken, for example It may be implemented as an operating system process or thread. In another embodiment, the ENMs 528 are not necessarily in the same rack as some of the hosts 529. cannot be co-located within a single ERG host, e.g., A separate rack or other physical container may be used for one or more ENMs. In at least one embodiment, the ENM has at least some Can perform DHCP (Dynamic Host Configuration Protocol) operations (for example, ENMs , can function as a DHCP server). In some embodiments, the context of FIG. One or more of the operations discussed in sends a VCS control plane command to the ERG host. It may not necessarily be executed as part of a sending workflow, e.g., different combinations of intermediaries A combination of transformations may be used or different combinations of transformations may be applied to a message or command. may be applicable.

[0065] Example ERG Network Manager An extended resource group implemented using one or more computing devices in an ERG Loop Network Manager (ENM) (ERG135 and its functions introduced in the context of Figure 1 and similar in characteristics) may be responsible for several different types of operations in various embodiments. FIG. 6 illustrates an extended resource group network, according to at least some embodiments. Illustrates exemplary elements of the ENM 605 provider network connection manager. In the illustrated embodiment, the server 603 is a provider network data center resource. Initiate or provide access to one or more secure communication channels to or from a resource In at least some embodiments, for example, a VPN tunnel can be established. The kernel can be used for a secure channel.

[0066] The local premises connection manager 607 of the ENM 605 is, in effect, The VCS-managed virtual machines for the ERG and the non-VCS-managed virtual machines that reside in the location where the ERG is set up In at least some embodiments, the CS resource may function as an edge device between the CS resource and the edge device. You can assign one or more public IP addresses to an ERG, and you can Allows resources in an ERG to communicate with VMs in the ERG. Local Premises Connection Manager For example, in at least some embodiments, the JA may IP address mapping and private to public IP address mapping For example, if the destination VM is part of a plan within an ERG, A public IP address is assigned as the destination address to which the private IP address is assigned. For packets addressed to a VM in an ERG using a public P address, A public-to-private address translation is performed, as well as a corresponding private-to-public IP address translation to a link can be performed for packets flowing in the reverse direction. Other encapsulation protocols for inbound / outbound data plane traffic Protocol packet processing may be performed in different embodiments. The ENM may, for example, be a private entity that includes an ERG host, as discussed in more detail below. To help set up a bet network, we need at least some Able to perform HCP functions.

[0067] The control plane command forwarder 612 of the ENM 605 in the illustrated embodiment For example, management commands (set up in ERGs) are sent from the VCS control plane. Receives the command (sent via the OCC), analyzes the command content, and selects the ERG VCS Control Plane Error in Enhanced Security Virtualization Host (EVH) It is responsible for forwarding commands to the appropriate destination, such as an agent. In some implementations, the command forwarder 612 may Then, it performs one or more validation / verification operations on the received command. and / or apply one or more transformations to the received version of the command. can.

[0068] The ERG host bootstrap manager 617 of the ENM 605 in the illustrated embodiment The ERG's EVH and / or other resources (such as caches for virtual machine boot images) bootstrap / initialize / start In at least some embodiments, the E shown in FIG. Not all elements of NM may run on the same computing device, but instead Instead, one or more of the elements may be implemented in an embodiment where a distributed ENM architecture is used. It should be noted that the various components may be executed on the respective computing devices. In at least some embodiments, one or more of the ENM elements shown in FIG. H. In some embodiments, the virtual machine may be instantiated in H. , ENM 605 may include a combination of elements different from those shown in FIG.

[0069] Example Virtual Machine and ERG Families VMs (using provider network datacenter resources or using ERGs) When a request is made to establish a virtual machine (a VM set up using In some embodiments, you may choose from a variety of supported VM types or categories. Similarly, in at least one embodiment, different combinations of features may be used. Various ERG types with different capabilities may also be supported, from which clients can choose the best fit for their needs. FIG. 7 illustrates an ERG type that, according to at least some embodiments, Virtual machines and extended resources that may be supported by virtualized computing services. 1 illustrates an exemplary family of service groups.

[0070] The supported virtual machine types of VCS similar to VCS110 introduced in the context of Figure 1 The standardized set of VMs 705 may, in some embodiments, include a generic VM family 710, Compute Optimized VM Family 720, Memory Optimized VM Category 730, Storage Optimized VM M family 740. The number of VMs that a given VM can use by a client. Select a VM type from the appropriate family depending on the estimated needs of the application to be used. The relative resource capabilities of different VM types can be selected or or a defined unit, e.g., in computing, something that runs at a particular clock speed. Use units such as virtual CPUs that are roughly equivalent in performance to a specific known processor that runs The family can be selected based on qualitative considerations. (e.g., the application mix may consume more memory or Does it consume a lot of compute?), or a particular VM type within a family (e.g., in the general purpose family) (Small 712 vs. Medium 714 vs. Large 716) are based on desired performance levels, cost, and other considerations. You can choose between low CO VM 722, medium CO VM 724, or high CO VM 725. A compute-optimized VM 720, such as M726, in the illustrated embodiment, may be implemented using a computationally optimized VM, such as It may be implemented using a host with many and / or fast processors or cores. Memory optimization such as MO small VM732, MO medium VM734, or MO large VM736 The VM730 uses a ratio of memory to computing power (e.g., per virtual CPU). gigabytes of memory) than a Compute Optimized VM720 or a General Purpose VM Similarly, SO small VM 742, SO medium VM 744, and SO large VM The storage-optimized 740 family, which includes the 746, offers a high ratio of local storage to compute power. Ratio (for example, local solid-state device (SSD) storage per virtual CPU) The number of gigabytes of data per second (GB) may be higher than other families. A predefined family of microVMs (mVMs)770, each of which is Based on the requirements of the application, it may be possible to qualify as a micro-VM and be supported. For example, mVM small category 772, mVM medium category 774, and / or mVM large category The Gol776 can be included in the Micro VM family. In this state, the client may also, for example, send a message that does not match one of the predefined types. By identifying some combination of processing, storage, and memory resources, In some cases, we can define and use our own VM types. In some embodiments, it can be set up in a provider network data center. Any instance of a VM type that can be deployed in an ERG can also be set up in an ERG. In other embodiments, each different set of VM types is associated with an ERG and a provider network. In at least some embodiments, the As mentioned above, bare metal instances can be used in addition to the VM types shown in Figure 7. , can be set up at the VCS or the host of its ERG.

[0071] In some embodiments, some ERG categories 755 are determined by the VCS operator. and allow clients to request services on their behalf at locations outside the provider network. This allows you to select a specific combination of ERG features that have been set up using the RG types (small ERG758, medium ERG759 and large ERG760) are the illustrated embodiments. In some embodiments, the embodiments may differ from each other according to any of several aspects. In terms of form, the physical footprint (required for a rack containing a collection of ERG devices) The required floor space (PF1, PF2, or PF3) may vary for each ERG type. may have different power demands (PD1, PD2, or PD3) and different computing power (CC1 , CC2, CC3) may differ and / or the cached VM image The sets of CIS1, CIS2, or CIS3 may differ between different ERG types. In at least one embodiment, when requesting an ERG, the client notifies the VCS Specifications (physical footprint) that do not necessarily match the specific ERG type supported by the (in terms of resources, power demand, computing power, cached VM images, etc.) In one such scenario, a VCS operator may choose to accept and comply with the ERG specifications. Either hand over the equipment to be used or show the client the closest supported ERG configuration. (which can be ordered by the client as needed). In one embodiment, a single type Only ERGs of this type may be supported by a VCS operator.

[0072] Example Program Interactions In various embodiments, the VCS client may be the ERG 135 introduced in the context of FIG. To submit requests associated with setting up and using similar ERGs Several program interfaces can be used. Expanded resource groups for virtualized computing services according to some embodiments 1 illustrates an exemplary program interaction for the VCS 812, e.g., Application Programming Interface (API), web-based console, graphs In the illustrated embodiment, one The above program interfaces 877 may be implemented, and any combination thereof may be used. The match may be used by the VCS client 810 to interact with the VCS.

[0073] The client 810 may, for example, access the Setup program via a program interface 877. Submit pERG request 814 to have the VCS perform the installation at a designated location outside of its own data center. You may wish to configure an Extended Resource Group. Multiple categories of ERGs In embodiments where is supported (similar to ERG type 755 discussed in the context of FIG. 7 ), The particular type of ERG to be set up is determined by the parameters of request 814 (the ERG Other parameters may be indicated, for example, via the ERG. Constraints such as where the ERG device / equipment collection must be delivered and the time frame within which the ERG device / equipment collection must be delivered In response to the SetupERG request 814, the provider network may A workflow may be initiated in some embodiments that includes various preliminary tasks being performed. A response message such as setup started 815 may be sent to the client. In one embodiment, an Outbound Command Communicator (OCC) instance A virtual machine (such as a virtual machine in a VCS-extended service of the type considered above) is As part of the workflow, the ERG may be instantiated or assigned to the requested ERG. Such an OCC may, in some embodiments, effectively be an ERG network manager. Of course, you can also actively wait for the ERG network manager to contact you. The sensor, in at least some embodiments, is configured to detect whether the ERG device is connected to a subject's external location. Communication with OCC until the device is handed over, turned on, connected to the Internet, etc. You may not be able to try this because the SetupERG request is 12. In some embodiments, the OC of the VCS The C manager 892 may instantiate an OCC instance, and / or It may be responsible for determining which particular OCC instance is used for a given ERG. In at least one embodiment, a given OCC instance may be, for example, a multitenant In the client operation mode, you can use several different ERGs for one or more VCS clients. It is possible.

[0074] In the illustrated embodiment, the VCS 812 is handed over the ERG and initialized in the desired location. After that, an ERGOnline message 816 can be sent to the client 810. , indicating that the client can begin using the ERG. Uses implicit communication to communicate the requested ERG's usage instead of explicit messages. You can notify clients that you are ready to start a new session, for example by using Entry of a requested ERG will display a visual icon indicating that the ERG is now available for use. In at least some embodiments, the ERGOnline message One or more prerequisites must be met before the equivalent of a message is sent to the client. For example, an OCC instance may need to send a management command to an ERG. If you need to establish a secure communication channel that can be used to Also, there may be some minimum number of virtualization hosts that must boot successfully.

[0075] In the illustrated embodiment, the VCS client 810 includes a program interface 87 7 to match the specified ERG (for example, indicated by the parameter ERGID) A request 817 may be submitted to start one or more virtual machines using the same virtual machine ID. 17, in at least some embodiments, sends the corresponding command(s) to the ERG. The VCS control plane is connected to the VCS via a path different from the one-way path used to receive A command to start the VM(s) is sent from the OCC to the ERG. After the message 821 is displayed, the message 821 indicates that the boot has started. In at least some embodiments, the LaunchIn initiated message 821 indicates that the command to start the VM(s) is It can simply show that the VM was sent to the ERG and not actually perform the work to start the VM on the ERG's EVH. In such an embodiment, the VCS agent process that runs the Note that the ISP may not necessarily send an outbound management message to check whether I want to be.

[0076] In some embodiments, the DescribeERG request 823 is sent to the client 810 to obtain information about the content and / or status of the ERG. The requested information may be provided in the form of an ERGInfo message 825. In some embodiments, the information is, for example, using ERG (e.g., V A list of entities authorized to launch or terminate the M, ERG's EVH and and / or set of VMs, set of VM image caches in ERGs (if any) , the date the ERG was set up, ERG invoice related data, etc.

[0077] In at least some embodiments, the client 810 may, for example, indicate a desired change. Request a modification to the ERG by submitting a modifyERG request 828. Such modifications could include, for example, increasing or decreasing the number of EVHs in the ERG. Request for authorization to use the ERG, set of users / groups / entities authorized to use the ERG The requested modification may include a request to modify a VCS and if a policy-based response is possible, in the illustrated embodiment, the change is implemented. A corresponding workflow can be started to make the change and a change workflow is in progress. The client sends a ModifyInitiated message 833 to the client indicating that In one embodiment, the client 810 may receive a notification that the ERG is no longer needed. An ERG request 841 may be submitted to indicate that the ERG is not configured. In response, the VCS terminates access to the ERG and deletes any Initiate workflow to shut down, remove, and transport away any installed equipment. In some embodiments, the ERGUnconfigured message 843 can be clicked. to inform the client that the workflow has started and that it is no longer possible to In at least some embodiments, the ERG may be inaccessible. It is possible that a different set of program interactions than those shown in Figure 8 may be used for ERG-related manipulations. can be supported for

[0078] Example VM Startup Workflow FIG. 9 illustrates an example of a virtual machine with extended resource groups, according to at least some embodiments. 9 illustrates an example workflow associated with initiating In such scenarios, the ERG934 can be deployed in customer data centers at the request of the VCS client. The client happens to be in data center 930. Using a device 932 (e.g., laptop, desktop, etc.) that may be located Send a request to launch a virtual machine (LaunchVM(ERGID) request 971) The VM can then be started on the ERG934 as shown in step 1 of the workflow. The network path through which the LaunchVM request is submitted is This is the same as the path used for any VCS request that the client submits through the VCS's public API. Therefore, this path is not supported by the VCS control plane via OCC911. A one-way secure channel set up specifically to send commands to the ERG. In at least one embodiment, the VCS may not include a regional data center 9 10, may be implemented using multiple regional data centers in a provider network. Individual data centers in regional data centers are, for example, supported by provider networks. They can be set up in various metropolitan areas, states, or countries that provide services. In some embodiments, when a particular VCS client submits a program request, For example, based on the source IP address from which the request is submitted, or on the client's Specific geographic data sets are created based on the "home" region that a count is associated with by default. The call may be routed to a center.

[0079] In the embodiment depicted in FIG. 9, the control plane in a regional data center 910 The device 912 receives the LaunchVM request 971, processes the request, and The internal version of hVM command 972 is used when the VM is instantiated (workflow (as shown in step 2 of the above) to the OCC911 associated with the ERG. In at least some embodiments, it is possible to start virtual machines in individual ERGs. A database 931 indicating the users / groups / entities that are authorized to As part of the processing of the LaunchVM request in the control plane device 912, uses database 731 to allow the requester to request a In one embodiment, the method may include verifying that the user is authorized to launch a VM of the user. OCC911 will respond to requests submitted by the client and / or LaunchVM Can participate in processing / validating the internal version of commands.

[0080] After the processing and / or verification operations related to the VM startup have been successfully completed, in the illustrated embodiment As part of step 3 of the workflow, a converted version of the command is ERG Network Manager on ERG934 via secure one-way channel 973 from C911 As part of the operations performed by the OCC, several In some embodiments, one or more security tokens may be deleted or replaced on command. You can create a message authentication code, such as HMAC, for your command. , command entries can be stored in a log by OCC911, etc. .

[0081] In the illustrated embodiment (step 4 of the workflow), the ENM 935 The specific EVH937 selected for the machine will transmit the received command to the VCS controller. In some embodiments, the EN The M and / or agent 935 may request that the VM requester provide local data to the ERG. The launcher can then be re-verified as being among the authorized launchers shown in database 938. In at least one embodiment, the ENM and / or agent also comprises an OCC. It can store log entries indicating which commands were received from which particular EVH93 The decision as to which of the following to use can, in different embodiments, be one of several approaches: In one embodiment, the client may use, for example, the first In the uchVM request, indicate the specific EVH to be used for the VM in the specified ERG. In another embodiment, the VCS control plane device 912 One or more deployment managers running on the same cluster determine the specific EVH where the requested VM should be launched. 937 may be selected. In some embodiments, OCC911 and / or The ENM 935 may be responsible for selecting a particular EVH for a requested VM. The aunchVM command is EVH937 and the VCS control plane agent 93 5, a new VM 922 of the requested type is sent to, for example, In the illustrated embodiment, the same techniques may be used to instantiate the In various embodiments, message patterns similar to those shown in FIG. 9 may be used in other types of For example, the client may request a This differs from the one used for the corresponding VCS control plane issued termination command. A request to terminate a VM in the ERG can be submitted using different routes. In some embodiments, a different workflow than that shown in FIG. 9 may be used to generate the ERG You can start virtual machines and / or send other types of client requests Different routes may be used for the messages involved in fulfilling a request.

[0082] Example Use of Additional Provider Network Services in an ERG In some embodiments, a provider in which the virtualization computing service is implemented The network is also more flexible than other networks that use VCS virtual machines as building blocks. It may provide access to a high level of network-accessible services, e.g. The database instance may be implemented using a virtual machine and may have network access. A database service may be provided for use by a client of the database. In the form, such higher level services also implement features of the service, e.g. Use a local VM set up within the ERG to make it available to the ERG. In addition, other sub-systems of the provider network that do not depend directly on the VCS virtual machine can be The service also provides, in some embodiments, that such services are provided through a provider network. The same way that it can be accessed from a VM set up in a data center , which can be accessed from VMs set up in the ERG. According to some embodiments, in an extended resource group of a virtualized computing service Illustrates an example of the use of additional provider network services.

[0083] In the illustrated embodiment, the provider network 1001 includes at least a virtualized computer. VCS (similar in function and features to VCS110 in Figure 1) , storage services 1030, and database services 1042. The portion 1010 uses resources located in the provider network data center. and an extension of the VCS may be implemented as a customer data repository that also includes one or more non-VCS servers 1023. Outside the provider network, such as ERG 1035 located within center 1032 can be set up in place.

[0084] In the exemplary scenario depicted in FIG. 10, an isolated virtual network of the type previously discussed is An Intranet Network (IVN) 1015 is established for the VCS clients. IVN 101 5 is a diagram showing a VCS provider network data center part 1010 for multiple virtual machines. VMs (such as VM1017A and 1017B), and located in customer data center 1032 Contains multiple virtual machines (such as 1018J and 1018K) within the ERG1035 In the illustrated embodiment, a program running in one of the VMs may access other providers. Network service resources can be used. For example, storage services Storage instance 1022 in 1030 is connected to VM 101 in the provider network. 7B and from VM 1018K in ERG 1035, and similarly, the data The database instance 1042 of the database service 1040 is It can be accessed from VM 1017A in the network and from VM 1018J in the ERG. In the illustrated embodiment, the VMs in the ERG 1035 communicate with other provider network services. , which can be provided with access to The access provided to the VCS virtual machine is logically equivalent to that provided to the virtual machine. Services built on top of virtual machines (i.e. using VMs as building blocks) The configuration and use of the service (which is provided by the ERG) is done in a seamless manner without the need to access resources outside the ERG. In the illustrated embodiment, it may be executed locally. In one embodiment where the database instance comprises a VCS virtual machine, the new database instance The stance is to use the ERG VM1018J to transfer the data from another ERG VM1018K. In response to a request from an instance of B, it may be created locally within the ERG. In the illustrated embodiment, for VCS customers on whose behalf the ERG 1035 is set up, ERG-based virtual machines such as 1018J and 1018K are not part of the provider network. It is functionally equivalent to VMs such as 1017A and 1017B that are instantiated within the However, the control plane components of the VCS It is implemented within provider network data centers and is not replicated in the ERG. In at least some embodiments, if an ERG and a provider network data center If the connection between the ERG and the ERG is interrupted, the VMs already instantiated in the ERG and the ERG Resources for other services set up in RG (such as the database service mentioned above) The instance can continue to function for at least some period of time, so that clients A continuous connection to the provider network data center for the ERG to be used may not be necessary.

[0085] How to expand your provider network services FIG. 11 illustrates a method for establishing an extended resource group and temporarily To allow virtual machines to be set up in extended resource groups, the provider FIG. 1 is a flow diagram illustrating aspects of operations that may be performed in a data network. 1101, can be set up on premises outside the provider network Expanded resource group of virtualized computing services of provider networks A table of families of loop (ERG) categories (e.g., "small" ERGs, "large" ERGs, etc.) The indication may optionally be provided, for example, via a programmatic interface of the VCS. ERGs can, for example, provision VCS managed compute instances ( Virtual machines and bare metal instances, etc., that can access the Internet. Anywhere available (customer data centers, colocation facilities, telecommunications companies, energy - Set up and securely manage your organization's remote field locations A given ERG may, for example, in at least some embodiments, , one or more pre-configured hardened virtualization hosts (EHs), The resource pool may include each set of resources that have been allocated.

[0086] In at least one embodiment, VCS control plane commands are sent to the provider Outbound command code for secure transmission from the network data center to the ERG For example, one implementation can set up an OCC instance. In this form, one or more provider network data centers are dedicated to ERG management. The virtual machines are connected to the network by the client in a specific isolated virtual network (IVN) that is set up in the Outbound Command Communicator assigned to the specific ERG requested by In some embodiments, a stance OCC1 may be established (element 1104). VCS compute instances may be used to implement at least a portion of the functionality of the OCC. In one embodiment, OCC1 is configured to: For example, delivery / deployment of ERG resources at external locations as indicated by the client can be established before.

[0087] In OCC1, (a) client resources are located in the provider network’s data center. (b) a secure network established for communication between the ERG and some set of An indication of the call channel may be obtained (element 1107) in the illustrated embodiment. For example, ERG network managers and provider network data centers VPN tunnels set up between the client IVNs in the A connection established over a dedicated physical link designated for client traffic. After the secure channel is established, program requests for ERG operation (e.g. For example, a request to launch a compute instance on an ERG) may, in various embodiments, be Instead, the VCS control plane is set up by a client. A device (e.g., A running on a server in a provider network data center) PI handler).

[0088] According to various embodiments, in response to a client request, the VCS control plane When a generated VM start command (or other management command) is sent to the ERG A determination can be made in OCC1 (element 1110) as to whether the command is to be executed. Address Addr1 is at least one ISP address located in the provider network data center. Part of the address range used by the VCS control plane on some hosts In one exemplary scenario, OCC1 may have an associated There may be one or more virtual network interfaces, and the display of such commands may include: A specific virtual network associated with the VCS control plane and connected to OCC1. The OCC1 can be obtained by the OCC1 through the VNI1 network interface. As discussed above (e.g., by setting the address of VNI1), In some embodiments, the address of the target host EH may be assigned. In some embodiments, the client that generated the VCS control plane command The client request is sent via a secure network channel set up between OCC1 and the ERG. Note that the VCS control plane may receive packets using paths that do not include the channel. In at least one embodiment, the security settings of VNI1 are may not allow messages from OCC1 to be sent to the VCS control plane. .

[0089] In the illustrated embodiment, OCC1 transmits the converted version of the command using a secure channel. version (e.g., to reduce the possibility of misuse of security artifacts in the ERG) Some security artifacts were removed / replaced / re-signed to ensure G (element 1113). In one embodiment, the converted version of the command The implementation may, for example, include one or more additional intermediaries similar to those considered in the context of Figure 5. The ERG's network manager (ENM) may then send the request to the ERG via a path that includes the The address used to deliver messages containing commands to the ENM may vary depending on the In an embodiment, the address range of a first network set up on an external premises, e.g. For example, to a client regardless of the address used within the VCS control plane. The command may be part of the address range selected by the ENM. In some embodiments, one or more address translations may be sent to The execution along the path traversed by the message(s) used to communicate to the ERG In some embodiments, the VCS control plane command may be It does not necessarily need to be converted before being sent to an ENM or other resource.

[0090] In the ERG, in some embodiments, the translated version of the command is It can be obtained by the instantiated VCS control plane agent. In one embodiment, the EH is connected to a second network set up at an external premises. In some embodiments, the ENM may be assigned an address Addr1 within the Both networks at the local premises, i.e. those configured by the client, and addresses from the range used by the VCS control plane are assigned to the host. An agent in the EH can be part of a second network that is managed by a new You can start a compute instance and terminate an existing compute instance. The EVH can then take the requested action, such as by sending a request to the EVH (element 1116). In the illustrated embodiment, a similar workflow is performed for additional requests submitted by the client. For example, the operations corresponding to elements 1110 to 1113 are It can be repeated for various client requests as long as it remains operational. Also, in various embodiments, various requirements regarding the configuration of the ERG itself, e.g., the text of FIG. As discussed in the previous section, requests to view, modify, or terminate the ERG are handled by the VCS control. The request can then be submitted to the trawl plane and the VCS can carry out the requested operation. Cut.

[0091] FIG. 12 illustrates a virtualized computing service, in accordance with at least some embodiments. Element 12 is a flow diagram illustrating aspects of operations that may be performed on an extended resource group. As shown in Figure 01, the provider network VCS Extended Resource Group (ER G) (e.g., hardened with pre-installed software / firmware One or more racks or other containers of security servers or hosts ( and set up at a location in response to a program request submitted to the ERG. The location where the equipment is set up may, at least in some embodiments, be a customer device. data centers, colocation facilities, or organizations such as telecommunications or energy companies Premises outside the provider network's own data centers, such as remote work locations of the organization may include.

[0092] One or more trigger signals (at least one device in the ERG is switched on) and / or an indication that the ERG has enabled connectivity to the Internet At least an ERG network manager (ERG) responds to the NM) in some embodiments, using one or more resources of the ERG, In one embodiment, the functionality of the ENM may be instantiated or initiated (element 1204). At least a portion may be implemented on a hardware card (e.g., a PCI card). In other embodiments, the NM may be one or more computing devices or hosts of the ERG. An ENM may include one or more processes or threads of execution in a Now, bootstrap or re-boot at least some of the other hosts / servers in the ERG. In at least some embodiments, the ENM For example, boot from the VCS control plane host network address range. One or more DHCP functions, including allocating addresses to hosts that are strapped to them The ENM itself can, in some embodiments, network, i.e., one network set up by a customer on whose behalf the ERG was delivered A second ( It can be configured as a member of a private network.

[0093] Secure network channels (e.g., VPN connections / tunnels and / or dedicated A connection that uses a physical link, for example, an interface from an ENM to the VCS control plane. - Various executions occur as a result of a workflow initiated by a request submitted via the Internet. In an embodiment, a resource may be established / configured between the ENM and one or more resources of the VCS (element 1 207). In at least some embodiments, the VCS control plane to the ERG An outbound command server that is specifically set up to send / forward management commands to The communicator instance OCC1 has access to use a secure channel. In at least one embodiment, as described above, OCC1 is a VCS component. One VNI that is used only to receive commands from the control plane, and one VNI that is used only to receive commands from the control plane. Send commands (possibly modified versions) to the ERG via a secure channel. Multiple virtual network interfaces, including separate VNIs that can be used to A VNI may include a virtual machine with an attached VNI.

[0094] After the secure channel is established, the ERG is set up on behalf of the VCS client. The account (and / or other users / groups that are granted access to the ERG) may vary In one embodiment, a program request (to launch a compute instance on an ERG virtualization host) In the illustrated embodiment, the ENM of the ERG may start sending Control plane commands from OCC1 (e.g. ERG is set up on your behalf) It is generated by the VCS control plane in response to requests submitted by clients The generated version of the launch VM command may be received (element 1210). In some embodiments, a path similar to that described above may include one or more message transformations, address In at least one embodiment, the command E The network address received by the NM is part of the customer configured network above. Possible. The ENM sends commands to the control plane agent on the ERG host. The command can then be implemented there (e.g. to instantiate a VM). If the command was generated in response to a client request for (element 1213) The host may use various In an embodiment, the ERG may be part of a private network set up Used in the VCS control plane for other hosts in the second network A network address can be assigned from the range that is created by the VCS control plane. If additional commands generated by the ENM are received, the ENM forwards them to the appropriate virtualization host. In the illustrated embodiment, the operations corresponding to elements 1210 and 1213 are repeat.

[0095] The ENM, in some embodiments, transmits the information from the VCS control plane to the ERG host. In addition to transferring commands, several other types of functions can be performed. If necessary, for example, ENMs can also be public-to-private or private It performs the IP address translation from the IP address range to the public address range and other encapsulation related operations. Traffic between the deployed compute instance and devices in the external premises where the ERG is deployed. It allows traffic to flow (element 1216).

[0096] In various embodiments, some of the operations shown in FIG. 11 and / or FIG. The steps may be performed in a different order than shown, or may be performed in parallel rather than sequentially. Additionally, some of the operations shown in FIG. 11 and / or FIG. may not be required in one or more implementations.

[0097] Use Cases Network access for provider networks such as virtualized computing services The data plane of the accessible services is logically distributed to premises outside the provider network. The above techniques for scaling efficiently and securely can be very useful in a variety of scenarios. Providers for features such as availability, reliability, security, and affordability. Many organizations use network resources (such as virtual machines at various levels of capacity) Locations of your choice, e.g. customer data centers, colocation facilities, and / or temporary You may wish to use similar resources within your own fieldwork site. Thus, such a location may be a valuable resource for large amounts of application data that may be stored in that location. For access to the Internet, it may be preferred, and the provider network clients may Such large amounts of data are delivered to the provider network's own data centers via the Internet. If you prefer to avoid the costs and / or delays associated with transferring data and, where possible, ensure that data does not leave the customer's own premises. Clients may wish to use the TLS protocol when their Internet connection is unreliable or highly Remote locations where data may be valuable (e.g., oil rigs, cell phone towers, scientific data collection centers) Virtual machines and / or virtual machines in a building near a sensor array There may be cases where you need to use a service to act as a lock. For example, Organizations have many engineers / designers in offices or other physical locations and Near / Designer's ultra-low latency virtual machine for rendering or or other computationally intensive operations may be required. Anywhere with an Internet connection Provided in provider network data centers using local equipment at desired locations The ability to support the same capabilities as those provided by the provider network allows efficient The range of applications that can be run efficiently and safely can be greatly expanded.

[0098] Exemplary Computer System At least some embodiments support the types of service enhancement techniques described herein. Ports (e.g., at selected locations outside the provider network's data centers) Setting up and configuring extended resource groups for virtualized computing services The control plane and data plane components used to A server implementing one or more of the following functions may provide one or more computer-accessible media: may include a general-purpose computer system configured to include or access FIG. 13 illustrates such a computing device 9000. In an embodiment, the computing device 9000 includes an input / output (I / O) interface. System memory 9020 (non-volatile and volatile memory modules) via interface 9030 The computer includes one or more processors 9010 coupled to a processor (which may include both a processor and a processor). The operating device 9000 is connected to a network The device further includes a network interface 9040.

[0099] In various embodiments, the computing device 9000 includes one processor 90 10, or a single processor system 9010 (e.g., 2 The system may be a multi-processor system including multiple processors (eg, 1, 2, 3, 4, 8, or another suitable number). The processor 9010 may be any suitable processor capable of executing instructions. For example, in various embodiments, the processor 9010 may be an x86, PowerPC, SPAR Any of a variety of instruction sets, such as the C, or MIPS ISA, or any other suitable ISA. The processor may be a general-purpose or embedded processor implementing an Intel Integrated Services Architecture (ISA). In a multiprocessor system, each of the processors 9010 typically, but not necessarily In some implementations, the graphics processor may implement the same ISA. Graphics processing units (GPUs) are used instead of or in addition to traditional processors. It is possible.

[0100] The system memory 9020 is an instruction memory accessible by the processor(s) 9010. In at least some embodiments, the system may be configured to store commands and data. The stem memory 9020 may include both volatile and non-volatile portions, and in other embodiments In various embodiments, the system memory 9020 may use only volatile memory. The volatile part is static random access memory (SRAM), synchronous dynamic The memory may use any suitable memory technology, such as a microprocessor-based RAM, or any other type of memory. It may be implemented in a non-volatile portion of system memory (e.g., including one or more NVDIMMs). In some embodiments, a flash device, including a NAND flash device, In at least some embodiments, a memory device based on the The non-volatile portion of the system memory is powered by a supercapacitor or other power storage device ( In various embodiments, the power source may include a memristor-based resistor. ReRAM, 3D NAND technology, ferroelectric RAM, magnetoresistance either magnetic random access memory (MRAM), or various types of phase change memory (PCM) At least one non-volatile portion of the system memory may be used. These methods, techniques, and data include program instructions that implement one or more desired functions. The instructions and data are stored in the system memory 9020 as code 9025 and data 9026. The information is stored and displayed in the memory.

[0101] In one embodiment, the I / O interface 9030 is Memory 9020, such as various types of persistent and / or volatile storage devices, Devices that include a network interface 9040 or other peripheral interfaces The I / O traffic between any of the peripheral devices in the In some embodiments, the I / O interface 9030 may include a single component (e.g., For example, a data signal from a system memory 9020 to another component (e.g., a processor Any protocol required to convert the data into a format suitable for use by the In some embodiments, the I / O The interface 9030 may be, for example, a peripheral component interconnect (PCI) bus standard. or variants of the Universal Serial Bus (USB) standard In some embodiments, the I The functions of the / O interface 9030 are, for example, the north bridge and the south bridge. Also, in some embodiments, , an I / O interface 9030, such as an interface to system memory 9020; Some or all of the functionality may be incorporated directly into the processor 9010.

[0102] The network interface 9040 is connected to the computing device 9000. For example, other computer systems or devices such as those illustrated in FIGS. Other devices 906 connected to a network or networks 9050 0. In various embodiments, The network interface 9040 may be, for example, an Ethernet network type. Supports communication over any suitable wired or wireless general data network, such as a In addition, the network interface 9040 can be used to connect to an analog voice network. Telecommunications / telephone networks, such as digital fiber optic networks over a storage area network such as a Fibre Channel SAN, or communication over any other suitable type of network and / or protocol. can be supported.

[0103] In some embodiments, the system memory 9020 may be any of the memory types discussed in the context of FIGS. At least one program instruction and data used to implement the method and apparatus 1 illustrates an embodiment of a computer-accessible medium configured to store a subset. However, in other embodiments, the program instructions and / or data may be The data may be received, transmitted, or stored on different types of computer-accessible media. Generally speaking, the computer accessible medium may be any type of medium, such as a magnetic or optical medium. A non-transitory storage medium or memory medium, such as a computer, is connected via an I / O interface 9030. A disk or DVD / CD may be coupled to the computing device 9000. The non-transitory computer-accessible storage medium may also be the system memory 9020 or another In some embodiments of the computing device 9000, RAM (e.g. SDRAM, DDR SDRAM, RDRAM, SRAM, etc.) The memory may include any volatile or non-volatile medium, such as a ROM, a memory controller, or the like. In this embodiment, the non-transitory computer readable storage media may be implemented on one or more processors. When run across them, they implement at least a subset of the methods and techniques described above. The computer-accessible medium may collectively store program instructions for: , a network and Any electrical, electromagnetic or other signals transmitted over a communications medium, such as a radio link, The present invention may further include a transmission medium or signal, such as a digital signal. Various implementations may be implemented using parts or all of multiple computing devices such as In certain embodiments, the described functionality may be implemented, for example, on a variety of different devices and servers. Software components operating in conjunction with each other may provide functionality. In some embodiments, portions of the described functionality may be implemented using a general-purpose computer system. In addition to or instead of a storage device, network device, or dedicated computer As used herein, "computer" refers to a method and apparatus for performing a programmable logic operation using a computer system. The term "operating device" refers to at least all of these types of devices. , but is not limited to these types of devices.

[0104] Embodiments of the present disclosure may be described in light of the following clauses. 1. A system comprising: One or more compute instances of the provider network's virtualized computing services a provider network including one or more data centers; , a virtualization host for virtualized computing services located in one or more data centers; The set of lists is a list of each network address in the first network address range. is assigned, The one or more computing devices include instructions, which are executed by one or more processors. When executed on or across a server, the server may cause one or more computing devices to The Outbound Command Communicator associated with the extended resource group and obtaining a virtual machine start command, the outbound communicator receiving the first a first network address within a network address range of A group consists of one or more virtualized hosts located on-premises outside one or more data centers. the virtual machine start command includes an indication of the first network address. To do, The virtual machine startup command is sent to the extended resource group over a secure network channel. sending a message indicating the first network address, The destination address of the message is the secondary network address used at the premises. The target virtualization host is part of the address range established in the extended resource group. The target is assigned a first network address in the private network. Processing the virtual machine start command on the source virtualization host is synchronized with the virtual machine start command on the target virtualization host. One or more computers that cause the instantiation of a machine System. 2. A secure network channel is established between one or more data centers and an extended resource group. The secure network channel established between the two parties may be (a) a VPN tunnel, or (b) b) Includes one or more dedicated physical links between the premises and the provider network. 4. The system described in clause 1. 3. The one or more computing devices further include instructions, the instructions comprising one or more When executed on or across multiple processors, it represents one or more computing devices. To the vice, A request to establish an extended resource group is submitted via a program interface. determining that a command has been issued, the outbound command communicator comprising: determining whether the extended resource group is established in response to a request to establish the extended resource group; The system according to any one of claims 1 to 2, 4. The one or more computing devices further include instructions, the instructions comprising one or more When executed on or across multiple processors, it represents one or more computing devices. To the vice, prior to determining that a request to establish a secure network channel is received, The provision further causes an outbound command communicator to be instantiated. The system according to any one of items 1 to 3. 5. The outbound command communicator runs on a virtual machine on a separate virtualization host. and other virtualization hosts are configured to use a second network address within the first network address range. 5. A system according to any one of clauses 1 to 4, in which addresses are assigned. 6. A method comprising: by one or more computing devices in the provider network; The control plane of the provider network's computing services receiving a request to establish a computing instance; The first outbound command communicator in the provider network responds to requests. In response, you get an indication that a compute instance is established on the target host. The target host is a provider network located on the client premises. The first network address is part of the first extended resource group of the The role plane associates with the target host and issues the first outbound command The communicator is assigned at least a first network address, and, Sending a message indicating the second network address as the destination and the second network address is a first network address established at the client premises. network, and the message establishes a compute instance on the target host. a command to establish a first network address at the client premises; The target host is assigned to the second network established in the Processing the command results in the establishment of a computation instance, sending, and executing The method includes: 7. The method of clause 6, wherein the compute instance comprises a virtual machine. 8. by one or more computing devices in the provider network; Supported by hosts in the provider network via a programmatic interface providing a representation of a set of virtual machine categories that are The method further includes providing a set of virtual machines belonging to a particular virtual machine category. 8. The method according to any one of clauses 6 to 7. 9. The computation instance obtained by the first outbound command communicator The indication that the authentication is established includes a first security artifact, and the method further comprises: by one or more computing devices of the internetwork, A second security artifact is included in messages sent to a second network address. a second security artifact including a first out Further including executing the include generated by the bound command communicator. , the method described in clauses 6 to 8. 10. By one or more computing devices in the provider network, The first outbound command communicator is connected to the first virtual network interface. The goal was to programmatically connect to a virtual network interface and a second virtual network interface. The first virtual network interface is connected to the control plane and the compute is used to obtain an indication that a second virtual network instance is established. The interface is a secure network through which messages are sent to the first extended resource group. Performs a programmatic connection used to access a work channel The method according to any one of clauses 6 to 9, further comprising: 11. A secure network channel is: (a) a virtual private network (VPN) ) tunnel, or (b) a dedicated tunnel between the client premises and the provider network. 11. The method of claim 10, comprising one or more of the physical links. 12. The first outbound command communicator is at least partially a virtual machine. 11. The method according to any one of clauses 6 to 10, implemented using 13. The first extended resource group is the provider network computing The first outbound command communication established on behalf of the first client of the service The communicators are configured in isolated virtual networks of the computing service, The virtual network is established on behalf of a second client of the computing service. At least a second outgoing connection established to communicate with the second extended resource group Includes Bound Command Communicator, as described in any one of clauses 6-10 or 12 How to. 14. The first extended resource group is the provider network computing The method comprises: establishing on behalf of a client of the service one or more computing devices; By Vice, The compute instance is assigned a unique IP address that is established within the provider network on behalf of the client. Assign a network address from the network address range of the isolated virtual network 6-10 or 12-13, which further includes carrying out the Method of posting. 15. A request to establish a compute instance is made to a provider network compute Public Application Programming Interfaces for the The message is received over a first network path associated with the first network and the message is transmitted over a second network path. 6-10 or 12-13, which is sent to a second network address via a network path. 14. The method according to any one of claims 1 to 14. 16. A system comprising: Computing devices running on one or more computing devices in a service provider environment Troll plane and An outbound command controller coupled to a first network in a service provider environment. a first network using a range of network addresses; The outbound command communicator sends a command to the first network address in the range of network addresses. A network address is assigned to the first network in the service provider environment. Outbound command, including multiple servers configured to host computing instances and a communicator, The control plane is A first message including a request to launch a compute instance is sent to a first network configured to send to the network address, The outbound command communicator Receiving a first message; Send a second message to a device that is assigned a second network address. and wherein the second message comprises at least one request to launch a computation instance. and the second network address includes a portion of the first network address established at the customer data center. The device is part of a third network established in the customer data center. configured to transmit a second message to a target server coupled to the network. and the third network comprises at least a portion of the range of network addresses. The target server is assigned the first network address and sends and a system configured to: 17. The second message is sent via (a) the VPN tunnel, or (b) the customer data center. and the service provider environment via one or more of the dedicated physical links. The system of claim 16, wherein the system is transmitted to a network address. 18. Compute Instances are the same as those described in clauses 16-17, including bare metal compute instances. The system is as follows: 19. Outbound command communicators: A second message is sent to a second network address, which is then forwarded to the target server. The condition is further configured to include a security object that can be authenticated by Item 19. The system according to any one of items 16 to 18. 20. A device that is assigned a second network address can communicate with the second network. and a third network, the device being causing a destination of the second message to be the first network address; and , and transmitting the second message to the target server. The system according to any one of clauses 16 to 19. 21. A system comprising: Network manager and Extended resource groups for provider network virtualized computing services one or more computing devices, Virtualization host including secure storage devices and Trusted Platform Modules The virtualization hosts are located on premises outside the provider network. The first network configured at the premises has a first range of network addresses. Virtualized computing resources located in one or more data centers in the provider network. The virtualization hosts for the computing service are assigned from a second range of network addresses. One or more computing devices, each of which is assigned a network address and The network manager includes instructions, the instructions being executed on or by one or more processors. When the network manager is The endpoint associated with the extended resource group is connected to the A message sent by the bound command communicator containing a command to start a virtual machine. The virtual machine start command is to obtain a virtualized computing device page. provided through the public application programming interface of the Messages generated in response to requests made and assigned to the network manager The destination address of the network address to be obtained is within the first range of network addresses. And, Virtual machine start command from network manager for extended resource group virtualization The virtual machine start command is sent to the virtualization host, and the processing of the virtual machine start command on the virtualization host is This results in the instantiation of a virtual machine on the virtualization host and the creation of the virtual machine. Within the private network, the virtualization hosts are assigned addresses in a second range. A system that transmits and causes to be performed. 22. At least a portion of the network manager (a) is responsible for the extended resource group A power-on signal on a specific device, or (b) a specific device in an extended resource group. is accessing the Internet , the system described in clause 21. 23. The network manager further includes instructions, the instructions including instructions for controlling one or more processors When run on or across them, the network manager is informed Perform one or more transformation operations on network packets generated by the virtual machine And, Sends a transformed version of the network packet to a network endpoint. a network endpoint that is (a) a part of a provider network; and (b) is not part of an extended resource group. The system according to any one of clauses 21 to 22. 24. The network manager further includes instructions, the instructions being for controlling one or more processors. When run on or across them, the network manager is informed Initiate the bootstrap operation on the virtualization host, as described in any one of clauses 21 to 23. The system described. 25. At least a portion of the network manager is responsible for the peripheral components of the extended resource group. The system according to any one of clauses 21 to 24, implemented in a 26. A method comprising: In the Network Manager for the Extended Resource Group of the Provider Network, A message containing the command to launch a compute instance on a host in the source group. and obtaining a host on a premises outside the provider network. The message is sent to a first address range of a first network configured at the premises. obtaining an address obtained by a network manager; Send commands from the network manager to the hosts to create compute instances. wherein in a second network configured at the premises, The host is assigned an address in the second address range and receives one The above addresses are sent to the provider by the provider network control plane. and transmitting the assigned IP address to each host in the network. 27. Before instantiating a compute instance, use the host security module. Clause 26 further includes verifying that the Host meets the Acceptance Criteria by The method described above. 28. The host includes a storage device, and the method comprises: Store an encrypted version of the compute instance's state information on a storage device. decryption of the encrypted version is performed by the storage device Any of Articles 26 to 27, which require the data to be physically connected to the computer or which further includes storing the data The method according to any one of the preceding claims. 29. The extended resource group includes a compute instance image cache and To instantiate an instance, Provisions, including using a particular compute instance image retrieved from a cache 29. The method according to any one of 26 to 28. 30. One or more instances that are authorized to request the launch of a compute instance in an extended resource group. Obtaining a representation of the entity on the Before you can instantiate a compute instance, you must ensure that the compute instance is associated with one or more entities. and verifying that the request has been made by an entity of the entity. 30. The method according to any one of items 26 to 29. At least a portion of the network manager is connected to the peripheral interface. Any of clauses 26 to 30 implemented on cards connected to hosts in the extended resource group The method according to any one of the preceding claims. 32. Hosts are organized into racks in extended resource groups and are managed by network managers. At least a portion of the equipment is implemented in one or more devices that are not installed in the rack. The method according to any one of clauses 26 to 31. 33. Instantiating a Network Manager in response to one or more trigger signals And, A network manager communicates with at least a portion of a provider network. and initiating configuration of a secure network channel to 6. The method according to any one of claims 6 to 32. 34. One or more trigger signals may be (a) a power supply for a particular device in an extended resource group; (b) a power-on signal that a particular device in the extended resource group has access to the Internet 34. The method of claim 33, comprising one or more indications that the device is accessing the 35. The network manager can now be used to initiate bootstrap operations on hosts. 34. The method according to any one of clauses 26 to 33, comprising: 36. A system comprising: One or more computing devices in the extended resource group of the provider network At least one host in the extended resource group is in the provider network. Located on external premises, The one or more computing devices include instructions, which are executed by one or more processors. When executed on or across a server, the server may cause one or more computing devices to Network manager for extended resource groups to launch compute instances and receiving a message including a command for The network manager can then use the address range to determine which network address is being used. Obtained by, obtaining, Sending a command from a network manager to a host, the processing of the command being The process results in the instantiation of compute instances on the hosts and the Within the second network, hosts are assigned addresses within a second address range; One or more addresses in the second address range are used by the control processor of the provider network. A lane assigns a transmission bandwidth to each host in the provider network. A system including one or more computing devices for performing and causing the above to be performed. 37. The one or more computing devices further include instructions, the instructions comprising: When executed on or across processors, one or more computing On your device, Before instantiating a compute instance, use the host security module to and the system described in clause 36, which allows the host to verify that the acceptance criteria are met. . 38. The one or more computing devices further include instructions, the instructions comprising: When executed on or across processors, one or more computing On your device, Store an encrypted version of the compute instance's state information on a storage device. decryption of the encrypted version is performed by the storage device The further requirement of storing the information is that it be logically connected to the information, and Articles 36-37 13. The system according to claim 12 . 39. The extended resource group includes a compute instance image cache and To instantiate an instance, Provisions, including using a particular compute instance image retrieved from a cache 39. A system according to any one of 36 to 38. 40. The message is sent over (a) a dedicated physical connection between the premises and the provider network; link, or (b) a virtual private network (VPN) tunnel. The information in any one of clauses 36 to 39, obtained by the network manager via the above The system is as follows:

[0105] conclusion Various embodiments are implemented in accordance with the foregoing description on a computer-accessible medium. It may further include receiving, transmitting, or storing instructions and / or data. Generally speaking, the computer accessible medium may be a magnetic or optical medium, e.g. disk or DVD / CD-ROM, RAM (e.g. SDRAM, DDR, RDRAM, SRAM, volatile or non-volatile media such as ROM, via communications media, such as memory media, and networks and / or wireless links; This includes transmission media or signals, such as electrical, electromagnetic, or digital signals, This can be done.

[0106] The various methods as illustrated in the drawings and described herein are exemplary embodiments of the methods. The method may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed and various elements may be added, rearranged, combined, It may be omitted or modified.

[0107] Various modifications and changes may be made, as will be apparent to those skilled in the art having the benefit of this disclosure. It is intended to include all such modifications and alterations and therefore to consider the above description in an exclusive sense. The present disclosure is intended to be regarded in an illustrative sense rather than a restrictive sense.

Claims

1. one or more computing devices of an extended resource group of a provider network, where at least a first host of the extended resource group comprises one or more computing devices located at a first premises outside the provider network; The one or more computing devices may include: obtaining a message including a command to launch a compute instance at a first network manager of the extended resource group, the message being obtained by the first network manager at an address within a first address range of a first network configured at the first premises; sending the command from the first network manager to the first host, where processing of the command results in instantiation of a compute instance at the first host, where within a second network configured at the first premises, the first host is assigned an address within a second address range, and where one or more addresses in the second address range are assigned by a control plane of the provider network to each host in the provider network; comprising instructions which, when executed on a processor, cause system.

2. at least a portion of the first network manager is implemented on a card connected to a host of the extended resource group via a peripheral interface; The system of claim 1 .

3. The first host is embedded in a first rack of the extended resource group, and at least a portion of the first network manager is implemented in one or more devices that are not embedded in the first rack. The system of claim 1 .

4. the message is received at the first network manager via one or more of: (a) a dedicated physical link connecting the external premises to the provider network; or (b) a virtual private network (VPN) tunnel; A system according to any one of claims 1 to 3.

5. the one or more computing devices further include instructions, when executed on the processor, that cause the one or more computing devices to verify that the first host meets an acceptance criteria before instantiating the computing instance using a security module of the first host. A system according to any one of claims 1 to 4.

6. the one or more computing devices further include further instructions, when executed on the processor, that cause the one or more computing devices to store an encrypted version of the state information of the computational instance in a first storage device, and decrypting the encrypted version requires the first storage device to be physically connected to the first host. A system according to any one of claims 1 to 5.

7. the extended resource group includes a compute instance image cache, and instantiating the compute instance includes utilizing a particular compute instance image obtained from the compute instance image cache. A system according to any one of claims 1 to 6.

8. the one or more computing devices further include instructions, when executed on the processor, that cause the one or more computing devices to verify, prior to instantiating the compute instance, that the compute instance has been requested by an entity that is authorized to request activation of compute instances in the extended resource group. A system according to any one of claims 1 to 7.

9. The first network manager performing a first set of one or more transformation operations on a first network packet generated by the virtual machine; sending a version of the first network packet on which the first set of one or more transformation operations has been performed to a first network endpoint, the first network endpoint being (a) not part of the provider network, and (b) not part of the extended resource group; A system according to any one of claims 1 to 8.

10. Executed by one or more computing devices of an extended resource group of a provider network, wherein at least a first host of the extended resource group is located at a first premises outside the provider network; obtaining a message including a command to launch a compute instance at a first network manager of the extended resource group, the message being obtained by the first network manager at an address within a first address range of a first network configured at the first premises; sending the command from the first network manager to the first host, where processing of the command results in instantiation of a compute instance at the first host, and within a second network configured at the first premises, the first host is assigned an address within a second address range, and one or more addresses in the second address range are assigned to each host in the provider network by a control plane of the provider network. method.

11. and verifying that the first host meets an acceptance criteria prior to instantiating the computing instance using a security module of the first host. The method of claim 10.

12. the first host comprises a first storage device, the method further comprising storing an encrypted version of the state information of the computing instance on the first storage device, and decrypting the encrypted version requiring the first storage device to be physically connected to the first host.

12. The method according to claim 10 or 11.

13. instantiating the first network manager in response to one or more trigger signals; the one or more trigger signals include one or more of a power-on signal at a particular device of the extended resource group, or an indication that a particular device of the extended resource group has access to the Internet. The method according to any one of claims 10 to 12.

14. and initiating, by the first network manager, one or more of: configuring a secure network channel for communicating with at least a portion of the provider network; or a bootstrap operation at the first host. The method according to any one of claims 10 to 13.

15. obtaining a message including a command to start a compute instance at a first network manager of an extended resource group of a provider network, the message being obtained by the first network manager at an address within a first address range of a first network configured at a first premises outside the provider network; storing program instructions executable on or between the one or more processors to cause one or more processors to: send the command from the first network manager to a first host of the extended resource group, the first host being located at the first premises outside the provider network, processing of the command resulting in instantiation of a compute instance at the first host, and within a second network configured at the first premises, the first host being assigned an address in a second address range, and one or more addresses in the second address range being assigned by a control plane of the provider network to each host in the provider network; One or more non-transitory computer-readable storage media.

Citation Information

Patent Citations

  • Thin client system, server device, policy management apparatus, control method, and control program

    JP2016224484A