Virtualized Block Storage Servers in Cloud Provider Substrate Extensions

Abstract

To provide a method for enabling customers to deploy applications in a location-independent manner using provider-managed infrastructure, at sites selected by the customers.SOLUTION: A method includes: receiving, by a computer system of an extension of a provider network, a first request to launch a first virtual machine to host a block storage server application; provisioning a portion of a storage capacity of storage devices of a host computer system to the first virtual machine as a provisioned storage device; and executing the block storage server application with the first virtual machine. The execution includes: creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network; and receiving a third request to perform an input / output operation with the logical volume.SELECTED DRAWING: Figure 12

Description

[Background technology]

[0001] To support their operations, many businesses and other organizations operate computer networks that interconnect many computing systems, e.g., the computing systems may be co-located (e.g., as part of a local network) or alternatively located in multiple different geographic locations (e.g., connected via one or more private or public intermediate networks). For example, data centers housing a significant number of interconnected computing systems have become commonplace, including private data centers operated by and on behalf of a single organization, and public data centers operated by entities as businesses to provide computing resources to customers. Some public data center operators provide network access, power, and secure installation facilities for various customer-owned hardware, while others offer "full-service" facilities that also include hardware resources made available for use by their customers. However, as the size and scope of a typical data center increases, the tasks of provisioning, overseeing, and managing physical computing resources become increasingly complex.

[0002] The emergence of general-purpose hardware virtualization technologies has brought benefits in managing large-scale computing resources for many customers with diverse needs, enabling various computing resources to be shared efficiently and securely by multiple customers. For example, virtualization technologies may enable a single physical computing machine to be shared among multiple users by providing each user with one or more virtual machines hosted by the single physical computing machine. Each such virtual machine is a software simulation that functions as a unique logical computing system that gives the user the illusion of being the sole operator and manager of a given hardware computing resource, while also providing application isolation and security among the various virtual machines. Furthermore, some virtualization technologies are capable of providing virtual resources that span more than one physical resource, such as a single virtual machine with multiple virtual processors that span multiple unique physical computing systems. As another example, virtualization technologies may enable data storage hardware to be shared among multiple users by providing each user with a virtualized data store that may be distributed across multiple data storage devices, each of which functions as a unique logical data store that gives the user the illusion of being the sole operator and manager of the data storage resource.

[0003] Various virtual machine types optimized for various types of applications, such as compute-intensive applications, memory-intensive applications, etc., may be set up in some cloud computing provider network data centers in response to client requests. Furthermore, higher-level services that rely on such provider network virtual computing services, such as some database services in which database instances are instantiated using virtual machines of the virtual computing service, may also be made available to provider network clients. However, for some types of applications, such as applications that process significantly large amounts of data that need to be stored on customer premises outside the provider network, services that are limited to providing virtualized resources using hardware located in the provider network data centers may not be optimal, for example, for latency-related and / or other reasons.

[0004] Various embodiments according to the present disclosure are described with reference to the following drawings. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a block diagram illustrating an example provider network extended with a provider substrate extension located in a network external to the provider network, according to at least some embodiments. [Figure 2] FIG. 1 is a block diagram illustrating an exemplary provider substrate extension, according to at least some embodiments. [Figure 3] FIG. 1 is a block diagram illustrating an example connection between a provider network and a provider substrate extension, according to at least some embodiments. [Figure 4] FIG. 1 is a block diagram illustrating an example of a virtualized block storage system, according to at least some embodiments. [Figure 5]FIG. 1 is a block diagram illustrating an example of a virtualized block storage system, according to at least some embodiments. [Figure 6] FIG. 1 is a block diagram illustrating an example system for booting a virtualized block storage server using a first technique, according to at least some embodiments. [Figure 7] FIG. 1 is a block diagram illustrating an example system for booting a virtualized block storage server using a second technique, according to at least some embodiments. [Figure 8] FIG. 1 is a block diagram illustrating an example system for booting additional compute instances of a provider substrate extension from a block storage server, according to at least some embodiments. [Figure 9] FIG. 1 is a block diagram illustrating an example system for managing virtualized block storage servers, according to at least some embodiments. [Figure 10] FIG. 1 is a block diagram illustrating an example system for providing volume mapping to block storage clients, according to at least some embodiments. [Figure 11] FIG. 1 is a block diagram illustrating an exemplary system for tracking volume mapping, according to at least some embodiments. [Figure 12] FIG. 1 is a flow diagram illustrating operations of a method for starting a virtualized block storage server, according to at least some embodiments. [Figure 13] FIG. 1 is a flow diagram illustrating operations of a method for using a virtualized block storage server, according to at least some embodiments. [Figure 14] FIG. 10 is a flow diagram illustrating operations of a method for managing virtualized block storage servers in a provider substrate extension, according to at least some embodiments. [Figure 15] 1 illustrates an exemplary provider network environment, according to at least some embodiments. [Figure 16]FIG. 1 is a block diagram of an exemplary provider network that offers storage and hardware virtualization services to customers, according to at least some embodiments. [Figure 17] FIG. 1 is a block diagram illustrating an exemplary computing device that may be used in at least some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0006] The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for configuring a provider substrate extension to communicate with networks outside the provider network and for providing virtualized resources on the substrate extension that are the same as or similar to resources available in the provider network. A provider network operator (or provider) offers its users (or customers) the ability to utilize one or more of various types of computing-related resources, including, for example, compute resources (e.g., running virtual machines (VMs) and / or containers, running batch jobs, running code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks including groups of compute resources, content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other compute resources may be provided as services.

[0007] Provider network operators often offer these and other computing resources as services that rely on virtualization technology. For example, virtualization technology may be used to provide users with the ability to control or utilize compute instances (e.g., VMs that use guest operating systems (OSs) running using a hypervisor that may or may not further run on an underlying host OS; containers that may or may not run in a VM; instances that can run on “bare metal” hardware without an underlying hypervisor), allowing one or more compute instances to be implemented using a single electronic device. Thus, users may directly utilize compute instances provided by an instance management service (sometimes referred to as a hardware virtualization service) hosted by the provider network to perform various computing tasks. Additionally or alternatively, users may indirectly utilize compute instances by submitting code to be executed by the provider network (e.g., utilized by an on-demand code execution service), which then utilizes the compute instances to execute the code (typically without the user having any control over or awareness of the associated underlying compute instance(s)).

[0008] Resources that support both the services that provide computing-related resources to users and the computing-related resources provisioned to users may be generally referred to as a provider network substrate. Such resources typically include hardware and software in the form of many networked computer systems. In various embodiments, provider network substrate traffic and operations may be roughly subdivided into two categories: control plane traffic carried on a logical control plane and data plane operations carried on a logical data plane. The data plane represents the movement of user data through the distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. The control plane generally includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic generally includes management operations such as establishing isolated virtual networks for various customers, monitoring resource usage and status, identifying specific hosts or servers on which requested compute instances should be launched, and provisioning additional hardware as needed. The data plane includes customer resources (e.g., computing instances, containers, block storage volumes, databases, file storage) implemented on the provider network. Data plane traffic generally includes non-management operations such as forwarding data to and from customer resources. Control plane components are typically implemented on a set of servers separate from the data plane servers, and control plane traffic and data plane traffic may be transmitted over separate / different networks. In some embodiments, control plane traffic and data plane traffic may be supported by different protocols. In some embodiments, messages (e.g., packets) transmitted over the provider network include a flag indicating whether the traffic is control plane traffic or data plane traffic.In some embodiments, the payload of the traffic can be examined to determine its type (e.g., control plane or data plane). Other techniques for distinguishing between traffic types are possible.

[0009] While some customer applications easily migrate to a provider network environment, some customer workloads need to remain on-premise (“on-premises”) due to low latency, large data volumes, data security, or other customer data processing requirements. Exemplary on-premises environments include customer data centers, robotics integration, field locations, colocation facilities, telecommunications facilities (e.g., near cell towers), etc. To satisfy customer requirements, this disclosure relates to the deployment of board-like resources on-premises. The term “provider board extension” (PSE) refers to a collection of resources (e.g., hardware, software, firmware, configuration metadata, etc.) that a customer can deploy on-premises (e.g., geographically remote from the provider network) but that provides the same or similar functionality (e.g., virtualized computing resources) as those offered in the provider network. Such resources may be physically delivered as one or more computer systems or servers delivered in racks or cabinets, as commonly found at on-premises locations. A PSE can offer customers a set of features and capabilities that can be deployed on-premises, similar to the provider network features described above. In effect, from the perspective of a customer of a provider network, a PSE represents a local extension of the provider network's capabilities that can be set up in any physical location that can accommodate a PSE (e.g., in terms of physical space, power, Internet access, etc.). From the perspective of the provider network itself, the PSE may be considered to be virtually located in the same provider network data center as the core provider network substrate, while being physically located at a deployment site selected by the customer.In at least some embodiments, the customer physically hosting the PSE grants permission to its own customers (e.g., other users of the provider network) to launch instances and host their respective workloads in the PSE at the customer's on-premises location, potentially allowing the workloads to access the customer's network.

[0010] In at least some embodiments, the PSE may be pre-configured, e.g., by a provider network operator, with an appropriate combination of hardware, software, and / or firmware elements to support various types of computing-related resources in a manner that meets various local data processing requirements without compromising the security of the provider network itself or any other customers of the provider network. In at least some embodiments, the PSE is generally managed through the same or similar set of interfaces that customers use to access computing-related resources in the provider network. For example, a customer can provision, manage, and operate computing-related resources in an on-premise PSE or multiple PSEs at various deployment sites via the provider network using the same application programming interfaces (APIs) or console-based interfaces that they otherwise use to provision, manage, and operate computing-related resources in the provider network.

[0011] In at least some embodiments, provider network resources instantiate various networking components to ensure secure and reliable communications between the provider network and the PSE. Such components can establish one or more secure tunnels (e.g., VPNs) with the PSE. Such components can further separate control plane traffic and data plane traffic and handle each type of traffic differently based on factors including the direction of the traffic (e.g., to and from the PSE). In at least some embodiments, control plane services dynamically provision and configure these networking components for deployed PSEs. Such control plane services can monitor each PSE's network components and invoke self-healing or repair mechanisms designed to prevent loss of communications with the PSE due to failures occurring within the provider network.

[0012] One service commonly offered to customers of a provider network is a block storage service, where an instance is presented with a logical view of physical storage resources, acting as a virtualized persistent disk for the instance, but the mapping of this logical storage space to the storage's actual physical location is handled by the virtualization system. Volumes may be replicated one or more times to provide high availability and durability to customers, with replicas typically stored on different servers. A customer can attach one or more block storage volumes to an instance, and clients supporting the instance can use the virtualized block storage volumes to perform block-based operations on the instance. For example, a customer may specify that a particular instance should boot from a given boot volume (e.g., a volume containing the operating system) and have a separate attached volume to support data stored by customer applications executed by the instance. To provide high flexibility regarding available storage, provider networks decouple the physical storage devices supporting a given attached volume from the computing resources supporting a given compute instance. Traditionally, a fleet of block storage servers supporting a block storage service would divide the attached physical storage devices (e.g., solid-state drives, magnetic drives, etc.) into many logical volumes. A single block storage server can support the storage resources of hundreds or even thousands of instances. These block storage servers typically run in a "bare metal" configuration: the server software runs within an operating system environment that runs directly on dedicated server hardware, instead of, for example, on a virtual machine or in a container.

[0013] This bare-metal block storage server design may not be well suited to board expansion because the capacity of a provider board's expansion capabilities may be significantly limited compared to the capacity of an availability zone. For example, some board expansions may have only a single server (or another small number of servers), making it impossible to dedicate an entire server to block storage resources, thereby rendering the server unavailable for compute instances. To address this challenge, embodiments of the present disclosure virtualize block storage services so that they can be launched within compute instances, thereby enabling more adaptable and efficient use of limited capacity. For example, a single server can be configured to host both compute instances and instances (e.g., block storage servers) that virtualize their attached volumes, providing greater adaptability in the use of the PSE's finite set of resources. Also, a single server may be divided into isolated failure domains to host multiple block storage servers (and even multiple replicas of volumes). A failure domain generally refers to a logical portion of a system that can fail without affecting other parts of the system. When running on a bare-metal system, the failure domain of a block storage system generally corresponds to the entire bare-metal computer system. By using virtualization to separate failure domains from a per-server basis to a sub-server basis, the number of failure domains within a PSE can be increased, leveraging the redundancy of the underlying hardware used to support the block storage server instances. As a result, block storage servers can individually manage smaller amounts of data so that when a failure occurs, the workload to recover the data associated with the failure is reduced. These block storage instances can be created as virtual private clouds (VPCs), and instance clients communicating with the block storage volumes can also communicate in this same VPC. Beneficially, this allows for VPC encryption to be leveraged to make communications more secure across the board.

[0014] However, virtualizing a block storage service in a provider substrate extension presents certain technical challenges, including initializing the block storage service from a boot volume (also referred to as a block storage service machine image) before the block storage service is launched in a substrate extension where the boot volume could be stored. While a PSE provides low-latency compute resources to on-premises facilities, those resources experience increased latency when reverting to the provider network. While a PSE may rely on a block storage server in the provider network's domain, such a dependency is undesirable because it suffers from increased latency. As described in further detail below, embodiments of the present disclosure address this challenge with a local boot technique that can load a block storage server's boot volume in the PSE's local storage from data stored in the provider network's domain, then boot the block storage server and provide the volume to other instances launched within the PSE. Thus, the disclosed local boot technique enables customers to launch instances in a substrate extension using a block storage service machine image, even when the block storage service itself does not yet exist in the substrate extension.

[0015] The disclosed systems and techniques also protect the provider network from potential security issues that may be enabled by connecting the PSE to the provider network. In some embodiments, to operate, the PSE may require secure networking tunnels from the customer site where the PSE is installed to the provider network substrate (e.g., the machine's physical network). These tunnels may include virtual infrastructure components hosted on both the virtualized computing instances (e.g., VMs) and the substrate. Examples of tunnel components include VPCs and proxy computing instances and / or containers running on the computing instances. Each server in the PSE may use at least two tunnels (one for control plane traffic and one for data plane traffic). As described in more detail below, intermediate resources located along the network path between the provider network substrate and the PSE can securely manage traffic flowing between the substrate and the PSE.

[0016] In at least some embodiments, the provider network is a cloud provider network. A cloud provider network, or "cloud," refers to a large pool of accessible virtualized computing resources (such as compute, storage, and network resources, applications, and services). A cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads. Thus, cloud computing can be viewed as both applications delivered as services over publicly accessible networks (e.g., the Internet, cellular communication networks) and the hardware and software of cloud provider data centers that provide those services.

[0017] A cloud provider network can be formed as several regions, where a region is a geographic area where the cloud provider clusters its data centers. Each region can include two or more availability zones connected to each other, for example, via a private high-speed network of fiber optic connections. An availability zone refers to an isolated failure domain that includes one or more data center facilities with separate power sources, separate networks, and separate cooling from those in other availability zones. Preferably, availability zones within a region are located far enough apart from each other so that the same natural disaster does not cause two or more availability zones to go offline at the same time. Customers can connect to availability zones in the cloud provider network via a publicly accessible network (e.g., the Internet, a cellular communication network). The PSE described herein can also connect to one or more availability zones via a publicly accessible network.

[0018] The cloud provider network may include a physical network (e.g., sheet metal boxes, cables) referred to as the substrate. The cloud provider network may also include an overlay network of virtualized computing resources running on the substrate. Thus, network packets can be routed along the substrate network according to the configuration of the overlay network (e.g., VPC, security groups). A mapping service can coordinate the routing of these network packets. The mapping service is a regionally distributed lookup service that maps combinations of overlay IPs and network identifiers to substrate IPs so that distributed substrate computing devices can look up the packet's destination.

[0019] To illustrate, each physical host may have an IP address on the substrate network. Hardware virtualization technology may enable multiple operating systems to run simultaneously on a host computer, for example, as virtual machines on the host. The host's hypervisor or virtual machine monitor allocates the host's hardware resources among the host's various virtual machines and monitors the execution of the virtual machines. Each virtual machine may be provided with one or more IP addresses on the overlay network, and the host's virtual machine monitor may recognize the IP addresses of the host's virtual machines. The virtual machine monitor (and / or other devices or processes on the network substrate) may use encapsulation protocol technology to encapsulate and route network packets (e.g., client IP packets) over the network substrate between virtualized resources on different hosts within the cloud provider network. The encapsulation protocol technology may be used in the network substrate to route the encapsulated packets between endpoints on the network substrate via overlay network paths or overlay routes. The encapsulation protocol technology may be viewed as providing a virtual network topology overlaid on the network substrate. The encapsulation protocol technology may include a mapping service that maintains a mapping directory that maps IP overlay addresses (public IP addresses) to substrate IP addresses (private IP addresses), and the mapping service can be accessed by various processes on the cloud provider network to route packets between endpoints.

[0020] As one skilled in the art will appreciate in light of this disclosure, particular embodiments may be capable of achieving various advantages, including some or all of the following: (a) enabling customers of a provider network operator to deploy various applications in a location-independent manner using provider-managed infrastructure (e.g., PSEs) at sites selected by the customer, while still maintaining the scalability, security, availability, and other operational advantages enabled by the provider network; (b) reducing the amount and resulting volume of application data that needs to be transferred over long distances, such as over links between customer data centers and provider network data centers; (c) moving applications closer to data sources / destinations, thereby improving overall latency and responsiveness for applications that may consume potentially large amounts of data as input or produce as output; and / or (d) improving the security of sensitive application data.

[0021] 1 is a block diagram illustrating an exemplary provider network extended by a provider substrate extension located in a network external to the provider network, according to at least some embodiments. Within the provider network 100, customers can create one or more isolated virtual networks 102. Customers can launch compute instances 101 within an IVN to run their applications. These compute instances 101 are hosted by substrate addressable devices (SADs) that are part of the provider network substrate (not shown). Similarly, SADs that are part of the provider network substrate can host control plane services 104. Exemplary control plane services 104 include instance management services (sometimes referred to as hardware virtualization services) that enable customers or other control plane services to launch and configure instances and / or IVNs, object storage services that provide object storage, block storage services that provide the ability to attach block storage devices to instances, database services that provide various database types, etc.

[0022] Note that the components depicted within provider network 100 can be treated as logical components. As previously described, these components are hosted by a SAD of a provider network substrate (not shown). For example, the provider network substrate can host instances 101 using containers or virtual machines running within an isolated virtual network (IVN). Such containers or virtual machines are executed by the SAD. As another example, the provider network substrate can host one or more control plane services 104 using a SAD in a bare-metal configuration (e.g., without virtualization). In at least some embodiments, a SAD refers to software (e.g., a server) executed by hardware that is addressable via a network address of the provider network, rather than another network (e.g., a customer network, IVN, etc.). In at least some embodiments, a SAD may also refer to the underlying hardware (e.g., a computer system) that executes the software.

[0023] As shown, provider network 100 communicates with provider substrate extensions (PSEs) 188 deployed within customer network 185 and PSEs 198 deployed within customer network 195. Each PSE includes one or more substrate addressable devices (SADs), such as SADs 189A-189N shown within PSE 188. Such SADs 189 facilitate provisioning of computing-related resources within the PSE. Note that solid box-oval-dashed box combination illustrations of components, such as SADs 189A-189N, are generally used in this and subsequent figures to indicate that one or more of those components may be present (although corresponding textual references may refer to the singular or plural of the component, with or without a letter suffix). Customer gateway / router 186 provides connectivity between provider network 100 and PSEs 188, as well as between PSEs 188 and other customer resources 187 (e.g., other on-premises servers or services connected to customer network 185). Similarly, customer gateway / router 196 provides connectivity between provider network 100 and PSE 198, as well as connectivity between PSE 198 and other customer resources 197. Various connectivity options exist between provider network 100 and PSE 198, such as a public network like the Internet as shown in PSE 188 or a direct connection as shown in PSE 198.

[0024] Within the provider network 100, control plane traffic 106 is typically (but not always) directed to a SAD, while data plane traffic 104 is typically (but not always) directed to an instance. For example, some SADs may sell APIs that allow instances to be launched and terminated. A control plane service 104 can send commands via the control plane to the API of such a SAD to launch a new instance on the IVN 102.

[0025] As suggested by its name, an IVN may include a set of hosted (e.g., virtualized) resources that are logically separated or isolated from other resources (e.g., other IVNs) of the provider network. A control plane service can set up and configure IVNs, including assigning an identifier to each IVN to distinguish it from other IVNs. The provider network can offer various ways to enable communication between IVNs (e.g., a gateway in one IVN is configured to communicate with a gateway in another IVN), such as by setting up peering relationships between IVNs.

[0026] IVNs can be established for a variety of purposes. For example, an IVN can be set up for a particular customer by reserving a set of resources exclusively for that customer, with considerable flexibility regarding the network configuration of that set of resources provided to the customer. Within the IVN, the customer can set up subnets, assign desired private IP addresses to various resources, set up security rules governing inbound and outbound traffic, etc. In at least some embodiments, by default, a set of private network addresses set up within one IVN may be inaccessible from another IVN (or, more generally, from outside the IVN).

[0027] Tunneling techniques facilitate the traversal of IVN traffic between instances hosted by different SADs in provider network 100. For example, a newly launched instance in IVN 102 may have IVN address A and be hosted by the SAD with board address X, while instance 101 may have IVN address B and be hosted by the SAD with board address Y. To facilitate communication between these compute instances, SAD X encapsulates packets sent from the newly launched instance to instance 101 (from IVN address A to IVN address B) within the payload of the packet (from board address X to board address Y) that include address information of the SAD hosting each instance. Packets sent between SADs may further include an identifier for IVN 102 to indicate that the data is destined for IVN 102, as opposed to another IVN hosted by the SAD with board address Y. In some embodiments, the SADs further encrypt packets sent between instances within the payload of packets sent between SADs using an encryption key associated with the IVN. In at least some embodiments, the encapsulation and encryption is performed by a software component of the SAD that hosts the instance.

[0028] In the case of a PSE, provider network 100 includes one or more networking components that effectively extend the provider network substrate outside of provider network 100 to the PSEs connected to the customer's on-premises network. Such components can ensure that data plane and control plane operations targeted at the PSE are securely, reliably, and transparently communicated to the PSE. In the illustrated embodiment, PSE interface 108, PSESAD proxy 110, and PSESAD anchor 112 facilitate data plane and control plane communications between provider network 100 and PSE 188. Similarly, PSE interface 118, PSESAD proxy 120, and PSESAD anchor 122 facilitate data plane and control plane communications between provider network 100 and PSE 198. As described herein, the PSE interface receives control plane and data plane traffic from the provider network, transmits such control plane traffic to the PSESAD proxy, and transmits such data plane traffic to the PSE. The PSE interface also receives data plane traffic from the PSE and sends such data plane traffic to the appropriate destination within the provider network. The PSESAD proxy receives control plane traffic from the PSE interface and sends such control plane traffic to the PSESAD anchor. The PSESAD anchor receives control plane traffic from the PSESAD proxy and sends such control plane traffic to the PSE. The PSESAD anchor also receives control plane traffic from the PSE and sends such control plane traffic to the PSESAD proxy. The PSESAD proxy also receives control plane traffic from the PSESAD anchor and sends such control plane traffic to the appropriate destination within the provider network.Other embodiments may employ different combinations or configurations of network components to facilitate communication between the provider network 100 and the PSE (e.g., the PSE interface, the functionality of the PSESAD proxy, and / or the PSESAD anchor may be combined in various ways, with both the PSE interface and the PSESAD proxy, with both the PSESAD proxy and the PSESAD anchor, with an application performing the operations of all three components, etc.).

[0029] As shown above, each PSE has one or more board network addresses for its SADs (e.g., SADs 189A-189N). Because those board addresses are not directly reachable through provider network 100, PSE interfaces 108, 118 masquerade with attached virtual network addresses (VNAs) that match the board addresses of each PSE. As shown, PSE interface 108 has attached VNA(s) 150 that match the SAD address(es) of PSE 188, and PSE interface 118 has attached VNA(s) 152 that match the SAD address(es) of PSE 198. For example, traffic destined for an SAD with Internet Protocol (IP) address 192.168.0.10 in PSE 188 is sent to PSE interface 108 with attached virtual address 192.168.0.10, and traffic destined for an SAD with IP address 192.168.1.10 in PSE 198 is sent to PSE interface 118 with attached virtual address of 192.168.1.10. Note that IPv4 or IPv6 addressing may be used. In at least some embodiments, a VNA is a logical construct that allows various network-related attributes, such as IP addresses, to be programmatically transferred between instances. Such transfers may be referred to as "attaching" a VNA to an instance and "detaching" a VNA from an instance.

[0030] At a high level, the PSE interface is essentially a packet forwarding component that routes traffic based on whether it is control plane traffic or data plane traffic. Given the substrate addressing and encapsulation techniques described above, note that both control and data plane traffic are routed to the PSE interface because both are destined for the SAD. For control plane traffic, the PSE interface routes traffic to a PSESAD proxy based on the SAD address. For data plane traffic, the PSE interface establishes and serves as an endpoint for one or more encrypted data plane traffic tunnels between the provider network 100 and the PSE (e.g., tunnel 191 between PSE interface 108 and PSE 188, tunnel 193 between PSE interface 118 and PSE 198). For data plane traffic received from the provider network 100, the PSE interface encrypts the traffic for transmission through the tunnel to the PSE. For data plane traffic received from the PSE, the PSE interface decrypts the traffic, optionally verifies the SAD addressing of the packets, and sends the traffic to the identified SAD destination via the provider network 100. Note that if the PSE interface receives traffic from the PSE that does not conform to the expected format (e.g., protocol) used to carry data plane traffic, the PSE interface can drop such traffic. Additionally, note that the PSE interface can validate the addressing of encapsulated packets to ensure that the originator of the traffic (e.g., an instance hosted by the PSE within a particular IVN) is authorized to send traffic to the addressed destination (e.g., an instance hosted by a provider network within the same or a different IVN).

[0031] Each SAD of a PSE has a corresponding group of one or more PSE interfaces, and each member of the group establishes one or more tunnels for data plane traffic with the PSE. For example, if a PSE with four SADs has four PSE interfaces, each PSE interface establishes a secure tunnel (e.g., 16 tunnels) with a respective data plane traffic endpoint of the SAD. Alternatively, a group of PSE interfaces may be shared by multiple SADs by attaching an associated VNA to each member of the group.

[0032] Each PSE has one or more PSESAD proxies and one or more PSESAD anchors that handle control plane traffic between the provider network 100 and the PSE's SAD. Control plane traffic typically has a command-response or request-response format. For example, a control plane service in the provider network 100 can issue a command to the PSESAD to start an instance. Because management of PSE resources is facilitated from the provider network, control plane commands sent through secure tunnels typically do not originate from the PSE. At a high level, the PSESAD proxy acts as a stateful security boundary between the provider network 100 and the PSE (such a boundary is sometimes referred to as a data diode). To that end, the PSESAD proxy can employ one or more techniques, such as applying various security policies or rules to received control plane traffic. Note that other control plane services 104 may indirectly or directly provide public APIs, allowing instances hosted by the PSE to issue commands to the provider network 100 via non-tunneled communications (e.g., over a public network such as the Internet).

[0033] For traffic originating within the provider network 100 and destined for a PSE, the PSESAD proxy can provide a control plane endpoint API for its corresponding SAD in the PSE. For example, a PSESAD proxy for a PSESAD that can host an instance can provide an API that matches the API that can receive control plane operations to launch, configure, and terminate the instance. Depending on the API calls and associated parameters that are destined for the PSESAD and received by the PSESAD proxy, the PSESAD proxy can perform various operations. For some operations, the PSESAD proxy can pass the operation parameters and associated parameters to the destination SAD without modification. In some embodiments, the PSESAD proxy can validate that the parameters of API calls received from within the provider network 100 are legal for the API before completing those operations.

[0034] For some API calls or associated parameters, the PSESAD can act as an intermediary to prevent sensitive information from being transmitted outside the provider network 100. Exemplary sensitive information includes cryptographic information such as encryption keys, network certificates, and the like. For example, the PSESAD proxy can decrypt data using a confidential key and re-encrypt the data using a key that can be exposed to the PSE. As another example, the PSESAD proxy can terminate a first secure session (e.g., a Transport Layer Security (TLS) session) originating from within the provider network 100 and create a new secure session with the corresponding SAD using a different certificate, preventing the provider network certificate from leaking to the PSE. Thus, the PSESAD proxy can receive certain API calls from within the provider network 100 that contain sensitive information and issue a replacement or exchange API call to the PSESAD that exchanges the sensitive information.

[0035] With respect to traffic originating from the PSE and destined for the provider network 100, the PSESAD proxy may, for example, drop all control plane commands or requests originating from the PSE, or only commands or requests that are not destined for exposed control plane endpoints within the provider network.

[0036] In some embodiments, the PSESAD proxy can process responses to control plane operations depending on the nature of the expected response, if any. For example, for some responses, the PSESAD proxy can simply drop the response without sending any message to the originator of the corresponding command or request. As another example, for some responses, the PSESAD proxy can sanitize the response to ensure that it conforms to the expected response format of the corresponding command or request before sending the sanitized response to the originator of the corresponding command or request via control plane traffic 107. As yet another example, the PSESAD proxy can generate a response (either immediately or upon receipt of the actual response from the SAD) and send the generated response to the originator of the corresponding command or request via control plane traffic 107.

[0037] As part of serving as a security boundary between the provider network 100 and the PSE, the PSESAD proxy can track the state of communications between components of the provider network (e.g., control plane service 104) and each SAD in the PSE. The state data can include session keys for the duration of a session, pending outbound API calls with associated sources and destinations to track outstanding responses, the relationship between API calls received from within the provider network 100 and those API calls issued to the SAD along with any sensitive information exchanged or replaced, etc.

[0038] In some embodiments, the PSESAD proxy can provide stateful communication for network communications from other PSEs to providers in addition to control plane traffic, which may include Domain Name System (DNS) traffic, Network Time Protocol (NTP) traffic, and operating system activation traffic (e.g., for Windows activation).

[0039] In some embodiments, only certain components of a PSE may serve as endpoints of encrypted control plane traffic tunnels with provider network 100. To provide redundancy and reliability for connections between provider network 100 and the PSEs, a PSESAD anchor may serve as the provider network-side endpoint for each of the PSE's available tunnel endpoints. As shown, PSESAD anchor(s) 112 serve to tunnel control plane traffic to PSE 188 via tunnel 190, and PSESAD anchor(s) 122 serve to tunnel control plane traffic to PSE 1198 via tunnel 192.

[0040] Various embodiments can limit the radial impact of any planned attacks originating from outside the provider network (e.g., from within a PSE) by both using the traffic processing techniques described above and isolating those network components exposed to the traffic from other portions of provider network 100. Specifically, network components can operate within one or more IVNs to limit the scope of an attacker's penetration, thereby protecting the operation of both the provider network and other customers. Accordingly, various embodiments can instantiate PSE interfaces, PSESAD proxies, and PSESAD anchors as applications executed by virtual machines or containers running within one or more IVNs. In the illustrated embodiment, groups of PSE interfaces of different PSEs launch within a multi-tenant IVN (e.g., PSE interface IVN 132 of PSE 188 and PSE 198). In other embodiments, each group of PSE interfaces can launch within a single-tenant IVN. Furthermore, each group of PSESAD proxies and each group of PSESAD anchors for a given PSE runs within a single tenant IVN (e.g., PSESAD proxy IVN 134 of PSE 188, PSESAD anchor IVN 136 of PSE 188, PSESAD proxy IVN 138 of PSE 198, and PSESAD proxy IVN 40 of PSE 198).

[0041] Note that the redundancy provided by running multiple instances of each of the network components (e.g., PSE interface, PSESAD proxy, and PSESAD anchor) allows the provider network to periodically recycle the instances hosting those components without interrupting network communications from the PSE to the provider. Recycling may include, for example, restarting the instance or launching a new instance and reconfiguring other instances with, for example, the recycled instance's address. Periodic recycling limits the time frame in which an attacker can exploit a compromised network component if it becomes compromised.

[0042] The PSE connection manager 180 manages the setup and configuration of the network components that provide connectivity between the provider network 100 and the PSE. As previously described, the PSE interfaces 108, 118, the PSESAD proxies 110, 120, and the PSESAD anchors 112, 122 can be instantiated and hosted by the provider network board. The PSE connection manager 180 can request or initiate the activation of the PSE interface(s), the PSESAD proxy(ies), and the PSE's PSESAD anchor(s) when the PSE is shipped to a customer and / or when the PSE comes online and exchanges configuration data with the provider network. Additionally, the PSE connection manager 180 can further configure the PSE interface(s), the PSESAD proxy(ies), and the PSESAD anchor(s). For example, PSE attach manager 180 can attach VNA(s) corresponding to the PSE's SAD to the PSE interface(s), provide the PSE interface(s) with the address of the PSESAD's proxy / proxies, and provide the PSESAD's proxy / proxies with the address of the PSESAD anchor(s) for the PSE's PSESAD. Additionally, PSE connection manager 180 can configure the IVNs of various components to enable, for example, communication between PSE interface IVN 132 and the PSE's PSESAD proxy IVN and communication between the PSESAD proxy IVN and the PSE's PSESAD anchor IVN.

[0043] Note that to facilitate the establishment of tunnels 190-193, the tunnel endpoints may have one or more attached VNAs or assigned physical network addresses that can receive traffic from outside their respective networks (e.g., from outside the provider network for the PSE interface and PSESAD anchor, and from outside the customer network for the PSE's tunnel endpoint). For example, the PSE 188 may have a single outgoing network address and manage communications to multiple SADs using port address translation (PAT) or multiple outgoing network addresses. Each of the PSESAD anchors 112, 122 may have or share (e.g., via PAT) an outgoing network address, and each of the PSE interfaces 108, 118 may have or share (e.g., via PAT) an outgoing accessible network address.

[0044] 2 is a block diagram illustrating an exemplary provider substrate extension, according to at least some embodiments. In the illustrated embodiment, PSE 188 includes one or more PSE frameworks 202 and one or more hosts 220. At a high level, each of hosts 220 may be functionally (and possibly structurally) similar to at least a portion of the computer systems that form part of the provider network substrate (e.g., that substrate resource hosting an instance within the provider network), while PSE framework(s) 202 provide the supporting infrastructure for emulating the provider network substrate within the PSE and for providing connectivity to the provider network via control plane traffic tunnels and data plane traffic tunnels (e.g., tunnels 190-193 of FIG. 1).

[0045] In at least some embodiments, each of the PSE frameworks 202 can send or receive control plane traffic or data plane traffic from each of the hosts 220, and vice versa, in a mesh-like architecture, as illustrated by PSE control plane traffic 240 and PSE data plane traffic 242. Such redundancy enables a level of reliability that customers can expect from a provider network.

[0046] PSE framework 202 includes one or more control plane tunnel endpoints 204 that terminate encrypted tunnels (e.g., tunnel 190, tunnel 192) carrying control plane traffic. In some embodiments, provider network 100 hosts a PSESAD anchor for each of the control plane tunnel endpoints 204. Back in the provider network, a proxy or proxies (e.g., proxy 110) of the PSESAD can distribute the control plane traffic to the PSESAD anchors (e.g., anchor 112), effectively spreading the control plane traffic load across the PSE frameworks 202 of PSE 188. PSE framework 202 further includes one or more data plane tunnel endpoints 206 that terminate encrypted tunnels (e.g., tunnel 191, tunnel 193) carrying data plane traffic from the provider network's PSE interfaces; PSE frameworks 202 may further be connected in a mesh-like architecture (e.g., a given PSE interface 108 establishes a tunnel with each of the data plane tunnel endpoints 206 of PSE framework 202).

[0047] As shown above, control plane traffic packets and data plane traffic packets may include SADs as both the source and destination. The latter (data plane traffic packets) are encapsulated in packets with SAD-based addressing. As shown, PSE framework 202 is SAD 289, and host 220 is SAD 290. Note that SADs within PSE 188 (e.g., SADs 289, 290) can also provide secure session termination (e.g., TLS termination) for secure sessions established with a corresponding PSESAD proxy or proxies (e.g., PSESAD proxy 110) within the provider network.

[0048] The SAD sells one or more control plane APIs to handle control plane operations directed to the SAD, which manages the SAD's resources. For example, the PSE manager 210 of the PSE framework 202 can sell control plane APIs for managing components of the PSE framework 202. One such component is the PSE gateway 208, which routes control plane traffic and / or data plane traffic to and from the PSE 188, such as control plane traffic directed to the SAD 289 to the PSE manager 210 and control plane or data plane traffic directed to the SAD 290 to the host manager 222. The PSE gateway 208 can also facilitate communication with customer networks, such as to and from other customer resources 187 accessible via the PSE deployment site's network (e.g., customer network 185).

[0049] The API of PSE manager 210 may include one or more commands for configuring PSE gateway 208 of PSE framework 202. Other components 212 of PSE framework 202 may include various applications or services involved in the operation of the PSE infrastructure for host 220, such as DNS, Dynamic Host Configuration Protocol (DHCP), and / or NTP services.

[0050] Host manager 222 can provide control plane APIs for managing components of host 220. In the illustrated embodiment, host manager 222 includes instance manager 224 and network manager 226. Instance manager 224 can process API calls related to managing host 220, including commands to launch, configure, and / or terminate instances hosted by host 220. For example, an instance management service in a provider network (not shown) can issue control plane commands to instance manager 224 to launch instances on host 220. As shown, host 220 hosts customer instances 232, which launch within customer IVN 233, third-party (3P) instances 234, which launch within 3P IVN 235, and service instances 236, which launch within service IVN 237. Note that each of these IVNs 233, 234, 235 can extend existing IVNs established within the provider network. Customer instance 232 may run some customer application or workload, 3P instance 234 may run another party application or workload that the customer has allowed to launch an instance of within PSE 188, and service instance 236 may run provider network services (e.g., block storage services, database services, etc.) that are provided locally to PSE 188.

[0051] The network manager 226 can process SAD-addressed data plane traffic received by the host 220. For such traffic, the network manager can perform any necessary decapsulation of the IVN packets before sending them to the addressed hosted instance. Additionally, the network manager 226 can handle the routing of traffic sent by the hosted instance. When a hosted instance attempts to send traffic to another locally hosted instance (e.g., on the same host), the network manager 226 can forward the traffic to the addressed instance. When a hosted instance attempts to send traffic to a non-local instance (e.g., not on the same host), the network manager 226 can find the board address of the device hosting the non-local instance, encapsulate the corresponding packet, and optionally encrypt it into a SAD-addressed packet, which can then be sent over the data plane (e.g., either to another host in the PSE or back to the provider network via the PSE gateway 208). Note that the network manager 226 may contain or have access to various data that facilitates routing of data plane traffic (e.g., to look up the address of the SAD hosting an instance that has an IVN network address as the destination of a packet received from the hosted instance).

[0052] FIG. 3 is a block diagram illustrating an example connection between a provider network and a provider substrate extension, according to at least some embodiments. Specifically, FIG. 3 illustrates an example connection between a provider network and a PSE. With respect to FIG. 3 and as shown at the top of the figure, it should be noted that the term “inbound” refers to traffic received by the provider network from the PSE, and the term “outbound” refers to traffic transmitted by the provider network to the PSE. Although not shown, this example assumes that the PSE includes two PSE frameworks 202 and two hosts 220 (four total) of SADs. The PSE frameworks provide tunnel endpoints 204A, 204B for control plane traffic and tunnel endpoints 206A, 206B for data plane traffic. Outbound traffic is decrypted and transmitted to its destination within the PSE substrate via PSE gateways 208A, 208B.

[0053] For each of the four SADs, the provider network includes a VNA, one or more PSE interfaces, and one or more PSESAD proxies. In this example, the provider network includes a PSESADVNA 304, two PSE interfaces 108A, 108B, and two PSESAD proxies 110A, 110B for a given PSESAD. Together, the PSE interface(s) and the proxy / proxies of the PSESAD may be referred to as a slice as shown, with each slice corresponding to a particular SAD within the PSE. In other embodiments, the PSE interface(s) may be shared by all VNAs for a VPN, rather than a single VNA for one of the SADs.

[0054] The PSESADVNA 304 acts as a front for a given PSE, allowing other components of the provider network to send traffic to and receive traffic from the PSE's corresponding SAD. A load balancer (not shown) can route outbound traffic sent to the PSESADVNA 304 to one of the PSE interfaces 108A, 108B. The illustrated PSE interfaces 108A, 108B for a given slice and those of other slices (not shown) operate within the PSE interface IVN 132. The PSE interfaces 108A, 108B send data plane traffic to the PSE through data plane traffic tunnels and control plane traffic to the PSE by forwarding the control plane traffic to the slice's PSESAD proxy 110A, 110B. The PSE interface 108A, 108B stores (or has access to) the network address(es) of the PSESAD's proxy / proxies for the associated SAD, the network addresses of the data plane tunnel endpoint(s), and one or more keys or associated keys for the PSE's data plane tunnel endpoint(s) for protecting communications with those endpoint(s).

[0055] In at least some embodiments, PSE interfaces 110A, 110B establish secure tunnels for data plane traffic with each of data plane tunnel endpoints 206A, 206B, resulting in N data plane tunnels, where N is the number of PSE interfaces per SAD (assuming each SAD has the same number of interfaces) multiplied by the number of data plane tunnel endpoints multiplied by the number of SADs. In this example, 16 data plane tunnels are established between the PSE interfaces and the data plane tunnel endpoints (i.e., 2 PSE interfaces per SAD x 2 data plane tunnel endpoints x 4 SADs).

[0056] The PSESAD proxies 110A, 110B receive control plane traffic from the PSE interfaces 108A, 108B, perform various operations described elsewhere herein, and transmit the control plane traffic to the PSE via one of the two PSESAD anchors 112A, 112B. Similarly, the PSESAD proxies 110A, 110B receive control plane traffic from one of the two PSESAD anchors 112A, 112B, perform various operations described elsewhere herein, and transmit the control plane traffic 107 to a destination within the provider network. The illustrated PSESAD proxies 110A, 110B of a given slice and proxies of other slices (not shown) operate within the PSESAD proxy IVN 134. The PSE interfaces 108A, 108B store (or have access to) the network addresses of the PSESAD anchor(s).

[0057] In at least some embodiments, PSESAD proxies can access or otherwise exchange information with a shared data store 306. Such information exchange can be used for several reasons. For example, recall that a PSESAD proxy can sell an API interface to emulate the API interface of an associated SAD in a PSE. Because some communications may be stateful and various load balancing techniques may prevent the same PSESAD proxy from processing all communications for a given set of operations, one PSESAD proxy may need to access communication state previously processed by a different PSESAD proxy (e.g., PSESAD proxy 110A sends control plane operations to the PSE, and PSESAD proxy 110B receives responses to the control plane operations from the PSE). For inbound control plane traffic, the PSESAD proxy checks whether the inbound message matches the expected state, and if so, can transmit the message via control plane traffic 107, as described elsewhere herein. If there is no match, the PSESAD proxies 110A, 110B can drop the traffic. As another example, recall that the PSESAD proxy can bridge a separate secure session (e.g., a TLS session) so that the provider network certificate is not sent to the PSE. Again, because the PSESAD proxy handling the outbound message can be different from the PSESAD proxy handling the response to that message, the PSESAD proxy handling the response message can use the same key established between the originator of the outbound message and the PSESAD proxy that processed the outbound message to send the secure response message to the originator via control plane traffic 107.

[0058] In this example, each PSE framework provides a single control plane tunnel endpoint 204. For each available control plane tunnel endpoint 204, the provider network includes a PSE anchor. In this example, the provider network includes two PSE anchors 112A, 112B. The PSESAD anchors 112A, 112B operate within the PSESAD anchor IVN 136. The PSE anchor 112 receives control plane traffic from each of the eight PSESAD proxies (two per slice in each of the four SADs) and sends that traffic to the PSE. The PSE anchor also receives control plane traffic from the PSE and sends that traffic from the PSE to one of the two PSESAD proxies associated with the SAD from which the traffic originated. The PSE anchor 112A, 112B stores (or has access to) the network addresses of the proxy / proxies of each PSESAD of the SAD, the network addresses of the control plane tunnel endpoint(s), and one or more keys or associated keys for the PSE's control plane tunnel endpoint(s) for protecting communications with those endpoint(s).

[0059] In at least some embodiments, a network component or provider network can employ load balancing techniques to distribute the workload of routing control and data plane traffic between the provider network and the PSE. For example, traffic sent to PSESADVNA 304 can be distributed among PSE interfaces 108A, 108B. As another example, each of PSE interfaces 108 can distribute traffic among data plane tunnel endpoints 206A, 206B. As yet another example, each of PSE interfaces 108 can distribute traffic among PSESAD proxies 110A, 110B. As yet another example, each of PSESAD proxies 110 can distribute outbound traffic among PSESAD anchors 112A, 112B. As yet another example, PSESAD anchor 112 can distribute inbound traffic among PSESAD proxies 110A, 110B. In either case, such load balancing can be performed by a transmitting entity or load balancer (not shown). Exemplary load balancing techniques include employing a load balancer with a single VNA that distributes traffic to multiple components "behind" that address, providing each data sender with multiple recipient addresses, and distributing selected recipients at the application level, etc.

[0060] 1-3 illustrate the establishment of separate tunnels for control plane traffic and data plane traffic, other embodiments may employ one or more tunnels for both control plane traffic and data plane traffic. For example, the PSE interface may route data plane traffic to the PSESAD anchor for transmission to the PSE through a shared tunnel, bypassing the additional operations performed by the PSESAD proxy on the control plane traffic.

[0061] 4 and 5 are block diagrams illustrating an example of a virtualized block storage system, according to at least some embodiments. As shown in FIG. 4, provider network 400 or PSE 488 includes host 420A. Host 420A hosts block storage server instance 450. In hosting block storage server instance 450, host 420A provisions some amount of host storage device(s) 421 as storage devices 447 (e.g., virtual drives or virtual disks) provisioned for block storage server instance 450. Block storage server instance 450 can carve out provisioned storage devices 447 to create and host volumes on behalf of other instances. An instance management service (not shown) of provider network 400, via instance manager 426, can initiate the launch of such block storage server instances on either the hosts of provider network 400 or the hosts of PSE 488. The instance manager 426 may be a virtual machine manager (VMM), hypervisor, or other virtualization program that can provision host resources (e.g., compute, memory, network, storage) for instances (e.g., virtual machines) and launch and terminate instances.

[0062] Host 420A also hosts instance A 430 within instance IVN 432. Block storage server instance 450 provides block storage volume A 434 to instance A 430 and provides block storage volume B 440 to another instance (not shown). Host 420A includes block storage client 460A through which the hosted instances access block storage volumes hosted by the block storage server (e.g., by providing a virtual block storage interface to the instances). Specifically, block storage client 460 may be a software program that intercepts or otherwise receives block storage operations issued to volumes “attached” to the instances (e.g., block storage volume A 434 attached to instance A 430).

[0063] 5, provider network 400 or PSE 488 includes two hosts 420A and 420B. Again, host 420A hosts block storage server instance 450. Host 420B hosts instance B 436 within instance IVN 438. Block storage server instance 450 provides block storage volume B 440 to instance B 436. Host 420B includes block storage client 460B, whose hosted instances access block storage volumes hosted by the block storage server.

[0064] When a volume is attached to an instance, the block storage client obtains a mapping that associates the volume with the block storage server instance(s) that host the replicas. The block storage client can open a communication session (e.g., via a network-based data transfer protocol) with each of the block storage server instances. Upon receiving a block storage operation from an instance, the block storage client issues the operation to the appropriate block storage server(s) to fulfill the operation for the instance. For example, with reference to FIG. 4, instance A 430 can issue a read of a block of data at a given block address (e.g., a logical block address or LBA) from volume A 434. Block storage client 460A receives the read operation and issues an operation including the requested block address to block storage server instance 450 based on the mapping data of instance-attached volume A 434 to block storage server instance 450. Upon receiving the data from block storage server instance 450, block storage client 460A provides the data to instance A 430. A similar series of operations can be performed between instance B 436, block storage client 460B, block storage server instance 450, and volume B 440 in FIG.

[0065] 4 and 5, block storage messaging between block storage clients and block storage server instances can be transmitted using a network-based data transfer protocol such as Global Network Block Drive (GNDB). As described elsewhere herein, such messages can be encapsulated as data plane traffic and routed between block storage clients 460 and block storage servers 450 by the network manager 424 of each host. In some embodiments where the block storage server instances are hosted by the same host as the instance issuing the block storage operation, the block storage client can issue the relevant operation to the network manager over a network-based data transfer protocol, which can then route the operation internally to the hosted block storage server instance. In other embodiments where the block storage server instances are hosted by the same host as the instance issuing the block storage operation, the block storage client can issue the relevant operation over a PCIe or other interconnect residing between the block storage client and the hosted instance, thereby avoiding protocol packing associated with network-based transfers.

[0066] 4 and 5, the system can deploy various levels of encryption to protect block storage data. To provide end-to-end encryption of block storage traffic, block storage server instance 452 runs within a block storage service (BSS) IVN 452, which may include any number of block storage server instances and extend from the provider network to the PSE. Block storage client 460 has a VNA 462 associated with block storage service IVN 452. In this manner, IVN-level encryption using block storage service IVN encryption key 484 can be used to encrypt and decrypt traffic sent between block storage server instance 450 and block storage client 460. Note that encryption and decryption of traffic carried over the block storage service IVN can be performed by a network manager at the IVN endpoint. For example, network manager 424B encrypts traffic sent from block storage client 460B via VNA 462B. As another example, network manager 424A decrypts traffic sent to instances hosted within the IVN 452 of the BSS, including block storage server instance 450. To provide encryption at rest for data, block storage volumes may also have volume-level encryption. For example, volume A 434 may be encrypted with volume A encryption key 480, and volume B 440 may be encrypted with volume B encryption key 482. Using encryption key 480, block storage client 460A encrypts data written to volume A 434 by instance A 430 and decrypts data read from volume A 434 by instance A 430. Using encryption key 482, block storage client 460B can perform similar encryption and decryption operations between instance B 436 and volume B 440.Note that in some embodiments, there may be policies in place that prevent a block storage server instance that hosts a volume from being hosted on the same host as the instance attached to that volume, and that prevent a volume encryption key from being hosted on the same host as the volume it encrypts (e.g., volume A encryption key 480 and volume A 434 will no longer be on host 420A).

[0067] In some embodiments, the instance and host manager are executed by different processor(s), as shown in Figures 4 and 5. For example, block storage server instance 450 can be executed by a first processor (e.g., processor 1710), and host manager 422A can be executed by a second processor (e.g., processor 1775). The processor(s) executing the block storage server instance access the host's storage device(s) through one or more interconnects. For example, the processor(s) executing block storage server instance 450 (e.g., processor 1710) may access storage device(s) 421 (e.g., storage device 1780) through a PCIe interconnect between the two. As another example, the processor(s) running the block storage server instance 450 may access the storage device(s) 421 via a first PCIe bus to a processor or system-on-chip running the host manager 422A (e.g., processor 1775), which then bridges those communications to the storage device(s) 421 via a second PCIe bus.

[0068] Although depicted as a single volume in Figures 4 and 5, each instance volume 434, 440 may be mirrored across multiple replicas stored on different block storage servers to provide high reliability for the system. For example, one replica may be referred to as the primary replica, which processes reads from and writes to the volume (input and output operations, or "input / output (I / O)"), and the server hosting the primary replica can synchronously propagate writes to other secondary replicas. If the primary replica fails, one of the secondary replicas is selected to serve as the primary replica (referred to as a failover). Replicas may be subdivided into partitions (e.g., striped), with each partition of a given replica stored on a different server to facilitate parallel reads and writes. Replicas may also be encoded with redundancy (e.g., parity bits) so that when a block storage server becomes unavailable (e.g., due to a server or disk failure), lost data can be recovered from available data. As a result, volumes attached to an instance may be hosted by many block storage servers.

[0069] One challenge associated with hosting block storage servers in a virtualized context is booting those servers. If there is no operational block storage server instance, how does a block storage server instance boot from a boot volume hosted by a block storage server instance? This is particularly challenging in the context of a PSE, which may not include dedicated (or bare metal) block storage servers. Figures 6 and 7 show example local boot techniques for booting the first or "seed" block storage server instance (a block storage server instance that can launch other instances) within a PSE. Once launched, other instances can be launched from the seed block storage server instance.

[0070] FIG. 6 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using the first technique, according to at least some embodiments. At a high level, FIG. 6 illustrates an example of remotely booting a block storage server 650 in an instance 690 via a proxy server hosted within a provider network 600. A block storage service (BSS) 606 in the provider network 600 provides users with the ability to create block storage volumes and attach those volumes to instances. For example, a user (e.g., a customer of the provider network, another service in the provider network) can submit commands via an API or console relayed to the BSS 606 to create, resize, or delete volumes managed by the BSS and attach or detach those volumes to instances. The BSS 606 maintains a BSS object store 608 that contains volume snapshots. A snapshot can be considered a type of object that, similar to a file, can include object data and metadata about the object (e.g., creation time, etc.). A volume snapshot is a copy (or backup) of a volume at a given point in time, enabling the recreation of that volume on a physical or virtual drive. Snapshots can generally be divided into boot volumes and non-boot volumes, the former (boot volumes) facilitating the booting of software running within an instance. A machine image can be a group of one or more snapshots of a given instance, including the boot volume.

[0071] In some embodiments, BSS 606 applies object-level encryption of objects in BSS object store 608. To prevent compromise of encryption keys used within provider network 600, BSS 606 can also manage PSE object store 610 and re-encrypt objects sent to a given PSE using an encryption key different from that used to encrypt objects in BSS object store 608. In this example, block storage server instance machine image 612A is used to boot block storage server 650 on PSE's host 620. As shown in circle A, BSS 606 decrypts block storage server instance machine image 612A using a key maintained within provider network 600 and re-encrypts it as block storage server instance machine image 612B using key 613, which can be sent to the PSE.

[0072] The IVN can provide security for traffic between the block storage server instance and the block storage client. In some embodiments, the BSS 606 manages its IVN, shown as the BSS's IVN 652. The host 620 on which the block storage server 650 is launched includes a host manager 622 having a block storage client 660 with an associated VNA 662 of the BSS's IVN 652. To facilitate communication between the block storage client 660 and the provider network 600 through the BSS's IVN 652, the BSS 606 launches a proxy server instance 642 within the BSS's IVN 652. The proxy server instance 642 provides a block storage-like interface to the block storage client 660 and may allow the client 660 to access the block storage server instance's machine image 612B. The BSS 606 launches the proxy server instance 642 as follows:

[0073] As shown in circle B, the BSS 606 first sends a message to the bare metal block storage server 614 to create a new logical volume to serve as the boot volume for the proxy server instance 642. The bare metal block storage server 614 creates a new logical volume 619 on its one or more storage devices 618 of the bare metal server 614 and loads the proxy server machine image 616 stored in the BSS object store 608 into the volume 619.

[0074] As shown in circle C, BSS 606 initiates the launch of proxy server instance 642 with instance management service 640. Instance management service 640 is another control plane service of provider network 600 that facilitates the launch of instances within provider network 600 or a PSE. In some embodiments, instance management service 640 may track or have access to host utilization metrics (i.e., metrics such as CPU utilization, memory utilization, and network utilization) that indicate how “hot” a host is. For example, instance management service 640 or another service may periodically poll hosts to obtain utilization metrics. By accessing metrics associated with a pool of potential hosts (subject to any constraints regarding the location of hosts within a given PSE, etc.), instance management service 640 can identify hosts to host the instance in response to the launch request by selecting those with low utilization (e.g., below a threshold or thresholds). In requesting the launch of proxy server instance 642, BSS 606 specifies that the instance should be launched from volume 619 within the BSS's IVN 652. As shown in circle D, instance management service 640 identifies host 646 that will host the instance and sends a message to host 646 to launch the instance. An instance manager (not shown) on host 646 provisions the host's resources for proxy server instance 642 and launches it from boot volume 619.

[0075] BSS 606 also initiates the launch of block storage server 650 via instance management service 640. BSS 606 specifies that the instance should boot from a boot volume made available by proxy server instance 642 within the BSS's IVN 652 of a particular PSE (e.g., using a PSE identifier). As shown in circle E, instance management service 640 identifies the host 620 that will host the block storage server instance 650 and sends a message to the host 620 (e.g., via a control plane traffic tunnel) to launch the instance. As shown, the request to launch the instance is received by instance manager 626, which can provision host resources for the instance and launch and terminate the instance. Here, instance manager 626 creates instance 690 to host the block storage server.

[0076] As part of configuring instance 690, instance manager 626 can configure basic input / output system (BIOS) 692 for instance 690 and load block storage server instance machine image 612B into virtual boot volume 694 via proxy server instance 642. BIOS 692 includes block storage device drivers (e.g., non-volatile memory express (NVMe) drivers) and can attach two block storage devices. The virtualized block storage devices are presented by proxy server instance 642 via block storage client 660 and boot volume 694. Note that although shown within instance 690, boot volume 694 corresponds to a virtual drive volume allocated to instance 690 by instance manager 626 from a storage device (not shown) of host 620 accessible via block storage client 660 or another component of host manager 622. Other embodiments may include an interface other than BIOS (e.g., Unified Extensible Firmware Interface (UEFI)) for connecting instance 690 to the two block devices.

[0077] During execution, BIOS 692 (or other firmware, such as UEFI) can load boot volume 694 from block storage server instance machine image 612B. When block storage server instance machine image 612B is encrypted using key 613, block storage client 660 decrypts block storage server instance machine image 612B during the load process. Once the load operation is complete, BIOS continues the boot process, booting instance 690 from boot volume D94, which contains the block storage server software. In some embodiments, BIOS 692 can detach the block storage device corresponding to proxy server instance 642 before booting block storage server software 650.

[0078] While shown and described using proxy server instance 642, other embodiments can use a block storage server instance (not shown) launched from block storage server instance machine image 612A to provide block storage devices to instance 690 via block storage client 660. BSS 606 creates a volume on a virtual drive or virtual disk of the launched block storage server instance and loads the block storage server instance's machine image 612A onto the volume. During the loading process, the block storage server instance can decrypt the machine image using a provider network key and optionally encrypt the volume using a volume-specific key (e.g., how key 480 encrypts volume B440) and provide it to block storage client 660 via BSS 606.

[0079] FIG. 7 is a block diagram illustrating an exemplary system for booting a virtualized block storage server using the second technique, according to at least some embodiments. At a high level, FIG. 7 illustrates an example of remotely booting an instance 790 to run block storage server software 750 via pre-boot software executed by the instance 790 prior to running the block storage server software 750. Specifically, storage device 721 of PSE 720 is pre-loaded (e.g., prior to shipping of the PSE) with pre-boot instance boot volume 723. Pre-boot instance boot volume 723 includes software that can be booted on the instance during block storage server pre-boot phase 798 to load the block storage server software onto a separate boot volume. The instance can then reboot and boot from the other boot volume to launch the block storage server instance during block storage server boot phase 799.

[0080] As shown in circle A, BSS 606 initiates the launch of a block storage server instance with instance management service 640 and specifies that the instance should launch within the IVN of the BSS of a particular PSE. Again, instance management service 640 identifies the PSE's host 620 that hosts the block storage server instance 650 and sends a message (e.g., via a control plane traffic tunnel) to the host's host manager 722 to launch the instance. The request to launch the instance is received by the instance manager (not shown) of host manager 722.

[0081] As shown in circle B, host manager 722 (or instance manager) provisions host resources for instance 790 running block storage server 750. Additionally, host manager 722 configures two attached block storage devices (block storage device interfaces presented by a block storage client (not shown) to pre-boot instance boot volume 723 and virtual boot volume 794 (such as boot volume 694 described above)) in BIOS 792A (or UEFI) to enable booting from pre-boot instance boot volume 723.

[0082] As shown in circle C, BIOS 792A then enables booting of pre-boot software 796 from pre-boot instance boot volume 723. If pre-boot instance boot volume 723 is based on a generic machine image, host manager 722 may also need to update the configuration of instance 790 to facilitate communication between pre-boot software 796 and provider network 700, as shown in circle D. Specifically, host manager 722 can attach a VNA to instance 790 and provide pre-boot software 796 with credentials to use when accessing PSE object store 610 and keys to decrypt the machine image loaded onto boot volume 794. The VNA configuration, credentials, and keys may have been passed to host manager 722 as part of a request from instance management service 640 to launch the instance.

[0083] As shown in circle E, once instance 790 is configured and pre-boot software 796 runs, pre-boot software 796 copies block storage server instance machine image 612B from PSE object store 610 and loads it onto boot volume 794. As part of loading boot volume 794, pre-boot software 796 can decrypt block storage server instance machine image 612B with key 613, assuming it was encrypted. Host manager 722 can detect, or pre-boot software can signal, the completion of loading boot volume 794. For example, pre-boot software can initiate a restart of the instance when boot volume 794 is loading. As another example, network manager 724 can detect the end of a session between instance 790 and PSE object store 610 (e.g., by detecting when a TCP session is closed).

[0084] As shown in circle F, host manager 722 can update or otherwise reconfigure BIOS 792A to 792B either during the data transfer or once the data transfer is complete. Such reconfiguration includes changing the boot volume from pre-boot instance boot volume 723 to boot volume 794 and removing the block storage device containing pre-boot instance boot volume 723. Before restarting instance 790, host manager 722 can also clear memory provisioned to instance 790, which may include residual data written by pre-boot software 796. Upon restarting instance 790, BIOS 792B boots block storage server 750 from boot volume 794.

[0085] While Figures 6 and 7 illustrate exemplary techniques for booting a block storage server instance within a PSE, other techniques are possible. For example, an instance manager can create a volume on the storage device of the PSE as part of the provisioning resources of the instance to be launched. Before launching the instance, the instance manager can send a message to a control plane service (e.g., a block storage service or an instance management service) to request a bootable snapshot to load into the volume. The control plane service can send the bootable snapshot to the instance manager over a control plane traffic tunnel. Once loaded, the instance manager allows the instance software to start.

[0086] FIG. 8 is a block diagram illustrating an exemplary system for booting additional compute instances of a provider substrate extension from a block storage server, according to at least some embodiments. Once a block storage server instance is launched, such as using the techniques described with reference to FIG. 6 or FIG. 7, that “seed” block storage server instance can provide boot volumes to other instances (including other block storage server instances). This can be particularly useful in the context of a PSE where the seed block storage server instance is launched from a machine image transferred or hosted over a relatively slow connection between the provider network and the PSE compared to the PSE's substrate interconnect. In this example, instance 890 is launched on host 820A of PSE 888. Instance 890 is running block storage server software 850 and operates within IVN 844 of the BSS. Additionally, block storage clients 860A and 860B have attached VNAs 862A and 862B that enable communication across IVN 844 of the BSS.

[0087] As shown in circle A, the instance management service 640 of the provider network 800 can request that the BSS 606 create a new volume from a specified snapshot. As shown in circle B, the BSS 606 can instruct the block storage server instance 890 to create a volume based on the specified snapshot. In some embodiments, the instance management service 640 can provide a volume-specific encryption key (e.g., key 680) to encrypt the volume.

[0088] As shown in circle C, the block storage server instance 890 can create a volume by fetching a specified snapshot from an object store 810, such as the PSE object store 610. Although not shown, in other embodiments, the object store 810 can be an object store hosted by a BSS cache instance of the PSE 888 for caching volume snapshots and / or machine images. The BSS 606 can manage the BSS cache instance, and a customer can specify to the BSS 606 that a particular snapshot be loaded into the cache prior to any request to start an instance. In this way, the block storage server instance 890 can create a volume from a cached boot volume snapshot, dramatically reducing the boot time of an instance of the PSE by avoiding delays associated with transferring data from an object store in the provider network 800 to the block storage server instance 890.

[0089] As shown in circle C, instance management service 640 of provider network 600 can issue commands to a host of PSE 888 to attach or detach volumes hosted by block storage server instance 890 to other instances hosted by PSE 888. For example, volume A 832 can be loaded with a bootable snapshot. Instance management service 640 can instruct host 820A to boot instance A 830 using the volume identifier associated with volume A 832. In response, block storage client 860A can attach volume A 832 hosted by block storage server instance 890 to instance A 830 via IVN 844 of the BSS, thereby enabling instance A 830 to boot from volume A 832. As another example, volume B 836 can be loaded with a non-bootable snapshot containing other data. Instance management service 640 can instruct host 820B to attach volume B 836 to hosted instance 834. In response, block storage client 860B can attach volume B 836, hosted by block storage server instance 890, to instance B 834 via IVN 844 of the BSS.

[0090] FIG. 9 is a block diagram illustrating an exemplary system for managing virtualized block storage servers, according to at least some embodiments. Running a block storage server within a virtualized environment provides several benefits, including allowing an instance management service to automatically scale the number of running block storage server instances as needed. Rather than a BSS operating on a predefined pool of manually scaled bare-metal servers, the instance management service can monitor host resource (e.g., CPU, memory, network, etc.) utilization and automatically adjust the number of running block storage server instances. For example, if two hosts of block storage server instances are reporting high resource utilization, the instance management service can launch additional block storage server instances on hosts with lower resource utilization. The block storage service can then create volumes on the newly launched instances, thereby avoiding increased workload and possible performance degradation on the running block storage server instances.

[0091] Furthermore, running block storage servers as instances allows the number of failure domains to exceed the number of host computer systems by decoupling the failure domain from the overall computer system. Increasing the number of failure domains allows for an increased number of block storage servers to run on a fixed set of hardware, and increasing the number of block storage servers reduces the overall footprint of data managed by any particular server. For example, assume a system contains nine host computer systems that are not subdivided into smaller failure domains (i.e., one computer system has one failure domain). To avoid data loss, each host or failure domain runs a single block storage server. If those nine block storage servers host 90 terabytes (TB) of data across the instance's volumes (including encoding to enable data recovery), each block storage server would store approximately 10 TB of data. If one of the hosts failed, that 10 TB would need to be recovered (e.g., from the other 80 TB). Such data recovery incurs costs for compute, transfer, and storage of data. If those nine hosts are each subdivided into two failure domains, increasing the number of block storage servers to 18, each block storage server will store approximately 5TB of data, roughly halving the amount of data (and corresponding cost) that would need to be recovered if one component in a failure domain fails.

[0092] In this example, PSE 988 or provider network 900 includes three hosts 920A, 920B, and 920C. Depending on the host's hardware design, the host can either be treated as a single failure domain or subdivided into two or more failure domains. Here, host 920 includes two failure domains. Host 920A includes failure domain 922 and failure domain 924, such that components in one failure domain can continue to operate even if components in another failure domain fail. For example, failure domain 922 may correspond to a first processor in a multiprocessor system connected to a first memory bank (e.g., RAM) and using a first set of one or more storage drives (e.g., SSDs), while failure domain 924 may correspond to a second processor in the system connected to a second memory bank and using a second set of one or more storage drives. Note that some components may be shared across failure domains, such as power supplies, but this is again subject to redundancy in the hardware design and how the failure domains are overlaid on the hardware.

[0093] Host 920A runs block storage server instance 950A in failure domain 922 and block storage server instance 950B in failure domain 924. Host 920B runs block storage server instance 950C in failure domain 926 and block storage server instance 950F in failure domain 928. Volume A includes a primary replica 660 served by block storage server instance 950A and a secondary replica 662 served by block storage server instance 950B. Volume B includes a primary replica 664 served by block storage server instance 950C and a secondary replica 666 served by block storage server instance 950A. Volume C includes a primary replica 668 served by block storage server instance 950B and a secondary replica 670 served by block storage server instance 950C. Although shown as having two replicas per volume, in reality each volume may have a fewer or greater number of replicas, and each replica may be divided among many different block storage server instances (e.g., by striping).

[0094] As shown in circle A, the instance management service 940 monitors the utilization of the host's physical resources, such as processor utilization, memory utilization, storage utilization, and network utilization, which may be aggregated across the host, isolated by failure domain, or per resource (e.g., the instance management service 940 may receive metrics from host 920A indicating that the average CPU utilization is 50%, that the CPU utilization of the processor(s) in failure domain 922 is 50%, or that the CPU utilization of a particular processor supporting failure domain 922 is 50%). The instance management service 940 may track the host's resource utilization in a database (not shown). Note that in addition to block storage server instances, the illustrated failure domains can host other instances (e.g., customer instances) that contribute to the host's physical resource utilization. The instance management service 940 may periodically update the BSS 906 with resource utilization metrics, allowing the BSS 906 to select less utilized block storage server instances when creating new volumes, as shown in circle B.

[0095] As shown in circle C, the instance management service 940 can launch new block storage server instances when resource utilization of the hosts supporting the block storage server instances exceeds one or more thresholds. The thresholds can be defined in various ways, such as per resources aggregated across hosts (e.g., average processor utilization of all processors running block storage servers is 50%), some combination of resources across hosts (e.g., storage utilization above 80% and processor utilization above 50%), or on an individual resource and / or host basis. To avoid launching block storage server instances in failure domains that are already hosting block storage server instances, the instance management service 940 can track (e.g., in a database containing resource usage metrics) which failure domains are occupied and which are available. Here, the instance management service 940 determined that resource utilization of hosts 920A, 920B, and / or 920C exceeded certain thresholds and launched new block storage server instances 950D and 950E in previously unoccupied failure domains 930 and 932 of host 920C, respectively. Although failure domain 928 includes block storage server 950F that is not hosting a volume, the resource utilization of other instances hosted within that failure domain or the aggregate resource utilization of host 920B may be too high to support another block storage server instance.

[0096] As shown in circle D, instance management service 940 can update BSS 906 with updated identification information for operational block storage instance 950 (now including block storage servers 950D and 950E). Based on the identified block storage instance 950 and resource utilization metrics received from instance management service 940, BSS 906 can create a new volume on host 920C, as shown in circle E. Now, assuming that host 920C's resource utilization, as reported by instance management service 940, indicates low, BSS 906 creates new volume D, including primary replica 670 provided by block storage server instance 950D and secondary replica 672 provided by block storage server instance 950E.

[0097] FIG. 10 is a block diagram illustrating an exemplary system for providing volume mapping to block storage clients, according to at least some embodiments. As alluded to above, a single volume may be associated with multiple replicas, such as a primary replica and several secondary replicas. Each replica may be distributed across several block storage servers. The associations between a volume, its replicas, and the servers hosting the replicas (or some of the replicas) can change over time when hardware failures occur or when data migrates between servers as part of background load balancing operations. When deploying block storage servers in a PSE, it is preferable to be able to change those associations even when the PSE is disconnected from or unreachable from the provider network in order to maintain high data availability (e.g., from a primary replica to a secondary replica in the event of a failure) and high durability (e.g., immediately beginning the process of recreating lost data in the event of a server failure).

[0098] As shown, PSE 1088 includes a component that tracks how volumes are distributed (or mapped) across block storage server instances, so that when a block storage client needs to access a volume, the block storage client can find the block storage server that hosts that volume. Exemplary volume A mapping data 1098 includes several items for each entry, including a server identifier (e.g., a unique identifier associated with the instance or host hardware), a server address (e.g., an IP address), and the volume type (e.g., primary or secondary) for volume A. The mapping data may include different or additional items, such as a block identifier (e.g., if replicas are split or striped across multiple block storage server instances). Exemplary volume A mapping data 1098 shows that volume A includes a primary replica 1052 provided by block storage server instance 1050A and two secondary replicas 1054 and 1056 provided by block storage server instances 1050B and 1050C, respectively. Note that PSE 1088 can, and likely does, host many other block storage server instances (not shown).

[0099] To reliably store the volume mapping data 1098, a distributed data store 1064 can store the volume mapping data 1098. In some embodiments, each distributed data store corresponds to a cluster of nodes that separately maintains the state of the volume mapping(s). Each node of a cluster exchanges messages with other nodes of the cluster to update its state based on the state seen by the other cluster nodes. One of the nodes of a cluster can be designated as a leader or primary node to which changes to volume mappings are proposed. The nodes of a cluster can implement a consensus protocol, such as the Paxos protocol, to propose and agree on changes to the volume mapping data for a given volume. A cluster may track volume mapping data for one or more volumes. As shown, cluster 1066 tracks volume mapping data 1098 for volume A, while another cluster 1067 tracks volume mapping data for other volumes.

[0100] In some embodiments, the nodes of the cluster are instances executed by hosts in the provider network. Such instances persist their respective views of volume mapping data to the host's non-volatile storage (e.g., via a block storage client to volumes hosted by the block storage server instance). In other embodiments, the nodes of the cluster are part of block storage server software executed by the block storage server instance. As shown in the exemplary node software environment 1090, the nodes can run as containers 1082 hosted within a container engine process included in the block storage server software 1080. Such nodes can persist their views of volume mapping data directly to volumes provided by the block storage server software. Preferably, the nodes of the cluster are hosted by separate instances or within separate failure domains.

[0101] Like other software run by the PSE host, nodes are subject to hardware failures. In such cases, the remaining nodes (or the provider network's block storage service) can detect the loss of a node, create a new node to replace the lost node, and update the new node's volume mapping data based on the consensus view of the other nodes' volume mapping data. As a result, not only can the volume mapping data change, but the IDs of the instances hosting the cluster's nodes can also change. The cluster mapping data can be used to track the cluster. Exemplary cluster mapping data 1099 includes several items, such as a node identifier (e.g., VOL_A_NODE1), a node address (e.g., an IP address), and a node type (e.g., a leader node). In this example, the cluster is formed with five nodes.

[0102] Cluster mapping data can be determined and maintained by a cluster discovery service 1062. As shown in circle A, the cluster discovery service 1062 can monitor the location of nodes in a cluster for various volumes hosted by block storage server instances in the PSE. The cluster discovery service 1062 can monitor the location of nodes in various ways. For example, in an embodiment in which nodes execute in the environment 1090, the cluster discovery service 1062 can periodically poll all of the block storage server instances 1050 hosted by the PSE 1088 to obtain the identity of any resident nodes. As another example, a network manager on the host of the PSE 1088 can be configured to route a special broadcast message to any of the hosted cluster nodes (e.g., hosted directly or indirectly by the block storage server instances, etc.). The cluster discovery service 1062 can periodically broadcast a query to obtain the identity of any of the hosted cluster nodes.

[0103] In some embodiments, the cluster discovery service 1062 is an instance hosted by one of the hosts of the PSE 1088. Such an instance may have a VNA with a reserved IP address in the IVN 1052 of the BSS so that it can be reached even if a host needs to be changed due to a hardware failure. In other embodiments, the cluster discovery service 1062 can be integrated into the DNS service of the PSE. For example, a volume cluster can be associated with a domain, and the DNS service can resolve name resolution requests for that name to the IP addresses of one or more nodes in the cluster.

[0104] As shown in circle B, the instance management service 1040 can send a message to a block storage client 1060 of a particular host to attach a volume to the hosted instance (not shown). For example, the instance management service 1040 can send a message to the host that includes the instance identifier and volume identifier for volume A. As shown in circle C, the block storage client 1060 can query a cluster discovery service 1062 to obtain the identities of one or more nodes in volume A's cluster 1066. In some embodiments, the block storage client 1060 can cache the cluster mapping data in a cluster mapping data cache 1066. Note that in some embodiments, the cluster discovery service 1062 can be omitted, and the block storage client 1060 is configured to query block storage server instances in the PSE 1088 (e.g., via the broadcast mechanism described above) to identify the nodes in volume A's cluster 1066.

[0105] As shown in circle D, the block storage client 1060 can obtain a current view of volume mapping data for volume A from the cluster 1066 and, based on the volume mapping data, connect to the block storage server 1050 that hosts volume A, as shown in circle E. Although not shown, in some embodiments, upon receiving a connection request from a client, the block storage server can send a message to the volume cluster indicating whether the block storage server still hosts the volume (or at least a portion of the volume). Despite receiving a connection request from a block storage client, the block storage server may not host the volume for various reasons. For example, recent changes to the set of servers hosting the volume may not yet have propagated to or through the volume cluster, or the block storage client sending the connection request may be relying on outdated cached volume mapping data. Regardless of whether the block storage server hosts the volume, the block storage server receiving the connection request from the client can obtain the identities of one or more nodes for the volume from the cluster discovery service 1062. If a block storage server no longer hosts the volume, the block storage server can propose an update to the datastore maintained by the cluster to remove the block storage server from the volume mapping data. Additionally, the block storage server can send a response to the block storage client that initiated the request indicating that the connection attempt failed and, optionally, indicating that the volume is no longer hosted by the server. If the block storage server still hosts the volume, the block storage server can send an acknowledgement to the cluster indicating to the cluster that at least that portion of the mapping data is still valid.

[0106] FIG. 11 is a block diagram illustrating an exemplary system for tracking volume mapping, according to at least some embodiments. As described above, connectivity between a PSE and a provider network cannot be guaranteed. To meet desired levels of data availability and data durability, the PSE includes a facility that allows for changing the mapping between a given volume and the block storage server instance that hosts that volume. As described with reference to FIG. 10, volumes for a given volume can be tracked using a cluster that implements a distributed data store. FIG. 11 illustrates a phased approach to volume placement, or the process of selecting block storage server instances to host replicas or portions of replicas of a volume. Specifically, a BSS volume placement service 1107 of a BSS 1106 makes the initial placement decision and associated volume mapping upon volume creation, and a PSE volume placement service 1108 manages subsequent changes to the volume mapping over the volume's lifetime. The PSE volume placement service 1108 can be implemented in various ways, such as by an instance hosted by a PSE 1188 integrated as a component of a PSE framework (e.g., PSE framework 202).

[0107] As shown in circle A, PSE volume placement service 1108 monitors the status of block storage server instances hosted by PSE 1188. For example, PSE volume placement service 1108 can periodically poll the hosted block storage server instances for responses and / or collect metrics related to the hosted block storage server instances' resource usage, their failure domains, and / or hosts (e.g., as described with reference to FIG. 9). As shown, PSE volume placement service 1108 can send the collected server status to BSS volume placement service 1107, as shown in circle B. Note that in other embodiments, BSS volume placement service 1107 can obtain metrics related to resource utilization from an instance management service (e.g., as described with reference to FIG. 9).

[0108] Upon receiving a request to create a new block storage volume for an instance hosted by PSE 1188, BSS 1106 can request a volume placement recommendation from BSS placement service 1107. Depending on the profile of the new volume (e.g., number of replicas, whether replicas are striped, etc.), BSS placement service 1107 can provide the identification of recommended block storage server instances. In this example, BSS placement service 1107 recommends block storage server instances 1150A and 1150B to host new volume A with two replicas. Using the recommended block storage server instances, BSS 1106 creates a new cluster 1166 and tracks volume mapping data for volume A, as shown in circle C. The mapping data initially identifies block storage server instance 1150A as hosting the primary replica block and storage server instance 1150B as hosting the secondary replica. Further, BSS 1106 sends one or more messages to the identified block storage servers to create storage volumes (e.g., a storage volume hosted by block storage server instance 1150A and a storage volume hosted by block storage server instance 1150B), as shown in circle D. The storage volumes can be backed by the capacity of the underlying host storage devices provisioned to each block storage server instance. The storage volume created on block storage server instance 1150A can host primary replica 1152 of volume A, and the storage volume created on block storage server instance 1150B can host secondary replica 1154A of volume A. In some embodiments, the newly created storage volumes can be loaded from a volume snapshot or machine image, as described with reference to FIG. 8.In this example, an instance (not shown) attached to volume A performs block storage operations for some time, causing block storage server instance 1150A to communicate with block storage server instance 1150B (e.g., to propagate writes from primary replica 1152 to secondary replica 1154B, as shown in circle E).

[0109] At some point, block storage server instance 1150B may experience a problem, as shown in circle F. For example, block storage server instance 1150B may become slow or unresponsive (e.g., due to a memory leak, hardware failure, etc.). Problem detection can occur in a variety of ways. In some embodiments, block storage server instance 1150A detects a problem due to, for example, a failure to acknowledge propagated writes. In such cases, block storage server instance 1150A may include a policy that includes one or more actions to perform in response to the detected problem. For example, block storage server instance 1150A may wait for several consecutive propagated writes to go unacknowledged, or may wait for a certain period of time. At that point, block storage server instance 1150A can request a replacement block storage server instance for secondary replica 1154 from PSE volume placement service 1108. In other embodiments, PSE volume placement service 1108 detects the problem during monitoring (e.g., based on collected metrics or responsiveness) as described above with reference to circle A. Again, PSE volume placement service 1108 may include a policy that includes one or more actions to perform in response to the detected problem, including initiating replacement of the block storage server instance of secondary replica 1154. Regardless of the detector, PSE volume placement service 1108 provides block storage server instance 1150A with the identity of the replacement block storage server instance.

[0110] In this example, PSE volume placement service 1108 identifies block storage server instance 1150C to block storage server instance 1150A, as shown in circle G. A message is sent to block storage server instance 1150C to create a storage volume that backs the replica data being relocated. For example, upon identifying block storage server instance 1150C, PSE volume placement service 1108 can send a message to block storage server instance 1150C to create the storage volume. As another example, upon receiving identification information from PSE volume placement service 1108, block storage server instance 1150A can send a message to create the storage volume. Once created, block storage server instance 1150A can begin mirroring replica 1152 to block storage server instance 1150C as replica 1154B, as shown in circle H. Note that if block storage server instance 1150B is responsive (but, for example, exhibits poor performance), a mirroring operation can be performed by copying replica 1154A to replica 1154B. While remirroring from either replica 1154 or replica 1152 is feasible in this scenario because the replicas are not distributed across storage servers, in other embodiments, it may be necessary to access the various storage servers across which the replicas are distributed to recreate or otherwise generate the lost data by using redundancy, etc. (e.g., parity bits, error correction codes, etc.) encoded in the stored data. For example, if replica data is encoded and distributed across 10 block storage server instances and one of the 10 is lost, the lost data can be recreated by reading the data associated with the replica from the remaining 9 block storage server instances.As another example, if another 10 block storage servers host other replicas of the volume using the same distribution pattern, the lost data can be copied from the block storage server instances that host the corresponding portions of the other replicas.

[0111] As shown in circle I, block storage server instance 1150A can submit a request to cluster 1166 to update the volume mapping for volume A to replace block storage server instance 1150B with block storage server instance 1150C as the host of the secondary replica. Block storage server instance 1150A may submit the request to cluster 1166 once it has initiated or completed remirroring to block storage server instance 1150C. In other embodiments, another entity (such as PSE volume placement service 1108 or block storage server instance 1150C) may submit a request to cluster 1166 to update the volume mapping for volume A.

[0112] FIG. 12 is a flow diagram illustrating method operations for launching a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes described herein, or variations, and / or combinations thereof) are performed under the control of one or more computer systems comprised of executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors through hardware or a combination thereof. The code is stored on a computer-readable storage medium, e.g., in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of an extension of a provider network. The extension of a provider network includes one or more physical computing devices or systems and is located remotely from a data center of the provider network (e.g., outside of a data center network), such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the data centers of the provider network. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems in the provider network. In some embodiments, one or more (or all) of the operations are performed by components of a host (e.g., host 420A) in another illustration.

[0113] In block 1205, the operations include receiving, by a computer system of a provider network extension, a first request to start a first virtual machine to host a block storage server application, the provider network extension communicating with the provider network over at least a third-party network. In providing block storage devices to instances hosted by the PSE, the block storage server can be hosted using virtualization provided by a host in the PSE. For example, the block storage service in the provider network can initiate the start of a block storage server virtual machine with an instance management service, which can then issue a start request to a host manager of a selected host in the PSE over a secure communication channel between the provider network and the PSE.

[0114] The operations may further include provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as provisioned storage devices, at block 1210. As part of booting the virtual machine, a host manager of the host system may allocate or provision a portion of the host system's computing resources to the virtual machine. Such resources may include, for example, storage capacity of a storage device (e.g., SSD), memory capacity (e.g., RAM), processors or processor cores, etc. of the host system.

[0115] At block 1215, the operations further include executing a block storage server application on the first virtual machine. As part of executing the block storage server application, at block 1220, the operations further include creating a logical volume on the provisioned storage device in response to a second request from the block storage service of the provider network to create the logical volume. For example, the block storage service of the provider network can send one or more messages to create a volume (using the provisioned storage capacity) that can be attached to other instances, thereby allowing the instances to access the volume via the block storage interface.

[0116] As part of executing the block storage server application, the operations further include receiving a third request to perform an I / O operation on the logical volume at block 1225 and performing the requested I / O operation on the logical volume at block 1230. For example, with reference to FIG. 4, instance A 430 can issue a command to read a block of data from a block address of a virtual block device attached to the instance and backed by block storage volume A 434. Block storage server instance 450 can receive the command (e.g., via IVN 452 of the BSS) and process it for block storage volume A 434. For example, with reference to FIG. 5, instance B 436 can issue a command to write a block of data to a block address of a virtual block device attached to the instance and backed by block storage volume B 440. Block storage server instance 450 can receive the command (e.g., via IVN 452 of the BSS) and process it for block storage volume A 440.

[0117] FIG. 13 is a flow diagram illustrating method operations for using a virtualized block storage server, according to at least some embodiments. Some or all of the operations (or other processes described herein, or variations, and / or combinations thereof) are performed under the control of one or more computer systems comprised of executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors through hardware or a combination thereof. The code is stored on a computer-readable storage medium, e.g., in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of an extension of a provider network. The extension of a provider network includes one or more physical computing devices or systems and is located remotely from a data center of the provider network (e.g., outside of a data center network), such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the data centers of the provider network. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems in the provider network. In some embodiments, one or more (or all) of the operations are performed by a host in another figure (e.g., host 420A).

[0118] At block 1305, the operations include executing, by the computer system, a first block storage server virtual machine for hosting a first volume using one or more storage devices of the computer system. As shown in FIG. 4, for example, host 420A hosts a virtual machine (i.e., block storage server instance 450). At block 1310, the operations further include executing, by the computer system, a second virtual machine that can access the virtual block storage device. Host 420A also hosts another virtual machine (i.e., instance 430). At block 1315, the operations further include executing, by the computer system, a block storage client. Host 420A includes block storage client 460A, which can facilitate attaching block storage devices to hosted virtual machines. As part of executing the block storage client, the operations further include receiving, from the second virtual machine, a first block storage operation to perform on the virtual block storage device, at block 1320, and sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the first block storage operation on the first volume, at block 1325. The virtual machine can issue block storage operations (e.g., block reads, writes, burst reads, writes, etc.) to the attached storage device via the block storage client. The block storage client can, for example, relay those block storage operations across a network to a block storage server hosting a volume containing the block addresses.

[0119] FIG. 14 is a flow diagram illustrating method operations for managing virtualized block storage servers in a provider substrate extension, according to at least some embodiments. Some or all of the operations (or other processes described herein, or variations, and / or combinations thereof) are performed under the control of one or more computer systems comprised of executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that collectively execute on one or more processors through hardware or a combination thereof. The code is stored on a computer-readable storage medium, e.g., in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) operations are performed by a computer program or application executed by one or more components of an extension of the provider network. The extension of the provider network includes one or more physical computing devices or systems and is located remotely from (e.g., outside of) the data center of the provider network, such as on the premises of a customer of the provider network. A provider network, such as a cloud provider network, includes various services performed by computer systems located within the provider network's data center. One or more components of the extension communicate with the provider network by receiving management operations, etc. from services executed by computer systems in the provider network. In some embodiments, one or more (or all) of the operations are performed by components of the PSEs in other figures (e.g., PSE 1088, PSE 1188).

[0120] In block 1405, the operations include receiving a first request by a first block storage server instance to create a first storage volume and store a first portion of the first logical volume, and in block 1410, receiving a second request by a second block storage server instance to create a second storage volume and store a second portion of the first logical volume. As shown in FIG. 11, for example, initial placement of volumes (e.g., replicas, striped replicas, etc.) can originate from block storage service 1106 in the provider network and be received by components of PSE 1188. In the example shown in FIG. 11, volume A is initially stored using block storage server instances 1150A and 1150C hosted by PSE 1188.

[0121] At block 1415, the operations further include sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume. At some point, it may be necessary to change the block storage server instance that hosts a portion of the volume. As described herein, the PSE volume placement service 1108 or one of the other block storage server instances that host the volume can send a message to another block storage server instance to replace the instance that is being changed.

[0122] At block 1420, the operations further include storing, by the third block storage server instance, the second portion of the first logical volume in a third storage volume. At block 1425, the operations further include updating a data store containing identification information of each block storage server instance hosting a portion of the first logical volume to remove the identification information of the second block storage server instance and add the identification information of the third block storage server instance. To track volume migration across instances, PSE 1188 can host a data store that maps volumes (including replicas, if any, striping across servers, etc.). Such a data store can be a cluster as described with reference to FIG. 10.

[0123] FIG. 15 illustrates an exemplary provider network (or “service provider system”) environment, according to at least some embodiments. Provider network 1500 may offer resource virtualization to customers via one or more virtualization services 1510, which enable customers to purchase, rent, or acquire instances 1512 of virtualized resources (including, but not limited to, compute and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 1516 may be associated with resource instance 1512. The local IP address is the internal network address of resource instance 1512 in provider network 1500. In some embodiments, provider network 1500 may also offer public IP addresses 1514 and / or public IP address ranges (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that customers may acquire from provider 1500.

[0124] Conventionally, provider network 1500, via virtualization service 1510, may enable a customer of the service provider (e.g., a customer operating one or more client networks 1550A-1550C including one or more customer device(s) 1552) to dynamically associate at least some public IP addresses 1514 assigned or allocated to the customer with particular resource instances 1512 assigned to the customer. Provider network 1500 may also enable a customer to remap a public IP address 1514 previously mapped to one virtualized computing resource instance 1512 allocated to the customer to another virtualized computing resource instance 1512 also allocated to the customer. Using the virtualized computing resource instances 1512 and public IP addresses 1514 provided by the service provider, a customer of the service provider, such as an operator of customer network(s) 1550A-1550C, may, for example, implement customer-specific applications and present the customer's applications over an intermediate network 1540, such as the Internet. Other network entities 1520 in intermediate network 1540 may then generate traffic to destination public IP address 1514 exposed by customer network(s) 1550A-1550C. The traffic is routed to the service provider data center, where it is routed through the network fabric to the local IP address 1516 of the virtualized computing resource instance 1512 that is currently mapped to destination public IP address 1514. Similarly, response traffic from virtualized computing resource instance 1512 may be routed through the network fabric to intermediate network 1540 back to source entity 1520.

[0125] As used herein, a local IP address refers to the internal or "private" network address of a resource instance, for example, in a provider network. The local IP address may be within an address block reserved by Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or may be in an address format specified by IETF RFC 4193, and may be changeable within the provider network. Network traffic originating from outside the provider network is not directly routed to the local IP address. Instead, the traffic uses a public IP address that is mapped to the local IP address of the resource instance. The provider network may include a network device or appliance that provides network address translation (NAT) or similar functionality to map from public IP addresses to local IP addresses (or vice versa).

[0126] A public IP address is an Internet-variable network address assigned to a resource instance by either the service provider or the customer. Traffic routed to the public IP address is translated, for example, by a 1:1 NAT, and forwarded to the local IP address of each resource instance.

[0127] Some public IP addresses may be assigned to particular resource instances by the provider network infrastructure. These public IP addresses may be referred to as standard public IP addresses or simply standard IP addresses. In some embodiments, the mapping of standard IP addresses to the local IP addresses of the resource instance is the default launch configuration for all resource instance types.

[0128] At least some public IP addresses may be allocated to or acquired by customers of provider network 1500. The customers may then assign the allocated public IP addresses to specific resource instances allocated to the customers. These public IP addresses may be referred to as customer public IP addresses or simply customer IP addresses. Instead of being assigned to resource instances by provider network 1500, as is the case with standard IP addresses, customer IP addresses may be assigned to resource instances by customers, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are allocated to customer accounts and can be remapped to other resource instances by each customer as needed or desired. Customer IP addresses are associated with a customer's account, not a specific resource instance, and the customer controls the IP address until they choose to release it. Unlike traditional static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping their public IP addresses to any resource instances associated with their account. For example, customer IP addresses allow for troubleshooting customer resource instance or software problems by remapping the customer IP address to a replacement resource instance.

[0129] 16 is a block diagram of an exemplary provider network that offers storage services and hardware virtualization services to customers, according to at least some embodiments. Hardware virtualization service 1620 offers multiple computational resources 1624 (e.g., VMs) to customers. Computational resources 1624 may be rented or leased, for example, to customers of provider network 1600 (e.g., customers implementing customer network 1650). Each of computational resources 1624 may be provided with one or more local IP addresses. Provider network 1600 may be configured to route packets from the local IP addresses of computational resources 1624 to public Internet destinations and from public Internet sources to the local IP addresses of computational resources 1624.

[0130] Provider network 1600 may provide, for example, customer network 1650, coupled to intermediate network 1640 via local network 1656, the ability to implement virtual computing systems 1692 via hardware virtualization service 1620, which is coupled to intermediate network 1640 and provider network 1600. In some embodiments, hardware virtualization service 1620 may provide one or more APIs 1602, e.g., web services interfaces, through which customer network 1650 may access functionality provided by hardware virtualization service 1620, for example, via console 1694 (e.g., web-based application, standalone application, mobile application, etc.). In some embodiments, in provider network 1600, each of virtual computing systems 1692 in customer network 1650 may correspond to computational resources 1624 leased, rented, or otherwise provided to customer network 1650.

[0131] From an instance of virtual computing system 1692 and / or another customer device 1690 (e.g., via console 1694), a customer may access functionality of storage service 1610, e.g., via one or more APIs 1602, to access data from and store data in storage resources 1618A-1618N in a virtual data store 1616 (e.g., folders or "buckets," virtualized volumes, databases, etc.) provided by provider network 1600. In some embodiments, a virtualized data store gateway (not shown) may be provided in customer network 1650, which may cache at least some data, e.g., frequently accessed or important data, locally and communicate with storage service 1610 via one or more of a number of communication channels to upload new or modified data from the local cache, thereby maintaining a primary store of data (virtualized data store 1616). In some embodiments, a user may mount and access volumes of a virtual data store 1616 through a virtual computing system 1692 and / or on another customer device 1690 through a storage service 1610 acting as a storage virtualization service, and these volumes may appear to the user as local (virtualized) storage 1698.

[0132] 16, the virtualization service(s) may also be accessed from resource instances within provider network 1600 via API(s) 1602. For example, a customer, appliance service provider, or other entity may access the virtualization service from within each virtual network in provider network 1600 via API 1602 to request allocation of one or more resource instances within the virtual network or within another virtual network.

[0133] FIG. 17 is a block diagram illustrating an exemplary computing device that may be used in at least some embodiments. In at least some embodiments, such a computer system may be used as a server implementing one or more of the control plane and / or data plane components, various virtualization components (virtual machines, containers, etc.), and / or SEDs used to support the provider board and / or PSE described herein. Such a computer system may include a general-purpose or special-purpose computer system that includes or is configured to access one or more computer-accessible media. In at least some embodiments, such a computer system may also be used to implement components outside of the provider board and / or provider board extensions (e.g., customer gateway / router 186, other customer resources 187, etc.). In the illustrated embodiment of a computer system, computer system 1700 includes one or more processors 1710 coupled to system memory 1720 via input / output (I / O) interface 1730. Computing device 1700 further includes a network interface 1740 coupled to I / O interface 1730. FIG. 17 depicts computer system 1700 as a single computing device, in various embodiments, computer system 1700 may include one computing device or any number of computing devices configured to operate together as single computer system 1700.

[0134] In various embodiments, computer system 1700 may be a uniprocessor system including one processor 1710, or a multiprocessor system including several processors 1710 (e.g., two, four, eight, or another suitable number). Processor 1710 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as the x86, ARM, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of processors 1710 may typically, but not necessarily, implement the same ISA.

[0135] System memory 1720 may store instructions and data accessible by processor(s) 1710. In various embodiments, system memory 1720 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data that implement one or more desired functions, such as those methods, techniques, and data described above, are shown stored in system memory 1720 as code 1725 and data 1726.

[0136] In one embodiment, I / O interface 1730 may be configured to coordinate I / O traffic between processor 1710, system memory 1720, and any peripheral devices of a device including network interface 1740 or other peripheral interfaces. In some embodiments, I / O interface 1730 may perform any necessary protocol, timing, or other data conversion to convert data signals from one component (e.g., system memory 1720) into a format appropriate for use by another component (e.g., processor 1710). In some embodiments, I / O interface 1730 may include support for devices attached by various types of peripheral buses, such as, for example, variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I / O interface 1730 may be split between two or more separate components, such as, for example, a northbridge and a southbridge. Also, in some embodiments, some or all of the functionality of I / O interface 1730, such as the interface to system memory 1720, may be incorporated directly into processor 1710.

[0137] Network interface 1740 may be configured to allow data to be exchanged between computer system 1700 and other devices 1760 (e.g., other computer systems or devices as shown in FIG. 1) attached to a network or networks 1750. In various embodiments, network interface 1740 may support communications over any suitable wired or wireless general data network, such as, for example, an Ethernet network type. Additionally, network interface 1740 may support communications over a telecommunications / telephony network, such as an analog voice network or a digital fiber communications network, over a storage area network (SAN), such as a Fibre Channel SAN, or over any other suitable type of network and / or protocol.

[0138] In some embodiments, computer system 1700 includes one or more offload cards 1770 (including one or more processors 1775 and possibly one or more network interfaces 1740) connected using an I / O interface 1730 (e.g., a bus implementing the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect version such as the QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). For example, in some embodiments, computer system 1700 may function as a host electronic device (e.g., operating as part of a hardware virtualization service) that hosts compute instances, and one or more offload cards 1770 execute a virtualization manager that can manage the compute instances executing on the host electronic device. By way of example, in some embodiments, offload card(s) 1770 can perform compute instance management operations such as pausing and / or unpausing compute instances, launching and / or terminating compute instances, performing memory transfers / copies, etc. These management operations may, in some embodiments, be performed by offload card(s) 1770 in coordination with (e.g., in response to a request from) a hypervisor executed by another processor 1710A-1710N of the hypervisor computer system 1700. However, in some embodiments, the virtualization manager implemented by the offload card(s) 1770 may service requests from other entities (e.g., from the compute instances themselves) and may not coordinate with (or service) any other hypervisor. Referring to FIG. 2, in at least some embodiments, at least some of the functionality of PSE framework 202 and host manager 222 executes on one or more processors 1775 of offload card 1770 while the instances (e.g., 232, 234, 236) execute on one or more processors 1710.

[0139] In some embodiments, computer system 1700 includes one or more storage devices (SD) 1780. Exemplary storage devices 1780 include solid-state drives (e.g., with various types of flash or other memory) and magnetic drives. Processor(s) 1710 may access SD 1780 via interface(s) 1730 or, in some cases, via offload card(s) 1770. For example, offload card(s) 1770 may include a system-on-chip (SoC) that includes multiple interconnect interfaces and bridges the interface 1730 with the interface to SD 1780 (e.g., performs a PCIe-to-PCIe bridge).

[0140] In some embodiments, system memory 1720 may be one embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer-accessible media. Generally speaking, computer-accessible media may include non-transitory storage or memory media, such as magnetic or optical media (e.g., disks or DVDs / CDs), coupled to computing device 1700 via I / O interface 1730. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, double data rate (DDR), SDRAM, SRAM, etc.), read-only memory (ROM), etc., that may be included in some embodiments of computing device 1700 as system memory 1720 or another type of memory. Additionally, computer-accessible media may include signals, such as electrical, electromagnetic, or digital signals, conveyed over a transmission medium or a communication medium, such as a network and / or wireless link, such as may be implemented via network interface 1740.

[0141] Various embodiments as described or suggested herein can be implemented in a variety of operating environments, which may include one or more user computers, computing devices, or processing devices that can be used to run any number of applications. User or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, as well as mobile, wireless, and handheld devices capable of running mobile phone software and supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as dummy terminals, thin clients, gaming consoles, and other devices capable of communicating over a network.

[0142] Most embodiments utilize at least one network that will be familiar to those skilled in the art to support communications using any of a variety of commercially available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The network(s) may include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0143] In embodiments utilizing a web server, the web server may run any of a variety of server or middle-tier applications, including an HTTP server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a data server, a Java server, a business application server, etc. The server(s) may also be capable of executing programs or scripts in response to requests from user devices by executing one or more web applications, etc., which may be implemented as one or more scripts or programs written in any programming language, such as Java, C, C#, or C++, or any scripting language, such as Perl, Python, or TCL, and combinations thereof. The server(s) may also include database servers, including, but not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM, etc. Database servers may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0144] The environment may include various data stores and other memory and storage media as described above. These may reside in a variety of locations, such as storage media local to (and / or resident on) one or more computers or remote from any or all of the computers across a network. In a particular set of embodiments, information may reside within a storage area network ("SAN") familiar to those skilled in the art. Similarly, any files necessary to perform functions belonging to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computerized devices, each such device may include hardware elements that may be electrically coupled via a bus, including, for example, at least one central processing unit ("CPU"), at least one input device (e.g., a mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable storage devices, memory cards, flash cards, etc.

[0145] Such devices may also include computer-readable storage medium readers, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.), and working memory, as described above. A computer-readable storage medium reader is configured to connect to or receive a computer-readable storage medium and represents a remote, local, fixed, and / or removable storage device, as well as a storage medium for temporarily and / or permanently containing, storing, transmitting, and retrieving computer-readable information. Systems and various devices also typically include several software applications, modules, services, or other elements located within at least one working memory device, including operating systems and application programs, such as client applications or web browsers. It should be understood that alternative embodiments may have many variations from those described above. For example, customized hardware may also be used, and / or particular elements may be implemented in hardware, software (including portable software, such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0146] Storage media and computer-readable media for containing code or portions of code may include any suitable media known or used in the art, including, but not limited to, storage media and communication media such as volatile and nonvolatile media, removable and non-removable media, implemented in any method or technology for storing and / or transmitting information such as computer-readable instructions, data structures, program modules, or other data, including RAM, ROM, Electrically Erasable Programmable Read-Only Memory ("EEPROM"), Flash memory, or other memory technology, Compact Disc Read-Only Memory ("CD-ROM"), Digital Versatile Disk (DVD), or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage device, or any other medium that can be used to store the desired information and that can be accessed by a system device. Based on the disclosure and teachings provided herein, one of ordinary skill in the art will recognize other means and / or methods for implementing the various embodiments.

[0147] In the foregoing description, various embodiments are described. For purposes of explanation, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that the embodiments may be practiced without the specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0148] Bracketed text and blocks with dashed borders (e.g., large dashes, small dashes, dotted lines, and dotted lines) are used herein to describe optional actions that add additional features to some embodiments. However, such notations should not be interpreted to mean that these are the only options or optional actions and / or that blocks with solid borders are not optional in certain embodiments.

[0149] A reference number with a suffix (e.g., 101A, 102A, etc.) may be used to indicate that there may be one or more instances of the referenced entity in various embodiments; when multiple instances are present, each need not be identical, but may instead share some common characteristic or function in a common manner. Furthermore, the use of a particular suffix is not meant to imply that a particular quantity of the entity is present, unless specifically indicated to the contrary. Thus, two entities using the same or different suffixes may or may not have the same number of instances in various embodiments.

[0150] References to "one embodiment," "an embodiment," "exemplary embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but not all embodiments necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is believed to be within the knowledge of one of ordinary skill in the art to affect such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described.

[0151] Additionally, in the various embodiments described above, unless otherwise specified, disjunctive language, such as the phrase "at least one of A, B, or C," is intended to be understood to mean either A, B, or C, or any combination thereof (e.g., A, B, and / or C). Thus, such disjunctive language is not intended to, and should not be understood to, imply that a given embodiment requires the presence of at least one of A, at least one of B, or at least one of C, respectively.

[0152] At least some embodiments of the disclosed technology can be explained in view of the following examples. 1. A computer-implemented method comprising: executing, by a first one or more processors of a first computer system in a provider network, a block storage server virtual machine to host a first storage volume using one or more storage devices of the first computer system; executing, by the first one or more processors, a customer virtual machine that has access to a virtual block storage device; executing, by a second one or more processors of the first computer system, a block storage client; The block storage client: receiving a first block storage operation from the customer virtual machine to perform on the virtual block storage device; sending a message to the block storage server virtual machine to cause the block storage server virtual machine to perform the first block storage operation on a first storage volume; The computer-implemented method. 2. The computer-implemented method of clause 1, wherein the message is transmitted over a secure virtual network using a key to encrypt and decrypt traffic transmitted over the virtual network. 3. The computer-implemented method of clause 2, wherein a first virtual network address of the virtual network is associated with the block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client. 4. A computer-implemented method comprising: running, by a computer system, a first block storage server virtual machine to host a first volume using one or more storage devices of said computer system; executing, with the computer system, a second virtual machine that has access to a virtual block storage device; executing, by the computer system, a block storage client; executing the block storage client, receiving, from the second virtual machine, a first block storage operation to be performed on the virtual block storage device; sending a message to the first block storage server virtual machine to cause the first block storage server virtual machine to perform the block storage operation on the first volume; The computer-implemented method. 5. The computer-implemented method of clause 4, wherein the message is transmitted over a secure virtual network using a key to encrypt and decrypt traffic transmitted over the virtual network. 6. The computer-implemented method of clause 5, wherein a first virtual network address of the virtual network is associated with the first block storage server virtual machine and a second virtual network address of the virtual network is associated with the block storage client. 7. The computer-implemented method of any one of clauses 4 to 6, wherein the first block storage server virtual machine and the second virtual machine are executed by a first one or more processors of the computer system, and the block storage client is executed by a second one or more processors of the computer system. 8. The computer-implemented method of any one of clauses 4 to 7, wherein the first block storage server virtual machine is a first virtual machine hosted by the computer system, and the second virtual machine is a second virtual machine hosted by the computer system. 9. The computer-implemented method of clause 4, further comprising running, by the computer system, a second block storage server virtual machine to host a second volume using the one or more storage devices of the computer system, the second volume being a replica associated with the first volume. 10. The computer-implemented method of clause 9, wherein the first block storage server virtual machine executes using a first physical component of the computer system and the second block storage server virtual machine executes using a second physical component of the computer system that is different from the first physical component. 11. The computer-implemented method of any one of clauses 4 to 10, wherein the first block storage operation is writing a block of data, and the block storage client further encrypts the block of data using an encryption key associated with the virtual block storage device to generate an encrypted block of data, and the message sent to the first block storage server virtual machine includes the encrypted block of data, causing the first block storage server virtual machine to write the encrypted block of data to the first volume. 12. The computer system is included in an extension of a provider network, and the extension of the provider network communicates with the provider network through at least a third-party network; 12. The computer-implemented method of any one of clauses 4 to 11, wherein the provider network instance management service initiates the execution of the first block storage server virtual machine and the second virtual machine by the computer system. 13. One or more storage devices in a host computer system; a first one or more processors of the host computer system executing a first block storage server application and a second application that can access a virtual block storage device, the first one or more processors executing the first block storage server application including instructions that, when executed, cause the first block storage server application to host a first volume using the one or more storage devices; a first one or more processors of the host computer system executing a block storage client application, the block storage client application including instructions that, when executed, cause the block storage client application to: receiving, from the second application, a first block storage operation to perform on the virtual block storage device; sending a message to the first block storage server application to cause the first block storage server application to perform the first block storage operation on the first volume; Including, the system. 14. The system of clause 13, wherein the message is transmitted over a secure virtual network using a key to encrypt and decrypt traffic transmitted over the virtual network. 15. The system of clause 14, wherein a first virtual network address of the virtual network is associated with the first block storage server application and a second virtual network address of the virtual network is associated with the block storage application. 16. The system described in any one of clauses 13 to 15, wherein the first block storage server application runs within a first virtual machine hosted by the host computer system, and the second application runs within a second virtual machine hosted by the host computer system. 17. The system described in clause 13, wherein the first one or more processors of the host computer system further execute a second block storage server application, the second block storage server application including instructions that, when executed, cause the second block storage server application to host a second volume using the one or more storage devices of the host computer system, the second volume being a replica associated with the first volume. 18. The system described in clause 17, wherein the first block storage server application is executed using a first physical component of the computer system and the second block storage server application is executed using a second physical component of the computer system that is different from the first physical component. 19. The system of any one of clauses 13 to 18, wherein the first block storage operation is writing a block of data, and the block storage client application further includes instructions that, when executed, cause the block storage client application to encrypt the block of data using an encryption key associated with the virtual block storage device to generate an encrypted block of data, and the message sent to the first block storage server application includes the encrypted block of data, causing the first block storage server application to write the encrypted block of data to the first volume. 20. The host computer system is included in an extension of a provider network, and the extension of the provider network communicates with the provider network through at least a third-party network; 20. The system of any one of clauses 13 to 19, wherein the provider network instance management service initiates the execution of the first block storage server application and the second application by the host computer system. 21. A computer-implemented method comprising: receiving a first request to start a first virtual machine at a host computer system of a provider network extension to host a block storage server application, the provider network extension communicating with the provider network over at least a third-party network; provisioning at least a portion of the storage capacity of one or more physical storage devices of the host computer system to the first virtual machine as provisioned storage devices; running the block storage server application on the first virtual machine; executing the block storage server application creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume; and receiving a third request from a block storage client application over a virtual network to perform an I / O operation on the logical volume; performing the requested I / O operation on the logical volume. The computer-implemented method. 22. Before executing the block storage server application on the first virtual machine, the method further comprises: running another application on the first virtual machine to load a boot volume from a machine image retrieved from a data store on the provider network, the boot volume being another logical volume on the provisioned storage device; modifying the first virtual machine to boot from the boot volume; restarting the first virtual machine; 22. The computer-implemented method of claim 21, comprising: 23. The computer-implemented method of clause 21, wherein before executing the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device. 24. A computer-implemented method comprising: receiving, by a computer system of a provider network extension, a first request to start a first virtual machine to host a block storage server application, the provider network extension communicating with the provider network over at least a third-party network; provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as provisioned storage devices; running the block storage server application on the first virtual machine; executing the block storage server application creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume; and receiving a third request to perform an I / O operation on the logical volume; performing the requested I / O operation on the logical volume. The computer-implemented method. 25. Before executing the block storage server application on the first virtual machine, the method further comprises: running another application on the first virtual machine to load a boot volume from a machine image retrieved from a data store in the provider network, the boot volume being another logical volume on the provisioned storage device; modifying the first virtual machine to boot from the boot volume; restarting the first virtual machine; 25. The computer-implemented method of claim 24, comprising: 26. The computer-implemented method of clause 25, further comprising clearing memory of the first virtual machine before executing the block storage server application. 27. The computer-implemented method of clause 24, wherein before executing the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device. 28. The computer-implemented method of clause 27, wherein the loading is performed by at least one of a basic input / output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine. 29. The third request is received from a block storage client application over a virtual network; 29. The computer-implemented method of any one of clauses 24-28, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network. 30. The first virtual machine is one of a plurality of virtual machines, each of which executes a block storage server application, and the method further comprises: determining that a resource utilization associated with one or more of the plurality of virtual machines exceeds a threshold; provisioning at least a portion of the storage capacity of one or more storage devices of another host computer system to a second virtual machine; running another block storage server application on the second virtual machine; and 30. The computer-implemented method of any one of clauses 24 to 29, comprising: 31. Provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system to a second virtual machine as a separate provisioned storage device; running another block storage server application on the second virtual machine, wherein the first virtual machine executes using a first physical component of the computer system and the second virtual machine executes using a second physical component of the computer system that is different from the first physical component; and 31. The computer-implemented method of any one of clauses 24 to 30, further comprising: 32. The computer-implemented method of clause 31, wherein the I / O operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested I / O logical volume on the logical volume includes writing the block of data to the block address of the logical volume, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network includes one or more physical computing devices located outside a data center of the provider network and located at the premises of a customer of the provider network. 33. One or more storage devices of a host computer system of an extension of a provider network, the extension of the provider network communicating with the provider network through at least a third-party network; one or more processors of the host computer system for executing the host manager application, the host manager application including instructions that, when executed, cause the host manager application to: receiving a first request to start a first virtual machine to host a block storage server application; provisioning at least a portion of the storage capacity of the one or more storage devices to the first virtual machine as a provisioned storage device; executing the block storage server application on the first virtual machine, the block storage server application including instructions that, when executed, cause the block storage server application to: creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume; and receiving a third request to perform an I / O operation on the logical volume; performing the requested I / O operation on the logical volume; Including, the system. 34. The host manager application includes further instructions that, when executed, cause the host manager application, before executing the block storage server application on the first virtual machine, to: running another application on the first virtual machine to load a boot volume from a machine image retrieved from a data store in the provider network, the boot volume being another logical volume on the provisioned storage device; modifying the first virtual machine to boot from the boot volume; restarting the first virtual machine; 34. The system of claim 33, 35. The system of clause 34, wherein the host manager application includes further instructions that, when executed, cause the host manager application to clear memory of the first virtual machine before executing the block storage server application. 36. The system of clause 33, wherein a component of the first virtual machine includes instructions that, when executed, cause the component to load a boot volume of the first virtual machine from a machine image stored by the first block storage device before executing the block storage server application on the first virtual machine, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device. 37. The system of clause 36, wherein the component is at least one of a basic input / output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine. 38. The third request is received from a block storage client application over a virtual network; 38. The system of any one of clauses 33 to 37, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network. 39. The host manager application includes further instructions, which, when executed, cause the host manager application to: provisioning at least another portion of the storage capacity of the one or more physical storage devices of the host computer system to a second virtual machine as a separate provisioned storage device; running another block storage server application on the second virtual machine, the first virtual machine running using a first physical component of the computer system and the second virtual machine running using a second physical component of the computer system different from the first physical component; The system according to any one of clauses 33 to 38, 40. The system of clause 39, wherein the I / O operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested I / O logical volume on the logical volume includes writing the block of data to the block address of the logical volume, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network includes one or more physical computing devices located outside a data center of the provider network and located at the premises of a customer of the provider network. 41. A computer-implemented method comprising: receiving, by a first block storage server instance, a first request from a block storage service of a provider network, the first request to create a first storage volume and store a first portion of the first logical volume; receiving, by a second block storage server instance, a second request from the block storage service, the second request to create a second storage volume and store a second portion of the first logical volume; determining that the second block storage server instance is unresponsive; and sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume; storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume; updating a data store containing identification information of each block storage server instance that hosts a portion of the first logical volume to remove the identification information of the second block storage server instance and add the identification information of the third block storage server instance; the first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems of an extension of the provider network, the extension of the provider network communicating with the provider network over at least a third-party network. 42. Receiving, by a block storage client, from the data store, the identification of each block storage server instance hosting a portion of the first logical volume; establishing, by the block storage client, a connection to at least one block storage server instance included in the identification information; receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; sending, by the block storage client, a message over the connection to the at least one block storage server instance to cause the at least one block storage server instance to write the block of data to the block address of the first storage volume; 42. The computer-implemented method of claim 41, further comprising: 43. The computer-implemented method of any one of clauses 41-42, further comprising polling the first block storage server instance and the second block storage server instance for a response. 44. A computer-implemented method comprising: receiving, by a first block storage server instance, a first request to create a first storage volume and store a first portion of a first logical volume; receiving, by a second block storage server instance, a second request to create a second storage volume and store a second portion of the first logical volume; sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume; storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume; updating a data store containing identification information of each block storage server instance that hosts a portion of the first logical volume to remove the identification information of the second block storage server instance and to add the identification information of the third block storage server instance; the computer-implemented method comprising: 45. Receiving, by a block storage client, from the data store, the identification of each block storage server instance hosting a portion of the first logical volume; establishing, by the block storage client, a connection to at least one block storage server instance included in the identification information; receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; sending, by the block storage client, a message over the connection to the at least one block storage server instance to cause the at least one block storage server instance to write the block of data to the block address of the first storage volume; 45. The computer-implemented method of claim 44, further comprising: 46. The computer-implemented method of clause 45, wherein the data store is a distributed data store including multiple nodes that independently execute a consensus protocol to update the identity information. 47. The computer-implemented method of clause 46, further comprising obtaining, by the block storage client, an identification of each node of the plurality of nodes from a service. 48. The computer-implemented method of clause 46, wherein one node of the plurality of nodes runs within a container hosted by the first block storage server instance. 49. The computer-implemented method of clause 48, further comprising broadcasting, by the block storage client, a request for identity information of any one of the plurality of nodes to a plurality of block storage server instances. 50. The computer-implemented method of any one of clauses 44-49, further comprising polling the first block storage server instance and the second block storage server instance for a response, wherein the third request is sent in response to a determination that the second block storage server instance is not responsive. 51. The computer-implemented method of any one of clauses 44-50, wherein the first storage volume is striped across multiple block storage server instances, each of the multiple block storage server instances being included in the identification information. 52. The computer-implemented method of any one of clauses 44-51, wherein the first block storage server instance, the second block storage server instance, and the third block storage server instance are hosted by one or more computer systems of an extension of a provider network, the first request and the second request are sent from a block storage service of the provider network, and the extension of the provider network communicates with the provider network via at least a third-party network. 53. One or more computing devices of an extension of a provider network, the extension of the provider network communicating with the provider network through at least a third-party network, the one or more computing devices including instructions that, when executed on one or more processors, cause the one or more computing devices to: receiving, by a first block storage server instance, a first request to create a first storage volume and store a first portion of a first logical volume; receiving, by a second block storage server instance, a second request to create a second storage volume and store a second portion of the first logical volume; sending a third request to a third block storage server instance to create a third storage volume and store the second portion of the first logical volume; storing, by the third block storage server instance, the second portion of the first logical volume in the third storage volume; updating a data store containing identification information of each block storage server instance that hosts a portion of the first logical volume to remove the identification information of the second block storage server instance and to add the identification information of the third block storage server instance; Including, the system. 54. The one or more computing devices include further instructions that, when executed on the one or more processors, cause the one or more computing devices to: receiving, by a block storage client, from the data store, the identification of each block storage server instance hosting a portion of the first logical volume; establishing, by the block storage client, a connection to at least one block storage server instance included in the identification information; receiving, by the block storage client, a first block storage operation from another instance, the first block storage operation writing a block of data to a block address of a virtual block storage device; sending, by the block storage client, a message over the connection to the at least one block storage server instance to cause the at least one block storage server instance to write the block of data to the block address of the first storage volume; 54. The system of claim 53, 55. The system of clause 54, wherein the data store is a distributed data store including multiple nodes that independently execute a consensus protocol to update the identity information. 56. The system of clause 55, wherein the one or more computing devices include further instructions that, when executed on the one or more processors, cause the one or more computing devices to obtain, by the block storage client, identification information for each node of the plurality of nodes from a service. 57. The system of clause 55, wherein one node of the plurality of nodes runs within a container hosted by the first block storage server instance. 58. The system of clause 57, wherein the one or more computing devices include further instructions that, when executed on the one or more processors, cause the one or more computing devices to broadcast, by the block storage client, a request for identity information of any of the plurality of nodes to a plurality of block storage server instances. 59. The system of any one of clauses 53 to 58, wherein the one or more computing devices include further instructions that, when executed on the one or more processors, cause the one or more computing devices to poll the first block storage server instance and the second block storage server instance for a response, and the third request is sent in response to a determination that the second block storage server instance is not responding. 60. The system of any one of clauses 53-59, wherein the first storage volume is striped across multiple block storage server instances, each of the multiple block storage server instances being included in the identification information.

[0153] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present disclosure as set forth in the appended claims.

Claims

1. 1. A computer-implemented method comprising: receiving, by a host computer system of an extension of a provider network, a first request to start a first virtual machine to host a block storage server application, the extension of the provider network communicating with the provider network over at least a third-party network; provisioning at least a portion of the storage capacity of one or more storage devices of the host computer system to the first virtual machine as provisioned storage devices; running the block storage server application on the first virtual machine; executing the block storage server application creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume; receiving a third request to perform an I / O operation on the logical volume; performing the requested I / O operation on the logical volume. Computer-implemented method.

2. Before executing the block storage server application on the first virtual machine, the method further comprises: running another application on the first virtual machine to load a boot volume from a machine image retrieved from a data store on the provider network, the boot volume being another logical volume on the provisioned storage device; modifying the first virtual machine to boot from the boot volume; restarting the first virtual machine; The computer-implemented method of claim 1 , comprising:

3. The computer-implemented method of claim 2 , further comprising clearing memory of the first virtual machine before executing the block storage server application.

4. 2. The computer-implemented method of claim 1, wherein before executing the block storage server application on the first virtual machine, the method further includes loading a boot volume of the first virtual machine from a machine image stored by a first block storage device, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device.

5. The computer-implemented method of claim 4 , wherein the loading is performed by at least one of a basic input / output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine.

6. the third request is received from a block storage client application over a virtual network; The computer-implemented method of any one of claims 1 to 5, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.

7. The first virtual machine is one of a plurality of virtual machines, each of the plurality of virtual machines executing a block storage server application, and the method further comprises: determining that a resource utilization associated with one or more of the plurality of virtual machines exceeds a threshold; provisioning at least a portion of the storage capacity of one or more storage devices of another host computer system to a second virtual machine; running another block storage server application on the second virtual machine; The computer-implemented method of any one of claims 1 to 6, comprising:

8. provisioning at least another portion of the storage capacity of the one or more storage devices of the host computer system to a second virtual machine as a separate provisioned storage device; running another block storage server application on the second virtual machine, wherein the first virtual machine executes using a first physical component of the host computer system and the second virtual machine executes using a second physical component of the host computer system that is different from the first physical component; The computer-implemented method of any one of claims 1 to 7, further comprising:

9. 9. The computer-implemented method of claim 8, wherein the I / O operation is a write operation for writing a block of data to a block address of the logical volume, performing the requested I / O operation on the logical volume includes writing the block of data to the block address of the logical volume, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network includes one or more physical computing devices located outside a data center of the provider network and located at the premises of a customer of the provider network.

10. one or more storage devices of a host computer system of an extension of a provider network, the extension of the provider network communicating with the provider network through at least a third-party network; one or more processors of the host computer system for executing a host manager application, the host manager application including instructions that, when executed, cause the host manager application to: receiving a first request to start a first virtual machine to host a block storage server application; provisioning at least a portion of the storage capacity of the one or more storage devices to the first virtual machine as provisioned storage devices; executing the block storage server application on the first virtual machine, the block storage server application including instructions that, when executed, cause the block storage server application to: creating a logical volume on the provisioned storage device in response to a second request from a block storage service of the provider network to create the logical volume; receiving a third request to perform an I / O operation on the logical volume; performing the requested I / O operation on the logical volume; Including, the system.

11. The host manager application includes further instructions that, when executed, cause the host manager application, before executing the block storage server application on the first virtual machine, to: running another application on the first virtual machine to load a boot volume from a machine image retrieved from a data store on the provider network, the boot volume being another logical volume on the provisioned storage device; modifying the first virtual machine to boot from the boot volume; restarting the first virtual machine; The system of claim 10 .

12. 12. The system of claim 11, wherein the host manager application includes further instructions that, when executed, cause the host manager application to clear memory of the first virtual machine before executing the block storage server application.

13. 11. The system of claim 10, wherein the component of the first virtual machine includes instructions that, when executed, cause the component to load a boot volume of the first virtual machine from a machine image stored by a first block storage device before executing the block storage server application on the first virtual machine, the boot volume being another logical volume of the provisioned storage device, and the first block storage device being a virtual block storage device.

14. 14. The system of claim 13, wherein the component is at least one of a basic input / output system of the first virtual machine and a unified extensible firmware interface of the first virtual machine.

15. the third request is received from a block storage client application over a virtual network; The system of any one of claims 10 to 14, wherein traffic sent through the virtual network is encrypted using a key associated with the virtual network.

16. The host manager application includes further instructions that, when executed, cause the host manager application to: provisioning at least another portion of the storage capacity of the one or more storage devices of the host computer system to a second virtual machine as a separate provisioned storage device; running another block storage server application on the second virtual machine, the first virtual machine running using a first physical component of the host computer system and the second virtual machine running using a second physical component of the host computer system different from the first physical component; The system according to any one of claims 10 to 15,

17. 17. The system of claim 16, wherein the I / O operation is a write operation for writing a block of data to a block address of the logical volume, and performing the requested I / O operation on the logical volume includes writing the block of data to the block address of the logical volume, the first physical component is a first memory device, the second physical component is a second memory device, and the extension of the provider network includes one or more physical computing devices located outside a data center of the provider network and located at the premises of a customer of the provider network.

Images