Distributed data processing application services in a provider network

Abstract

Techniques for distributed data processing application services in a cloud provider network are described. A virtual machine bundle is obtained, the virtual machine bundle including a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime. A request to start a virtual machine to execute the distributed data processing application runtime is received, the request to start the virtual machine including a bundle identifier identifying the virtual machine bundle. The virtual machine is started based at least in part on the virtual machine bundle. The virtual machine sends a message to register the virtual machine with a virtual machine cluster.

Description

Background Art

[0001] Various distributed data processing applications or tools allow users to complete computationally demanding tasks. In order to complete such tasks in a reasonable amount of time, such applications often adopt a "divide and conquer" approach: breaking the task into smaller chunks of work and distributing those chunks across a cluster of computer systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure will be described with reference to the following drawings.

[0003] Figure 1 is a diagram illustrating a cloud provider network environment including a distributed data processing application service that enables rapid creation of computing clusters, according to some embodiments.

[0004] Figure 2 is a diagram illustrating a method according to some embodiments Figure 1 Another diagram of a distributed data processing application service starting a new member of a compute cluster.

[0005] Figure 3 Depicted is an example network configuration of two computing clusters launched within a cloud provider network environment, according to some embodiments.

[0006] Figure 4 Depicted is an exemplary network addressing scheme used by a distributed data processing application service to form a cluster, according to some embodiments.

[0007] Figure 5 Depicted is a method for initiating a Figure 1 An exemplary call flow for the first member of the cluster is shown in FIG.

[0008] Figure 6 Depicted is a method for initiating a Figure 2 An exemplary call flow for another member of the cluster is shown in FIG.

[0009] Figure 7 Depicted is an exemplary cloud provider network computer system hosting members of a computing cluster according to some embodiments.

[0010] Figure 8 An agent is depicted capturing a state of a virtual machine after launching at least a portion of an application according to some embodiments.

[0011] Fig. 9 A bundle manager service that facilitates distribution of bundles to computer systems is depicted according to some embodiments.

[0012] Fig.10Depicted are methods for launching new members into a computing cluster according to some embodiments.

[0013] Fig.11 A method for establishing network communications between executors of a distributed data processing application is depicted according to some embodiments.

[0014] Fig.12 Depicted is an exemplary provider network environment according to some embodiments.

[0015] Fig.13 is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers according to some embodiments.

[0016] Fig.14 is a block diagram illustrating an exemplary computer system that may be used in some embodiments. DETAILED DESCRIPTION

[0017] The present disclosure relates to methods, devices, systems, and non-transitory computer-readable storage media for distributed data processing application services in a cloud provider network. The distributed data processing application services described herein provide users with a rapidly scalable computing cluster on which to perform computing tasks. Cluster members are hosted in a virtual environment, and the number of members in a cluster can range from a small number to thousands (or more). The embodiments described herein enable the distributed data processing application service to quickly add new members to the cluster.

[0018] Distributed data processing applications allow users to accomplish computationally demanding tasks. To complete such tasks in a reasonable amount of time, such applications employ a "divide and conquer" approach: breaking down tasks into smaller chunks of work and distributing those chunks across a cluster of computer systems. Distributed data processing applications (or distributed applications) include analytical and database applications such as Spark, Hadoop, and Presto (to name a few). Existing cloud-based distributed application services typically require someone to manage the clusters on which the applications run. Such management activities are often the responsibility of information technology administrators or application developers, and include increasing and decreasing server capacity, tuning hidden settings, and managing software updates. Even experienced administrators or users may find it challenging to properly tune a cluster in advance due to unpredictable factors such as application complexity, application data models, and fluctuations in the usage of machines within the cluster depending on the time of day or day of the week. As a result, cluster management often requires a trade-off between either over-provisioning resources to increase cluster responsiveness at an increased cost, or spinning up clusters on demand to reduce costs but suffer delays in cluster responsiveness.

[0019] One of the factors that impacts cluster responsiveness to latency associated with on-demand management policies is the need for cluster members to communicate with each other. Existing techniques for creating and maintaining cluster networks within a cloud provider network have latency costs. Such techniques involve using the provider network's control plane for operations, including creating or modifying dedicated virtual private clouds, subnets, and security groups. The total latency using such techniques can be significant, often reaching 8 to 10 seconds or more. Therefore, for a task that originally ran across 10 members with a one-minute runtime on an existing cluster, if adding each member takes 8 to 10 seconds, the total runtime on the newly formed cluster will more than double.

[0020] Additionally, many of the tasks performed by distributed data processing applications are temporary. That is, the cluster can complete these tasks in a short period of time - often within a minute or two. Clusters are often used to support many such tasks, but there can often be significant idle time between submissions. Therefore, the overall cost of maintaining a cluster is high relative to the time the cluster is in use.

[0021] The distributed data processing application service disclosed herein eliminates many of these tradeoffs by allowing users to cost-effectively run distributed data processing applications without the hassle of managing server capacity, tuning hidden settings, managing software updates, or any other chores associated with managing cluster infrastructure. Specifically, the architecture and methods of the distributed data processing application service described herein utilize multiple techniques to allow cluster members to be quickly started and to create isolated networks for cluster members to communicate, while also scaling to allow a large number of client requests to be handled (e.g., thousands of networks formed per second). The distributed data processing application service largely hides the creation and configuration of task clusters from users, making it easy for data analysts and application developers to run interactive, large-scale distributed data processing applications without having to plan, set up, tune, and manage clusters.

[0022] One aspect of the present disclosure that facilitates rapid creation of clusters is that there is no need to maintain global state in the startup flow for adding new members to a cluster. Calls between entities in the startup path are generally local and unidirectional, because the issuing entity does not need to check any global state of the cluster or other clusters to which the new member is being added, and further does not wait for the recipient to confirm the successful completion of the requested operation. Doing so avoids delays such as those that occur when there is contention for global state resources between multiple entities.

[0023] Another aspect of the present disclosure that facilitates rapid cluster creation is a "pre-warmed" virtual machine bundle for a particular data processing application (also referred to as a data processing tool). As described in more detail below, cluster members are hosted in a virtualized environment (e.g., distributed data processing application software is executed by a virtual machine). A virtual machine bundle refers to a grouping of software, virtual machine configuration, and virtual machine state by which a virtual machine is started to host a cluster member. To pre-warm the bundle, the virtual machine is started, including the beginning stages of execution of a distributed data processing tool runtime (e.g., Spark, Hadoop, Presto, etc.), until some time before the client computation begins. The virtual machine is paused in this penultimate state, and a copy of the virtual machine's memory and processor registers is included in the bundle. Future virtual machines started to host members of a computing cluster will be resumed using this bundle, thereby eliminating a significant portion of the startup time that would otherwise exist if the virtual machine and associated data processing tool runtime were started without a pre-warmed bundle.

[0024] Another aspect of the present disclosure that facilitates rapid cluster creation is to use the 128-bit address space of IPv6 (Internet Protocol version 6) to create a session network for secure cluster communication. Distributed data processing application services use this space to embed information in addresses that can be used to isolate individual networks supporting different clusters from each other. Specifically, the distributed data processing application service uses a portion of the address space to embed a unique network identifier (also referred to as a session network identifier) ​​that uniquely identifies the cluster and permits services restricted to each cluster. The networking technology described herein permits cluster members to be added to or removed from a session network. The session network identifier determines a portion of the IPv6 address that will be assigned to each cluster member. Such computer systems are also referred to as "bare metal" or simply "metal" systems. This delegation allows the metal system to assign IPv6 addresses to cluster members and configure access to the cluster network without updating the networking configuration data of other existing members, and typically does not involve the cloud provider network control plane, thereby reducing potential bottlenecks in network configuration.

[0025] Figure 11 is a diagram illustrating a cloud provider network environment including a distributed data processing application service that enables rapid creation of computing clusters according to some embodiments. Components 101 of the distributed data processing application service are hosted within a provider network 100. Provider network 100 (or "cloud" provider network) provides users with the ability to use one or more of multiple types of computing-related resources. For example, a cloud provider network (or just "cloud") generally refers to a large pool of accessible virtualized computing resources (such as computing, storage and networking resources, applications and services). Exemplary computing-related resources include computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including multiple groups of computing resources), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers and other data processing resources, etc. These and other computing resources can be provided as services, such as hardware virtualization services that can execute computing instances, storage services that can store data objects, etc. The provider network 100 can span multiple geographic regions, and in some cases can even control or provide access to resources outside the provider network (such as edge locations, on-site locations, etc.).

[0026] To provide these and other computing resource services, the provider network 100 often relies on virtualization technology. For example, virtualization technology can provide users with control or use of computing resources (e.g., "computing instances," such as VMs using guest operating systems (O / Ss) that operate using a hypervisor that may or may not further operate on top of an underlying host O / S; containers that may or may not operate in VMs; computing instances that can be executed on "bare metal" hardware without an underlying hypervisor), where a single electronic device can be used to implement one or more computing resources. As part of "launching" a virtualized resource (such as a computing instance or virtual machine), the underlying host computer system allocates some portion of its memory and computing resources to that virtualized resource. Thus, users can directly use computing resources hosted by a provider network (e.g., provided by a hardware virtualization service) to perform a variety of computing tasks. Additionally, or alternatively, users can indirectly use computing resources by submitting code to be executed by a provider network (e.g., via an on-demand code execution service), which in turn uses one or more computing resources to execute the code, typically without the user having any control or knowledge of the underlying computing instances involved.

[0027] Users (or "customers") of the provider network 100 may use one or more user accounts associated with a customer account, although these terms may be used somewhat interchangeably depending on the context of use. Users may interact with the provider network 100 across one or more intermediate networks (e.g., the Internet) via one or more interfaces 104, such as by using application programming interface (API) calls, via a console implemented as a website or application, and the like. In this way, the cloud can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to variable loads. Therefore, cloud computing can be viewed as both applications delivered as services over a publicly accessible network (e.g., the Internet, a cellular communications network) and the hardware and software in a cloud provider data center that provides those services.

[0028] An API refers to an interface and / or communication protocol between a client and a server such that if a client issues a request in a predefined format, the client will receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for clients to access cloud infrastructure by allowing them to obtain data from a cloud provider network or cause actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted in the cloud provider network. An API may also enable different services of a cloud provider network to exchange data with each other. An interface 104 may be part of or act as a front end to a control plane 102 of a provider network 100, which includes "backend" services that support and implement services that may be provided more directly to clients.

[0029] In the case of a distributed data processing application service, interface 104 provides a set of client-facing APIs to facilitate submitting computations to the distributed data processing application service, and implicitly or explicitly creating sessions to perform computations while isolating the distributed data processing application service from external networks (e.g., the Internet). As used herein, a "session" is a logical entity of a distributed data processing application service that can be used to track a cluster of one or more executors (or more generally, members) and client-submitted computations executed by the cluster. Clients submit "computations" for session processing. In this context, a computation refers to client code (e.g., Spark Scala code, SQL queries, etc.) that will be executed by the cluster.

[0030] The Distributed Data Processing Application Service can provide several types of sessions, including interactive sessions, standalone sessions, and custom sessions. Interactive sessions are targeted at notebook use cases and promote the stateful nature of such sessions. Standalone sessions do not require explicit lifecycle management by the customer. They are automatically created for a single computation submission and are stopped once the computation reaches its logical conclusion. Custom sessions are created explicitly using start and stop commands issued by the customer to support use cases such as the customer's own long-lived applications.

[0031] Executor (or member of a cluster) is a separate computing unit that can be added to and / or removed from a cluster. Executor is implemented using virtualization technology such as a virtual machine, and includes an application runtime for supported specific distributed data processing applications (e.g., Spark, Presto, etc.). In some embodiments, executor is implemented as a lightweight "microVM" running specific software and having a specific configuration. As used herein, microVM generally refers to the following VM: initialized to have a limited device model and / or a minimum OS kernel supported by a lightweight virtual machine manager (VMM), and each microVM can have a low memory overhead of <5MiB, so that thousands of microVMs can be packaged on a single host. For example, microVM can have a streamlined version of the OS kernel (e.g., only having the required OS components and their dependent libraries) to minimize startup time and memory footprint. In one implementation, each process of a lightweight VMM encapsulates one and only one microVM. The process can run the following threads: API, VMM, and vCPU. The API thread is responsible for the API server and the associated control plane. The VMM thread exposes the machine model, the minimal legacy device model, the microVM metadata service (MMDS), and the VirtIO device emulation network and block devices. In addition, there are one or more vCPU threads (one vCPU thread per guest CPU core).

[0032] The subsequent description of distributed data processing application services generally refers to virtual machines, but other virtualization technologies may be used, such as microVMs, containers, or other technologies. In addition, the size of the executor may vary in multiples of data processing units ("DPUs"). For example, one DPU may represent 4 virtual CPUs and 16 gigabytes of memory.

[0033] A special type of executor is a "session coordinator". A session coordinator is an executor responsible for distributing the work associated with a task across the cluster. Using Spark as an example, the session coordinator executor would correspond to the Spark driver. The session coordinator is responsible for adding / removing other executors to the session, as well as executing computations submitted to it via the distributed data processing application service.

[0034] Component 101 of the distributed data processing application service includes a session manager service 110, a network manager service 112, and a resource manager service 114, which work together to form clusters in response to user requests via interface 104, and automatically scale clusters up and down to facilitate the execution of distributed data processing applications. The cluster associated with a given session operates within an isolated network that allows only executors (also known as peers) participating in the same session to communicate freely, and prevents communication with entities that are not part of the session. This isolated network is called a "session network."

[0035] Executors (e.g., cluster members) that are part of a session are hosted by computer systems 150 selected from a warm pool of assets of a distributed data processing application service that is ready to initiate the session. For example, the warm pool may include computer systems 150-1 through 150-M (also referred to as assets or metal), where M represents the number of assets in the pool (e.g., hundreds, thousands, or more). A capacity manager service (not shown) can dynamically adjust M by requesting additional assets from a hardware virtualization service (when demand is high) or releasing assets from the warm pool to a hardware virtualization service (when demand is low). Figure 7 Additional details regarding computer system 150 are described.

[0036] In general, session manager service 110, network manager service 112, and resource manager service 114 may be viewed as "microservices" that together constitute a distributed data processing application service. Each of these microservices may itself be a distributed service. That is, each distributed microservice may be composed of multiple components that can be accessed behind a load balancer that distributes requests for the microservice. In order to quickly create a session network, the dependencies of these microservices on other services and, in the distributed case, the internal dependencies between entities of the microservices are minimized, as described herein.

[0037] The network manager service 112 issues a session network identifier while ensuring that there are never two active sessions with the same identifier. The session network identifier is used to obtain the IPv6 address of the virtual machine supporting the executor. By using the session network identifier, a separate session network can be easily (quickly) created to isolate traffic between executors in a particular cluster using associated firewall rules.

[0038] The session manager service 110 acts as a coordinator for the lifecycle of sessions, computations, and executors. In some embodiments, the session manager service 110 tracks these entities and their relationships (e.g., which executors are part of a given session) in session metadata stored in the data store 113. To avoid these tracking operations interfering with the speed at which the cluster can be started and resized, the fine-grained orchestration of sessions (e.g., adding and removing executors) is delegated to the application runtime. When the application runtime executing on the session coordinator wants to scale up or down a session, the session manager service 110 acts as a bridge to the resource manager service 114, as described below.

[0039] The resource manager service 114 extracts a warm pool of assets (e.g., 150-1 through 150-M) from the requestor of the virtual machine. The resource manager service 114 makes a placement decision—that is, it selects which asset in the warm pool to attempt to launch a new virtual machine to host the requested executor. In some embodiments, the resource manager service 114 is implemented as a fully in-memory architecture with minimal external dependencies.

[0040] In some embodiments, the resource manager service 114 is part of a hardware virtualization service. Such hardware virtualization services (referred to as elastic computing services, virtual machine services, computing cloud services, computing engines, or cloud computing services in various implementations) can enable users of the provider network 100 to provision and manage computing resources, such as virtual machine instances. Virtual machine technology can use one physical server, for example, to run the equivalent of many servers (each of which is referred to as a virtual machine) using a hypervisor, which can run at least on an offload card of the server (e.g., a card connected to a physical CPU via PCI or PCIe), and other components of the virtualization host can be used for some virtualization management components. Such an offload card of the host may include one or more CPUs that are not available for customer instances, but are dedicated to instance management tasks, such as virtual machine management (e.g., hypervisor), input / output virtualization to network attached storage volumes, local migration management tasks, instance health monitoring, etc.). Virtual machines are generally referred to as computing instances or simply "instances." As used herein, provisioning a virtual computing instance generally includes: reserving the resources (e.g., computing and memory resources) of an underlying physical computing instance for a client (e.g., from a pool of available physical computing instances and other resources); installing or starting required software (e.g., an operating system); and making the virtual computing instance available to the client to perform tasks specified by the client.

[0041] The service programming interface 116 issues an API that application runtime developers (e.g., Spark or other distributed data processing application developers) will use to integrate with the functionality provided by the distributed data processing application service. Just as the interface 104 isolates the distributed data processing application service from external networks, the service programming interface 116 isolates the session manager service 110 (and other non-customer-specific components of the distributed data processing application service) from the session network when third-party software is executing in those environments. Exemplary API requests issued by the service programming interface 116 include requests to register a newly launched executor with a session, increase or decrease the number of executors in the session network, obtain recommended settings for a given logical plan from a recommendation service (not shown), and execute a new computation (e.g., for an interactive session).

[0042] The high-level flow of the startup of the first executor or session coordinator of a session is described with reference to the circled numbers 1 to 9. At circle 1, the network manager service 112 prepares the session network identifier space. As mentioned above, the session network identifier defines how the session network of the session is configured. Specifically, the session network identifier is used to form a portion of the IPv6 address that is later assigned to the virtual machine hosting the executor. For example, a 52-bit session network identifier allows over a quadrillion combinations. In order to efficiently allocate session network identifiers while avoiding conflicts, the network manager service 112 can divide the space into smaller groups of identifiers. For example, the network manager service 112 can divide the combinations into multiple groups of 1,024 session network identifiers. When a new session is created, these groups can be assigned to the session manager service 110. In addition, in order to provide a certain degree of obfuscation of the assignment of session network identifiers (to delay potential attackers from compromising the session network), the network manager service 112 can shuffle the session network identifiers before or as part of dividing the session network identifiers into multiple groups (also referred to as session network identifier blocks).

[0043] Because the number of blocks of session network identifiers may be large and the session manager service 110 may be a distributed service that issues multiple requests for blocks, the network manager service 112 may also generate a data structure to track which blocks have been allocated. The network manager service 112 may use this block tracking metadata to ensure that it does not provide the same block twice.

[0044] The network manager service 112 may store the prepared chunks and tracking data (also referred to as chunk metadata) in a data store 113, which may be provided by another service of the provider network. Figure 4 Additional details regarding the shuffle and chunk operations and chunk tracking operations of the network manager service 112 are described.

[0045] By preparing blocks of session network identifiers and distributing those blocks to the session manager service 110 (or, if distributed, to each distributed session manager service component), the network manager service 112 does not need to be part of the startup flow path of new executors. Thus, startup time for executors is reduced.

[0046] At circle 2, session manager service 110 issues a request to allocate one or more session network identifier blocks to network manager service 112. Network manager service 112 returns session network identifier blocks previously prepared by network manager service 110. In addition, network manager service 112 marks the allocated session network identifier blocks as used in the block tracking metadata to prevent the session network identifier blocks from being reused in later allocation requests from session manager service 110.

[0047] At this stage, the session manager service 110 has one or more allocated blocks of session network identifiers that it can use when creating clusters in response to client requests and when servicing client computations. The session manager service 110 can store the allocated blocks of session network identifiers in local memory (not shown) so that when a request to create a session or run a computation is received, the session manager service 110 can perform operations locally to select an unused session network identifier without issuing an external network request. In this way, network latency can be eliminated from this portion of the cluster member startup path.

[0048] At circle 3, a client submits a request to a distributed data processing application service via an electronic device 108 and interface 104. Various types of requests are supported using interface 104 exposed to one or more intermediate networks between the electronic device 108 and provider network 100, effectively proxying the request to session manager service 110. For example, a client may request to create a session. As another example, a client may request to perform a computation (which may implicitly result in a session being created). Figure 5 Describes additional details about the requests made by these initial clients.

[0049] In some embodiments, upon receiving a request to run a computation or create a session, the session manager service 110 first checks with an admission control service (not shown) to verify whether the request should be allowed to start or denied. The admission control service can make this determination based on a variety of factors, including system capacity, customer account limits, etc. Exemplary customer account limits include limits on the rate and concurrency of sessions, computations, executors, DPUs, etc. on a customer account.

[0050] At circle 4, the session manager service 110 creates a new session by assigning a session identifier to the session. As indicated above, a session is a logical entity that can be used to track a cluster. In addition, the session manager service 110 selects a session network identifier from the block of session network identifiers it previously obtained at circle 2. At this point, the session manager service 110 can update the session metadata 190 to associate the assigned session identifier with the selected session network identifier. In this example, the session identifier is XYZ and the session network identifier is 123456789ABCD. The exemplary session metadata 190 can also include, for example, the identification of each executor that is part of the session (if any) and various configuration settings. Exemplary configuration settings include software configuration settings (such as the identification of the application runtime (e.g., Spark) executed by the cluster) and hardware configuration settings for members that can be launched into the cluster (e.g., the number of virtual CPUs and the amount of memory to be allocated to each virtual machine).

[0051] At circle 5, the session manager service 110 sends a request including the selected session network identifier to the resource manager service 114 to launch a new virtual machine that will become the session coordinator (first executor) in the session. At circle 6, the resource manager service 114 evaluates its inventory of assets in the warm pool and selects an asset that is believed to have the capacity to host the requested virtual machine. At circle 7, the resource manager service 114 sends a request including the session network identifier to the agent 160 running on the selected asset 150 to launch the new virtual machine. In this example, the resource manager service 114 selects computer system 150-1. Assuming that the asset has the capacity to host the requested executor, the agent launches and configures executor 170-1, in this case the session coordinator, as indicated at circle 8. Refer below to Figure 7 Provides additional details about the startup and configuration of the executor. Now, it is worth noting that the agent 160 forms the IPv6 address assigned to the executor by combining the host address of computer system 150-1, the session network identifier, and the local network identifier. In addition, the agent configures a firewall between the executor and the outside world to permit communication with all other executors (if any) that share the session network identifier. Figure 3 , Figure 4 and Figure 7 Provide additional details describing the form and formation of this address.

[0052] In some embodiments, the agent 160 may request customer credentials from an authentication service (not shown) prior to launching the virtual machine. The credentials typically include a unique identifier associated with a secret access key, which when used together can be used to cryptographically sign programmatic requests to services such as the session manager service 110. The agent 160 may load the customer credentials into the virtual machine so that the software executed by the virtual machine can access the customer credentials. For example, when making a request to a service of the provider network to perform an operation on behalf of the customer (such as adding an additional actuator to a session), the software executed by the virtual machine may include the customer credentials.

[0053] At circle 9, the now up and running session coordinator 170-1 registers itself with the session manager service 110 via the service programming interface 116. The session manager service 110 may then update the session metadata 190 to associate the newly started session coordinator with the session. In some embodiments, actuators are identified in the session metadata 190 by network address and type. For example, the session coordinator 170-1 may be added to the session XYZ based on its network address (described in more detail below) and with an indication that it is a session coordinator (rather than a generic actuator).

[0054] Data store 113 is intended to represent a variety of data store types and locations. Session manager service 110 and network manager service 112 may each use the same or different data stores, local or remote stores, stores provided via another store or database service of a provider network, and the like.

[0055] In some embodiments, the control plane 102 also includes an executor proxy service (not shown). The executor proxy service effectively provides a path by which the session manager service 110 or other distributed data processing application service control plane 102 components can issue commands directly to the executor. For example, for an interactive session type, a client can submit a series of calculations to the session manager service 110 via the interface 104, and the session manager service 110 can relay those calculations to the session coordinator via the executor proxy service.

[0056] The session coordinator will typically enlist the help of attached executors to perform the work required for the computations performed by the session. Figure 2 Describes adding an executor to a session network.

[0057] Figure 2 is a diagram illustrating a method according to some embodiments Figure 1 Another diagram of a distributed data processing application service starting a new member of a compute cluster. Figure 1Continuing where the description of 170 was interrupted, at circle 1, the application runtime of session coordinator 170-1 has determined to distribute a portion of the computation to another executor. Because no other executors have been added to the session, session coordinator 170-1 issues a request to session manager service 110 via service programming interface 116 to add a new executor.

[0058] In some embodiments, the distributed data processing application service includes a recommendation service (not shown) that can provide recommended virtual machine settings for an application to the session coordinator (or user, if requested via interface 104). For example, the recommendation service can be able to understand the logical plan (e.g., Spark plan) associated with the application runtime and / or evaluate the historical computational performance of comparable (abstracted) computations to recommend how many executors to use at each stage of the computation or what size executors to use for the computation. If such a recommendation service is available, the application developer can configure the session coordinator 170-1 to obtain recommended hardware configurations from the recommendation service to include in requests to add new executors.

[0059] After receiving the request to add an executor from session coordinator 170-1, session manager service 110 may issue a request to resource manager service 114 to start a new executor, as indicated at circle 2. Figure 2 At circles 3 and 4, follow the same steps as reference Figure 1 In this case, the resource manager service 114 selects the request to start a new virtual machine and sends the request to the agent 160 running on the selected asset 150-4. At circle 5, the agent 160 of the computer system 150-4 starts and configures the executor 170-2. Figure 7 Additional details are provided regarding the startup and configuration of the executor. The agent 160 of the computer system 150-4 forms an IPv6 address assigned to the executor by combining the host address of the computer system 150-4, the session network identifier, and the local network identifier. In addition, the agent 160 of the computer system 150-4 configures a firewall between the executor and the outside world to permit communication with all other executors (if any) of the same shared session network identifier. Due to the independent address formation and addressing scheme described herein, no modification to the configuration of the session coordinator 170-1 is required to enable communication with the newly started executor 170-2.

[0060] At circle 6, the now up and running executor 170-2 registers itself with the session manager service 110 via the service programming interface 116, such as reference Figure 1 As described by circle 9.

[0061] Figure 3 An exemplary network configuration of two computing clusters launched within a cloud provider network environment according to some embodiments is depicted. The IPv6 protocol uses a 128-bit address space, which supports an additional approximately 7x10^38 addresses in addition to the approximately 4 billion addresses of IPv4 compared to the 32-bit address space of IPv4. The additional space greatly reduces the probability of two networks with conflicting IP ranges affecting each other. In fact, the IPv6 specification has a mechanism to completely avoid the possibility of conflicts by assigning subnets from the Internet registry if all parties comply.

[0062] The 128-bit address space is represented as 8 groups of 16 bits each, usually written in hexadecimal form. (In contrast, the IPv4 space is represented by 4 groups of 8 bits each.) The address space is divided into two halves. The first (leftmost) 64 bits are the routing prefix or network identifier, which is 64 bits for the smallest single IPv6 subnet. The second (rightmost) 64 bits are the interface identifier.

[0063] The IPv6 address space is divided using prefixes, in a format such as "fe80:: / 10", where the leading address identifies a fixed number in the prefix, and the number following the slash ( / ) identifies how many of the leftmost bits of the 128-bit address space are fixed. The network / routing portion of the IPv6 address range is large enough and can be divided into approximately 1.8x10^19 " / 64" subnets. This means that ownership of sufficiently large and unique prefixes can be easily obtained (compared to IPv4, which only had a total of about 4 billion addresses).

[0064] A provider network 300 typically has a large address space assigned to it (e.g., one or more / 56 or / 64 network prefixes), which can be divided among components of the provider network. Figure 1 and Figure 2The host computer systems (e.g., computer system 150) in the warm pool shown are assigned one or more / 68 subnet prefixes. Here, computer system 350-1 has been assigned two subnet prefixes: 123:4567:89AB:CDEF:1000::: and 123:4567:89AB:CDEF:0000::: (note that, by convention, leading zeros are often omitted), computer system 350-2 has been assigned one subnet prefix 123:4567:89AB:4567:1000:::, and computer system 350-3 has been assigned one subnet prefix 123:4567:89AB:0123:0000:::. Computer system 350 - 1 hosts three executors 370 - 1 , 370 - 2 , and 380 - 2 , computer system 350 - 2 hosts one executor 380 - 3 , and computer system 350 - 3 hosts two executors 370 - 3 and 380 - 1 .

[0065] As shown, three executors 370 form session network 372, and three executors 380 form session network 382, ​​all within provider network 300. In this example, the 128-bit executor network address is formed by combining the 68-bit subnet prefix assigned to the host computer system, the 52-bit session network identifier, and the 8-bit local network identifier that uniquely identifies executors hosted on the same computer system that are part of the same session network and share the same subnet prefix. In this way, the session network is identified by the middle portion of the IPv6 address—session network 372 is identified by the 123:4567:89AB:CD portion of the IP address of a member of a cluster in session network 372, and session network 382 is identified by the 123:4567:89AB:CE portion of the IP address of a member of a cluster in session network 382.

[0066] The differences and similarities between the actuator network addresses are Figure 3. For example, executors 370-1 and 370-2, hosted on the same computer system 350-1, share the same prefix (0123:4567:89AB:CDEF:0), and the same portion of the IPv6 address specified by the session network identifier (123:4567:89AB:CD), while having different local network identifiers (00 versus 01) to distinguish the executors because they are located within the same subnet prefix on the same host computer system. As another example, executors 380-1 and 380-2, hosted on computer systems 350-3 and 350-1, respectively, share the portion of their IPv6 addresses specified by the session network identifier (123:4567:89AB:CE), while having different subnet prefixes and local network identifiers. The local network identifiers of executors in the same session hosted by different computer systems can be the same or different; the local network identifier prevents ambiguity between executors in the same session on the same host computer system and shares the same subnet prefix assigned to the host computer system.

[0067] Use something like Figure 3 With the addressing scheme shown, the launch of the configured virtual machine and the configured agent can make only local firewall policy updates in constant time without coordination with other computer systems (if any) hosting actuators that are part of the same session. Moreover, those local updates actually add actuators to and remove actuators from the session network by permitting traffic to other members of the session network. Additionally, the large size of the session network identifier space avoids reuse (subject to certain low-frequency changes described herein) and improves traceability of events that occur on the provider network, such as the possibility that malicious or buggy software executed by an actuator may attempt to violate a session network other than the session network in which the actuator was launched.

[0068] Intrusion detector 390 can monitor and aggregate traffic flow logs. Such flow logs can include packet header information, such as the source and destination of each packet traversing a session network such as session networks 372 and 382. Intrusion detector 390 can analyze the flow logs to find patterns that indicate unwanted behavior of an actuator. For example, various thresholds for pattern descriptions can be set empirically or using machine learning algorithms. One such pattern would be an attempt to communicate from a virtual machine instance within one session network to another session network. A single packet traversing from one session network to another session network may indicate an intruder, as sessions should not generally communicate. In response to detecting a potential intruder, intrusion detector 390 can trigger an intrusion response. Assuming that the asset is not compromised, an exemplary intrusion response includes issuing an actuator termination request to session manager service 112 or to agent 160 (if a remote kill switch is implemented). If the asset is potentially compromised, the intrusion response can include, for example, terminating the asset itself via resource manager service 114.

[0069] Figure 4 An exemplary network addressing scheme used by distributed data processing application services to form a cluster according to some embodiments is depicted. As described above, a 128-bit IPv6 address 400 is represented as 8 groups of 16 bits, each group typically written in hexadecimal form. The address space is divided into two halves. The first (leftmost) 64 bits are a routing prefix or network identifier 402, and the second (rightmost) 64 bits are an interface identifier 404.

[0070] Generally speaking, an IPv6 address can be divided into three (or more) parts for use by distributed data processing application services. The three parts include an M-bit subnet prefix, an N-bit session network identifier, and an O-bit local network identifier, where M+N+O=128. In some embodiments, M+N+O can be less than 128, where one or more bits of the IPv6 address space are reserved for other purposes. In these cases, the entire IPv6 address of the executor is still formed by combining the subnet prefix, the session network identifier, and the local network identifier, as well as any bits used for other purposes.

[0071] exist Figure 4 In the embodiment illustrated in FIG. 4 , one or more / 68 subnet prefixes 406 are assigned to a host computer system, such as computer system 150. The rightmost 8 bits represent a local network identifier 410 that distinguishes executors hosted by the same computer system for the same session network and sharing the same subnet prefix. The remaining middle portion of the 52 bits creates a session network identifier space 412.

[0072] A 52-bit session network identifier space can support more than a quadrillion session network identifiers 418. If collisions are avoided and session network identifiers are used up, this would be equivalent to having exhausted the number of sessions that can run on the provider network before reusing from all potential session network identifiers. If the provider network averages 5 million new jobs per day, it would take millions of years to run out of combinations.

[0073] In some embodiments, the network manager service 112 prepares the session network identifier space by optionally shuffling and chunking 419 the session network identifiers within the space prior to use by the distributed data processing application service. Shuffling the session network identifiers serves several purposes. Randomizing the order in which the session network identifiers are consumed can improve collision avoidance (assuming other mechanisms may fail) and obfuscate the launch rate. Additionally, the randomness increases the time an intrusion detector (e.g., intrusion detector 390) has to detect a possible intrusion by increasing the time required for a potential intruder to identify a vulnerable session. Chunking permits blocks of session network identifiers to be assigned to the session manager service 110 prior to launching a session, thereby removing the network manager service 112 from the session / executor launch path.

[0074] As illustrated, in one embodiment, the network manager service 112 shuffles the 52-bit session network identifier space 418 and then divides the shuffled space into blocks of 1,024 session network identifiers each, resulting in 2^42 blocks 420.

[0075] Another example of shuffling and chunking according to some embodiments is as follows. To reduce the amount of shuffling computation, P bits may be set aside to define chronological partitions. When the supply of session network identifiers in the remaining (52-P) bits becomes low or exhausted, the chronological partitions may be slowly passed (e.g., from partition 0x00000 to partition 0x00001 if not shuffled). Values ​​from 0 to 2^P may be shuffled to create a sequence of chronological partitions.

[0076] Various techniques may be used to shuffle and chunk the remaining 52-P-bit session address space. In this example, assuming that a first shuffle and chunk operation on a first chronological partition is performed before enabling a distributed data processing application service, a subsequent shuffle and chunk operation on the next 52-P-bit session address space of the next chronological partition may be performed at some time before advancing to the next partition. Assuming a 20-bit chronological partition, the possible session network identifiers in the remaining 32 bits may be shuffled. The shuffled identifiers may be combined with the chronologically partitioned bits and then divided into groups or "chunks" for assignment to session manager services. An exemplary block size is 1,024. After 1,024 52-bit session network identifiers have been accumulated, the resulting "chunk" is written to a storage area and assigned a chunk identifier.

[0077] In order to track which blocks have been allocated in response to requests from the session manager service 110, the network manager service 112 may store metadata to indicate which blocks have been allocated. In some embodiments, the method for representing block allocations in metadata will allow blocks to be allocated in parallel without contention (e.g., when the network manager service 112 is executed across multiple threads or systems). One such method uses a binary tree to track block entries in the metadata 422 as follows. The blocks are divided into groups of 1,024. A binary tree is created on the group. Each node in the tree will contain metadata indicating whether any / both of the child nodes have been fully allocated. Any number of machines / threads can traverse the tree at startup by randomly walking through the tree to locate leaves with unallocated blocks. Once selected, the thread can scan the group of 1,024 blocks until the next unallocated block is found and start allocating from that position. If a machine / thread detects repeated allocation contention (e.g., conditional update failure), it can randomly back off on the tree until a node with less contention is found. If a group is found to be fully allocated, the node above can be marked so that future traversals skip the subtree / group. Once marked, the machine / thread will randomly walk the tree to find another group.

[0078] Figure 5 Depicted is a method for initiating a Figure 1 An exemplary call flow for the first member of the cluster is depicted in FIG. Entities involved in the launch path include session manager service 110, resource manager service 114, and agent 160 of computer system 150-1.

[0079] The client-oriented API can take many forms. Two examples are described herein. In the first example, the client-oriented API allows explicit creation of a session (call 501, operation 503, and call 505). Here, the session manager service 110 receives a create session call 501 (e.g., from the electronic device 108). The call 501 includes a session type (SESSION_TYPE), a runtime identifier (RUNTIME_ID), and a virtual machine configuration (VM_CONFIG). As described above, the session types can be divided into persistent sessions, manual sessions, independent sessions, and interactive sessions. The runtime identifier identifies the data processing tools supported by the cluster. The virtual machine configuration may include the default hardware configuration of the virtual machine that is started to support the members of the cluster, such as the number of virtual processors (or other processing units) and the amount of memory allocated to the virtual machine from the host computer system. After receiving the create session call 501, the session manager service 110 performs operations 503 associated with creating a new session. Such operations include generating a new session identifier, selecting an unused session network identifier for the session, and updating session metadata to track the session, the associated session network identifier, and the configuration data received with the call 501. The session manager service 110 may return a new session identifier to the caller.

[0080] At some later time, the session manager service 110 receives a submit computation call 505 (eg, from the electronic device 108). The call 505 includes a session identifier and a computation (CALCULATION), which may include or identify the location of the client code to be executed by the cluster.

[0081] Returning to the second example of a client-facing API, a session may also be implicitly created (call 509, operation 511). Here, the session manager service 110 receives a create submit compute call 509 (e.g., from the electronic device 108). The call 501 includes a runtime identifier (RUNTIME_ID), a virtual machine configuration (VM_CONFIG), and a calculation (CALCULATION). Although these parameters are similar to those described above, this call 501 implicitly creates a session. Upon receiving the submit compute call 509, the session manager service 110 performs operations 511 associated with creating a new session, which are similar to those described above for operation 503, with the primary difference being that the session identifier is not communicated to the submit compute caller.

[0082] In either of the above two examples, the session manager service 110 receives a computation to be performed on a cluster that has zero members to date. To start the first member of the cluster, the session manager service 110 issues a create virtual machine call 515 to the resource manager service 114. In the illustrated example, the call 515 includes a session identifier (SESSION_ID), an executor identifier (EXECUTOR_ID), a session network identifier (SESSION_NETWORK_ID), a virtual machine configuration (VM_CONFIG), and a bundle identifier (BUNDLE_ID). The session identifier is a session identifier previously assigned to the session by the session manager service 110. The executor identifier can be uniquely assigned to an outbound request for accounting purposes, which will become apparent later after the session manager service 110 learns of the successful start of a cluster member. The session network identifier corresponds to the session network identifier selected by the session manager service 110 when creating the session. The virtual machine configuration represents the hardware configuration of the virtual machine to be started, and can be or be based on the virtual machine configuration received in the call 501 or 509. The bundle identifier identifies the pre-warmed bundle with the correct data processing tool for the virtual machine (determined by the previously received runtime identifier).

[0083] Upon receiving call 515, resource manager service 114 performs operation 517. Operation 517 includes evaluating the available capacity of assets in the distributed data processing application service warm pool on which the requested virtual machine can be launched. This includes identifying which assets in the warm pool have the hardware capacity to host the virtual machine and the appropriate bundle for launching the virtual machine. Once one or more assets that can satisfy the request are identified, resource manager service 114 selects one of the identified assets and issues a create virtual machine call 519 to an agent on the selected asset. In this example, call 519 is issued to agent 160 on computer system 150-1 (associated with Figure 1 ). The parameters of call 519 largely mimic the parameters of call 515. In this embodiment, call 519 also includes a local network identifier. Resource manager service 114 may randomly select a local network identifier (a portion of the session network address that prevents conflicts between two cluster members hosted by the same computer system; see Figure 3 ).

[0084] In some embodiments, after receiving the call 519, the agent 160 can reject the request. The rejection may occur for several reasons. First, in some embodiments, scaling can limit the resource manager service 114 to the final consistent view of the warm pool because the various components of the distributed resource manager service do not maintain the "global" state of the assets on which the virtual machine can be started. Therefore, the resource manager service may inadvertently select a computer system that does not have enough resources to host a virtual machine of the requested size. For example, the resource manager service 114 itself can be a distributed service, and another component of the distributed service may have requested the capacity of the host computer system that the agent has consumed, so that the request in the call 519 cannot be satisfied. In this case, the agent 160 can reject the request, causing the resource manager service 114 to repeat the identification and selection operation 517 and issue a request to different agents on different computer systems. Another reason for rejecting the request is related to the local network identifier. In order to limit the complexity of the resource manager service 114 (and therefore increase the speed), the resource manager service 114 can randomly select a local network identifier. If a local network identifier that is used by a member of a cluster hosted on the selected computer system happens to be selected, the agent can reject the request. In this case, the resource manager service 114 may randomly select a different local network identifier and resubmit the call to the same proxy that rejected the previous request, or, such as due to insufficient capacity, identify and select a different computer system to which to direct the new request to launch the virtual machine.

[0085] Assuming the proxy 160 does not reject the call 519 (e.g., returns or "throws" an error), the proxy initiates the launch of the virtual machine, as indicated at 521. In this case, the virtual machine hosts the session coordinator 170-1. Once the session coordinator 170-1 is launched or during the launch of the session coordinator 170-1, the session coordinator 170-1 can issue a register executor call 523 to the session manager service 110. The call 523 includes an executor identifier that notifies the session manager service 110 that the request initiated at call 515 was successful.

[0086] In this way, the agent 160 is authoritative for operations performed on the underlying computer system. Therefore, any "global" state of the distributed data processing application service actually varies with many local states of the computer system controlled by the agent 160, rather than having some global state continuously updated and managed as a shared resource. In addition, the indication of the successful startup of the session coordinator 170-1 does not need to be signaled backward through the previous startup flow (e.g., from the agent 160 back to the resource manager service 114, from the resource manager service 114 back to the session manager service 110), which would introduce unwanted delays. In this way, the calls in the startup flow of the session coordinator 170-1 are generally one-way, and the session manager service 110 is ultimately notified of the successful completion of the call 515 in the call 523.

[0087] After receiving call 523, session manager service 110 performs operation 525. Operation 525 includes updating session metadata to associate the identity of session coordinator 170-1 with the session identifier. For example, session manager service 110 may identify session coordinator 170-1 by the source network address of call 523.

[0088] Now, with the first member added to the cluster for the session, the session manager service 110 issues a submit computation call 527 to the session coordinator 170, which includes the computation previously submitted at call 505 or 509. Call 527 may be issued by the executor proxy service (not shown) mentioned above.

[0089] To facilitate placement decisions by the resource manager service 114, warm pools (e.g., Figure 1 The agents of computer systems in the warm pool (shown in FIG. 1 ) can periodically report their available capacity for hosting instances to the resource manager service 114. For example, the agent 160 of computer system 150-1 issues a report capacity call 530 to the resource manager service 114. The content of the call 530 may include, for example, the amount of free capacity available on computer system 150-1 for launching virtual machines (e.g., in terms of processor and memory capacity, DPU, etc.) and the available bundles that have been downloaded locally to computer system 150-1. Figure 8 and Fig. 9 Provides additional details about the bundle.

[0090] In some embodiments, the distributed data processing application service may allow the user to further customize the cluster. For example, the create session call 501 may include a bundle identifier in addition to or in lieu of a virtual machine configuration. Figure 8As explained in more detail, some bundles may be pre-warmed with customer-specific configurations to further reduce the time to launch cluster members for a particular user. The session manager service 110 may store the specified bundle identifier in the session metadata 190 for that session and send the bundle identifier specified by that user in the create virtual machine call 515. As another example, some users may want other provider network services to be directly addressable on the session network. Such other services may provide other computing resources, data / storage resources, network-related resources, and application resources such as those described above. In these cases, the create session call 501 may further identify other provider network services to connect to the cluster's session network, and the session manager service 110 may store the identities of those other provider network services stored in the session metadata 190 for that session. The session manager service 110 may then send additional networking configuration data in the create virtual machine call 515 to be relayed to the proxy 160 for configuring the launched executor environment to be able to access other provider network resources within the session network, as described in reference to FIG. Figure 7 Describe in more detail.

[0091] Figure 6 Depicted is a method for initiating a Figure 2 An exemplary call flow for another member of the cluster is shown in FIG. Figure 5 Continuing where we left off, at some point during the processing of a computation, the session coordinator 170-1 may determine that it wants to distribute a portion of the computation to another member of the cluster, but no other cluster members exist at this time. The session coordinator 170-1 may request additional members via the session manager service 110. To add a new member, the session coordinator 170-1 issues a create executor call 601 to the session manager service 110. The call 601 may include a session identifier (as described above). The call 601 may also include a virtual machine configuration and a bundle identifier. Alternatively, the session manager service 110 may determine the virtual machine configuration and / or bundle identifier to use when issuing a request for a new virtual machine based on configuration data associated with the session identifier in the session metadata. The call 601 may also include client credentials (obtained from an authentication service and loaded into the virtual machine environment executing the session coordinator 170-1).

[0092] At this point, the start flow mimics the previous reference Figure 5170-2. Upon startup or during startup of executor coordinator 170-2, executor 170-2 may issue a register executor call 623 to session manager service 110. Call 623 includes an executor identifier that notifies session manager service 110 that the request initiated at call 615 was successful.

[0093] After receiving call 623, session manager service 110 performs operation 625. Operation 625 includes updating session metadata to associate the identity of executor 170-2 with the session identifier. For example, session manager service 110 may identify executor 170-2 by the source network address of call 523.

[0094] Now, with the second member added to the cluster for the session, the session manager service 110 can provide a response 627 including an identification (e.g., via a network address) of the new cluster member to call 601. After receiving the identity of the new cluster member, the distributed data processing application tool runtime can distribute a portion of the computation to the new member.

[0095] Figure 7 An exemplary cloud provider network computer system of a member of a hosted computing cluster according to some embodiments is depicted. In this example, a computer system 750 runs an agent 760 (e.g., agent 160). Agent 760 interfaces with resource manager service 114 and manages the life cycle of the hosted virtual machine. Specifically, agent 160 arranges the creation and destruction of virtual machines and their containment layers within the host environment to limit the access of processes executed within the virtual machine to the larger host computer system environment. Agent 160 also interfaces with an agent running within the started virtual machine to load VM-specific configuration data. Such configuration data may include customer credentials, executor identifiers, session identifiers, session network identifiers, etc. In some embodiments, agent 760 is considered to be a lightweight VMM and the started virtual machine is a microVM.

[0096] The computer system stores the bundle 714 in a local storage device. The bundle typically includes a pair of a root file system (rootfs) image 710 and a virtual machine snapshot 712 containing the state of the pre-warmed virtual machine. Fig. 9As depicted, the agent 760 may periodically update the locally stored bundle 714 by requesting updates from a bundle manager service (not shown).

[0097] After receiving a request to start an executor from the resource manager service 114, the agent 760 may perform one or more checks to, for example, verify that the computer system 750 has sufficient capacity to start the virtual machine and may then start the virtual machine. As described above, such a request from the resource manager service 114 may include software and / or hardware configuration, wherein the software configuration identifies one of the bundles 714 from which the virtual machine is to be started, and the hardware configuration identifies the characteristics of the virtual machine to be started (e.g., the number of DPUs). The agent 770 may track the amount of available memory and computing capacity based on the total amount of memory and computing capacity available to start the virtual machine and any existing virtual machines hosted by the computer system 750. If the agent 760 determines that the computer system 750 does not have sufficient resources to host the requested virtual machine, the agent 760 may reject the create virtual machine call from the resource manager service 114.

[0098] In this example, computer system 750 also includes network interface 702-1 and network interface 702-2. Network interface 702-1 carries traffic to and from distributed data processing application service components, such as resource manager service 714, authentication service from which credentials are obtained, and bundle manager service from which bundles are obtained, as described below. Network interface 702-2 carries traffic to and from other sources / destinations. Such other traffic includes traffic to / from other executors within the same session as the executor hosted by computer system 750. Such other traffic may include traffic to / from "outbound" virtual private clouds with endpoints that allow hosted applications to connect to data sources hosted by the provider network or outside the provider network (e.g., accessible via the Internet). For example, the customer who initiated the launch of the hosted executor may store data in a virtualized storage service, in a database (e.g., SQL or NoSQL), or elsewhere within the provider network. Such other traffic may include traffic to / from other customer-specific virtual private clouds (VPCs). For example, a customer may request that a session network be connected to a customer's VPC. A secure tunnel is established to carry traffic between a customer's specific executor and that customer's VPC endpoint.

[0099] After determining that the request to create a virtual machine can be satisfied, the agent 760 can proceed to start the requested virtual machine as follows. The agent 760 starts the process of starting the virtual machine by creating an isolated area (jail) in the host operating system environment, and starts the virtual machine in the isolated area. The isolated area refers to one or more containment layers provided by the host operating system to limit (or prevent) the process executed in the isolated area from accessing the wider host environment. Such isolation technology can also be referred to as "sandboxing" the virtual machine environment. The agent 760 then identifies the snapshot 712 as part of the configuration data for startup, and starts the virtual machine from the snapshot. In some embodiments, the agent 760 provides a copy of the snapshot 712 to the virtual machine to be started to prevent side channel attacks between virtual machines using the same snapshot. The agent 760 can also make the virtual machine to be started symbolically linked to the read-only shared copy of the rootfs 710 corresponding to the snapshot 712 from which the virtual machine will be started. The agent 760 can also create a read-write temporary, sparse file system file 724 for the virtual machine to be started. In some embodiments, agent 760 may also configure virtual network devices within the isolated zone with an overly restrictive initial set of firewall rules to allow agent 760 to perform actions on the virtual machine to be launched before exposing the virtual machine to the session network.

[0100] At this point, the agent 760 starts the virtual machine as a dedicated user in the previously created isolated area. In some embodiments, the agent 760 performs a post-startup check on the newly started virtual machine. The check may include various activities, such as generating a probe service to ensure that the firewall appropriately blocks access that is not in the virtual machine's explicit allow list.

[0101] As depicted, agent 760 has started two virtual machines 721 on computer system 750. Virtual machine 721-1 is executing guest environment 726-1 with rootfs 710-3, snapshot 712-3, and temporary file system 724-1. The exemplary guest environment includes an operating system, a distributed data processing application runtime, and an agent that interfaces with distributed data processing application services (e.g., reporting local heartbeats to agent 760, providing hooks for suspending and resuming computations, and / or providing a point of presence for agent 760 to further configure the virtual machine after startup and before handing control over to the client session). The virtual machine itself is isolated within an isolation zone 720-1 that includes a firewall 722-1.

[0102] The virtual machine 721-2 is executing a guest environment 726-2 with a rootfs 710-4, a snapshot 712-4, and a temporary file system 724-4 within the virtual machine. Again, the guest environment may include (although not shown) an operating system, a distributed data processing application runtime, and optionally an agent that helps the application runtime interface with the distributed data processing application service. The virtual machine itself is isolated within an isolation area 720-2 that includes a firewall 722-2.

[0103] The agent 760 may also load any application data included with or identified in a request from the resource manager service 714 to the agent operating within the guest environment. For example, the agent within the guest environment may then make such configuration data available as environment variables to the application runtime.

[0104] As part of starting the virtual machine, agent 760 forms an IPv6 address and assigns it to the virtual machine. Specifically, agent 760 forms the IPv6 address of the virtual machine based on the IPv6 subnet assigned to computer system 750 and the session network and local network identifiers received in the request to start the virtual machine from resource manager service 114. In some embodiments, agent 760 forms the IPv6 address of the new virtual machine by concatenating the IPv6 subnet prefix assigned to computer system 750, the session network identifier, and the local network identifier.

[0105] In some embodiments, instead of having the resource manager service 114 generate a local network identifier and include it in the request, the proxy 760 can determine a local network identifier so that it can uniquely identify multiple virtual machines hosted by the computer system 750 as part of the same session network, thereby eliminating one of the reasons why a request to launch a virtual machine may be denied. The proxy 760 can randomly select a local network identifier and verify that the local network identifier does not conflict with another virtual machine with the same session network identifier. Such verification can be done using an in-memory hash.

[0106] As depicted, firewalls 722 - 1 and 722 - 2 control communications between respective virtual machines 721 - 1 and 722 - 2 and any linked network interfaces 702 .

[0107] As part of starting the virtual machine, the proxy 760 configures the firewall 722 associated with the virtual machine to allow traffic to and from other virtual machines that are part of the same session network. To do this, the proxy 760 uses bit positions appropriate to the particular addressing scheme (e.g., Figure 3 Bits 47-8 in the addressing scheme shown in Figure 445-9 in the addressing scheme depicted in FIG. 4 and a mask that matches only those bit positions. More generally, the firewall will require that inbound and outbound traffic match the address bits corresponding to the mask bits having a value of "1" and ignore those address bits corresponding to the mask bits having a value of "0". For example, Figure 3 The firewall rules for session network 372 in will allow traffic matching address :::::4567:89AB:CD00 and mask :::::FFFF:FFFF:FF00. As another example, using Figure 4 The firewall rules for a session network with a session network identifier of 23456789AC will permit traffic with address :::::2345:6789:AC00 and mask :::::FFFF:FFFF:FE00. In this way, the firewall will not block inbound or outbound traffic that matches the session network identifier portion of the address associated with the virtual machine. In this way, the firewalls of existing executors in the session do not need to be updated even if other executors are added or removed from the session network.

[0108] As described above, a customer may wish to be able to access other provider network resources directly from within the session network of the cluster. The network configuration of the other resource may be identified in a request to launch a virtual machine received by proxy 760. Exemplary configuration data may include (1) an identification of the other provider network resource as an actual provider network address or an alias for the other resource, and (2) a network address based on a session network identifier associated with the other provider network resource. Proxy 760 may configure firewall 722 associated with the launched virtual machine to perform address translation between the session network address associated with the other provider network resource and the actual provider network address or alias for the other resource, and vice versa.

[0109] Figure 8An agent is depicted capturing the state of a running virtual machine from which a bundle is formed to start future virtual machines, thereby reducing startup delays. As described above, one aspect that helps to quickly start cluster members is to use a "pre-warmed" virtual machine bundle. A bundle refers to a grouping of virtual machine software, configuration, and state used to start a virtual machine instance. For example, a bundle may include a kernel, an operating system, an application runtime, and a copy of the virtual machine system memory and processor registers captured at a certain point in time. Pre-warmed bundles are so named because the virtual machine state is captured so that when a new virtual machine is started and resumed using the bundle, the distributed data processing application runtime is already "running." Different distributed data processing applications may allow different degrees of progress before the application is captured before each tool loads and processes session-specific configuration data (e.g., the identity of a cluster member, a portion of a customer-submitted calculation, etc.).

[0110] Figure 8 The right portion of pictorial 5 depicts a computer system 850 executing an agent 853. The agent 853 can manipulate and capture the operations of a virtual machine 855 hosted by the computer system 850. Such operations include pausing and resuming the execution of a virtual machine. In addition, the agent 853 can copy the state of the virtual machine (e.g., system memory and processor registers) in a suspended state and store that data as a "snapshot" 890.

[0111] Virtual machine 855 includes virtual file system 860 (which may be a read / write file system), virtual system memory 870, and virtual CPU (central processing unit) registers 880. The virtual machine may also use a read-only root file system (not shown).

[0112] During the startup of the virtual machine 855 and subsequent application execution, data may be loaded from the virtual file system 860 into the virtual system memory 870. Exemplary data loaded from the file system 860 into the system memory 870 may include a kernel, an operating system, an application (e.g., a distributed data processing tool) that started the virtual machine, and application data (e.g., libraries or other resources loaded by the application).

[0113] Figure 8The left portion of shows a simplified virtual machine startup sequence 800. As indicated by the vertical arrows, time proceeds from top to bottom. The startup sequence 800 may include an initial time period 801 during which the virtual machine executes a kernel and starts an operating system, resulting in associated data being loaded from a file system 860 into a system memory 870 of the virtual machine. The startup sequence 800 may include another time period 803 during which the virtual machine begins to execute an application runtime, resulting in the application and associated data being loaded from the file system 860 into the system memory 870 of the virtual machine. After one or more of these time periods, the agent 863 may suspend the execution of the virtual machine 855, as indicated at circle 1, and copy the state of the virtual machine into a snapshot, as indicated at circle 2. For example, the agent 853 may suspend the virtual machine 855 at a pause point 802 after time period 801 or at a pause point 804 after time period 803. Capturing snapshots of the virtual machine at one or more of these points effectively eliminates customer or session-unaware execution times from future launches of the virtual machine based on those captured snapshots. While eliminating all client and / or session agnostic startup times is beneficial, further optimizations are possible.

[0114] The startup sequence 800 may include another time period 805 during which the application runtime is configured to perform virtual computing, thereby loading additional libraries or other dependencies from the file system 860 into the system memory 870 of the virtual machine. The startup sequence 800 may include another time period 805 during which the application runtime is configured to execute customer-specific code, thereby loading customer-specific libraries or other dependencies from the file system 860 into the system memory 870 of the virtual machine. Note that in the latter case, the resulting snapshot and the bundle ultimately used to start the virtual machine will be customer-specific. Similarly, the agent 853 may pause the execution of the virtual machine 855 at pause points 806 and 808, respectively, after one or more of these time periods Y05 and 805, and capture snapshots of the state of the virtual machine at these points.

[0115] To provide an example of using virtual computation to load data, Spark itself runs using a Java virtual machine. The Spark runtime loaded into memory can be instructed to perform virtual computations. Virtual computations are written to exercise some common classes commonly used by client computations. By executing virtual computations, the application causes the associated classes to be compiled into the java virtual machine. At this point in the execution, the agent can pause the virtual machine and then terminate any classes created as part of the virtual computation, while retaining the compiled class footprint in memory. In this way, client computations subsequently performed by virtual machines started from snapshots can utilize classes already present in memory without introducing delays in loading that class into the java virtual machine.

[0116] Fig. 9 A bundle manager service that facilitates distribution of bundles to computer systems according to some embodiments is shown. The bundle manager service facilitates distribution of bundles including Figure 8 The bundles describing the captured state data are distributed to computer systems in the distributed data processing application service warm pool. As illustrated, the bundle manager service 920 interfaces with agents 853 that participate in virtual machine snapshot creation and with agents 160 that manage virtual machine lifecycles of distributed data processing application service cluster members hosted on computer system 150-1.

[0117] The bundle includes the virtual machine software, configuration, and state for launching a virtual machine instance. For example, a bundle for a specific virtual machine snapshot can be compiled by agent 853. In some embodiments, the bundle is a container that includes a uniquely identified bundle identifier, an application name (e.g., Spark), an application version (e.g., 3.0), a bundle version similar to a build number, a snapshot including memory and registers, a root file system (e.g., which includes an operating system and application runtime), and hardware configuration (e.g., the amount of system memory of the virtual machine, the number of virtual CPUs of the virtual machine, mount points, networks and other virtual devices, etc.).

[0118] At circle 1, the bundle manager service 920 obtains the bundle from the agent 853 and adds the bundle to a data store (not shown) that the bundle manager service 920 uses to store bundles. At circle 2, the agent 160 of the computer system 150-1 (or any agent of the computer systems in the warm pool cluster) obtains a manifest indicating which bundles 914 should be stored in the local storage device 913 of the computer system. For example, another component of the distributed data processing application service may periodically push the manifest to the agent 160. The agent 160 deletes any bundles not included in the manifest from the local storage device 913 and requests any bundles included in the manifest but not present in the local storage area from the bundle manager service 920. Once downloaded, the agent 160 sends a message to the resource manager service 114 announcing its capabilities, as indicated at circle 3 (e.g., similar to Figure 5 Resource manager service 114 may then update its view of the warm pool cluster (e.g., to consider computer system 150-1 when selecting a system to host a virtual machine from a newly downloaded bundle, or to exclude computer system 150-1 when selecting a system to host a virtual machine from a deleted bundle).

[0119] In some embodiments, the bundle includes an index that identifies each object that is part of the bundle (e.g., a root file system, a virtual machine memory snapshot, a virtual machine register snapshot, etc.). To improve performance, the agent 160 may announce the existence of the bundle to the resource manager service 114 after downloading only a portion of the bundle (e.g., the index and any objects required to start the virtual machine). The agent 160 may then download other objects that are part of the bundle (e.g., objects that will be used by the distributed data processing application once the computation has begun) after announcing the bundle to the resource manager service 114 or upon receiving a request to create a virtual machine from the partially downloaded bundle.

[0120] Fig.10A method for starting a new member to a computing cluster according to some embodiments is depicted. Some or all of operation 1000 (or other processes or variations and / or combinations thereof described herein) are performed under the control of one or more computer systems configured with executable instructions and implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program, which includes instructions that can be executed by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more operations (or all) in operation 1000 are performed by one or more (e.g., agent 160, agent 760) of an agent of a computer system hosting a virtual machine instance and a combination of a virtual machine started by that agent.

[0121] Operation 1000 includes obtaining a virtual machine bundle that includes a copy of memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime at block 1005. As described herein, a virtual machine bundle may include a copy of memory and registers of a virtual machine suspended at a certain point in execution. Figure 8 and associated descriptions include details of such bundles that can be pre-warmed for use in launching a cluster using a warm pool of computer systems in a distributed data processing application service. To eliminate delays associated with downloading a bundle from which to launch a virtual machine in response to a request to launch a virtual machine, an agent of the host computer system can occasionally update a locally stored bundle so that the bundle can be used after receiving a request to launch a virtual machine from the bundle, such as with reference to Fig. 9 described.

[0122] Operation 1000 also includes, at block 1010, receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request to launch a virtual machine including a bundle identifier identifying a virtual machine bundle. Figure 1 At circle 7 of FIG. 1 , agent 160 of computer system 150-1 receives a request to start a new member of a computing resource cluster to execute a distributed data processing application. In that example, the cluster has zero members at that point in time. Additional details of the call flow that results in the agent receiving a request to start a virtual machine can be found at Figure 5 and Figure 6 and the associated description.

[0123] Operation 1000 also includes, at block 1015, starting a virtual machine based at least in part on the virtual machine bundle. For example, agent 160 may allocate an amount of computing and memory capacity of the host computer system as part of starting the virtual machine. In some embodiments, starting the virtual machine includes the agent copying the suspended memory and registers of another virtual machine that was previously suspended and captured at a certain point in execution time and stored as part of the bundle to the started virtual machine, and resuming the started virtual machine from that point in execution time. Additional details of the startup process may be found at Figure 7 and Figure 8 and the associated description.

[0124] Operation 1000 also includes sending a message to register the virtual machine with the virtual machine cluster at block 1020. As described above, the session manager service 110 maintains metadata that can be used to track sessions. To reduce startup delays, successfully launched executors are registered with the session manager service 110 by sending a message from the virtual machine that executed the executor rather than relaying it back through the startup path. Additional details about registering cluster members such as session coordinators and other executors with the session manager service can be found in Figure 1 , Figure 2 , Figure 5 and Figure 6 and the associated description.

[0125] Fig.11 A method for establishing network communication between executors of a distributed data processing application program according to some embodiments is depicted. Some or all of operation 1100 (or other processes or variations and / or combinations thereof described herein) are performed under the control of one or more computer systems configured with executable instructions and implemented as a code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors. The code is stored on a computer-readable storage medium, for example, in the form of a computer program, which includes instructions that can be executed by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more operations (or all) in operation 1100 are performed, for example, by an agent (e.g., agent 160, agent 760) of a computer system hosting a virtual machine instance.

[0126] Operation 1100 includes receiving a first request to launch a virtual machine at block 1105, the first request including a session network identifier for identifying a distributed data processing application cluster hosted by a provider network. Figure 1At circle 7 of FIG. 1 , agent 160 of computer system 150-1 receives a request to start a new member of a computing resource cluster to execute a distributed data processing application. In that example, the cluster has zero members at that point in time. The session network identifier is used to establish a session network, such as Figure 3 and 4 Additional details of the call flow that results in the agent receiving a request to start a virtual machine can be found in Figure 5 and Figure 6 and the associated description.

[0127] Operation 1100 also includes starting the first virtual machine at block 1110. For example, agent 160 may allocate an amount of computing and memory capacity of the host computer system as part of starting the virtual machine. In some embodiments, starting the virtual machine includes the agent copying the suspended memory and registers of another virtual machine that was previously suspended and captured at a certain point in execution time to the started virtual machine, and resuming the started virtual machine from that point in execution time. Additional details of the startup process may be found at Figure 7 and Figure 8 and the associated description.

[0128] Operation 1100 also includes, at block 1115, setting an IP address for the first virtual machine, wherein the IP address for the first virtual machine is formed at least in part by combining an IP subnet assigned to the computer system with the session network identifier. Figure 3 and Figure 4 , the session network identifier is used to form a portion of an IP address assigned to a virtual machine. The entire IP address of the virtual machine may include a subnet prefix assigned to the host computer system and a local network identifier suffix that permits the virtual machine to be uniquely identified from other virtual machines that may be hosted by the same computer system as part of the same cluster having the same subnet prefix. The entire IP address may be formed by combining M bits of the subnet prefix, N bits of the session network identifier, and O bits of the local network identifier.

[0129] Operation 1100 also includes, at block 1120, configuring a firewall that controls network traffic to and from the first virtual machine to allow packets having a source address or a destination address that matches a portion of an IP address of the first virtual machine that includes a session network identifier. As described herein, the session network identifier allows a session network to be formed in a distributed manner: each agent that launches a virtual machine as part of a particular cluster can configure that virtual machine to communicate with other members of that cluster without having to update the other members of the cluster. Additional details of firewall configuration may be found at Figure 7 and the associated description.

[0130] Fig.12 An exemplary provider network (or "service provider system") environment is depicted in accordance with some embodiments. Provider network 1200 can provide resource virtualization to customers via one or more virtualization services 1210, which allow customers to purchase, rent, or otherwise obtain instances 1212 of virtualized resources (including, but not limited to, computing resources and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 1216 can be associated with the resource instance 1212; the local IP address is the internal network address of the resource instance 1212 on the provider network 1200. In some embodiments, the provider network 1200 can also provide public IP addresses 1214 and / or public IP address ranges (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that customers can obtain from the provider 1200.

[0131] Conventionally, the provider network 1200 can allow a customer of the service provider (e.g., a customer operating one or more customer networks 1250A-1250C (or "client networks") including one or more customer devices 1252) to dynamically associate at least some public IP addresses 1214 assigned or allocated to the customer with specific resource instances 1212 assigned to the customer via the virtualization service 1210. The provider network 1200 can also allow a customer to remap a public IP address 1214 previously mapped to one virtualized computing resource instance 1212 allocated to the customer to another virtualized computing resource instance 1212 also allocated to the customer. By using the virtualized computing resource instances 1212 and public IP addresses 1214 provided by the service provider, a customer of the service provider (such as an operator of the customer networks 1250A-1250C) can, for example, implement customer-specific applications and present the customer's applications over an intermediate network 1240 such as the Internet. Other network entities 1220 on the intermediate network 1240 can then generate traffic to the destination public IP address 1214 published by the customer networks 1250A to 1250C; the traffic is routed to the service provider data center, and at the data center is routed via the network underlay to the local IP address 1216 of the virtualized computing resource instance 1212 that is currently mapped to the destination public IP address 1214. Similarly, the response traffic from the virtualized computing resource instance 1212 can be routed back to the intermediate network 1240 via the network underlay to reach the source entity 1220.

[0132] As used herein, a local IP address refers to an internal or "private" network address of a resource instance, for example, in a provider network. A local IP address may be within an address block reserved by Request for Comments (RFC) 1918 of the Internet Engineering Task Force (IETF) and / or have an address format specified by IETF RFC 4193, and may be changed within a provider network. Network traffic originating from outside the provider network is not routed directly to a local IP address; instead, the traffic uses a public IP address mapped to the local IP address of the resource instance. A provider network may include a networking device or appliance that provides network address translation (NAT) or similar functionality to perform mapping from a public IP address to a local IP address (and vice versa).

[0133] A public IP address is an Internet mutable network address assigned to a resource instance by a service provider or customer. For example, traffic routed to a public IP address is translated via 1:1 NAT and forwarded to the corresponding local IP address of the resource instance.

[0134] Some public IP addresses can be assigned to specific resource instances by the provider network infrastructure; these public IP addresses can be referred to as standard public IP addresses, or simply standard IP addresses. In some embodiments, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0135] At least some public IP addresses may be assigned to or obtained by customers of the provider network 1200; then, the customer may assign the public IP address assigned to it to a specific resource instance assigned to the customer. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Unlike being assigned to a resource instance by the provider network 1200 as in the case of a standard IP address, a customer IP address may be assigned to a resource instance by a customer, for example, via an API provided by a service provider. Unlike a standard IP address, a customer IP address is assigned to a customer account and may be remapped to other resource instances by the corresponding customer when necessary or desired. A customer IP address is associated with a customer's account, rather than a specific resource instance, and the customer controls the IP address until the customer chooses to release the IP address. Unlike conventional static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping the customer's public IP address to any resource instance associated with the customer account. For example, a customer IP address enables a customer to solve a problem with a customer's resource instance or software by remapping the customer IP address to an alternative resource instance.

[0136] Fig.1313 is a block diagram of an exemplary provider network environment that provides storage services and hardware virtualization services to customers according to some embodiments. Hardware virtualization service 1320 provides multiple computing resources 1324 (e.g., computing instances 1325, such as VMs) to customers. For example, computing resources 1324 can be provided as a service to customers of provider network 1300 (e.g., to customers implementing customer network 1350). Each computing resource 1324 can have one or more local IP addresses. Provider network 1300 can be configured to route packets from the local IP addresses of computing resources 1324 to public Internet destinations and to route packets from public Internet sources to the local IP addresses of computing resources 1324.

[0137] The provider network 1300 may provide a customer network 1350 coupled to the intermediate network 1340, for example, via a local network 1356, with the ability to implement a virtual computing system 1392 via a hardware virtualization service 1320 coupled to the intermediate network 1340 and the provider network 1300. In some embodiments, the hardware virtualization service 1320 may provide one or more APIs 1302, for example, web service interfaces, via which the customer network 1350 may access functionality provided by the hardware virtualization service 1320, for example, via a console 1394 (e.g., a web-based application, a standalone application, a mobile application, etc.) of a customer device 1390. In some embodiments, at the provider network 1300, each virtual computing system 1392 at the customer network 1350 may correspond to a computing resource 1324 that is leased, rented, or otherwise provided to the customer network 1350.

[0138] A customer may access the functionality of the storage service 1310, e.g., via one or more APIs 1302 from an instance of a virtual computing system 1392 and / or another customer device 1390 (e.g., via a console 1394), to access and store data from storage resources 1318A through 1318N of virtual data stores 1316 (e.g., folders or “buckets,” virtualized volumes, databases, etc.) provided by the provider network 1300. In some embodiments, a virtualized data storage gateway (not shown) may be provided at the customer network 1350, which may cache at least some data locally, e.g., frequently accessed or critical data, and may communicate with the storage service 1310 via one or more communication channels to upload new or modified data from the local cache, such that a primary storage area for the data (the virtualized data store 1316) is maintained. In some embodiments, a user via the virtual computing system 1392 and / or on another client device 1390 may mount and access virtual data storage 1316 volumes via a storage service 1310 acting as a storage virtualization service, and these volumes may appear to the user as local (virtualized) storage devices 1398.

[0139] Although in Fig.13 1300, but the virtualization service may also be accessed from a resource instance within the provider network 1300 via API 1302. For example, a customer, appliance service provider, or other entity may access the virtualization service via API 1302 from within a corresponding virtual network on the provider network 1300 to request allocation of one or more resource instances within the virtual network or within another virtual network.

[0140] In some embodiments, a system implementing part or all of the techniques described herein may include a general-purpose computer system such as Fig.14 1400, which includes, or is configured to access, one or more computer-accessible media. In the illustrated embodiment, the computer system 1400 includes one or more processors 1410 coupled to a system memory 1420 via an input / output (I / O) interface 1430. The computer system 1400 also includes a network interface 1440 coupled to the I / O interface 1430. Although Fig.14 Computer system 1400 is shown as a single computing device, but in various embodiments, computer system 1400 may include one computing device or any number of computing devices configured to work together as a single computer system 1400 .

[0141] In various embodiments, computer system 1400 may be a uniprocessor system including one processor 1410 or a multiprocessor system including several processors 1410 (e.g., two, four, eight, or another suitable number). Processor 1410 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1410 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as an x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In a multiprocessor system, each processor 1410 may typically, but not necessarily, implement the same ISA.

[0142] The system memory 1420 may store instructions and data accessible by the processor 1410. In various embodiments, the system memory 1420 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, program instructions and data that implement one or more desired functions, such as those methods, techniques, and data described above, are shown as stored within the system memory 1420 as code 1425 and data 1426. The code 1425 may include, for example, executables for implementing, in whole or in part, components of the distributed data processing application services described herein. For example, the code 1425 may include executables for implementing the session manager service 110 (or members of the session manager service 110 in the case of a distribution), the network manager service 112, and / or the resource manager service 114 (or members of the resource manager service 114 in the case of a distribution). Exemplary executables also include executables for agents described herein (eg, agent 160, agent 760), application runtimes for distributed data processing tools, and the like.

[0143] In one embodiment, the I / O interface 1430 may be configured to coordinate I / O traffic between the processor 1410, the system memory 1420, and any peripheral devices in the device, including the network interface 1440 and / or other peripheral interfaces (not shown). In some embodiments, the I / O interface 1430 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 1420) into a format suitable for use by another component (e.g., the processor 1410). In some embodiments, the I / O interface 1430 may include support for devices attached through various types of peripheral buses, such as variants of the peripheral component interconnect (PCI) bus standard or the universal serial bus (USB) standard. In some embodiments, for example, the functionality of the I / O interface 1430 may be split into two or more separate components, such as a north bridge and a south bridge. Moreover, in some embodiments, some or all of the functionality of the I / O interface 1430 (such as an interface to the system memory 1420) may be directly incorporated into the processor 1410.

[0144] The network interface 1440 may be configured to allow data to be transferred between the computer system 1400 and other devices 1460 (e.g., such as Figure 1 In various embodiments, network interface 1440 may support communication via any suitable wired or wireless general purpose data network, such as various types of Ethernet, for example. Additionally, network interface 1440 may support communication via a telecommunications / telephone network, such as an analog voice network or a digital fiber optic communications network, via a storage area network (SAN) such as a Fibre Channel SAN, and / or via any other suitable type of network and / or protocol.

[0145] In some embodiments, computer system 1400 includes one or more offload cards 1470A or 1470B (including one or more processors 1475 and possibly one or more network interfaces 1440) connected using an I / O interface 1430 (e.g., a bus implementing a version of the Peripheral Component Interconnect-Express (PCI-E) standard or another interconnect such as Quick Path Interconnect (QPI) or Ultra Path Interconnect (UPI)). For example, in some embodiments, computer system 1400 may act as a host electronic device hosting computing resources (such as computing instances) (e.g., operating as part of a hardware virtualization service), and one or more offload cards 1470A or 1470B execute a virtualization manager that can manage computing instances executing on the host electronic device. As an example, in some embodiments, offload card 1470A or 1470B can perform computing instance management operations, such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In the context of a warm pool computer system serving distributed data processing applications, the offload card 1470 may execute an agent (e.g., agent 160, agent 760) that manages virtual machines executed by other processors (e.g., 1410) of the computer system 1400. In some embodiments, these management operations may be performed by the offload card 1470A or 1470B in cooperation with a hypervisor (e.g., pursuant to a request from the hypervisor) executed by the other processors 1410A-1410N of the computer system 1400. However, in some embodiments, the virtualization manager implemented by the offload card 1470A or 1470B may comply with requests from other entities (e.g., from the computing instances themselves) and may not cooperate with (or serve) any individual hypervisor.

[0146] In some embodiments, system memory 1420 may be an embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer-accessible media. In general, a computer-accessible medium may include any non-transitory storage medium or memory medium, such as a magnetic or optical medium, such as a disk or DVD / CD coupled to computing device 1400 via I / O interface 1430. Non-transitory computer-accessible storage media may also include any volatile or non-volatile medium, such as RAM (e.g., SDRAM, double data rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included in some embodiments of computer system 1400 as system memory 1420 or another type of memory. In addition, a computer-accessible medium may include a transmission medium or signal, such as an electrical signal, an electromagnetic signal, or a digital signal, communicated via a communication medium, such as a network and / or a wireless link such as may be implemented via network interface 1440.

[0147] The various embodiments discussed or proposed herein may be implemented in a wide variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that may be used to operate any of a number of applications. User or client devices may include: any of a number of general-purpose personal computers, such as desktop computers or laptop computers running standard operating systems; and cellular devices, wireless devices, and handheld devices running mobile software and capable of supporting a number of networking protocols and messaging protocols. Such systems may also include a number of workstations running a variety of commercially available operating systems and any of other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0148] Most embodiments utilize at least one network that will be familiar to those skilled in the art to support communications using any of a number of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The network may include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0149] In embodiments using a web server, the web server may run any of a variety of server or middle-tier applications, including HTTP servers, file transfer protocol (FTP) servers, common gateway interface (CGI) servers, data servers, Java servers, merchant application servers, etc. The server may also be capable of executing programs or scripts in response to requests from user devices, such as by executing a program that may be implemented in any programming language (such as The one or more servers may also include one or more web applications that use one or more scripts or programs written in SQL Server 2003 (C, C#, or C++) or any scripting language (such as Perl, Python, PHP, or TCL), and combinations thereof to execute programs or scripts. The one or more servers may also include a database server, including (without limitation) database servers available from Oracle (R), Microsoft (R), Sybase (R), IBM (R), etc. The database server may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0150] The environment disclosed herein may include a variety of data storage areas and other memories and storage media as discussed above. These may reside in a variety of locations, such as storage media residing in one or more computers local to the computer (and / or residing in one or more computers), or across a network away from any or all of the computers. In a specific set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server or other network device may be stored locally and / or remotely as appropriate. In the case where the system includes a computerized device, each such device may include hardware elements that can be electrically coupled via a bus, the elements including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touch screen or keypad) and / or at least one output device (e.g., display device, printer or speaker). Such systems may also include one or more storage devices, such as disk drives, optical storage devices and solid-state storage devices (such as random access memory (RAM) or read-only memory (ROM)) and removable media devices, memory cards, flash memory cards, etc.

[0151] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and a working memory, as described above. A computer-readable storage medium reader may be connected to a computer-readable storage medium or configured to receive a computer-readable storage medium, which represents a remote, local, fixed and / or removable storage device and a storage medium for temporarily and / or more permanently containing, storing, transmitting and retrieving computer-readable information. The system and various devices will generally also include many software applications, modules, services or other elements located in at least one working memory device, including an operating system and applications, such as a client application or a web browser. It should be understood that alternative embodiments may have numerous variants different from the above embodiments. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections to other computing devices such as network input / output devices may be used.

[0152] The storage medium and computer-readable medium for containing the code or part of the code may include any suitable media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk memory (CD-ROM), digital versatile disk (DVD) or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other medium that can be used to store the desired information and can be accessed by the system device. Based on the disclosure and teachings provided herein, those of ordinary skill in the art will understand other ways and / or methods of implementing various embodiments.

[0153] In the foregoing description, various embodiments are described. For the purpose of explanation, specific configurations and details will be set forth in order to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without the specific details. In addition, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0154] Bracketed text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used herein to illustrate optional operations that add additional features to some embodiments. However, such markings should not be taken to mean that these are the only options or optional operations, and / or that boxes with solid borders are not optional in certain embodiments.

[0155] In various embodiments, reference numerals with suffix letters (e.g., 1318A through 1318N) may be used to indicate that there may be one or more instances of the referenced entity, and when multiple instances exist, each instance need not be identical, but may instead share some general characteristics or act conventionally. Furthermore, the particular suffix used is not meant to imply that there is a particular number of entities unless specifically indicated to the contrary. Thus, in various embodiments, two entities using the same or different suffix letters may or may not have the same number of instances.

[0156] References to "one embodiment," "an embodiment," "exemplary embodiment," etc. indicate that the described embodiment may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be considered that it is within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0157] Furthermore, in the various embodiments described above, unless otherwise specifically noted, disjunctive language such as the phrase "at least one of A, B, or C" is intended to be understood to refer to A, B, or C, or any combination thereof (e.g., A, B, and / or C). Similarly, language such as "at least one or more of A, B, and C" (or "one or more of A, B, and C") should be understood to refer to A, B, or C, or any combination thereof (e.g., A, B, and / or C). As such, disjunctive language is not intended to, and should not be understood to, imply that a given embodiment requires that at least one of A, at least one of B, or at least one of C each be present.

[0158] Unless expressly stated otherwise, articles such as "a" or "an" are generally understood to include one or more of the recited items. Thus, phrases such as "a device configured to" or "computing device" are intended to include one or more of the recited devices. Such one or more recited devices may be collectively configured to perform the recited operations. For example, "a processor configured to perform operations A, B, and C" may include a first processor configured to perform operation A working in conjunction with a second processor configured to perform operations B and C.

[0159] At least some embodiments of the disclosed technology may be described in light of the following terms:

[0160] 1. A computer-implemented method, the method comprising:

[0161] An agent of a computer system in a pool of computer systems served by a distributed data processing application of a provider network performs the following steps:

[0162] obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime;

[0163] storing the virtual machine bundle in a local storage device of the computer system;

[0164] receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request comprising a bundle identifier identifying the virtual machine bundle, a session network identifier identifying a network of a cluster of virtual machines hosted by the provider network, and an executor identifier identifying a virtual machine to be launched in response to the request;

[0165] determining that the computer system has sufficient resources to satisfy the request to start a virtual machine to execute the virtual machine;

[0166] launching the virtual machine based at least in part on the virtual machine bundle stored in the local storage;

[0167] assigning an Internet Protocol (IP) address to the virtual machine based at least in part on the session network identifier; and

[0168] A message including the executor identifier is sent from the virtual machine to register the virtual machine with the cluster of virtual machines hosted by the provider network.

[0169] 2. The computer-implemented method of clause 1, further comprising:

[0170] The Session Manager service provided by the provider network:

[0171] receiving the message including the executor identifier from the virtual machine, the message identifying an Internet Protocol address of the virtual machine;

[0172] determining a session identifier identifying the cluster of virtual machines based at least in part on a portion of the Internet Protocol address that includes the session network identifier; and

[0173] The session metadata is updated to associate the identification of the virtual machine with the session identifier.

[0174] 3. A computer-implemented method according to any of clauses 1-2, the method also including sending a calculation to the virtual machine by a session manager service of a provider network after receiving a message including an executor identifier from the virtual machine, wherein the calculation is to be executed by the distributed data processing application runtime executed by the virtual machine.

[0175] 4. A computer-implemented method, the method comprising:

[0176] An agent for a computer system in the pool of computer systems served by the distributed data processing application performs the following steps:

[0177] obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime;

[0178] receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request to launch a virtual machine including a bundle identifier identifying a virtual machine bundle;

[0179] launching a virtual machine based at least in part on the virtual machine bundle; and

[0180] Sends a message from a virtual machine to register the virtual machine with the virtual machine cluster.

[0181] 5. The computer-implemented method of clause 4, further comprising, by a session manager service, performing the following steps:

[0182] receiving a message from the virtual machine to register the virtual machine; and

[0183] The session metadata is updated to associate the identity of the virtual machine with a session identifier that identifies the virtual machine cluster.

[0184] 6. The computer-implemented method of clause 5, further comprising: sending, by the session manager service, a computation to the virtual machine, the computation for execution at runtime by a distributed data processing application executed by the virtual machine.

[0185] 7. A computer-implemented method according to any of clauses 4-6, wherein the request to start a virtual machine also includes a session network identifier that identifies a network of a cluster of virtual machines, and the method also includes an agent of the computer system assigning an Internet Protocol (IP) address to the virtual machine based at least in part on the session network identifier.

[0186] 8. A computer-implemented method according to any of clauses 4-7, wherein the request to start the virtual machine further comprises a session identifier identifying a cluster of virtual machines, and the method further comprises performing, by the virtual machine, the following steps:

[0187] sending a request to a session manager service to add a new member to the virtual machine cluster, the request including a session identifier; and

[0188] Receives from the session manager service the identity of a new cluster member to start in response to the request.

[0189] 9. The computer-implemented method of any of clauses 4-8, wherein starting the virtual machine comprises resuming a suspended virtual machine executing the distributed data processing application runtime.

[0190] 10. The computer-implemented method of any of clauses 4-9, further comprising creating, by an agent of the computer system, an isolated area within which the virtual machine is hosted, and wherein the virtual machine is started within the isolated area.

[0191] 11. The computer-implemented method of any of clauses 4-10, wherein at least a portion of the bundle is stored in a local storage device of the computer system.

[0192] 12. A computer-implemented method according to any of clauses 4-11, wherein the virtual machine bundle includes an indication of an amount of memory to be allocated to the virtual machine and an indication of a number of virtual central processing units, and the method further comprises performing, by an agent of the computer system, the following steps:

[0193] determining an indication that an amount of available memory of the computer system meets or exceeds an amount of memory to be allocated to the virtual machine; and

[0194] A determination is made as to whether an amount of available computing capacity of the computer system meets or exceeds an indication of a number of virtual processing units to be allocated to the virtual machine.

[0195] 13. A system, comprising:

[0196] first one or more electronic devices for implementing a session manager service for a distributed data processing application service; and

[0197] A second one or more electronic devices of a computer system in a pool of computer systems of a distributed data processing application service of a provider network, the second one or more electronic devices comprising instructions that, when executed, cause the computer system to:

[0198] obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime;

[0199] receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request to launch a virtual machine including a bundle identifier identifying a virtual machine bundle;

[0200] launching the virtual machine based at least in part on the virtual machine bundle; and

[0201] Sends a message from the virtual machine to the Session Manager service to register the virtual machine with the virtual machine cluster.

[0202] 14. The system of clause 13, wherein the first one or more electronic devices include instructions that, when executed, cause the session manager service to:

[0203] receiving, from the virtual machine, the message to register the virtual machine; and

[0204] The session metadata is updated to associate the identification of the virtual machine with a session identifier that identifies the cluster of virtual machines.

[0205] 15. A system as described in any of clauses 13-14, wherein the first one or more electronic devices include other instructions that, when executed, cause the session manager service to send computations to the virtual machine, the computations being executed at runtime by a distributed data processing application executed by the virtual machine.

[0206] 16. A system as described in any of clauses 13-15, wherein the request to launch a virtual machine also includes a session network identifier that identifies a network of a cluster of virtual machines, and wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to assign an Internet Protocol (IP) address to the virtual machine based at least in part on the session network identifier.

[0207] 17. The system of any of clauses 13-16, wherein the request to launch the virtual machine further comprises a session identifier identifying the virtual machine cluster, and wherein the second one or more electronic devices comprise other instructions that when executed cause the computer system to:

[0208] sending a request to the session manager service to add a new member to the virtual machine cluster, the request including the session identifier; and

[0209] An identification of a new cluster member to be started in response to the request is received from the session manager service.

[0210] 18. The system of any of clauses 13-17, wherein starting the virtual machine comprises resuming a suspended virtual machine while executing the distributed data processing application runtime.

[0211] 19. A system as described in any of clauses 13-18, wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to create an isolated area in which to host the virtual machine, and wherein the virtual machine is launched in the isolated area.

[0212] 20. The system of any of clauses 13-19, wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to:

[0213] obtaining a manifest identifying one or more bundles to be stored in a local storage device of the second one or more electronic devices of the computer system;

[0214] deleting from local storage a first bundle not identified in the manifest;

[0215] downloading from a bundle manager service at least a portion of a second bundle identified in the manifest; and

[0216] At least a portion of the second bundle is stored in local storage.

[0217] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure as set forth in the claims.

Claims

1. A computer-implemented method, the method comprising: An agent of a computer system in a pool of computer systems served by a distributed data processing application of a provider network performs the following steps: obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime; storing the virtual machine bundle in a local storage device of the computer system; receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request comprising a bundle identifier identifying the virtual machine bundle, a session network identifier identifying a network of a cluster of virtual machines hosted by the provider network, and an executor identifier for identifying a virtual machine launched in response to the request; determining that the computer system has sufficient resources to satisfy the request to start a virtual machine to execute the virtual machine; launching the virtual machine based at least in part on the virtual machine bundle stored in the local storage; assigning an Internet Protocol (IP) address to the virtual machine based at least in part on the session network identifier; as well as A message including the executor identifier is sent from the virtual machine to register the virtual machine with the cluster of virtual machines hosted by the provider network.

2. The computer-implemented method of claim 1 , further comprising: The following steps are performed by the Session Manager service of the provider network: receiving the message including the executor identifier from the virtual machine, the message identifying an Internet Protocol address of the virtual machine; determining a session identifier identifying the cluster of virtual machines based at least in part on a portion of the Internet Protocol address including the session network identifier; as well as The session metadata is updated to associate the identification of the virtual machine with the session identifier.

3. The computer-implemented method of claim 2, further comprising: A computation is sent to the virtual machine by a session manager service of the provider network after receiving a message from the virtual machine including the executor identifier, the computation being executed at runtime by the distributed data processing application executed by the virtual machine.

4. A computer-implemented method, the method comprising: An agent for a computer system in the pool of computer systems served by the distributed data processing application performs the following steps: obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime; receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request to launch the virtual machine comprising a bundle identifier identifying the virtual machine bundle; launching the virtual machine based at least in part on the virtual machine bundle; as well as A message is sent from the virtual machine to register the virtual machine with a virtual machine cluster.

5. The computer-implemented method of claim 4, further comprising the steps of: receiving, from the virtual machine, the message to register the virtual machine; and The session metadata is updated to associate the identification of the virtual machine with a session identifier that identifies the cluster of virtual machines.

6. The computer-implemented method of claim 5, further comprising: A computation is sent by the session manager service to the virtual machine for runtime execution by the distributed data processing application executed by the virtual machine.

7. The computer-implemented method of claim 5, wherein the request to launch the virtual machine further comprises a session network identifier identifying a network of the virtual machine cluster, and the method further comprises: An Internet Protocol (IP) address is assigned to the virtual machine by the agent of the computer system based at least in part on the session network identifier.

8. The computer-implemented method of claim 7, wherein the request to launch the virtual machine further comprises identifying the session identifier of the virtual machine cluster, and the method further comprises performing, by the virtual machine, the following steps: sending a request to the session manager service to add a new member to the virtual machine cluster, the request including the session identifier; and An identification of a new cluster member to be started in response to the request is received from the session manager service.

9. The computer-implemented method of claim 4, wherein starting the virtual machine comprises: The suspended virtual machine is resumed while executing the distributed data processing application program is running.

10. The computer-implemented method of claim 4, further comprising: An isolated area is created by the agent of the computer system within which the virtual machine is hosted, and the virtual machine is started within the isolated area.

11. The computer-implemented method of claim 4, wherein at least a portion of the bundle is stored in a local storage device of the computer system.

12. The computer-implemented method of claim 4, wherein the virtual machine bundle includes an indication of an amount of memory to be allocated to the virtual machine and an indication of a number of virtual central processing units, and the method further comprises performing, by the agent of the computer system, the following steps: determining that an amount of available memory of the computer system meets or exceeds the indication of the amount of memory to be allocated to the virtual machine; and A determination is made as to whether an amount of available computing capacity of the computer system meets or exceeds the indication of the number of virtual processing units to be allocated to the virtual machine.

13. A system, comprising: a first one or more electronic devices, the first one or more electronic devices being used to implement a session manager service for a distributed data processing application service; as well as a second one or more electronic devices of a computer system in a pool of computer systems of the distributed data processing application service of a provider network, the second one or more electronic devices comprising instructions that, when executed, cause the computer system to: obtaining a virtual machine bundle, the virtual machine bundle comprising a copy of a memory and one or more registers of a suspended virtual machine executing a distributed data processing application runtime; receiving a request to launch a virtual machine to execute a distributed data processing application runtime, the request to launch the virtual machine comprising a bundle identifier identifying the virtual machine bundle; launching the virtual machine based at least in part on the virtual machine bundle; as well as A message is sent from the virtual machine to the session manager service to register the virtual machine with a virtual machine cluster.

14. The system of claim 13, wherein the first one or more electronic devices include instructions that, when executed, cause the session manager service to: receiving, from the virtual machine, the message to register the virtual machine; and The session metadata is updated to associate the identification of the virtual machine with a session identifier that identifies the cluster of virtual machines.

15. A system as described in claim 14, wherein the first one or more electronic devices include other instructions that, when executed, cause the session manager service to send calculations to the virtual machine, the calculations being used for runtime execution of the distributed data processing application executed by the virtual machine.

16. A system as described in claim 14, wherein the request to launch the virtual machine also includes a session network identifier that identifies the network of the virtual machine cluster, and wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to assign an Internet Protocol (IP) address to the virtual machine based at least in part on the session network identifier.

17. The system of claim 16, wherein the request to launch the virtual machine further comprises the session identifier identifying the virtual machine cluster, and wherein the second one or more electronic devices comprise other instructions that when executed cause the computer system to: sending a request to the session manager service to add a new member to the virtual machine cluster, the request including the session identifier; and An identification of a new cluster member to be started in response to the request is received from the session manager service.

18. The system of claim 13, wherein starting the virtual machine comprises: The suspended virtual machine is resumed while executing the distributed data processing application program is running.

19. The system of claim 13, wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to create an isolated area within which to host a virtual machine, and wherein the virtual machine is launched within the isolated area.

20. The system of claim 13, wherein the second one or more electronic devices include other instructions that, when executed, cause the computer system to: obtaining a manifest identifying one or more bundles for storage in a local storage device of the second one or more electronic devices of the computer system; deleting from the local storage a first bundle not identified in the manifest; downloading from a bundle manager service at least a portion of a second bundle identified in the manifest; as well as The at least portion of the second bundle is stored in the local storage.

Images