On-demand code execution computing resource management

Abstract

Systems and methods are provided for an on-demand code execution service that includes a set of computing devices to execute function code on demand while continuing to support execution of long run background processes. A subset of resources may be initialized based at least in part on application configuration data, including a request-response process, a background process, and a set of smaller computing resources for the background process. After execution of the background process has begun, a first request may be received. The on-demand code execution service may add computing resources to a larger set of computing resources to generate a first response to the first request. The first response may then be provided to a set of external computing resources. Upon determining that the queue does not include additional requests, the on-demand code execution service may reduce the level of computing resources to the smaller set of computing resources.

Description

By incorporating via reference

[0001] The following application is incorporated herein by reference and forms part of this specification: U.S. Patent Application No. 18 / 478,476, filed September 29, 2023, entitled “On-demand code execution computing resource management”. Background Technology

[0002] Computing systems can use communication networks to exchange data. In some implementations, a computing system can receive and process data provided by another computing system. For example, a computing system may receive data input from another computing system, store data, process data, and so on. Attached Figure Description

[0003] Embodiments of various inventive features will now be described with reference to the following accompanying drawings. Throughout the drawings, reference numerals may be used repeatedly to indicate the correspondence between referred elements. The drawings are provided to illustrate exemplary embodiments described herein and are not intended to limit the scope of this disclosure. For ease of identification of any particular element or action being discussed, the highest significant numeral in the reference numerals generally refers to the figure number in which that element was first introduced.

[0004] Figure 1 This is a block diagram of an illustrative computing environment in which an on-demand code execution system, according to some embodiments, can operate in conjunction with a data storage system.

[0005] Figure 2 This is a block diagram of an illustrative computing system configured to provide cross-function call data management in an on-demand code execution system, according to some embodiments.

[0006] Figure 3 Example routines of request / response cycles executed by an on-demand code execution system on a virtual execution environment (e.g., virtual machine instance, container, etc.) are shown.

[0007] Figure 4 A diagram depicting the interaction between the client device and components of the on-demand code execution system during the execution of the first request is shown.

[0008] Figure 5A A set of example interactions between components is shown, where queue 502 is used to handle multiple requests received from a client device.

[0009] Figure 5B Another set of example interactions between components is shown, in which queue 502 is used to handle multiple requests received from client device 402. Detailed Implementation

[0010] This disclosure relates to managing application instances using code that executes in an on-demand (“serverless”) manner, including code that executes synchronously (e.g., according to a request-response model) and additional code that executes continuously or asynchronously (e.g., as a background process outside the request-response model). To provide compute resources to continuous or asynchronous components of such application instances even when no synchronous processes are executing, a baseline amount or set of compute resources can be provisioned (e.g., less than the amount or set of compute resources provisioned for synchronous processes). When a synchronous process is to be executed, the amount or set of compute resources provisioned for the application can be increased to provide the desired performance level while the synchronous process is executing. In this way, application developers can obtain the benefits of an on-demand or “serverless” platform while ensuring that background processes or other asynchronous tasks running for longer periods can also be executed on the same platform.

[0011] introduction

[0012] Some data centers may include multiple interconnected computing systems used to provide computing resources to users within the data center. To facilitate increased utilization of data center resources, virtualization technology allows a single physical computing device to host one or more instances of virtual execution environments (VMs) for users within the data center, presenting and operating them as independent computing devices. Through virtualization, a single physical computing device can dynamically create, maintain, delete, or otherwise manage virtual execution environments, such as virtual machines (VMs), microVMs, containers, or other virtual computing components. Users can then request computing resources from the data center, including the configuration of a single computing device or a networked computing device, and can be provided with varying numbers of virtual resources.

[0013] In addition to computing resources, data centers offer a variety of other beneficial services to client devices. For example, data centers can provide data storage services configured to store data submitted by client devices and enable the retrieval of this data over a network. Various types of data storage services can be provided, typically varying depending on their input / output (I / O) mechanisms. Some data centers include on-demand code execution systems, sometimes referred to as serverless function execution systems.

[0014] In general, on-demand code execution systems enable the execution of any user-specified function or application code without requiring the user to create, maintain, or configure the execution environment (e.g., a physical machine or virtual machine) in which the function or application code executes. For example, conventional computing services typically require users to provision specific devices (virtual or physical), install operating systems on those devices, configure application settings, define network interfaces, and so on. On-demand code execution systems, however, allow users to submit code and provide application programming interfaces (APIs) that, when used, enable users to request code execution. Upon receiving a call via the API, the on-demand code execution system can dynamically generate an execution environment for the code, provision the code to the environment, execute the code, and provide the results. Therefore, on-demand code execution systems eliminate the need for users to manage the configuration and management of the environment used for code execution. Due to the flexibility of on-demand code execution systems in executing arbitrary functions or application code, such systems can be used to create a variety of network services. For example, such systems can be used to create "microservices," which implement a small number of functions (or just one function) and interact with other services to provide network services for an application. As another example, such systems can be used to implement Software as a Service (SaaS) platforms that enable customers to deploy entire applications (rather than individual functions, or more than individual functions) in a serverless, on-demand manner.

[0015] In the context of on-demand code execution systems, instances of function or application code that execute to provide such services are generally referred to as “called code,” or more specifically, “called function” or “called application,” or, for brevity, simply as “code,” “function,” or “application.” The terms “application,” “application code,” and “application software” are used herein according to their usual and conventional meaning in the field of computer technology, and refer to computer programs (or sets of computer programs) designed to perform a specific task (or set of tasks), not merely those tasks related to the operation of the computing system itself. A “function,” “function code,” or “function application” may define a “task” and, when executed on a virtual machine instance of an on-demand code execution system, implement a specific functionality corresponding to that task. Applications may include functions and may call external functions. Applications are typically executed by or at the instruction of an end user, but are not necessarily interactive. For example, an application may be executed on a schedule or triggered in response to an event, and may perform various processing tasks before termination without any end-user interaction.

[0016] The infrastructure or configuration of an on-demand code execution system may impose certain limitations on the functions or applications that run. In some cases, these limitations may interfere with or prevent the use of the on-demand code execution system to execute long-running processes (e.g., background processes, asynchronous processes) and to execute function or application code according to a request-response model or other synchronous execution model. For example, it may be desirable to run background processes that require minimal computational resources for relatively long periods (compared to execution according to a request-response model), while also being able to scale additional computational resources as needed for other operations (e.g., processing requests synchronously). However, conventional on-demand code execution systems allocate resources only according to a request-response model or other synchronous execution model to meet the requirements of the invoked functions. Furthermore, users who require long-running processes need to create, maintain, or configure a separate execution environment for this process, which significantly reduces the deployment and management benefits of using an on-demand code execution system for these users. To avoid deploying applications or systems across different environments and platforms, users can choose to create, configure, and maintain a single execution environment outside of the on-demand code execution service, which completely eliminates the cost, deployment, and management benefits of using on-demand code execution.

[0017] Some aspects of this disclosure utilize on-demand code execution systems to address some or all of the problems described above, which provision computational resources to applications in a configurable and dynamic manner based on the processing models of individual application components. In some embodiments, configurable and dynamic provisioning of computational resources to an application can provide the desired performance level for request-response or other synchronous operations, while also allowing background or asynchronous processes to run for longer periods in a resource-efficient and cost-effective manner (e.g., outside of request-response models). For example, the application can be provisioned with a non-zero “baseline” level of computational capacity sufficient to sustain the operation of background processes, and the system can dynamically increase computational resources as needed to meet the performance requirements of invoked synchronous processes. In one embodiment, this baseline amount of resources can be configured to achieve a minimum non-zero level of computational resources. Dynamically provisioning resources to maintain at least a non-zero baseline level of computational capacity can save time and computational resources for users who need to run background processes for extended periods without affecting (or materially affecting) the overall performance of the application. Furthermore, this type of configurable, dynamic resource provisioning allows applications previously developed for customer-managed environments (e.g., traditional server-based deployments) to be deployed to on-demand code execution environments without any changes to the application's architecture (or without any significant changes). For example, instead of redesigning the application as two separate applications running in two separate environments (e.g., a request-response component running in the on-demand code execution system and a continuously running background component on a traditional server-based deployment), the application can be deployed as a whole on the on-demand code execution system, whose resources can be dynamically scaled up and down in a cost-effective manner, but not below the minimum configurable resource count maintained for the background component.

[0018] In some embodiments, the on-demand code execution system may receive application configuration data, such as configuration data relating to a baseline non-zero level of computing power required to run background processes. For example, instructions relating to computing resources may include indications of the processing resources or network bandwidth required to run background processes. The configuration data may also include code and data required to run background processes and request / response processes.

[0019] As used herein, “baseline non-zero level of computing power” can also be referred to as “configurable lower level” computing power or computing resources (or simply “lower level” for brevity) to distinguish it from the level of computing power or computing resources required or desired by other parts of the application, such as synchronization processes. The level of computing power or computing resources required or desired by other parts of the application can be referred to as “higher level of computing power”. The term “computing power” can refer to a “set of computing resources” or can be used interchangeably with “set of computing resources”. For example, baseline or lower non-zero level of computing power may be a baseline or a smaller set of computing resources. As another example, higher level of computing power may be a larger set of computing resources.

[0020] In some embodiments, the configuration data may further include instructions related to a higher level of computing power for running one or more invoked functions. For example, configuration data related to the maximum level of computing power may include instructions to increase computing resources to a higher level of computing power when the application or function is invoked (e.g., to process a request). In some embodiments, configuration data related to a higher level of computing power may include instructions to expand resources to the amount of computing resources required to run the invoked application or function, within the range defined by the lower non-zero level of computing power and the higher level of computing power. For example, the invoked function may require computing power higher than the lower non-zero level of computing power but lower than the higher level of computing power to run. In some aspects, the system may determine which computing resources are required to run the invoked application or function and provide those resources, provided that they are less than the higher level of computing power.

[0021] Referring to the illustrative example, a client device can submit a request for a result to an application. This request can be stored in a queue monitored by the on-demand code execution system or its components, such as a supervisor component executing on the host device in which the application runs. In response to the request, the supervisor component of the on-demand code execution system or some other component can determine which function should be invoked to provide the result, prepare the necessary computing resources (if needed), and invoke the function required to obtain the result. Once the result is obtained, the supervisor component of the on-demand code execution system or another component can reduce the computing resources to a lower non-zero level of computing power specified in the application configuration data. The application configuration data can also specify a higher level of computing power required to execute one or more invoked functions or the application, background processes, and request / response processes. Alternatively, the supervisor component can determine that the queue does not contain additional requests before reducing the computing resources to a lower non-zero level of computing power. If the system determines that the queue does contain additional requests, the system can provide results for each request in the order they were received, using the process described above. For example, for each request, the system can determine which application or function should be invoked to provide the result for each request, provision the necessary computing resources (if not already provisioned), and invoke the function required to obtain the result. Once the queue is empty or the system otherwise determines that there are no pending incomplete requests, the supervisor component can reduce the computing resources to a low non-zero level of computing power sufficient to continue the background process.

[0022] In some cases, the result corresponding to a specific request may not be obtained by the system. For example, one or more invoked functions required to obtain the result may fail. In such cases, the supervisor component of the on-demand code execution system or another component can access instructions, including a set time period for responding to the request. The set time period can be provided in the configuration data. If the supervisor component fails to obtain the result within the set time period, it can cancel one or more invoked functions. The supervisor component can then determine whether the queue contains any additional requests and, based on this determination, decide whether to reduce or maintain computing resources accordingly.

[0023] Various aspects of this disclosure will be described with respect to certain examples and embodiments, which are intended to be illustrative and not limiting. While aspects of some embodiments described herein will focus on managing on-demand code execution systems to meet user needs for long-running background processes for illustrative purposes, these examples are merely illustrative and not intended to be limiting. In some embodiments, the techniques described herein can be applied to additional or alternative types of function code (e.g., subroutines), datasets, and on-demand code execution system configurations. Furthermore, any feature used in any embodiment described herein may be used in any combination with any other feature or in any other embodiment without limitation.

[0024] Example network environment

[0025] On-demand code execution systems provide network-accessible services that allow users to submit or specify computer-executable source code to be executed by virtual machine instances on the on-demand code execution system, which will be referred to as "code" in this document for simplicity.

[0026] Code on an on-demand code execution system can be limited to applications with interactive, request-response, or other short-running processes. Applications can also have long-running background processes. Users can submit the level of computing resources required to run a background process. This level can be a low, non-zero level of computing power allocated to a virtual machine instance to run the user-submitted computer-executable source code. Any background process submitted by a user can be initiated when the virtual machine instance of the on-demand code execution system starts, or triggered by some other event (e.g., after responding to an initial request, after a period of time, etc.). Long-running background processes can include, but are not limited to: monitoring a data storage device for changes in the number of items in the data storage device, or monitoring a queue containing user requests to determine if a threshold number of requests has been reached. For example, a background process can monitor the number of items in a cache and send an alert when it reaches a threshold. In a non-limiting example, the cache could be 10,000 items. If the background process notices that this threshold has been reached, it can respond by deleting the cache or calling another function / application or API to perform an action. In some aspects, the action could be transmitting a message to a client device. As another example, a background process can be used to monitor external storage units (e.g., cloud storage). In a non-restrictive example, the background process can be configured to run at intervals to determine if a new item (e.g., a file) exists in the container. If a new item is found, the background process can transmit a message to the client device.

[0027] Alternatively, code on an on-demand code execution system can define a "task" and implement a specific function corresponding to that task when executed on a virtual machine instance of the on-demand code execution system. A task can be implemented as a standalone on-demand function or as a feature of a larger application (e.g., an application with attached background processes as described above). Implementing a task independently on an on-demand code execution system can be referred to as "executing" the task (or "task execution"). For example, a task can be implemented using a request-response protocol, where a request is received and application or function code is invoked in response to the request.

[0028] In some embodiments, the on-demand code execution system allows users to directly trigger the execution of an application or individual task based on a variety of potential events, such as transmitting API requests or specially formatted Hypertext Transfer Protocol (“HTTP”) packets to the on-demand code execution system. The on-demand code execution system can therefore execute any specified executable code “on demand” without requiring the configuration or maintenance of the underlying hardware or infrastructure on which to execute code. Furthermore, the on-demand code execution system can be configured to execute tasks in a fast manner (e.g., within 100 milliseconds [ms]), thus enabling “real-time” (e.g., with virtually no latency perceptible to the end user) execution of tasks. To achieve this fast execution, the on-demand code execution system may include one or more virtual machine instances that are “pre-warmed” or pre-initialized (e.g., booted into an operating system and executing a full or substantially full runtime environment) and configured to execute user-defined code, allowing the code to be executed rapidly in response to a request for execution without the latency caused by the initialization of the virtual machine instances. Preheated or pre-initialized virtual machine instances can be allocated non-zero levels of compute power (such as the configurable lower levels of compute power described above) to run background processes and begin execution upon boot. Therefore, when an "on-demand" task is triggered, the code corresponding to that task can be executed within a pre-initialized virtual machine for a very short time. Background processes can continue running during and after the execution of the "on-demand" task.

[0029] Specifically, to execute applications or individual tasks, the on-demand code execution system described herein can maintain a pool of execution virtual machine instances that are immediately available upon receiving a request for execution. Due to the pre-initialized nature of these virtual machines, the latency (sometimes referred to as delay) associated with executing application code or individual task code (e.g., instance and language runtime startup time) can be significantly reduced, typically to below 100 milliseconds. Illustratively, the on-demand code execution system can maintain a pool of virtual machine instances on one or more physical computing devices, each virtual machine instance loading one or more software components (e.g., operating system, language runtime, libraries, etc.). Any virtual machine instance can be additionally configured to run background processes.

[0030] For example, a user can submit the code required to start a background process and provide a low non-zero level of computing power required to run the background process. The code required to start the background process may include startup code that runs during the startup of the virtual machine instance, as well as code containing data and instructions for the background process. In a non-limiting embodiment, the on-demand code execution system can bootstrap a virtual machine instance allocated a low non-zero level of computing power, and the virtual machine instance can begin executing a background process while executing the startup code provided by the user. The virtual machine instance running the background process can additionally perform tasks. For example, the virtual machine instance can be configured to run a background process to monitor the number of items in the cache and send an alert when it reaches a threshold. While this process is running, the on-demand code execution system can receive requests for executable code (“tasks”).

[0031] When an on-demand code execution system receives a request to execute a task, it can select a virtual machine instance to execute the user's program code based on one or more computational constraints associated with the task (e.g., a required operating system or runtime), and execute the task on the selected virtual machine instance. The task can be executed in an isolated container created on the virtual machine instance, or it can be executed within a virtual machine instance isolated from other virtual machine instances used as an environment for other tasks. Because the virtual machine instances in the pool are already booted and loaded with a specific operating system and language runtime, as well as optional background processes, when a request is received, the latency associated with finding computational power that can handle the request (e.g., by executing user code in one or more containers created on virtual machine instances) can be significantly reduced.

[0032] As used herein, the term "virtual machine instance" is intended to refer to the execution of software or other executable code that emulates hardware to provide an environment or platform (e.g., an "execution environment") on which software can be executed. Virtual machine instances are typically executed by hardware devices that may differ from the physical hardware emulated by the virtual machine instance. For example, a virtual machine may emulate a first type of processor and memory while executing on a second type of processor and memory. Thus, software intended for a first execution environment (e.g., a first operating system) can be executed on a physical device that is executing a second execution environment (e.g., a second operating system). In some cases, the hardware emulated by the virtual machine instance may be the same as or similar to the hardware of the underlying device. For example, a device with a first type of processor may implement multiple virtual machine instances, each emulating an instance of that first type of processor. Thus, a device can be divided into a number of logical sub-devices (each logical sub-device is called a "virtual machine instance") using virtual machine instances. Virtual machine instances on a host device may share network bandwidth to communicate with external computing resources. For example, multiple client devices may use allocated bandwidth controlled by an on-demand code execution system to communicate with different subsets of virtual machine instances within the host device. On-demand code execution systems can adjust network bandwidth allocation based on the processes running on virtual machine instances on each host device.

[0033] While virtual machine instances can typically provide a level of abstraction from the underlying physical hardware, this abstraction is not mandatory. For example, suppose a device implements multiple virtual machine instances, each emulating the same hardware provided by the device. In such scenarios, each virtual machine instance allows software applications to execute code on the underlying hardware without translation, while maintaining logical separation between software applications running on other virtual machine instances. This process, often referred to as "native execution," can be used to increase the speed or performance of virtual machine instances. Other techniques that allow direct utilization of the underlying hardware, such as hardware passthrough, can also be used.

[0034] While this article describes a virtual machine running an operating system as one example of an execution environment, other execution environments are possible. For example, applications, tasks, or other processes can execute within a software "container" that provides a runtime environment but does not itself provide hardware virtualization. Containers can be implemented within virtual machines to provide additional security, or they can run outside of virtual machine instances. Both virtual machine instances and containers can be configured to load and execute background processes.

[0035] Figure 1The diagram illustrates an illustrative operating environment 100 in which a service provider system 110 operates to enable a client device 102 to submit or request user-defined code to be retrieved by an on-demand code execution system 120.

[0036] For illustration, various example client devices 102 are shown communicating with service provider system 110, including desktop computers, laptop computers, and mobile phones. Generally, client device 102 can be any computing device, such as a desktop computer, laptop computer or tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA / mobile phone, mobile phone, e-book reader, set-top box, voice command device, camera, digital media player, etc.

[0037] In general description, data storage service 160 is operable to enable clients to read, write, modify, and delete data, such as files, objects, blocks, or records, each representing a set of data associated with an identifier (“object identifier” or “resource identifier”) that can be interacted with as a separate resource. For example, an object could represent a single file submitted by client device 102 (although data storage service 160 may or may not store such objects as single files). This object-level interaction contrasts with other types of storage services, such as block-based storage where data is manipulated at the block level, or database storage where data manipulation may occur at the table level.

[0038] Data storage service 160 illustratively includes one or more frontends 162 that provide an interface (command-line interface (CLI), application programming interface (API), or other programming interface) through which client devices 102 can interact with service 160 to configure service 160 on behalf of these client devices and perform I / O operations on service 160. For example, client device 102 can interact with frontend 162 to create a collection of data objects (e.g., a “bucket” of objects) on service 160 and configure permissions for that collection. Client device 102 can then create, read, update, or delete objects within the collection based on the interface of frontend 162. In one embodiment, frontend 162 provides a REST-compliant HTTP interface that supports various request methods, each corresponding to a requested I / O operation on service 160. As a non-limiting example, request methods may include: • A GET operation that requests to retrieve an object stored on service 160 by referencing an object's identifier; • A PUT operation that requests storage of an object to be stored on service 160, including the object's identifier and the input data to be stored as the object; • A DELETE operation that requests the deletion of an object stored on service 160 by referencing the object's identifier; and • The LIST operation requests a list of objects stored on service 160 by referencing the identifier of the collection.

[0039] It also supports a variety of other operations.

[0040] During normal operation, frontend 162 can be configured to receive a call to a request method and apply this request method to input data for said method. For example, frontend 162 can respond to a request to put (PUT) this input data as an object into service 160 by storing the input data as an object on service 160.

[0041] Data can be stored, for example, on data storage area 168, which corresponds to any persistent or substantially persistent storage device (including hard disk drive (HDD), solid-state drive (SSD), network accessible storage device (NAS), storage area network (SAN), non-volatile random access memory (NVRAM), or any of the various storage devices known in the art). As a further example, front-end 162 can respond to a request for access to a dataset or a portion thereof from service 160 by retrieving the requested data (e.g., an object representing input data for a GET resource request) from storage area 168 and returning that object to the requesting client device 102.

[0042] In some cases, a call to a request method can invoke one or more native data manipulations provided by service 160. For example, a SELECT operation can provide a query in SQL format to be applied to an object (also identified in the request), or a GET operation can provide a specific range of bytes of an object to be returned.

[0043] Service provider system 110 illustratively includes a caching service 170 configured to cache datasets of code executed by on-demand code execution system 120. Data can be cached on, for example, data caches 172, which correspond to any data storage device, such as hard disk drive (HDD), solid-state drive (SSD), network accessible storage device (NAS), storage area network (SAN), non-volatile random access memory (NVRAM), random access memory (RAM), or various storage devices known in the art. Although shown as separate from and outside of data storage service and on-demand code execution system 120, in some embodiments, caching service 170 may be implemented within or both of data storage service 160 and on-demand code execution system 120 (e.g., on a physical or logical communication system that is part of data storage service 160 or on-demand code execution system 120).

[0044] Client device 102, data storage service 160, and on-demand code execution system 120 can communicate via network 104, which can include any wired network, wireless network, or a combination thereof. For example, network 104 can be a personal area network (PAN), local area network (LAN), wide area network (WAN), over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or a combination thereof. As another example, network 104 can be a publicly accessible network of linked networks that can be operated by various different parties, such as the Internet. In some embodiments, network 104 can be a private or semi-private network, such as a corporate or university intranet. Network 104 can include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. Network 104 can use protocols and components for communication via the Internet or any of the aforementioned types of networks. For example, protocols used by network 104 may include Hypertext Transfer Protocol (HTTP), HTTP Secure (HTTPS), Message Queuing Telemetry (MQTT), Constrained Application Protocol (CoAP), and so on. Protocols and components used for communication via the Internet or any of the aforementioned types of communication networks are well known to those skilled in the art and therefore are not described in more detail herein.

[0045] System 120 includes one or more front-ends 130 capable of interacting with the on-demand code execution system 120. In an illustrative embodiment, front-ends 130 serve as "front doors" to other services provided by the on-demand code execution system 120, enabling users (via client device 102) to provide computer-executable code, request execution of the computer-executable code, and view the results of the computer-executable code. Front-ends 130 include various components for enabling interaction between the on-demand code execution system 120 and other computing devices. For example, each front-end 130 may include a request interface that provides client device 102 with the ability to upload or otherwise transmit user-specified code to the on-demand code execution system 120 and subsequently request execution of that code. In one embodiment, the request interface communicates with external computing devices (e.g., client device 102, front-end 162, etc.) via a graphical user interface (GUI), CLI, or API. Front-ends 130 process requests and ensure that requests are properly authorized. For example, front-ends 130 may determine whether the user associated with the request is authorized to access the user code specified in the request.

[0046] As used herein, references to user code can refer to any program code (e.g., program, routine, subroutine, thread, etc.) written in a particular programming language. In this disclosure, the terms “code,” “user code,” “function code,” “application code,” and “program code” are used interchangeably. Such user code can, for example, be executed in conjunction with specific data transformations developed by the user to achieve a specific function. As noted above, a separate collection of user code (e.g., used to achieve a specific function) is referred to herein as a “task,” and a specific execution of this code (including, for example, compiling, interpreting, or otherwise making the code executable) is referred to as “task execution” or simply “execution.” Tasks can be written in JavaScript (e.g., node.js), Java, Python, or Ruby (and / or another programming language) in a non-limiting manner.

[0047] To manage requests for code execution, frontend 130 may include an execution queue that maintains a record of requested task executions. Illustratively, the number of tasks that the on-demand code execution system 120 can execute simultaneously is limited, and therefore, new task executions initiated at the on-demand code execution system 120 (e.g., via API calls, via calls from executed or currently executing tasks, etc.) can be placed on the execution queue and processed, for example, in a first-in, first-out order. In some embodiments, the on-demand code execution system 120 may include multiple execution queues, such as individual execution queues for each user account. For example, a user of service provider system 110 may wish to limit the rate of task execution on the on-demand code execution system 120 (e.g., for cost reasons). Therefore, the on-demand code execution system 120 may utilize account-specific execution queues to regulate the rate of concurrent task execution for a particular user account. In some instances, the on-demand code execution system 120 may prioritize task executions such that task executions for a particular account or with a specified priority are bypassed or prioritized within the execution queue. In other instances, the on-demand code execution system 120 can execute a task immediately or substantially immediately after receiving a call to the task, and therefore, the execution queue can be omitted.

[0048] The front end 130 may further include an output interface configured to output information about the tasks performed on the on-demand code execution system 120. Illustratively, the output interface may send data about the task execution (e.g., the results of the task, errors related to the task execution, or details of the task execution, such as the total time required to complete the execution, the total data processed by the execution, etc.) to the client device 102 or the data storage service 160.

[0049] In some embodiments, the on-demand code execution system 120 may include a plurality of front-ends 130. In such embodiments, a load balancer may be provided to distribute incoming calls to the plurality of front-ends 130, for example, in a round-robin manner. In some embodiments, the load balancer may distribute incoming calls to the plurality of front-ends 130 based on the location or state of other components of the on-demand code execution system 120. For example, the load balancer may distribute calls to a geographically nearby front-end 130 or a front-end with the capability to serve the call. Where each front-end 130 corresponds to a separate instance of another component of the on-demand code execution system 120 (such as the activity pool 148 described below), the load balancer may distribute calls based on the capacity or load on those other components. In some instances, calls may be distributed deterministically among the front-ends 130 such that a given call to execute a task will always (or nearly always) be routed to the same front-end 130. This can, for example, help maintain an accurate execution record for tasks to ensure that tasks are executed only the expected number of times. For example, calls may be distributed to achieve load balancing among the front-ends 130. Those skilled in the art will appreciate other distribution techniques, such as anycast routing.

[0050] The on-demand code execution system 120 also includes one or more worker managers 140 that manage execution environments, such as virtual machine instances 150 (shown as VM instances 150A and 150B, generally referred to as "VMs") for servicing incoming calls to perform tasks. For example, the worker manager 140 may work with one or more supervisor processes. The supervisor process may be part of the worker manager 140, or there may be supervisor processes associated with other components of the on-demand code execution system (e.g., host devices, virtual machines, etc.). The supervisor process may allocate resources to the execution environment. Additionally or alternatively, the supervisor process may spawn an execution environment with a specific configuration and execute associated startup code (if present). The startup code may include instructions to begin executing background processes. While the following description refers to virtual machine instance 150 as an example of such an environment, embodiments of this disclosure may utilize other environments, such as software containers. Figure 1 In the example shown, each worker manager 140 manages an activity pool 148, which is a group (sometimes referred to as a pool) of virtual machine instances 150 running on one or more physical host computing devices. These virtual machine instances are initialized to execute a given task (e.g., by loading the task's code and any dependency data objects into the instance).

[0051] Although virtual machine instance 150 is described herein as being assigned to a specific task, in some embodiments, instances may be assigned to task groups, such that instances are bound to task groups and any task within said group can be executed within the instance. For example, tasks in the same group may (e.g., based on their security certificates) belong to the same security group, making it possible to execute a task in a container on a specific instance 150 (after another task has been executed in another container on the same instance) without posing a security risk. Tasks may be associated with permissions that cover various aspects of controlling how a task can be executed. For example, task permissions may define which network connections (if any) the task's execution environment can initiate. As another example, task permissions may define what authentication information is passed to the task, thereby controlling which network-accessible resources are available for task execution (e.g., objects on service 160). In one embodiment, a task's security group is based on one or more such permissions. For example, a security group may be defined based on a combination of permissions for initiating network connections and permissions for accessing network resources. As another example, tasks in a group may share common dependencies, allowing for rapid modification of the environment used to execute one task in the group to support the execution of another task within the group.

[0052] Alternatively or alternatively, in some embodiments, the instance may execute a background process. For example, a low level of resources may be allocated to the virtual machine instance to run a background process. The background process may begin execution during the virtual machine's boot process. Alternatively or alternatively, a user may submit startup code to initiate the execution of the background process. The background process may run during and / or after a specific task or group of tasks. For example, a virtual machine instance may be configured to run a background process to monitor the number of items in the cache and send an alert when it reaches a threshold. While this process is running, the front end 130 may receive requests to execute tasks. In some embodiments, the background process may continue running while a task is being processed. In some embodiments, multiple additional tasks may run concurrently while the background process continues to execute. Alternatively or alternatively, multiple requests may be received by the front end 130 at different times. A task associated with each request may run while the background process continues to execute.

[0053] Once the triggering event for executing a task has been successfully processed by frontend 130, frontend 130 passes the request to worker manager 140 for task execution. In one embodiment, each frontend 130 may be associated with a corresponding worker manager 140 (e.g., a worker manager 140 that is co-located with or geographically close to frontend 130), and thus, frontend 130 may pass most or all of the requests to that worker manager 140. In another embodiment, frontend 130 may include a location selector configured to determine the worker manager 140 to which the execution request is passed. In one embodiment, the location selector may determine the worker manager 140 to receive the call based on a hash of the call and distribute the call to the selected worker manager 140 based on the hash value (e.g., via a hash ring). In another embodiment, frontend 130 may determine whether the execution request will be passed to the worker manager 140 or a separate host (e.g., where the application has been instantiated and can receive requests for processing while continuing to execute background processes) based on previous invocations of the application on the host. Various other mechanisms for distributing calls among worker managers 140 will be apparent to those skilled in the art.

[0054] Subsequently, the worker manager 140 can modify the virtual machine instance 150 (if needed) and execute the task code within instance 150. As discussed above, instance 150 may already be running a background process. The worker manager 140 can modify the virtual machine instance to execute the task code without interrupting the execution of the background process. Figure 1 As shown, the corresponding instance 150 may have an operating system (OS) 152 (shown as OS 152A and 152B), a language runtime 154 (shown as runtimes 154A and 154B), and user code 156 (shown as user code 156A and 156B). OS 152, runtime 154, and user code 156 can work together to execute the user code to perform tasks. Therefore, by operating the on-demand code execution system 120, tasks can be executed quickly within the execution environment.

[0055] According to various aspects of this disclosure, each VM 150 further includes assembly code 157, which is executable to facilitate the aggregation of input data on VM 150 and the processing of output data written to VM 150, as well as VM data storage 158 accessible through the local file system of VM 150. Illustratively, assembly code 157 represents a process executing on VM 150 (or potentially a host device of VM 150) and is configured to obtain data from data storage service 160 or caching service 170 and place that data into VM data storage 158. Assembly code 157 may be further configured to obtain data written to a file within VM data storage 158 and send that data to data storage service 160 or caching service 170. Because such data is available at VM data storage 158, user code 156 does not need to obtain data over a network, thereby simplifying user code 156 and allowing user code 156 to further restrict network communication, thus improving security. Conversely, as discussed above, user code 156 can interact with input and output data as files on VM data store 158 by using file handles passed to code 156 during execution. In some embodiments, the input and output data may be stored as files within the kernel space file system of data store 158. In other instances, assembly code 157 may provide a virtual file system, such as a User Space File System (FUSE) interface, which provides an isolated file system accessible to user code 156, thereby restricting user code access to VM data store 158.

[0056] As used herein, the term "local file system" generally refers to a file system maintained within an execution environment, enabling software running within that environment to access data as files, rather than via a network connection. According to various aspects of this disclosure, a data storage device accessible via a local file system can be local (e.g., a local physical storage device) or remote (e.g., accessed via a network protocol such as NFS, or represented as a virtualized block device provided by a network-accessible service). Therefore, the term "local file system" is intended to describe the mechanism by which software accesses data, rather than the physical location of the data.

[0057] VM data store 158 may include any persistent or non-persistent data storage device. In one embodiment, VM data store 158 is a physical storage device of a host device, or a virtual disk drive hosted on a physical storage device of a host device. In another embodiment, VM data store 158 is represented as a local storage device, but is actually a virtualized storage device provided by a network-accessible service. For example, VM data store 158 may be a virtualized disk drive provided by a network-accessible block storage service. In some embodiments, data storage service 160 may be configured to provide file-level access to objects stored on data store 168, thus enabling VM data store 158 to be virtualized based on communication between assembly code 157 and service 160. For example, data storage service 160 may include a file-level interface that provides network access to objects as files within data store 168. The file-level interface may, for example, represent a network-based file system server (e.g., a network file system (NFS)) that provides access to objects as files, and assembly code 157 may implement a client of this server, thus providing file-level access to objects of service 160.

[0058] In some instances, VM data store 158 may represent virtualization access to another data store executed on the same host device as VM instance 150. For example, activity pool 148 may include one or more aggregated VM instances ( Figure 1 (Not shown in the image), these bundled VM instances can be co-leased with VM instances 150 on the same host device. Bundled VM instances can be configured to support retrieving and storing data from service 160 (e.g., data objects or portions thereof, input data passed by client device 102, etc.), and storing that data on a data store of the bundled VM instance. For example, a bundled VM instance can be designated as unavailable for supporting the execution of user code 156, and therefore has elevated privileges relative to instance 150 supporting user code execution. Bundled VM instances can allow other VM instances 150 within their host device (or potentially on a nearby host device) to access the data, such as by using a network-based file protocol (e.g., NFS). These other VM instances 150 can then act as clients of the bundled VM instance, enabling the creation of a virtualized VM data store 158, which, from the perspective of user code 156A, appears as a local data store. Advantageously, given that the data aggregation VM and VM instance 150 are both located on or near the host device, network-based access to the data stored at the data aggregation VM can be expected to occur very quickly.

[0059] While this document provides some examples of reading from or writing to VM data store 158 using IO stream handles, IO streams can also be used to read from or write to other interfaces of VM instance 150 (while still requiring user code 156 to perform operations other than stream-level operations, such as creating network connections). For example, assembly code 157 can "piped" input data as an input stream to the execution of user code 156, and its output can be "piped" as an output stream to assembly code 157. As another example, the hypervisor assembling VM instance or VM instance 150 can pass input data to the network port of VM instance 150, which can be read by assembly code 157 and passed as an input stream to user code 157. Similarly, data written to an output stream by task code 156 can be written to a second network port of instance 150A for retrieval by the assembly VM instance or hypervisor. In yet another example, the hypervisor for instance 150 can pass input data as data to be written to a virtualized hardware input device (e.g., a keyboard), and assembly code 157 can pass a handle to the I / O stream corresponding to this input device to user code 156. Similarly, the hypervisor can pass a handle to the I / O stream corresponding to a virtualized hardware output device to user code 156 and read data written to this stream as output data. Therefore, the examples provided herein with respect to file streams can generally be modified to relate to any I / O stream.

[0060] Data storage service 160, caching service 170, and on-demand code execution system 120 are in Figure 1 It is described as operating in a distributed computing environment, which includes the use of one or more computer networks ( Figure 1 Many interconnected computer systems (not shown). Data storage service 160, caching service 170, and on-demand code execution system 120 can also be connected in a manner that is more efficient than... Figure 1 The devices shown operate within a computing environment with fewer or more devices. Therefore, it should be... Figure 1The descriptions of data storage service 160, caching service 170, and on-demand code execution system 120 herein are intended to be illustrative and not limiting of this disclosure. For example, on-demand code execution system 120, or various configurations thereof, may implement various web service components, hosting or “cloud” computing environments, or peer-to-peer network configurations to implement at least a portion of the processes described herein. In some instances, data storage service 160, caching service 170, and on-demand code execution system 120 may be combined into a single service. Further, data storage service 160, caching service 170, and on-demand code execution system 120 may be implemented directly in hardware or software executed by hardware devices, and may include, for example, one or more physical or virtual servers implemented on physical computer hardware configured to execute computer-executable instructions for performing the various features described herein. The one or more servers may be geographically distributed or geographically co-located, for example, in one or more data centers. In some cases, the one or more servers may operate as part of a system of rapidly provisioned and released computing resources (often referred to as a “cloud computing environment”).

[0061] exist Figure 1 In the example, data storage service 160, caching service 170, and on-demand code execution system 120 are shown connected to network 104. In some embodiments, any component within data storage service 160, caching service 170, and on-demand code execution system 120 may communicate with other components of on-demand code execution system 120 via network 104. In other embodiments, not all components of data storage service 160, caching service 170, and on-demand code execution system 120 may be able to communicate with other components of virtual execution environment 100. In one example, only front-ends 130 and 162 (which may represent multiple front-ends in some cases) of on-demand code execution system 120 and data storage service 160 may be connected to network 104, and other components of data storage service 160 and on-demand code execution system 120 may communicate with other components of environment 100 via their respective front-ends 130 and 162.

[0062] While this document uses individual components of the data storage service 160, caching service 170, and on-demand code execution system 120 to describe some functionalities, other components or combinations of components can also implement such functionalities, either individually or in combination. Therefore, Figure 1 The specific configuration intent of the components is illustrative.

[0063] Figure 2 Depicting Implementation Figure 1 The overall architecture of the computing device of front-end server 200 with front-end 162. Figure 2The overall architecture of the front-end server 200 described herein includes the arrangement of computer hardware and software that can be used to implement various aspects of this disclosure. The hardware can be implemented on physical electronic devices, as discussed in more detail below. The front-end server 200 may include more than Figure 2 The diagram shows more (or fewer) components. However, it is not necessary to show all of these generally conventional components in order to provide a disclosure that can be implemented. Additionally, Figure 2 The overall architecture shown can be used to implement Figure 1 One or more of the other components shown.

[0064] As shown in the figure, the front-end server 200 includes a processing unit 290, a network interface 292, a computer-readable media drive 294, and an input / output device interface 296, all of which can communicate with each other via a communication bus. The network interface 292 provides connectivity to one or more networks or computing systems. The processing unit 290 can therefore receive information and instructions from other computing systems or services via network 104. The processing unit 290 can also communicate with main memory 280 or secondary memory 298 and further provide output information to an optional display (not shown) via the input / output device interface 296. The input / output device interface 296 can also accept input from optional input devices (not shown).

[0065] The main memory 280 or secondary memory 298 may contain computer program instructions (grouped into units in some embodiments) that are executed by the processing unit 290 to implement one or more aspects of this disclosure. These program instructions are... Figure 2 The data is shown as being included within main memory 280, but may be additionally or alternatively stored within secondary memory 298. Main memory 280 and secondary memory 298 correspond to one or more layers of memory devices, including (but not limited to) RAM, 3D XPOINT memory, flash memory, magnetic storage devices, etc. For descriptive purposes, it is assumed that main memory 280 represents the main working memory of worker manager 140, which is faster but has a smaller total capacity than secondary memory 298.

[0066] Main memory 280 may store operating system 284, which provides computer program instructions used by processing unit 290 in the general management and operation of front-end server 200. Memory 280 may also include computer program instructions and other information for implementing various aspects of this disclosure. For example, in one embodiment, memory 280 includes user interface unit 282, which generates a user interface (or instructions for that user interface) for display on the computing device, for example, via a navigation or browsing interface such as a browser or application mounted on the computing device.

[0067] Memory 280 may include control plane unit 286 and data plane unit 288, each executable to implement aspects of this disclosure. Illustratively, control plane unit 286 may include code executable to define or commit function code to be executed. Data plane unit 288 may illustratively include code enabling I / O operations on data storage service 160 or caching service 170, including retrieving datasets, generating data references for other functions to access datasets, caching datasets, etc.

[0068] Figure 2 The front-end server 200 is an illustrative configuration of such a device, although other configurations are possible. For example, although shown as a single device, in some embodiments, the front-end server 200 may be implemented as a logical device hosted by multiple physical host devices. Illustratively, a first device of such a front-end server 200 may implement control plane unit 286, while a second device may implement data plane unit 288.

[0069] Although Figure 2 The description is for a front-end server 200, but similar components can be used in some implementations. Figure 1 Other apparatuses shown in environment 100. For example, similar apparatuses may implement worker manager 140, as described in more detail in U.S. Patent No. 9,323,556 (“'556 Patent”), filed September 30, 2014, entitled “Programmable Event Detection and Message Generation for Requests to Execute Program Code”, the entire contents of which are hereby incorporated by reference.

[0070] Example lifecycle of a virtual machine

[0071] Figure 3 Example routine 300 is shown, which is used to manage computing resources allocated to a virtual execution environment (e.g., a virtual machine instance, container, or other virtual computing component) during a request / response cycle. The virtual execution environment described herein can be warmed up or pre-initialized to run background processes, as referenced above. Figure 1 As described. To recap, a virtual execution environment can be allocated a low non-zero level of computing power and can execute background processes. The execution routine begins at box 302.

[0072] At box 304, the on-demand code execution system (e.g., Figure 1The on-demand code execution system 120 can access the request. The request can be made from a client device (e.g., Figure 1 The request is submitted by the client device 102. Once submitted, the request can be sent by the frontend (e.g., ...). Figure 1 The request may be processed by the front-end (130). Alternatively, the request may also be received by a request broker of the virtual execution environment. The request broker may be a component of the virtual execution environment configured to monitor requests from client devices. The request broker may subsequently communicate with other components of the virtual execution environment to generate a response to each request. For example, requests received from client devices may be stored. For example, requests received from client devices may be stored in a queue. The request broker may retrieve and / or remove requests from the queue. The request broker may then determine what actions need to be taken to generate a response to the request. For example, the request broker may determine that the level of computing power allocated to the virtual execution environment needs to be increased and that certain functions should be invoked to obtain a response. The request broker may then generate instructions to take these actions, as will be discussed in detail at boxes 306 and 308.

[0073] At box 306, the on-demand code execution system can adjust the computing resources allocated to the virtual execution environment. In some embodiments, the adjustment may involve increasing the level of computing power allocated to the virtual execution environment. Such an adjustment can be referred to as “de-restraining” the computing resources, in contrast to “restraining” the computing resources (where the adjustment involves decreasing the level of computing power allocated to the virtual execution environment). For example, a virtual execution environment allocated a lower non-zero level of computing power can be de-restrained to a higher level of computing power, where the higher level of computing power is greater than the lower non-zero level of computing power. In some embodiments, this action can be performed by the request broker described at box 304. For example, in some embodiments, the request broker can communicate with a supervisor process to de-restrain the resources allocated to the virtual execution environment. Of course, in some embodiments, the supervisor process can access and / or remove requests from a queue and de-restrain the computing resources used for the virtual execution environment in response to the request.

[0074] In some embodiments, the lower non-zero level of computing power and the higher non-zero level of computing power can be specified by the client device. For example, the client device can provide configuration data for the virtual execution environment, including application and function code and the lower non-zero level of computing power. In some embodiments, the configuration data can also include a higher level of computing power greater than the lower level of computing power. The higher level of computing power can be configured to be sufficient to allow the execution of the invoked function or application. The application configuration data can also specify background processes and request / response processes. In another embodiment, the configuration data can define a range of computing resources bounded by the lower non-zero level of computing power and the higher level of computing power. In some embodiments, the configuration data can specify an intermediate level for each function or application configured to run by the virtual execution environment, wherein the intermediate level falls within the range of computing resources. In some embodiments, the lower non-zero level of computing power can be configured to achieve a minimum non-zero level. The higher level of computing power can also be configured to achieve a maximum level of computing power. In some embodiments, the lower non-zero level and the higher non-zero level can be provided in communications with the client device, such as configuration data. The client device can configure the lower non-zero level of computing power to the minimum non-zero level and the higher level of computing power to the maximum level.

[0075] In some embodiments, a higher level of computing power can be specified by a supervisor process. For example, a request broker or supervisor process can determine that an application needs to be invoked to generate a response to a request. The supervisor process can then determine a higher level of computing power based on the amount of resources required to run the invoked application. The supervisor process can configure this higher level of computing power to the maximum level. This determination can be based on past data relating to the computing resources allocated when the application successfully generated the requested response, rather than the computing resources allocated when the application failed to generate the requested response. While the preceding example specified an invoked application, the same process can also be applied to invoked functions. The process can also be applied to each function or application configured to run in a virtual execution environment.

[0076] As another example, the supervisor process can determine a higher level of computing power for a virtual execution environment based on resources allocated to other virtual execution environments under its supervision. For example, the supervisor process can be a component of a host device included in an on-demand code execution system. The host device can have a certain amount of computing resources, such as memory and / or network bandwidth, available for allocation to virtual execution environments that can run on the host device. The supervisor process can determine how these computing resources should be allocated to the virtual execution environments running on the host device. For example, the host device can have available computing resources including 10 megabits per second (Mbps) of network bandwidth and 10 vCPUs of processing power, where each vCPU can be equivalent to a CPU core. Alternatively, each vCPU can be equivalent to CPU time, which can be allocated as processing time on available processing resources. If 10 virtual execution environments are running on the host device, each device can be allocated 1 Mbps of network bandwidth and 1 vCPU.

[0077] In some embodiments, each execution environment may run background processes and is initially allocated a low non-zero level of computing power from the available computing resources of the host device. The remaining portion of the available computing resources can be allocated as needed to enable task execution in the virtual execution environments. For example, recalling the previous example, 10 virtual machines may run on the host device, which has available computing resources including 10 Mbps of network bandwidth and 10 vCPUs of processing power. The low non-zero level for each environment could be 0.1 Mbps and 0.1 vCPUs. Therefore, the remaining portion of the available computing resources could be 9 Mbps and 9 vCPUs. Typically, the computing power of a virtual execution environment can have a higher level of computing power than the remaining portion of the available computing resources (e.g., 2 Mbps or 2 vCPUs). However, the total higher level of computing power used for each virtual environment on the host computing device can exceed the remaining portion of the available computing resources (e.g., if 10 virtual execution environments have a total higher level of 12 vCPUs). However, such scenarios (which may be called oversubscription) can be managed so as not to negatively impact the virtual execution environment (e.g., by provisioning virtual execution environments that are unlikely to require their correspondingly high levels of computing power at the same time).

[0078] In some embodiments, a user can configure a virtual execution environment to have a higher level of computing power than the total available computing resources (e.g., a higher level of 9.1 Mbps and 9.1 vCPUs in the example above). If the requesting agent generates instructions to deconstrain (e.g., increase) the resources associated with the virtual execution environment to a higher level than the total available level, the supervisor process can allocate the remaining portion of the available computing resources to this virtual execution environment.

[0079] Although the above examples have the same lower non-zero level for each virtual execution environment (VEX), this lower non-zero level can vary between VEXs. The allocation of available computing resources can be transferred between VEXs on the host device, provided that the amount of computing resources allocated to all VEXs on the host device is within the range of available computing resources on the host device, and each VEX is allocated at least the lower non-zero level of computing power required for that VEX. For example, VEXs can be configured such that the computing resources allocated to a virtual machine fall within a range defined by a lower non-zero level and a higher level. Furthermore, multiple VEXs can be deregulated simultaneously, provided that the amount of computing resources allocated to all VEXs on the host device is within the range of available computing resources on the host device.

[0080] In a non-limiting embodiment, the computing resources used for the virtual execution environment may not be deregulated, since the computing resources have already been deregulated to actively process another request. For example, the first request may be accessed and / or removed from the queue by the request broker. Based at least in part on the request, the request broker may instruct the supervisor process to deregulate resources to a higher level and / or invoke the application to generate a first response to this request. While the virtual execution environment generates the first response, the request broker may access and remove the second request from the queue. Since computing resources are still deregulated to a higher level to generate the first response to the first request, no further deregulation of resources is required to generate a second response to the second request.

[0081] In some embodiments, the supervisor process may be a component of a worker manager (e.g., worker manager 140) and determines the level of compute capability allocated to each virtual execution environment (e.g., virtual machine, container, etc.) hosted by the worker manager. For example, the worker manager may have access to a certain amount of compute resources. These compute resources can be used to allocate to the virtual execution environments hosted by the worker manager. Each virtual execution environment may be allocated a lower non-zero level of compute capability. Each virtual execution environment may be allocated additional compute resources from the remainder of the available compute resources accessible to the worker manager, as described above in the example relative to the host device. In some embodiments, other components of the on-demand code execution system (e.g., host device, virtual machine, etc.) may include the supervisor process. The supervisor process may have access to available compute resources for the worker manager, host device, or similar entity. The amount of available compute resources may vary based on the needs of other virtual execution environments of the worker manager, host device, or similar entity. The supervisor process may be configured to request the allocation of compute resources greater than the minimum non-zero level required to generate a response to the request. For example, a higher level of compute capability may be defined in the configuration data for the virtual execution environment. The supervisor process can request higher levels of computing power from the host device to generate a response to the request. The host device or a component of the host device can be configured to provide higher levels of computing power to the virtual execution environment (if available). If higher levels of computing power are unavailable, the host device or a component of the host device can be configured to allocate available resources to the virtual execution environment.

[0082] At block 308, the on-demand code execution system can invoke the execution of an application or function on a virtual execution environment. In some embodiments, the on-demand code execution system can invoke the execution of multiple applications and / or functions in response to a request from a client device. In another embodiment, responses to each request can be generated in the order they are received. For example, a first request can be received from a client device. To generate a first response, the on-demand code execution system can invoke one or more functions. In embodiments with multiple functions, the functions can be executed in a predetermined order to complete a set of processing tasks and generate a first response. Alternatively or additionally, the on-demand code execution system can invoke an application to perform various processing tasks and generate a first response. The application can invoke one or more functions to perform these tasks. A second request can be received from the client device while the first response is being generated. The on-demand code execution system can wait until the first response has been generated or until a set response period has expired before invoking a function or application to generate a second response.

[0083] In some embodiments, responses to multiple requests can be generated simultaneously. For example, the on-demand code execution system can receive a first request from a client device and invoke a function or application to generate a first response to that request. While generating the first response, the on-demand code execution system can receive a second request from the client device. The on-demand code execution system can invoke a function or application process to generate the second response without waiting until the first response has been generated or a set response timeout has expired.

[0084] In some embodiments, requests can be executed in order of importance. In some embodiments, each request can include data indicating its importance relative to the execution of other requests. For example, a high-importance request can include an exclamation mark or the text "High Importance". A medium-importance request can include an exclamation mark or the text "Medium Importance", and a low-importance request can include an exclamation mark or the text "Low Importance". In a non-limiting embodiment, a first medium-importance request, a second high-importance request, and a third low-importance request can be received simultaneously or sequentially. The on-demand code execution system can generate a first response to the second request, followed by a second response to the first request, and then a third response to the third request.

[0085] At boxes 310 and 312, the on-demand code execution system can determine whether the first response has been generated by the invoked function or application. This can be determined by the request broker's receipt of the first response. In some embodiments, the virtual execution environment used to generate the first response can send the first response to the request broker when the first response is generated. The request broker can then forward the response to the client device. The client device can be the client device that initiated the request. Alternatively, the client device that initiated the request can belong to a group of client devices with a shared unique identifier. In some embodiments, the request broker can forward the first response to all client devices in this group. In other embodiments, the request broker can forward the first response to a subset of client devices in this group.

[0086] Alternatively or alternatively, after invoking a function or application as described above with respect to box 308, the request broker can determine whether a response has been generated by monitoring the execution of the function or application and determining whether it has executed successfully. If it has executed successfully, the request broker can retrieve the response and forward it to the client device. Alternatively or alternatively, the supervisor process can obtain the response from the function / application and then provide it to the request broker. Once the response is obtained, the on-demand code execution system can proceed to box 314 to determine whether the attached invoked function or application is still executing. In some embodiments, the application may not generate a response upon completion of a particular function or task, or the response may be an acknowledgment or completion message not intended to be forwarded to the client device (e.g., an acknowledgment message for logging purposes only).

[0087] If the requesting agent does not receive or receive a response, the on-demand code execution system may repeat boxes 310 and 312 until a response is received. Alternatively, the on-demand code execution system may repeat boxes 310 and 312 for a specified time period. If no response is received or received during this specified time period, the on-demand code execution system may proceed to box 314 to determine whether the invoked function or application is still executing. In some embodiments, the specified time period is provided in configuration data for the virtual execution environment. In some embodiments, the on-demand code execution system may have a standard specified time period for all virtual execution environments. In some embodiments, the specified time period may be 15 minutes.

[0088] At box 314, the on-demand code execution system can determine whether an append function or application is still executing after a set timeout period following the receipt of the first response or the acquisition of the first response. The append function or application may be associated with the generation of the first response. For example, the append function may be running to delete intermediate data created during the generation of the first response. Alternatively, the append function or application may have already been invoked by the on-demand code execution system during the generation of the second response, which is generated after the receipt of the second request.

[0089] After determining that no additional functions are being added or that the application is still executing, the on-demand code execution system can determine whether additional requests have been received. For example, the request broker can check the queue used to store received requests. If the queue is empty, the on-demand code execution system can proceed to conserve compute resources at box 316, where conserving compute resources reduces compute capacity to a lower non-zero level.

[0090] At box 316, the on-demand code execution system can restrict computing resources used for the virtual execution environment to a lower non-zero level. In some embodiments, the lower non-zero level can be specified by the user. For example, the user can transmit the lower non-zero level to the on-demand code execution system as part of a configuration instruction. In some embodiments, the user can configure the lower non-zero level as a minimum non-zero level in the configuration instruction. Alternatively, the user can transmit the lower non-zero level to the on-demand code execution system as part of a request. In some embodiments, the user can configure the lower non-zero level as a minimum non-zero level in the request. In some embodiments, the user can be associated with a customer account. For example, a customer account can be associated with a unique identifier. Each user associated with this customer account can share the unique identifier. Therefore, the on-demand code execution system can accept communication associated with a unique identifier, even if the communication comes from multiple users.

[0091] A lower non-zero level is sufficient for background processes in the virtual computing environment to continue execution. For example, the virtual execution environment may be executing background processes at the very beginning. The virtual execution environment can continue executing background processes while a request for an on-demand request is being generated. After a response is received or the set timeout period for generating a response has expired, the on-demand code execution system can check for additional requests, as discussed above with respect to box 314. If no request exists, the request broker can communicate with the supervisor process to confine the virtual execution environment's resources to a lower non-zero level.

[0092] At box 318, the on-demand code execution system can remain at a low non-zero level until the next request is received. As discussed above, background processes may continue to run. For example, background processes may include, but are not limited to: monitoring the data storage device for changes in the number of items in the data storage device, or monitoring a queue containing user requests to determine if a threshold number of requests has been reached. When a new request is received, boxes 302 through 318 can be repeated to generate a response to that request.

[0093] In some embodiments, the background process may communicate with another computing system during execution. For example, the background process may be used to monitor external storage locations (e.g., cloud storage). In a non-limiting example, the background process may be configured to run at intervals to determine if a new item (e.g., a file) exists in the external storage location. If a new item is found, the background process may process the file, move the file to another location, transmit a message to a client device, or perform some other operation. If a message is sent to a client device, the client device may take another action in response to the message. For example, the client device may transmit a new request to an on-demand code execution system in response to this message. Boxes 302 through 318 may be repeated to generate a response to this request.

[0094] In some embodiments, the background process may terminate. For example, the server including the virtual execution environment may fail. Another example is a virtual execution environment timeout. For example, the virtual execution environment may be hosted by a host device. The host device may host the virtual machine for a predetermined period of time. In a non-limiting embodiment, the predetermined period of time may be 6 hours. The background process may also stop executing when the client deletes the application. For example, the client may send a request to the front end stating that they no longer wish to have the virtual execution environment hosted on the host device.

[0095] Figure 4 A diagram illustrates an interaction 400 between a client device and components of the on-demand code execution system 406 during the processing of a first request. The client device 402 can be any computing device, such as a desktop computer, laptop or tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA / mobile phone, mobile phone, e-book reader, set-top box, voice command device, camera, digital media player, etc. In some embodiments, the client device 402 can be... Figure 1 One of the client devices in client device 102.

[0096] At [1], client device 402 may send a request. The request may include one or more tasks that client device 402 requests to be executed to generate a specific response. Alternatively, the request may include data indicating the importance of the request, such as a text string with the phrase "high importance". In some embodiments, the request may include additional data required for task execution. For example, the request may include the level of computing power required to provide a response to the request. The computing power level may be between a lower non-zero level of computing power specified in the configuration data for virtual execution environment 414 and a higher level of computing power. In some embodiments, virtual execution environment 414 may be a container for a client application, wherein the client application may include functions that can be invoked by request broker 412. The request may also include data locations that on-demand code execution can access any additional data required for task execution.

[0097] Client device 402 can transmit requests to a frontend, such as frontend 404. Frontend 404 can be located outside the on-demand code execution center. Alternatively, client device 402 can transmit requests to a data storage device, and frontend 404 can retrieve requests from this data storage device. In some embodiments, the data storage device can be a request queue, such as... Figures 5A to 5BThis is discussed in more detail in [2]. Then, the front end 404 can forward the request at [2] to a component of the on-demand code execution system 406, such as request broker 412. Alternatively, the front end 404 can transmit the request to a data storage device (such as a queue), from which request broker 412 can retrieve the request.

[0098] In some embodiments, the on-demand code execution system 406 may include one or more front-ends (e.g., Figure 1 The front end 130 (e.g., client device 402) can transmit a request to the on-demand code execution system. The front end 130 can receive and process the request before providing it to another component of the on-demand code execution system. Alternatively, client device 402 can transmit the request to a data storage device such as a queue, and the front end 130 can retrieve the request from this data storage device. The front end 130 can then transmit the request to another component of the on-demand code execution system 406, including but not limited to host device 408, virtual machine 410, request broker 412, or virtual execution environment 414. For example, the front end 130 can transmit the request to request broker 412. Alternatively, the front end 130 can transmit the request to a data storage device (such as a queue), and request broker 412 can retrieve the request from this queue.

[0099] In some embodiments, both an external frontend and an internal frontend can be used to receive requests from client device 402. For example, the client device can send a request to external frontend 404 at [1]. Frontend 404 can forward the request at [2] to the on-demand code execution system at [2], and the on-demand code execution system can do so through one or more internal frontends (e.g., Figure 1 The front end 130 receives the request. The request can be forwarded to a component of the on-demand code execution system for processing. For example, the request can be forwarded to the request broker 412 for processing. Alternatively, the internal front end can transmit the request to a data storage device (such as a queue), and the request broker 412 can retrieve the request from this queue.

[0100] The request agent can be a component of virtual machine 410, which is hosted on host device 408, and host device 408 is a component of on-demand code execution system 406. The request agent can generate instructions for one or more virtual execution environments (VEX) of virtual machine 410. For example, request agent 412 can access a request as described above, determine whether or not a response can or should be generated by VEX 414, and generate instructions for VEX 414 based on the request. VEX 414 may be running background processes using the minimum non-zero level of computing power allocated to VEX 414.

[0101] In some embodiments, virtual execution machine 410 may be an execution environment for a specific application, and the virtual machine may include one or more virtual execution environments 414 for the application, such as one or more containers for a specific application. The request broker may determine, at least in part, based on the request, that a specific application needs to be invoked to generate a response to the request. Alternatively, the request broker may determine, at least in part, based on the request, that a specific function configured to run in a particular pre-initialized virtual execution environment needs to be invoked to generate a response to the request. After making this determination, the request broker 412 may generate instructions for this particular virtual execution environment, wherein the instructions may include instructions for deregulating the compute resources of the virtual execution environment and for invoking the required functions and / or applications. For example, the request broker 412 may generate instructions for deregulating compute resources and invoking functions or applications of virtual execution environment 414 to generate a response to the request. In some embodiments, the request broker 412 may generate instructions for more than one virtual execution environment. For example, the request broker may determine that more than one virtual execution environment (e.g., more than one container) of the application is required to generate a response to the request. Alternatively, the request broker can receive multiple requests, each of which needs to be configured to execute as a function and / or application on a different virtual execution environment. Therefore, the request broker 412 can generate instructions for multiple virtual execution environments to generate responses to the multiple requests. Instructions for deregulating computing resources can be directed to the supervisor process, as described above relative to... Figure 3 As described in box 306. In some embodiments, the supervisor process may be a component of the on-demand code execution system 406 and control the allocation of resources to the host device (such as host device 408) of the on-demand code execution system 406.

[0102] In some embodiments, a supervisor process may exist, which is a component of the host device 408 and controls the resources allocated to virtual machines (such as virtual machine 410 hosted on the host device 408). The host's supervisor process may operate as a supplement to the supervisor process of the on-demand code execution system 406. Of course, the host device 408's supervisor process may be the sole supervisor process for controlling resource allocation.

[0103] In some embodiments, a supervisor process may exist, which is a component of the virtual machine 410 and controls the resources allocated to the virtual execution environment within the virtual machine 410. This supervisor process may operate as a supplement to the supervisor process of the host device 408 and / or the on-demand code execution system 406. Of course, the supervisor process of the virtual machine 410 may be the sole supervisor process for controlling resource allocation.

[0104] The request agent 412 of virtual machine 410 can work alone or with the supervisor process to generate a response to a request. For example, request agent 412 can access the request. Based at least in part on this request, request agent 412 can send instructions to the supervisor process of virtual machine 410 to deregulate computing resources. The supervisor process can deregulate computing resources as indicated by the dashed line in [3], where deregulating computing resources means increasing the allocation to the virtual execution environment (e.g., Figure 4 The virtual execution environment 414 contains computing resources. For example, when the supervisor process receives an instruction to deregulate computing resources, it can allocate a higher level of computing power to the virtual execution environment 414. This higher level of computing power can be configured as the maximum level of computing power. Of course, the supervisor process can also allocate an intermediate level of computing power between a lower non-zero level of computing power and a higher level of computing power, as described above relative to... Figure 3 The discussion is in box 306. Alternatively, the request agent 412 can directly deregulate the computing resources at [3] without using a separate supervisor process to do so.

[0105] In some embodiments, the supervisor process of virtual execution machine 410 may communicate with the supervisor process of host device 408 and / or the supervisor process of on-demand code execution system 406 to request a higher level of computing power to be allocated to virtual execution environment 414. The higher level of computing power may be defined in configuration data for virtual execution environment 414. In some embodiments, the higher level of computing power may be configured to a maximum level of computing power. Illustratively, the configuration data may configure the higher level of computing power to a maximum level of computing power. Once the resources allocated to virtual execution environment 414 are increased to a higher level of computing power, request agent 412 may generate instructions for invoking a function or application on virtual execution environment 414 at [4]. Of course, in some embodiments, request agent 412 may deconstrain computing resources at [3] and simultaneously generate instructions for invoking a function or application at [4].

[0106] The invoked function or application may generate a response to the request and forward this response to the request broker 412 at [5]. Alternatively, the supervisor process may receive the response to the request from the invoked function or application and forward this response to the request broker 412 at [5].

[0107] The request agent 412 may receive the requested response transmitted by the virtual machine or a component of the virtual machine at [5]. Once the response is received, in some embodiments, the request agent 412 may begin to restrict the computing resources to a lower non-zero level at [6]. For simplicity, the lower non-zero level may also be referred to as a lower level. Of course, in other embodiments, the request agent 412 may first forward the response to the front end 404 or the client device 402 at [7] before beginning to restrict the computing resources to a lower non-zero level at [6].

[0108] In some embodiments, the virtual execution environment may transmit the response to a data storage device. For example, virtual execution environment 414 may include an internal data storage device to store responses generated for multiple requests. Request broker 412 may obtain a response from this data storage device for a request sent by client device 402 at [1]. Alternatively, virtual execution environment 414 may transmit the response to a data storage device of on-demand code execution system 406 or a component of on-demand code execution system 406 (including, but not limited to, host device 408, virtual machine 410). Request broker 412 may obtain a response from this data storage device at [6]. Upon receiving the response, request broker 412 may forward the response to frontend 404 at [7]. Frontend 404 may then forward the response to client device 402, which receives the forwarded response at [8]. In some embodiments, virtual execution environment 414 may transmit the response to frontend 404, or otherwise send the response to client device 402, without using request broker 412.

[0109] Although the above is relative to Figure 4 The descriptive interactions discussed are in relation to generating a response to a single request, but multiple requests can also be handled. For example, client device 402 can send multiple requests simultaneously to frontend 404 at [1]. Frontend 404 can forward these requests to on-demand code execution system 406 at [2]. Frontend 404 can sort these requests before forwarding them to on-demand code execution system 406 at [2]. For example, each request can contain an indication of importance. For example, a request can include text strings such as “high importance”, “medium importance”, “low importance”. Alternatively, a request can include numbers generated by the client to indicate their importance. For example, a request including smaller numbers may be less important than a request including larger numbers. Frontend 404 does not forward requests simultaneously as is, but sends them in order of importance. Of course, frontend 404 can generate a table indicating the order in which requests should be executed and forward this table along with the requests to on-demand code execution system 406 at [2]. In addition, the frontend of on-demand code execution system 406 (e.g., Figure 1 The front end 130 can sort requests. For example, the front end of the on-demand code execution system 406 can queue requests according to the importance ranking mentioned above. Other components of the on-demand code execution system 406 (including but not limited to host device 408, virtual machine 410, and request broker 412) can sort requests in the manner described above relative to the front end 404.

[0110] In some embodiments, client device 402 may send additional requests while the request is being executed on the on-demand code execution system 406. For example, client device 402 may send a first request to front-end 404 at [1]. Front-end 404 may forward the request to on-demand code execution system 406. The request may be accessed by request broker 412. For example, on-demand code execution system 406 or one or more components of 406 may place the first request into a temporary data storage device (such as a cache or queue), and request broker 412 may retrieve the first request from this data storage device. Once the first request is retrieved by request broker 412, request broker 412 may begin to deconsolidate computing resources to a level of computing power greater than a lower non-zero level. This level may be a higher level of computing power, or an intermediate level of computing power between a higher level of computing power and a lower level of non-zero computing power. Request broker 412 may subsequently or simultaneously use virtual execution environment 414 to invoke functions or applications to generate a first response to the first request. While these functions or applications are being processed, request broker 412 may access a second request in the manner described above with respect to the first request. Request agent 412 may begin to deconsolidate additional computing resources to generate a second response to the second request. Alternatively, request agent 412 may not deconsolidate additional computing resources to generate a second response to the second request. For example, in some embodiments, computing resources used for virtual execution environment 414 may already be deconsolidated to a higher level of computing power, and no further deconsolidation may be necessary. In any case, request agent 412 may invoke a function or application to generate a second response to the second request and transmit the first and second responses back to client device 402. Request agent 412 may transmit the first response before transmitting the second response, or, in some embodiments, request agent 412 may transmit the responses simultaneously. For example, request agent 412 may receive the first and second responses from virtual execution environment 414 simultaneously, rather than receiving the first response first and then the second response, because in some embodiments, the first response takes longer to generate than the second response. Alternatively, request agent 412 may hold the responses for a period of time and transmit the responses received during this period in batches to frontend 404 or client device 402. In some embodiments, the request agent 412 may wait for a set period of time until no further requests are received before delimiting the computing resources used for the virtual execution environment 414 to a lower non-zero level at [6].

[0111] Figure 5A Example interactions 500 sets between components are shown, where queue 502 is used to handle multiple requests received from client device 402. Figure 5AThe virtual execution environment 414 shown can initially be allocated a low non-zero level of computing power and initialized to execute background processes. Responses to additional requests received through the processes described below can be generated not only by executing background processes, but also by executing the invoked functions and applications.

[0112] Return to Figure 5A The terms "a queue" or "multiple queues" are used herein in their usual and customary meaning in the field of computer technology, and refer to a linear data structure that is open at both ends to allow for execution of operations in the order of receipt. In some embodiments, each virtual execution environment 414 may have its own queue. For example, requests to be executed by invoking functions or applications on virtual execution environment 414 may be stored in the same queue. Alternatively, queues may also be provided for larger components of the on-demand code execution system (including, but not limited to, host devices, worker managers, or virtual machines), such as... Figure 4 As described. Queues for larger components can be handled by the front end of the on-demand code execution system 406 (e.g., Figure 1 The front-end 130) is processed and broken down into smaller queues for smaller components (including but not limited to virtual execution environments, such as virtual execution environment 414). As discussed above, virtual execution environment 414 may include containers.

[0113] In some embodiments, the client device 402 can be with Figure 1 The client device 402 is identical to the client device 102. The client device 402 can add a request to the queue 502 at any time after the data structure for the queue 502 is created. In some embodiments, the data structure for the queue 502 can be created when bootstrapping the virtual execution environment 414. In some embodiments, requests can be received simultaneously by the queue 502. For example, the client device 402 can transmit multiple requests simultaneously to add to the queue. In some embodiments, simultaneously received requests can be queued in a random order. However, in some embodiments, simultaneously received requests can be queued in order of importance. For example, a first request and a second request can be received simultaneously by the on-demand code execution system or a component of the on-demand code execution system. The first request may include an indication of high importance, such as the text string "high importance". The second request may include an indication of low importance, such as the text string "low importance". Based at least in part on these importance indications, the first request may be stored in the queue 502 before the second request.

[0114] In embodiments where client device 402 includes multiple client devices, the client devices 402 may share a unique identifier, which can be used to configure the handling of requests from client device 402. For example, requests from client devices sharing the unique identifier may be added to the same queue. Responses generated for these requests may be transmitted to all client devices sharing the unique identifier. Of course, in some embodiments, responses generated for these requests may be transmitted to a subset of client devices sharing the unique identifier. In some embodiments, a request may identify a subset of client devices to which a response should be provided.

[0115] Request agent 412 may remove a request from queue 502 at [1]. For example, request agent 412 may retrieve request A from queue 502 and simultaneously delete request A from queue 502. However, in some embodiments, request agent 412 may delete request A from queue 502 after retrieving it. In some embodiments, request agent 412 may instruct another component of the on-demand code execution system to delete request A from queue 502. For example, the on-demand code execution system may include a queue manager that may remove request A after request agent 412 transmits a notification that it has retrieved request A.

[0116] After retrieving request A, request agent 412 can communicate with virtual execution environment 414 at [2] to obtain response A to request A. In some embodiments, request agent 412 can generate instructions based at least in part on request A. Instructions can be directly transmitted to virtual execution environment 414. However, in some embodiments, instructions can be transmitted to a supervisor process. In some embodiments, the supervisor process can be a component of machine 410. Alternatively, the supervisor process can also be a subcomponent of another component of the on-demand code execution system.

[0117] The instructions may include instructions to: deconstrain resources and invoke functions or applications configured to execute on virtual execution environment 414 to obtain a response A to request A, as described above relative to... Figure 3 As discussed in boxes 306 and 308, instructions can be directly passed to the virtual execution environment 414. For example, request agent 412 can determine that generating a response A to request A requires invoking a specific function and / or application. Therefore, request agent 412 can generate instructions for virtual execution environment 414 to invoke the specific function or application.

[0118] In some embodiments, request agent 412 may also generate instructions to unconstrain computing resources to an increased level of computing power. This increased level may be a higher level of computing power. In some embodiments, the higher level of computing resources may be configured as the maximum level of computing power. The increased level may also fall within a range defined by a lower non-zero level of computing power and a higher level of computing power. The increased level of computing power may be obtained from a client device. For example, the increased level of computing power may be received in configuration data provided to the client device to configure the virtual execution environment 414. Alternatively, the increased level of computing power may be provided in a request. For example, request A may include an increased level of computing power allocated to complete a set of tasks. Request agent 412 may process this request and generate a request to unconstrain resources to an increased level of computing power and invoke a specific function or application.

[0119] Of course, in some embodiments, instructions to deconstrain computing resources may not be generated or transmitted. For example, in some embodiments, the virtual execution environment 414 may be in the process of generating a response A to request A. Computing resources may have already been deconstrained to a higher level to generate a response A to request A. Therefore, when request B is accessed, request broker 412 may not generate additional instructions to deconstrain computing resources. Instead, request broker 412 may generate instructions to invoke a specific function or application to generate a response B to request B. In other embodiments, response B and response A may be generated simultaneously and transmitted to client device 402 simultaneously. In some embodiments, response B and response A may be generated within a predefined time interval. For example, a first generated response may be held for a predefined time interval. Other responses generated during this period may also be held until the end of the predefined time interval. After the end of the predefined time interval, all held responses will be transmitted to client device 402. In some embodiments, the predefined time interval may be defined by the client device in configuration data.

[0120] In some embodiments, once response A has been generated by virtual execution environment 414, it can be transmitted to request broker 412. Once request broker 412 receives response A, it can provide response A to client device 402 at [3]. Request broker 412 can also check for additional requests in queue 502 at [4]. This can be done before, after, or simultaneously with transmitting the response to client device 402 at [3].

[0121] As an example of checking queue 502 and then transmitting response A to the client device at [3], request agent 412 may check queue 502 at intervals. Intervals may occur during processing request A and / or during generating response A, which are steps performed before generating response A. As an example of checking queue 502 and then providing response A to the client device at [3], request agent 412 may receive response A and transmit response A to client device 402 at [3]. Request agent 412 may then determine whether any requests still exist in the queue and then restrict the computing resources allocated to virtual execution environment 414 to a lower non-zero level, such as Figure 3 The discussion is at box 316.

[0122] At [4], request agent 412 may examine additional requests stored in queue 502 and determine that additional requests exist. For example, request agent 412 may determine that request B exists in queue 502. Request agent 412 may then remove request B from queue 502, as described above at [1] relative to request A. Response B for request B may then be generated in the manner described above relative to steps [1] through [4] for request A. This process may be repeated for all remaining requests in queue 502 (such as requests C and D). Request agent 412 may not generate instructions to conserve computing resources to a lower non-zero level while requests still exist in the queue and / or while responses are being generated for these requests in virtual execution environment 414.

[0123] If a response is not generated within the set time period, the invoked function or application associated with this request can be terminated or canceled. If no other function or application is executing and there are no remaining requests in the queue, the request broker 412 can generate instructions to restrict the computing resources allocated to the virtual execution environment 414 to a lower non-zero level. After the computing resources are restricted to a lower non-zero level, the background process can continue to execute. As an example, a set time period can be set in the configuration data for the virtual execution environment 414 to generate responses to requests within 15 minutes. The virtual execution environment 414 can generate responses A, C, and D within this set time period. However, response B may not be generated within the set time period. Therefore, the invoked function or application attempting to generate response B can be terminated. If there are no other requests remaining in the queue and no other function or application is being processed except for the background process, the request broker 412 can restrict the resources to a lower non-zero level. The background process is not canceled or terminated and will continue to execute using the lower non-zero level of computing resources.

[0124] Figure 5BAn example set of interactions 501 between components is shown, in which a queue 502 is used to process multiple requests received from a client device 402. Figure 5B The interaction shown can be in, for example Figure 5A As shown and as described above, this process follows requests A through C.

[0125] Queue 502 may include request D. At [I], request agent 412 may remove request D from queue 502. For example, request agent 412 may retrieve request D from queue 502 and simultaneously delete request D from queue 502. However, in some embodiments, request agent 412 may delete request D from queue 502 after retrieving it. In some embodiments, request agent 412 may instruct another component of the on-demand code execution system to delete request D from queue 502. For example, the on-demand code execution system may include a queue manager that may remove request D after request agent 412 transmits notification that it has retrieved request D. Responses A through C may be generated before request D is removed, and no other functions or applications may be processing in virtual execution environment 414.

[0126] At [II], request agent 412 may communicate with virtual execution environment 414 to release resources allocated to virtual execution environment 414 in response to request D (if needed). The instructions provided by request agent 412 to release resources may be implemented by the supervisor process of virtual machine 410. If virtual execution environment 414 has been allocated a higher level of computing power during the processing of a previous request, it may not be necessary to release resources. Request agent 412 may also invoke functions or applications to generate response D.

[0127] Once response D is generated, it can be provided to client device 402 at [III]. If response D is not generated within a set time period, the invocation of the function or application that generated response D can be canceled or terminated. In any case, request agent 412 can subsequently check for additional requests at [IV] and determine that no requests remain in the queue. After determining that no requests remain in the queue and that no requests are currently being processed by the invoked function or application in virtual execution environment 414, request agent 412 can begin to restrict the computing resources allocated to virtual execution environment 414 to a lower non-zero level at [V]. Request agent 412 can send instructions that can be executed by the supervisor process of virtual machine 410. After virtual execution environment 414 has been restricted to a lower non-zero level of computing power, background processes executing on virtual execution environment 414 can continue to use the lower non-zero level of computing power to execute at [VI]. Request agent 412 can continue to monitor queue 502 at intervals to check for any requests received. If a request is received, the request agent 412 can communicate with the virtual execution environment 414 to use the above-mentioned... Figures 5A to 5B The steps described are for generating a response to the request.

[0128] Terminology and Additional Notes

[0129] Some inventive aspects of this disclosure are set forth in the following clauses:

[0130] Clause 1. A system comprising: An on-demand code execution service includes a set of host computing devices for on-demand execution of application code, wherein the on-demand code execution service is configured to at least: Receive application configuration data for an application from a client device, the application including a request-response process and a background process, wherein the application configuration data indicates a baseline set of computing resources for the background process and a larger set of computing resources for the request-response process; The execution of a virtual computing component on a host computing device is initialized, wherein the application is executed within the virtual computing component, wherein the virtual computing component is configured with the baseline computing resource set at least in part based on the application configuration data, and wherein the background process begins to use the baseline computing resource set to execute. The request queue associated with the application is identified as including the first request; Retrieve the first request from the request queue, wherein the first request is to be processed by the application; The computing resources provisioned to the virtual computing component are added to the larger set of computing resources; The first request is sent to the virtual computing component, wherein the request-response process generates a response to the first request; Receive the response from the virtual computing component; The response is provided to the client device; It is determined that the request queue does not contain any additional requests awaiting processing by the application; and The computing resources allocated to the virtual computing component are reduced to the baseline computing resource set.

[0131] Clause 2. The system according to Clause 1, wherein the on-demand code execution service is further configured as follows: It is determined that the request queue includes a second request; Retrieve the second request from the request queue, wherein the second request is to be processed by the application; The computing resources provisioned to the virtual computing component are added to the larger set of computing resources; Receive the response from the virtual computing component; The response is provided to the client device; It is determined that the request queue does not contain any additional requests from the client device awaiting processing by the application; Generate a response to the second request; and The computing resources allocated to the virtual computing component are reduced to the baseline computing resource set.

[0132] Clause 3. The system according to Clause 1, wherein the on-demand code execution service is further configured as follows: It is determined that the request queue includes a second request; Retrieve the second request from the request queue, wherein the second request is to be processed by the application; The computing resources provisioned to the virtual computing component are added to the larger set of computing resources; Failure to receive the response from the virtual computing component for a set time period, wherein the set time period is provided in the application configuration data; and Reduce the computing resources to the baseline computing resource set.

[0133] Clause 4. The system described in Clause 3, wherein the computing resources include processing power in units of vCPUs.

[0134] Clause 5. A system comprising a computer-readable storage device and one or more processors, wherein the system is configured to: A host computing device for configuring a code execution system is configured using application code, the application code being configured to execute a first process according to a request-response protocol and a second process outside of the request-response protocol, wherein configuring the code execution system includes allocating a first number of computing resources of the host computing device to execute the second process, wherein the first number of computing resources is a non-zero level specified in configuration data associated with the application code; Receive the first request from the second computing system; A first response to the first request is generated at least in part based on the execution of the first process, wherein the code execution system provides on-demand execution of the first process by allocating a second number of computing resources to the host computing device, wherein the second number of computing resources is greater than the first number of computing resources; At least in part based on the generation of the first response to the first request, the allocation of computing resources is reduced to the first amount of computing resources, wherein the second process uses the first amount of computing resources to execute; and The first response is provided to the second computing system.

[0135] Clause 6. The system according to Clause 5, wherein the first number of computing resources of the host computing device includes a first number of vCPUs, and the second number of computing resources of the host computing device includes a second number of vCPUs, wherein each vCPU represents processing power allocated to the virtual execution environment of the host computing device.

[0136] Clause 7. The system according to Clause 5, wherein the first quantity of computing resources of the host computing device includes a first quantity of network bandwidth, and the second quantity of computing resources of the host computing device includes a second quantity of network bandwidth.

[0137] Clause 8. The system according to Clause 5, wherein the first number of computing resources of the host computing device are sufficient to allow at least one background process to execute.

[0138] Clause 9. The system pursuant to Clause 5, wherein the system configures the code execution system at least in part based on the configuration data provided by a client associated with the application code.

[0139] Clause 10. The system according to Clause 9, wherein the configuration data includes a baseline set of computing resources indicating the first number of computing resources.

[0140] Clause 11. The system according to Clause 10, wherein the configuration data further includes a larger set of computing resources, indicating the second number of computing resources.

[0141] Clause 12. The system according to Clause 9, wherein the configuration data includes an intermediate set of computing resources indicating a third number of computing resources, wherein the third number of computing resources is more than the first number of computing resources and less than the second number of computing resources.

[0142] Clause 13. A computer-implemented method, the computer-implemented method comprising: Under the control of a computing system including one or more computing devices configured to execute specific instructions: Configure a virtual execution environment for a host computing device to execute an application, wherein the application includes synchronous components and asynchronous components, wherein the synchronous components are to be executed using a first number of computing resources of the host computing device, wherein the asynchronous components are to be executed using at least a second number of computing resources, wherein the second number is a non-zero number less than the first number, and wherein the second number is specified in application configuration data associated with the application; The first number of computing resources are provided to the virtual execution environment, wherein the synchronization component uses at least a portion of the first number of computing resources to generate a response to a request; It is determined that the synchronization component has generated the response and there are no pending requests waiting for a response; and The computing resources provisioned to the virtual execution environment are reduced to the second number, wherein the asynchronous component uses at least a portion of the second number of computing resources to execute.

[0143] Clause 14. The computer-implemented method according to Clause 13, further comprising determining the second number of computing resources based at least in part on the application configuration data provided by the application owner.

[0144] Clause 15. The computer-implemented method according to Clause 14, further comprising: The container image to be used to instantiate the virtual execution environment is determined from among a plurality of container images, at least in part based on the application configuration data. Obtain the container image from the container image data storage area; and The container image is used to instantiate the virtual execution environment.

[0145] Clause 16. The computer-implemented method according to Clause 13, wherein providing the first number of computing resources to the virtual execution environment includes providing the first number of virtual central processing units (vCPUs) to the virtual execution environment.

[0146] Clause 17. The computer-implemented method according to Clause 13, the computer-implemented method further comprising configuring a second virtual execution environment of the host computing device to execute a second application different from the application, wherein the second application includes a second synchronous component and a second asynchronous component, wherein the second asynchronous component is to be executed using at least a third number of the computing resources, wherein the third number is a non-zero number different from the second number.

[0147] Clause 18. The computer-implemented method according to Clause 13, further comprising: The request is received from a second computing system, wherein the virtual execution environment provides the first number of computing resources in response to receiving the request; Send the request to the application; and The response is received from the application, wherein the determination that the synchronization component has generated the response is based on the following: receiving the response from the application.

[0148] Clause 19. The computer-implemented method according to Clause 18, further comprising: Receive the second request from the third computer system; Determine that the virtual execution environment is equipped with the first number of computing resources; and The second request is sent to the application without allocating additional computing resources to the virtual execution environment.

[0149] Clause 20. The computer-implemented method according to Clause 13, the computer-implemented method further comprising determining that a request queue associated with the application is empty, wherein determining that the synchronization component has no pending requests waiting for a response is based on: determining that the request queue associated with the application is empty.

[0150] All methods and tasks described herein can be performed and fully automated by a computer system. In some cases, a computer system may include multiple different computers or computing devices (e.g., physical servers, workstations, storage arrays, cloud computing resources, etc.) that communicate and interoperate via a network to perform the described functions. Each such computing device typically includes a processor (or processors) that executes program instructions or modules stored in memory or other non-transitory computer-readable storage media or devices (e.g., solid-state storage devices, disk drives, etc.). The various functions disclosed herein may be embodied in such program instructions or may be implemented in the computer system's dedicated circuitry (e.g., ASICs or FPGAs). In cases where a computer system includes multiple computing devices, these devices may, but need not, be located in the same location. The results of the disclosed methods and tasks can be persistently stored by transforming physical storage devices such as solid-state memory chips or disks into different states. In some embodiments, the computer system may be a cloud-based computing system whose processing resources are shared by multiple different business entities or other users.

[0151] Depending on the implementation, certain actions, events, or functions of any process or algorithm described herein may be performed in a different order, or may be added, combined, or omitted entirely (e.g., not all described operations or events are necessary for the practical algorithm). Furthermore, in some embodiments, operations or events may be performed simultaneously rather than sequentially, for example, through multithreading, interrupt handling, or on multiple processors or processor cores or other parallel architectures.

[0152] The various illustrative logic blocks, modules, routines, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware or a combination of electronic hardware and computer software. To clearly illustrate this interchangeability, the various illustrative components, blocks, modules, and steps have been generally described above in accordance with their functionality. Whether such functionality is implemented as hardware or as software running on hardware depends on the specific application and design constraints imposed on the system as a whole. The described functionality may be implemented in different ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

[0153] Furthermore, the various illustrative logic blocks and modules described in conjunction with the embodiments disclosed herein can be implemented or executed by machines designed to perform the functions described herein, such as processor devices, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic components, discrete hardware components, or any combination thereof. A processor device may be a microprocessor, but alternatively, it may be a controller, microcontroller, or state machine, a combination thereof, etc. A processor device may include an electronic circuitry system configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor device may also be implemented as a combination of computing devices (e.g., a combination of a DSP and a microprocessor), multiple microprocessors, one or more microprocessors along with a DSP core, or any other such configuration. While this document is primarily described with respect to digital technologies, a processor device may also primarily comprise analog components. For example, some or all of the algorithms described herein may be implemented in analog circuitry systems or hybrid analog and digital circuitry systems. A computing environment can include any type of computer system, including, but not limited to, microprocessor-based computer systems, mainframe computers, digital signal processors, portable computing devices, device controllers, or computing engines within appliances.

[0154] Elements of the methods, processes, routines, or algorithms described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, software modules executed by a processor device, or a combination of both. Software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disks, removable disks, CD-ROMs, or any other form of non-transitory computer-readable storage medium. Exemplary storage media may be coupled to a processor device such that the processor device can read information from and write information to the storage medium. Alternatively, the storage medium may be integrated with the processor device. The processor device and storage medium may reside in an ASIC. The ASIC may reside in a user terminal. Alternatively, the processor device and storage medium may reside as discrete components in the user terminal.

[0155] Unless otherwise specifically stated, or otherwise understood as used in the context herein, the conditional language used herein (such as, in addition to, "can," "may," "may," "may," "may," "for example," etc.) is generally intended to convey that certain embodiments include certain features, elements, or steps that are not included in other embodiments. Therefore, such conditional language is not generally intended to imply that one or more embodiments require features, elements, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, elements, and / or steps are included or to be performed in any particular embodiment, with or without further input or prompting. The terms "comprising," "including," "having," etc., are synonymous and used inclusively in an open-ended manner, and do not exclude additional elements, features, actions, operations, etc. Furthermore, the term "or" is used in an inclusive sense (rather than in an exclusive sense) such that when used, for example, to connect a series of elements, the term "or" means one, some, or all of the elements in the series.

[0156] Unless otherwise specifically stated, disjunctive languages ​​such as the phrase “at least one of X, Y, or Z” are understood in the context to generally represent that items, terms, etc., can be X, Y, or Z or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive languages ​​are generally not intended and should not imply that some embodiments require at least one of X, at least one of Y, or at least one of Z to be present respectively.

[0157] Unless otherwise explicitly stated, articles such as “a” or “an” should generally be understood to include one or more of the described items. Therefore, phrases such as “a device configured to…” are intended to include one or more of the described devices. Such one or more described devices may also be collectively configured to perform the stated description. For example, “a processor configured to perform descriptions A, B, and C” may include a first processor configured to perform description A, which works in conjunction with a second processor configured to perform descriptions B and C.

[0158] While the foregoing detailed description has shown, described, and pointed out novel features applicable to various embodiments, it will be understood that various omissions, substitutions, and changes in the form and details of the illustrated apparatus or algorithm may be made without departing from the spirit of this disclosure. As will be appreciated, certain embodiments described herein may be embodied in a form that does not provide all the features and benefits set forth herein, as some features may be used or practiced separately from other features. The scope of certain embodiments disclosed herein is indicated by the appended claims rather than by the foregoing description. All variations within the meaning and scope of the equivalents of the claims are to be included within the scope of the claims.

Claims

1. A system comprising a computer-readable storage device and one or more processors, wherein the system is configured to: A host computing device for configuring a code execution system is configured using application code, the application code being configured to execute a first process according to a request-response protocol and a second process outside of the request-response protocol, wherein configuring the code execution system includes allocating a first number of computing resources of the host computing device to execute the second process, wherein the first number of computing resources is a non-zero level specified in configuration data associated with the application code; Receive the first request from the second computing system; A first response to the first request is generated at least in part based on the execution of the first process, wherein the code execution system provides on-demand execution of the first process by allocating a second number of computing resources to the host computing device, wherein the second number of computing resources is greater than the first number of computing resources; The computing resource allocation is reduced to the first number of computing resources, at least in part based on the generation of the first response to the first request, wherein the second process uses the first number of computing resources to execute; and The first response is provided to the second computing system.

2. The system of claim 1, wherein the first number of computing resources of the host computing device includes a first number of vCPUs, and the second number of computing resources of the host computing device includes a second number of vCPUs, wherein each vCPU represents the processing power allocated to the virtual execution environment of the host computing device.

3. The system of claim 1, wherein the first quantity of computing resources of the host computing device includes a first quantity of network bandwidth, and the second quantity of computing resources of the host computing device includes a second quantity of network bandwidth.

4. The system of claim 1, wherein the first number of computing resources of the host computing device are sufficient to allow at least one background process to execute.

5. The system of claim 1, wherein the system configures the code execution system at least in part based on the configuration data, wherein the configuration data is provided by a client associated with the application code.

6. The system of claim 5, wherein the configuration data includes a baseline set of computing resources indicating the first number of computing resources.

7. The system of claim 6, wherein the configuration data further includes a larger set of computing resources, indicating the second number of computing resources.

8. The system of claim 5, wherein the configuration data includes an intermediate set of computing resources, indicating a third number of computing resources, wherein the third number of computing resources is more than the first number of computing resources and less than the second number of computing resources.

9. A computer-implemented method, the computer-implemented method comprising: Configure a virtual execution environment for a host computing device to execute an application, wherein the application includes synchronous components and asynchronous components, wherein the synchronous components are to be executed using a first number of computing resources of the host computing device, wherein the asynchronous components are to be executed using a second number of computing resources, wherein the second number is a non-zero number less than the first number, and wherein the second number is specified in application configuration data associated with the application. The first number of computing resources are provided to the virtual execution environment, wherein the synchronization component uses at least a portion of the first number of computing resources to generate a response to a request; Determine that the synchronization component has generated the response and there are no pending requests waiting for a response; as well as The computing resources provisioned to the virtual execution environment are reduced to the second number, wherein the asynchronous component uses at least a portion of the second number of computing resources to execute.

10. The computer-implemented method of claim 9, further comprising determining the second number of computing resources based at least in part on the application configuration data, wherein the application configuration data is provided by the application owner.

11. The computer-implemented method according to claim 10, further comprising: The container image to be used to instantiate the virtual execution environment is determined from among a plurality of container images, at least in part based on the application configuration data. Obtain the container image from the container image data storage area; as well as The container image is used to instantiate the virtual execution environment.

12. The computer-implemented method of claim 9, further comprising configuring a second virtual execution environment of the host computing device to execute a second application different from the application, wherein the second application includes a second synchronous component and a second asynchronous component, wherein the second asynchronous component is to be executed using at least a third number of computing resources, wherein the third number is a non-zero number different from the second number.

13. The computer-implemented method according to claim 9, further comprising: The request is received from a second computing system, wherein the virtual execution environment provides the first number of computing resources in response to receiving the request; Send the request to the application; as well as The response is received from the application, wherein the determination that the synchronization component has generated the response is based on the following: receiving the response from the application.

14. The computer-implemented method according to claim 13, further comprising: Receive the second request from the third computer system; It is determined that the virtual execution environment is equipped with the first number of computing resources; as well as The second request is sent to the application without allocating additional computing resources to the virtual execution environment.

15. The computer-implemented method of claim 9, further comprising determining that a request queue associated with the application is empty, wherein determining that the synchronization component has no pending requests waiting for a response is based on: determining that the request queue associated with the application is empty.

Images