Function as service (FAAS) system enhancements

Abstract

The invention discloses a function as a service (FAAS) system enhancement. A computing system configured to provide such enhanced FaaS services includes one or more control architecture subsystems, a software and orchestration subsystem, a network and storage subsystem, and a security subsystem. The computing system performs a function in an execution environment provided by an architecture subsystem in response to a user-triggered event, the architecture subsystem representing an abstraction of execution management and protecting the user from the burden of managing execution. A software and orchestration subsystem allocates computing resources for function execution by intelligently rotationally accelerating and rotationally decelerating containers for function code to achieve reduced instantiation latency and increased execution scalability while maintaining secure execution. Further, the computing system enables consumers to make payments only when their code is executed to achieve granularity charging to a millisecond-level increment.

Description

[0001] This application is a divisional application of the invention patent application with PCT international application number PCT / US2019 / 027659, international application date April 16, 2019, and application number 201980043306.6 entering the Chinese national phase, entitled "Function as a Service (FAAS) System Enhancement".

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of priority to PCT Provisional Patent Application No. PCT / CN2018 / 114602, filed on November 8, 2018. Background Art

[0004] Function as a Service (FaaS) is an event-oriented, highly scalable computer code execution model that typically pre-configures a single-purpose application programming interface (API) endpoint on a cloud computing infrastructure to receive and run code execution requests for a short period of time. Such code execution requests and / or the execution of the requested code are variously and collectively referred to as lambdas (λ), functions, actions, and / or run-to-completion processes. In this application, the term "function" may refer to any of these code execution requests and the execution of these requests, regardless of whether such execution is performed as software, as an action performed in hardware, or any combination of the above. That is, the term "function" as used herein may generally refer to FaaS functions unless otherwise explicitly and / or implicitly indicated by the context. Therefore, FaaS can be considered a step in the evolution of cloud computing. Sometimes also referred to as "serverless computing," FaaS enables software developers to write highly scalable code without having to pre-configure or otherwise predefine the cost, time, and expenditure required for the hardware or application software resources involved in executing the code. FaaS also enables cloud service providers (CSPs) to increase resource usage due to better distribution (e.g., box packaging).

[0005] Large-scale computing units have evolved over time, with increasing focus on business logic and decreasing concern / control over cloud stack implementations. FaaS abstracts the runtime environment, such as the Common Language Runtime (CLR) and the execution context lifecycle. Users such as application developers and CSPs can capture significant value through FaaS. For example, developers use high-level languages to build applications, such as function code, and upload this function code to the FaaS platform for execution. Developers simply use the FaaS platform without hosting the function code themselves, and the FaaS platform's infrastructure is opaque to users.

[0006] In the case of CSPs, while they can use their portfolio of functions as a service as a key differentiator, there's still considerable room for improvement. For example, concerns remain about proprietary lock-in, as function code written and deployed in the cloud becomes inextricably linked to the functionality provided by the CSP. This proprietary lock-in means that applications become optimized for specific cloud environments. Consequently, moving functionality between CSPs can compromise application performance and responsiveness, in addition to potentially causing some CSP-provided functionality to be unsupported by other CSPs.

[0007] Figure 2A A general existing serverless service platform 203 for providing function-as-a-service is illustrated, which receives serverless function code 201 uploaded by a computer application developer and executes the function code 201 triggered by a corresponding event. The application developer of the serverless function code 201 can pay an amount corresponding to the amount of time when the serverless function code 201 is called and / or executed in the serverless service platform 203. Figure 2AAs shown, the serverless service platform 203 has a serverless service manager 203a to receive the serverless function code 201, store the serverless function code 201 to a storage device, and schedule the execution of the serverless function code 201 using the required computer resources (such as containers). The serverless service platform 203 may also include a networking and messaging module 203b to trigger one or more events associated with the execution of the serverless function code 201 and route the one or more events between endpoints of the CSP network. The serverless service platform 203 may also include a security and access manager 203c and one or more storage services 203d. The security and access manager 203c is responsible for providing security in executing the serverless function code 201. For example, the security and access manager 203c can verify access to various computer resources and / or privileges associated with executing the serverless function code 201. The storage service 203d may include an object storage device (e.g., S3 of Amazon AWS Lambda), a key-value database (e.g., Dynamo database of Amazon AWS service), and / or a mobile backend database for providing dedicated backend services (e.g., Cloud Firestore of Google Cloud Function). The storage service 203d may also implement big data services, including big data query (e.g., AWS Athena and Google BigQuery) and big data transformation (e.g., AWS Glue and Google Cloud Dataflow). The serverless service platform 203 enables serverless services through a serverless service architecture 203e (e.g., the architecture of AWS Lambda, the architecture of Google CloudPlatform, and / or the architecture of Azure Functions) and one or more hardware-associated elements (such as hardware-assisted virtual machines, CPUs, GPUs, and accelerators).

[0008] For example, Figure 2A Existing FaaS solutions, as shown in [1], present challenges. For example, existing FaaS solutions often offer an immature ecosystem of developer tools. Furthermore, existing FaaS solutions frequently require unpredictable and costly cold starts, which not only run the code but also prepare the necessary execution environment, incurring high, variable, and unpredictable latency. For example, in the case of a running video encoding application, existing solutions using object storage may be overloaded and unable to support fine-grained communication and / or suffer from long latency before the serverless function code can execute.

[0009] Furthermore, existing FaaS solutions currently suffer from limited control and difficulty in managing the state of code execution. For example, large-scale machine learning training, such as Figure 2AExisting FaaS solutions shown in face challenges in achieving the desired high scalability due to the lack of fast and / or shared memory and the lack of adequate support for networking connectivity.

[0010] Furthermore, FaaS introduces an additional layer of abstraction, making it more difficult for existing FaaS solutions to expose and exercise different and / or new infrastructure features in processors, platforms, or systems (such as computer architectures used to support hardware heterogeneity). In other words, while existing FaaS solutions can abstract software development, such abstraction can lead to difficulties in effectively utilizing the hardware architecture to execute software code. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Various advantages of the embodiments will become apparent to those skilled in the art upon reading the following description and appended claims, and by referring to the following drawings, in which:

[0012] Figure 1 is an illustration of an example of a FaaS computing environment according to an embodiment;

[0013] Figure 2A Examples of common existing serverless services are illustrated;

[0014] Figure 2B is a block diagram of an example set of FaaS system components according to an embodiment;

[0015] Figure 3 is a block diagram of an example of a FaaS server configuration according to an embodiment;

[0016] Figure 4 is a block diagram of an example of an enhanced FaaS system according to an embodiment;

[0017] Figure 5 is a block diagram of an example of subsystems of an enhanced FaaS system according to an embodiment;

[0018] Figure 6A is a block diagram of an example of an enhanced FaaS architecture for resource allocation and control according to an embodiment;

[0019] Figure 6B is a flowchart for managing user-level capabilities using an enhanced FaaS architecture according to an embodiment;

[0020] Figure 6C is a flowchart for managing user-level capabilities using an enhanced FaaS architecture according to an embodiment;

[0021] Figure 7A is a block diagram of an exemplary FaaS compute node for monitoring various virtual power performance monitoring unit (vPMU) events according to an embodiment;

[0022] Figure 7B is a block diagram of an example of a vPMU buffer according to an embodiment; and

[0023] Figure 7C is a flow chart of an example of a method of monitoring performance of a function according to an embodiment;

[0024] Figure 8A is a block diagram of an example of an electronic processing system according to an embodiment;

[0025] Figure 8B is a block diagram of an example of a semiconductor packaging device according to an embodiment;

[0026] Figure 8C is a flow chart of an example of sharing memory among multiple FaaS functions according to an embodiment;

[0027] Figure 8D is a flow chart of an example of sharing memory among multiple FaaS functions according to an embodiment;

[0028] Figure 8E is a block diagram of another example of a FaaS system that provides memory sharing among multiple FaaS functions according to an embodiment;

[0029] Figure 8F is a block diagram of another example of a function lookaside buffer for facilitating communication between two FaaS functions, according to an embodiment;

[0030] Figure 8G is a block diagram of another example of a FaaS system for orchestrating distributed FaaS functions according to an embodiment;

[0031] Figure 9A is a flowchart of an example of a method for providing container execution prior to speculative execution of function-as-a-service according to an embodiment;

[0032] Figure 9B is a block diagram of another example of a FaaS system that supports speculative execution of previous container runs according to an embodiment;

[0033] Figure 9C is a block diagram of another example of a FaaS system that supports container execution before speculative execution of an image rotation function according to an embodiment;

[0034] Figure 10A is a flowchart of another example of a method for providing function as a service with feedback support according to an embodiment;

[0035] Figure 10B is a block diagram of another example of a FaaS system with feedback support according to an embodiment;

[0036] Figure 10C is a block diagram of another example of a FaaS system with feedback support according to an embodiment;

[0037] Figure 11A is an illustration of an example of a function having multiple options for instantiation according to an embodiment;

[0038] Figure 11B is a block diagram of another example of a FaaS system supporting functions with multiple options for instantiation according to an embodiment;

[0039] Figure 11C is a flow chart of another example of a method for providing a function as a service for a function having multiple options for instantiation according to an embodiment;

[0040] Figure 12 is a block diagram of another example of a FaaS system with a scheduler according to an embodiment;

[0041] Figure 13A is a block diagram of an example of a FaaS server architecture according to an embodiment;

[0042] Figure 13B is an example of an enhanced FaaS scheduling process according to an embodiment;

[0043] Figure 13C is a flow chart of the functionality according to an embodiment;

[0044] Figure 14A and Figure 14B is an example of an enhanced function execution sequence according to an embodiment;

[0045] Figure 14C is a flow chart of a function of scheduling a plurality of operations according to an embodiment;

[0046] Figure 15A is a block diagram of an example of a memory storage enhanced computing architecture for FaaS according to an embodiment;

[0047] Figure 15B is a flow chart of memory allocation for containers and functions of a FaaS platform according to an embodiment;

[0048] Figure 16A is an example of batch function requests for function execution according to an embodiment.

[0049] Figure 16B is an illustration of an example of a semiconductor package device according to an embodiment;

[0050] Figure 16C is a flow chart of batch processing function requests according to an embodiment;

[0051] Figure 16D is a flowchart of batch processing two or more function requests according to an embodiment;

[0052] Figure 16E is a flow chart of a scheduling function request according to an embodiment;

[0053] Figure 17A is an example of a redundant function implementation according to an embodiment;

[0054] Figure 17B is a flowchart of a redundant function implementation according to an embodiment;

[0055] Figure 18A illustrates a function generation diagram representing a scheduler for FaaS according to an embodiment;

[0056] Figure 18B An enhanced FaaS system with a scheduler according to an embodiment is illustrated;

[0057] Figure 18C is a flowchart of the FaaS function implementation according to an embodiment;

[0058] Figure 19A is an example of an enhanced FaaS architecture with shared data storage according to an embodiment;

[0059] Figure 19B is a flow chart of an exemplary FaaS data storage according to an embodiment;

[0060] Figure 19C is a flowchart of an exemplary method for implementing and enforcing a FaaS security protocol according to an embodiment;

[0061] Figure 20A is a block diagram of an example of an enhanced FaaS server architecture with a dedicated FaaS cache, according to an embodiment;

[0062] Figure 20B is a block diagram of an example of an enhanced FaaS server architecture with universal cache according to an embodiment;

[0063] Figure 20C is an example of a diagram illustrating the data amount of a data object according to an embodiment;

[0064] Figure 20D is a flow chart of an exemplary enhanced functional resource management according to an embodiment;

[0065] Figure 21A is an example of a method of prioritizing software threads according to an embodiment;

[0066] Figure 21B and Figure 21CIllustrate interactions between tasks and classes of service (CLOS) in a page table for providing page-level QoS in an enhanced FaaS architecture according to an exemplary embodiment;

[0067] Figure 21D Another architecture related to page-level QoS according to an exemplary embodiment is illustrated;

[0068] Figure 22 An example architecture for providing determinism and accuracy with respect to FaaS services according to an embodiment is illustrated;

[0069] Figure 23 is an example of a method for calculating resource usage for billing purposes according to an embodiment;

[0070] Figure 24A is a block diagram of an example of a distributed computing environment according to an embodiment;

[0071] Figure 24B is a flowchart of another example of providing function as a service according to an embodiment;

[0072] Figure 25A is a block diagram of an example of a FaaS system that enables memory reuse between multiple functions according to an embodiment;

[0073] Figure 25B is a flowchart of another example of a method for providing function as a service according to an embodiment;

[0074] Figure 26A is a flowchart of another example of providing function as a service according to an embodiment;

[0075] Figure 26B is an illustration of a function call graph according to an embodiment;

[0076] Figure 26C is an illustrative diagram of a split function according to an embodiment;

[0077] Figure 26D is another example diagram of a function call graph according to an embodiment;

[0078] Figure 26E is an illustrative diagram of a merge function according to an embodiment;

[0079] Figure 27A is a block diagram of another example of an enhanced FaaS system with memory shared by multiple functions according to an embodiment;

[0080] Figure 27B is a block diagram of another example of an enhanced FaaS system with memory shared by multiple functions according to an embodiment;

[0081] Figure 28A is a flow chart of providing function as a service according to an embodiment;

[0082] Figure 28B is a block diagram of another example of a FaaS system according to an embodiment;

[0083] Figure 29A is a flow chart providing a function as a service that supports a container reversal feature according to an embodiment;

[0084] Figure 29B is a block diagram of an example of a FaaS system supporting a container reversal feature according to an embodiment;

[0085] Figure 30A is a flowchart of providing a function as a service with a continuation application to achieve improved function execution performance according to an embodiment;

[0086] Figure 30B is a block diagram of an example of a FaaS system with continuation applications to achieve improved function execution performance according to an embodiment;

[0087] Figure 31A and Figure 31B is an example of enhanced container building and cache management for FaaS according to an embodiment;

[0088] Figure 31C is a flow chart of an exemplary cache eviction according to an embodiment;

[0089] Figure 31D is a flowchart of another exemplary cache eviction according to an embodiment;

[0090] Figure 31E is a flow chart of determining the time to live of a cached data object according to an embodiment;

[0091] Figure 32A and Figure 32B is an example of enhanced function distribution according to an embodiment;

[0092] Figure 32C and Figure 32D is an example of enhanced function distribution according to an embodiment;

[0093] Figure 32E is a flow chart of function distribution according to an embodiment;

[0094] Figure 33A This is an example of an enhanced functional configuration of the FaaS function according to an embodiment;

[0095] Figure 33Bis a flowchart of a moniker identified by an enhanced functional configuration according to an embodiment;

[0096] Figure 34A is an illustration of an example of using a call graph to prefetch functions based on the frequency of function calls according to an embodiment;

[0097] Figure 34B is an example of a method for enhancing the execution of FaaS functions according to an embodiment;

[0098] Figure 35A is a block diagram illustrating the previous functionality of the present functionality according to an embodiment;

[0099] Figure 35B is an example of a method for executing a FaaS function based on a predecessor function according to an embodiment;

[0100] Figure 36A An example of maintaining warmth of a container based on a probability of a function being executed according to an embodiment is illustrated;

[0101] Figure 36B is an example of a method for executing a FaaS function from a warm container according to an embodiment;

[0102] Figure 37A is a block diagram illustrating a size-based adaptive memory hierarchy according to an embodiment;

[0103] Figure 37B is a block diagram illustrating usage-based adaptive memory hierarchy according to an embodiment;

[0104] Figure 37C is an example of a method for adaptive memory tiering of functions according to an embodiment; Figure 38A An example of a fast class loading environment for FaaS according to an embodiment is provided;

[0105] Figure 38B is an example of a method for executing functions in an enhanced FaaS computing environment according to an embodiment;

[0106] Figure 38C is an example of a method for executing functions in an enhanced FaaS computing environment according to an embodiment;

[0107] Figure 39A Illustrate a FaaS environment that facilitates continuous application of both time-sequential and cloud-scale (horizontal) feedback according to an embodiment;

[0108] Figure 39B illustrates different vectors for each function type according to an embodiment;

[0109] Figure 39Cis an example of a method for reserving appropriate resources according to an embodiment;

[0110] Figure 40A illustrates various features related to code optimization according to embodiments;

[0111] Figure 40B is an example of a method for performing functions in enhanced FaaS computing according to an embodiment;

[0112] Figure 41A is a diagram illustrating a relationship between a demand fingerprint and function execution according to an embodiment;

[0113] Figure 41B illustrates the operation of a resource manager according to an embodiment;

[0114] Figure 41C is an example of a method for efficient FaaS resource management using demand fingerprints according to an embodiment;

[0115] Figure 41D is an example of a demand fingerprint according to an embodiment;

[0116] Figure 42A is an example of communication between a function client and a function execution engine according to an embodiment;

[0117] Figure 42B is an example of a method for performing FaaS functions using opaque tags according to an embodiment;

[0118] Figure 43A is an illustration of an example of server location selection based on a token that uniquely identifies a functional context, according to an embodiment;

[0119] Figure 43B is a flow chart of management function calls according to an embodiment;

[0120] Figure 43C is a flowchart of a detailed method for managing function calls according to an embodiment;

[0121] Figure 43D is a block diagram of an example FaaS system in which the location of function calls is selected based on the request source;

[0122] Figure 43E is a block diagram of an example of a FaaS system, in which the location of a function call is selected based on a function call tree;

[0123] Figure 44A is a block diagram of an example of cross-domain control transfer according to an embodiment;

[0124] Figure 44B is a flow chart of operating a remote procedure callee according to an embodiment;

[0125] Figure 44C is a flowchart of an operation remote procedure caller according to an embodiment;

[0126] Figure 45A is a block diagram of an example of a FaaS architecture in which application layer functions are collocated with data plane functions, according to an embodiment;

[0127] Figure 45B is a flowchart of an operating runtime framework according to an embodiment;

[0128] Figure 46A is an illustration of an example of a custom response object solution according to an embodiment;

[0129] Figure 46B is a flow chart of a response object for a customized call instance according to an embodiment;

[0130] Figure 47A is an illustration of an example of a parameter marshall solution according to an embodiment;

[0131] Figure 47B is a diagram of an example of a high-level architecture for uniformly calling cross-platform functions according to an embodiment;

[0132] Figure 47C is a flow chart of grouping function parameters according to an embodiment;

[0133] Figure 48A is a block diagram of an example of transferring capability information between functions according to an embodiment;

[0134] Figure 48B is a block diagram of an example of encoding inline capability (EIC) information according to an embodiment;

[0135] Figure 48C is a block diagram of an example of a hardware queue manager according to an embodiment;

[0136] Figure 48D is a flowchart of operating a hardware queue manager according to an embodiment;

[0137] Figure 48E is a flow chart of queuing capabilities according to an embodiment;

[0138] Figure 48F is a flow chart of dequeuing a capability according to an embodiment;

[0139] Figure 49A is an illustration of an example of a mapping between a key identifier and a key according to an embodiment;

[0140] Figure 49B is a block diagram of an example of a single address space according to an embodiment;

[0141] Figure 49C is a block diagram of an example of context switching according to an embodiment;

[0142] Figure 49D is a block diagram of an example of a key identifier mapping update according to an embodiment;

[0143] Figure 49E is a flow chart of updating a key identifier mapping according to an embodiment;

[0144] Figure 49F is a block diagram of an example of a translation lookaside buffer / page miss handler that maps between virtual addresses and physical addresses;

[0145] Figure 50A is a block diagram of an example of a protection key identifier update instruction according to an embodiment;

[0146] Figure 50B is a flow chart of updating a protection key identifier according to an embodiment;

[0147] Figure 51A is a block diagram of an example of an unprivileged component that is granted modify subpage permissions according to an embodiment;

[0148] Figure 51B is a flow chart of controlling sub-page permissions according to an embodiment;

[0149] Figure 52A is an illustration of an example of a deprivileged mode path including capability information constraints according to an embodiment; and

[0150] Figure 52B is a flow chart of controlling memory access according to an embodiment. DETAILED DESCRIPTION

[0151] Figure 1A FaaS computing environment according to one embodiment is shown. Developers write function code 100 (also referred to herein as "computer code") representing one or more computer functions, and function code 100 is uploaded to a FaaS platform 102, for example, in a CSP data center 104. A triggering event 106 (such as, for example, a use case or an Internet of Things (IoT) event) initiates the execution of function code 100 on the FaaS platform 102. The environment in which the function code is executed is called a container. A container can be any isolated execution entity, such as a process, a Docker container or a Kubernetes container, a virtual machine, etc. A container running in its own virtual machine is called a virtual container. Within the CSP data center 104 (including the FaaS platform 102, their data center, edge environment, and IoT (including mobile) devices, over which the CSP has control), infrastructure is "spun up" (e.g., activated and / or allocated) and scaled as needed. Function code 100 can execute on the CSP's physical infrastructure / edge / IoT devices and the underlying virtualized containers. Finally, the infrastructure is "spun down" (e.g., deactivated and / or deallocated) in response to completion of execution.

[0152] As discussed in more detail below, the techniques described herein reduce concerns about proprietary lock-in, costly cold starts, latency, and management / control of code execution state by providing enhanced FaaS features, such as running function code on demand in response to events and automatically scaling with the number of events based on atomic scale units that deploy function code and its corresponding functionality as compute-scale units. The techniques described herein further reduce these concerns by enabling increased breadth, allowing platform diversity, and / or enabling security directly with hardware and through standard libraries or runtimes. In addition, the techniques described herein make it easier to demonstrate and exercise new silicon features. By enabling developers to build applications using higher-level languages supported by CSP FaaS platforms and simplifying operations, optimization, flexibility, and scaling, the techniques described herein can further enable non-expert developers to implement robust, high-performance, and secure FaaS solutions that transparently benefit from the acceleration capabilities delivered in the hardware. The techniques described herein can further simplify coding and drive greater demand for computation performed by CSP FaaS platforms.

[0153] Figure 2B An example of components 200 (200a-200b) of an enhanced FaaS system 202 as described herein is shown. Figure 2ACompared to the challenges of existing FaaS services shown, the enhanced FaaS system 202 provides many enhancements, such as efficient function code execution in an event-oriented execution environment, which requires minimal management while providing high scalability and granular billing. In the illustrated example, the event-oriented component 200a ensures that function code runs on demand in response to events, the minimal management component 200b abstracts the management of the infrastructure that executes the function code from the user (e.g., code developer) through the FaaS service provider, the high scalability component 200c automatically scales function code execution with the number of events (e.g., user events or IoT-related events), the atomic scale unit component 200d deploys computer code and its corresponding functions as compute scale units, and the granular billing component 200e enables consumers (e.g., computer code developers) to pay only when their code is executed and consumers are billed in increments of, for example, 100 milliseconds. The enhanced FaaS system 202 may also include other advantageous components, as will be discussed in more detail below.

[0154] Users and CSPs will therefore gain significant value from the enhanced FaaS system 202. The value provided by the enhanced FaaS system 202 to users includes, for example, increased focus on developing operations of interest, shortened time to market (TTM), reduced infrastructure management, and lower total cost of ownership (TCO). For CSPs, the value provided by the enhanced FaaS system 202 includes, for example, driving adoption of high-margin services, earning more per computing unit, enabling new cloud workloads, improved cost amortization, elasticity in scheduling computing resources, and providing the ability to compete with other CSPs.

[0155] In an embodiment, there are five components of the enhanced FaaS system 202. The first component is a function format component for creating computer code for functions and executing functions. An example of a function format component is Figure 3 The illustrated FaaS executors 310, 324, 319, 315 (discussed below). The second component is an event handling API proxy component for routing event calls to functions. An example of an event handling API proxy is Figure 4 The third component is a function code store that receives, stores, and secures the function code package. An example of a function code store is Figure 4 The fourth component is the FaaS container that provides a function execution environment in which function code is downloaded and instantiated in a container or other isolated execution alternative. An example of a FaaS container is Figure 4The FaaS architecture 406 (discussed below) is described. After execution, the container is erased (or reset, reinitialized, etc.) for new functionality. For security and isolation, containers are often run within virtual machines (VMs). The final core component is a function / container orchestration component that optimizes the placement of containers within available computing resources by spinning up containers for functional code (e.g., "cold boot"). An example of a function / container orchestration component is Figure 4 404 (discussed below). If a function has recently been run, the new function can be placed in a container that is already "warm", thereby reducing instantiation latency. A warm container can be a container that has been recently used to execute a function, or a container that is frequently used for function execution. Instead of tearing down the container after the function is completed, the container can be retained for future function execution. Thus, a container can be considered "warm" because it does not have to be built and is ready to execute functions immediately.

[0156] Figure 3A FaaS server configuration 300 is shown for providing enhanced FaaS performance to users / CSPs. In the illustrated example, the FaaS server configuration 300 includes an intelligent network interface card (NIC) 302, which includes FaaS management logic 304 on an enhanced Faas substrate 306. In some embodiments, the intelligent network interface card 302 serves as a control center for the FaaS server configuration 300. The FaaS server configuration 300 includes one or more computer code execution units that are configured to perform multiple sets of functions in an orchestrated manner. In other embodiments, the FaaS management logic 304 and the enhanced FaaS substrate 306 can be implemented in various ways: in a baseboard management controller (BMC), a processor, a software module executed on a processor, dedicated functional logic, and / or implemented as any combination of the above. In the illustrated example, one or more central processing units (CPUs) 308a-308n (e.g., host processors) use a FaaS executor 310 to perform a first set of functions 312, one or more accelerators 314a-314n (e.g., fixed-function hardware logic) perform a second set of functions 316, one or more field programmable gate arrays (FPGAs) 318a-318n perform a third set of functions 320, and one or more graphics processing units (GPUs) 318a-318n perform a fourth set of functions 326. Accordingly, the illustrated FaaS server configuration 300 is powered by specialized silicon, including GPUs 322a-322n, FPGAs 318a-318n, and specialized accelerators 314a-314n. The FaaS server configuration 300 may include multiple CPUs 308a-308n, GPUs 322a-322n, FPGAs 318a-318n, and other specialized accelerators 314a-314n.

[0157] It is worth noting that the specialized accelerators 314a-314n can be of different types from one another. For example, each of the specialized accelerators 314a-314n can be specifically designed to accelerate certain functions or primitives (e.g., an accelerator specifically designed for matrix multiplication during training of a machine learning (ML) model, an accelerator for more efficient power and energy consumption, and / or an accelerator for better security and privacy of certain computations and data).

[0158] Similarly, the CPUs 308a-308n of the FaaS server configuration 300 can be of different architectures (e.g., manufactured by different vendors and have different schematics and architectures). For example, in some embodiments, the CPUs 308a-308n can be a combination of various processors selected from the group consisting of Intel architecture, advanced RISC processors, microprocessors without interlocked pipeline stages, and Reduced Instruction Set Computer-V (RISC-V) processors. In some embodiments, the CPUs 308a-308n can all have the same architecture (e.g., all Intel architecture). Similarly, the GPUs 322a-322n can be of the same architecture or different architectures (e.g., manufactured by different vendors and have different schematics). Similarly, the FPGAs 318a-322n can be of the same architecture or different architectures (e.g., manufactured by different vendors and have different schematics). Furthermore, the specialized accelerators 314a-322n can be of the same architecture or different architectures (e.g., manufactured by different vendors and have different schematics). Furthermore, the execution units, CPUs 308a-308n, GPUs 322a-322n, FPGAs 318a-318n, and specialized accelerators 314a-314n may utilize analog designs in addition to digital designs.

[0159] FaaS workloads may have different characteristics than traditional workloads, for example, in terms of finer (e.g., increased) granularity of the workload. The increased granularity of the workload makes it easier to take advantage of specialized silicon (such as accelerators 314a-314n and FPGAs 318a-318n). By decomposing multiple functional sets, the FaaS server configuration 300 enables the CSP to match each workload (e.g., transcoding, inference, specialized operations, etc.) with the optimal silicon component (such as accelerators 314a-314n, FPGAs 318a-318n, CPUs 308a-308n, GPUs 322a-322n, etc.). Hardware dedicated to FaaS, such as Figure 3 The FaaS server configuration 300 described in

[0045] enables improved FaaS performance and adoption by more users and CSPs. The enhanced FaaS solution provides high flexibility and heterogeneity to achieve improved execution characteristics compared to existing solutions.

[0160] The FaaS server configuration 300 may include more or fewer of the components illustrated. For example, in one example, the FaaS server configuration 300 may include a CPU, such as the first CPU 308a. In another example, the FaaS server configuration 300 may include an FPGA (such as the first FPGA 318A) and a CPU (such as the CPU 308a). In yet another example, the FaaS server configuration 300 includes the first accelerator 314a but does not include any of the CPUs 308a-308n. In yet another example, the FaaS server configuration 300 includes multiple CPUs 308a-308n, multiple GPUs 322a-322n, an FPGA such as the FPGA 318a, and multiple specialized accelerators 314a-314n of different types.

[0161] Now turn Figure 4 , illustrates an embodiment of an enhanced FaaS system 400. The illustrated system 400 includes a user interface handler 402, an orchestrator 404, a FaaS architecture 406, a network, storage, and memory manager 408, and a security manager 410. In an embodiment, the system 400 receives FaaS function code 412 to be executed using the FaaS architecture 406, which provides a container / function execution environment. The illustrated system 400 also detects one or more FaaS events 414 received by the system 400, such as a user request to execute a user's computer code, such as the FaaS function code 412. In response to the FaaS event, the corresponding function code 412 is downloaded and instantiated into a container or another isolated execution alternative of the system 400. The user interface handler 402 is an event handling API proxy that routes event calls to the corresponding function of the computer code 412. The orchestrator 404 optimizes the execution of functions within available computing resources (e.g., whether executed by a container, a virtual machine, a process, etc.) by spinning up a container or other execution engine for the function code 412. If a function has recently been run in a container, the container is marked as a "warm" container. The orchestrator 404 can place new functions in containers that are already "warm" (and more generally prepared execution engines), thereby reducing instantiation wait times. The term "container" as used herein can be considered to be any form of execution engine for code (such as function code 412). A container can be a process, a process group, a virtual machine, a sandbox environment within a process address space, a process within a virtual machine, etc. The network, storage, and memory manager 408 can receive and store the function code 412, which is secured by the security manager 410. In one example, the security manager 410 ensures that the container runs within the VM for security and isolation.

[0162] Orchestrator 404 may also include the following subcomponents: telemetry manager 416, profile manager 418, machine learning / artificial intelligence (ML / AI) advisor 420, and service level agreement / quality of service (SLA / QoS) manager 422. Orchestrator 404 may profile data such as, for example, Figures 13A-13C The resource requirements and / or demand profiles of the functions described in the various embodiments, such as with reference to Figures 24A-24B The various embodiments described herein function static and dynamic profile information, and as hereinafter referred to Figures 40A-40B Dynamic profile described.

[0163] Additionally, the telemetry manager 416 can monitor and record resource requirements of functions, such as metrics (e.g., cache usage over time), and / or Figures 13A-13C The requirements profiles described in the various embodiments of Figures 7A-7C Background performance monitoring and specific performance telemetry generated during function execution, as discussed in Figure 22 The out-of-band (OOB) telemetry described in the various embodiments of Figures 24A-24B The telemetry and profile information described in the various embodiments, as well as the reference Figures 36A-36B In addition, the SLA / QoS manager 422 can be used to determine the telemetry information as described in the various embodiments of the present invention. Figures 21A-21D The embodiments described in the present invention may include address-based QoS, service class, page-level QoS and / or thread-level QoS, as well as reference Figures 39A-39C Describes compliance with QoS implementations, QoS list vectors, QoS mixes, security vectors, and / or QoS specifications.

[0164] It will be understood that orchestrator 404, telemetry manager 416, profile manager 418, ML / AI advisor 420, and SLA / QoS manager 422 can be combined and utilized in various ways to implement any of the embodiments described herein. Furthermore, different arrangements of orchestrator 404 can be used, where only a subset of telemetry manager 416, profile manager 418, ML / AI advisor 420, and SLA / QoS manager 422 are part of orchestrator 404.

[0165] In some embodiments, the enhanced FaaS system 400 collects telemetry information associated with the function being executed via the telemetry manager 416. The telemetry information can be used as part of the profile information about the function and stored by the profile manager 418. In order to map the execution of the function to different execution units to enhance efficiency while meeting the requirements of future invocations of the function (e.g., time and / or cost constraints), the SLA / QoS manager 422 can also query the profile information of the profile manager 418 in addition to seeking advice and / or guidance from the ML / AI advisor 420. In one embodiment, the ML / AI advisor 420 can perform anomaly detection based on previously observed scenarios and take corrective actions and / or make recommendations to the SLA / QoS advisor 422.

[0166] Orchestrator 404 can efficiently place FaaS functions for execution. In addition to the algorithmic and rule-based mechanisms embedded in orchestrator 404, ML / AI advisor 420 can guide orchestrator 404 in scheduling functions. Due to the continuous operation of FaaS system 400, a large amount of "telemetry" data can be collected by telemetry manager 416. Such telemetry data can capture the execution characteristics of functions on different ranges of the system and at different times / locations. Telemetry information can include various attributes such as CPU utilization, memory consumption, cache hierarchy behavior, I / O and networking behavior, time / location of execution, power consumption / energy consumption, security / privacy aspects of functions and their data (e.g., security requirements and malicious attack detection), users and applications calling functions, parameters and data used by functions, etc.

[0167] Profile manager 418 can process the collected telemetry information (in real time and offline), generate information summaries, and find correlations and anti-correlations in function execution behavior. Orchestrator 404 can then query profile manager 418 and ML / AI advisor 420 before determining various aspects of function placement (such as time, node, accelerator usage, etc.).

[0168] For example, orchestrator 404 can avoid scheduling two or more CPU-intensive functions on the same processor at the same time. Instead, orchestrator 404 can schedule a CPU-intensive function and a memory-hungry function (e.g., a function that requires a large amount of memory or has a large number of processor cache misses, or both) on the same processor. Furthermore, orchestrator 404 can avoid scheduling multiple I / O-intensive functions on the same system at the same time, and so on. Thus, orchestrator 404 can ensure that resource consumption of FaaS servers is balanced.

[0169] Furthermore, in addition to such rules embedded in orchestrator 404, ML / AI recommender 420 can automatically learn from a wealth of past information collected by telemetry manager 416 and processed by profile manager 418. For example, ML models collected and built by profile manager 418 can assist ML / AI recommender 420 in automatically determining the placement of functions for execution. As an example, this information may include possible ML models collected and processed by profile manager 418. The ML models may reveal that certain sets of functions executing simultaneously on the same system will result in unusually poor execution characteristics (e.g., exceptions, high latency execution, etc.).

[0170] Orchestrator 404 can determine and record the functions running on all servers under its control and their status. Orchestrator 404 can also be a collection of distributed orchestrators, each responsible for a subset of FaaS servers. When a new function is to be placed for execution, orchestrator 404 can query the ML / AI recommender for guidance on function placement.

[0171] Thus, orchestrator 404 can be "adaptive". As telemetry data is continuously collected by telemetry manager 416, enhanced FaaS system 400 can continuously learn about function execution behavior. This aspect can enable enhanced FaaS system 400 to automatically learn about new functions that have not been previously analyzed by FaaS system 400 as well as learn about phase shifts (e.g., sudden and / or significant changes in function execution behavior, such as changes in CPU usage, memory consumption, I / O operations, and network interactions) and seasonal effects. Seasonal effects can include external events whose occurrence and frequency in time affect function execution and execution characteristics. For example, functions affected by seasonal effects can include internet-enabled GPS, which is driven by road traffic and may be executed more frequently during rush hour or national holidays. Another example can include a streaming application that executes video content after regular business hours. For example, enhanced FaaS system 400 can automatically detect a sudden surge in calls to execute a particular group of functions or access a particular data set. Such information can then lead to setting up more FaaS servers for execution, or moving data closer to where the functions are executed, including from the data center to the edge of the network, and / or allocating more memory in platforms with disaggregated memory architectures.

[0172] Figure 5An embodiment of an enhanced FaaS system 500 including subsystems is shown. In the illustrated example, the FaaS system 500 includes a user experience subsystem 502, a security subsystem 504, software subsystems 506 (506a-506c), and a hardware subsystem 508 (508a-508d). The software subsystem 506 may include libraries 506a, frameworks 506b, platform and orchestration modules 506c, and the like. In addition, the illustrated hardware subsystem 508 includes a CPU 508a, a GPU 508b, memory and storage 508c, a networking component 508d, an accelerator 508e (e.g., an FPGA), and the like. The enhanced FaaS system 500 receives FaaS function code 510 to be executed and executes the function code 510 in response to one or more associated FaaS events 512.

[0173] The enhanced FaaS system 500 is therefore suitable for a more diverse market by creating opportunities for feature monetization. For smaller service providers such as communication service providers (CoSPs) and regional CSPs that do not own edge locations, FaaS represents an opportunity to participate more in the next wave of end-to-end applications and services. Many next-generation applications and services may require that a portion of the application or service be executed or provided close to the consumer or enterprise. Hosting a small portfolio of functions at these edge sites is a much easier task for smaller players than providing a rich platform as a service (PaaS) with all the development and operations (DevOps) tools, APIs, and services such as those provided by hyperscale CSPs.

[0174] The enhanced FaaS system 500 improves upon existing FaaS architectures and broadens FaaS parameters by increasing the lease, scale, and utilization of computing resources. FaaS could become the default model for developing cloud-based technologies, freeing developers from backend infrastructure maintenance and opening programming to more people, to name a few examples. In this way, the enhanced FaaS system 500 has the potential to provide developers with entirely new ways of working. For example, the illustrated system 500 supports and / or directly provides software that augments and extends FaaS provider offerings. FaaS service providers benefit from lower barriers to entry to FaaS solutions with better tooling, while users experience the increased ease of use provided by the system 500.

[0175] Although some examples in this article are discussed with respect to FaaS functions, the concepts are more broadly applicable to other types of software pods, such as, for example, non-FaaS containers, desktop applications, etc.

[0176] Enhanced FaaS architecture

[0177] Enhanced FaaS systems (e.g. Figure 4and Figure 5 The FaaS system exemplified in

[15] can also provide user-level capabilities for directly controlling hardware features using software functions and / or acceleration functions (e.g., FPGAs or other configurable logic). Such an approach may be advantageous for supporting privileged maintenance functions for solid-state drive (SSD) block devices, management functions for accelerators, and the like. Because functions are non-persistent tasks that run until completion, hardware-based control over a function's use of hardware provides greater efficiency and transparency than requiring other platform-level software (e.g., platform-level software functions that are indistinguishable from any other functions of user-level software) to mediate such access.

[0178] Figure 6A An enhanced FaaS architecture 600 for resource allocation and control is shown. In the illustrated architecture 600, a first function 602 ("F1") includes code 604 to be executed within a CPU container 608 and a security attestation token 606. The illustrated first function 602 is also associated with metadata 610 that defines a set of user-level capabilities, e.g., user-level capabilities 614, 618, 622, 626. In one example, the set of user-level capabilities corresponds to one or more features external to the container 608. For example, features 614 in the OS 612 may include memory management, system calls, etc., or any combination thereof. Additionally, features 618 in the VMM 616 (virtual machine monitor, e.g., a hypervisor) may include memory management, device management, network reconfiguration, access to network paths, virtual software reconfiguration, etc., or any combination thereof. In an embodiment, features 622 in the CPU 620 include resource director technology (RDT) monitor counters, RDT cache and memory control, platform quality of service monitor counters, platform quality of service cache and memory control, hardware performance monitoring counters, more direct control of DRAM, 3D XPoint, storage devices, or any combination thereof. Additionally, features 626 in the accelerator 624 (e.g., FPGA) include hardware reconfiguration, bitstreaming (e.g., FPGA implementation images), device resets, etc., or any combination thereof.

[0179] 606 is difficult, if not impossible, to forge and is tied to some enumerable, discoverable, or verifiable property of the activation of the first function 602. Thus, if the verification module 628 in the OS 612 and / or the verification module 630 in the VMM 616 determines that the security proof token 606 is valid, then the first function 602 is permitted to use features 614, 618, 622, and 626 (e.g., corresponding to user-level capabilities). The illustrated curved arrows from the first function 602 represent various calls that the first function 602 can make within the container 608 and outside the container 608. In one example, the licensed features cannot be retained or otherwise blocked from being exposed to the first function 602 by the VMM 616, OS 612, or other software "master" (e.g., hypervisor host, guest OS). In practice, the architecture 600 can extend / expand the set of user-level capabilities, as long as the extension does not cause correctness violations (e.g., protocol and / or syntax errors) or security violations. Extensions can include, for example, filtered views of OS / VMM counters, customized controls for OS / VMM operations, such as "association" (e.g., implementing association rules that establish relationships between VMs and hosts), or policy hints for memory tiering. However, if it is determined that the security proof token 606 is invalid, the architecture 600 blocks the first function 602 from using the user-level capabilities and continues execution with default features. Thus, the first function 602 can directly call (e.g., command) features outside of the container 608, thereby bypassing the container 608.

[0180] Figure 6A The accelerator 624 illustrated in FIG. 6 also includes one or more virtual interfaces 632 (632a-632c) that implement separate functions for reconfiguring the accelerator 624. In one example, the first function 602 reconfigures the accelerator 624 via the first virtual interface 632a, the second function 634 ("F2") reconfigures the accelerator 624 via the second virtual interface 632b, and the third function 636 ("F3") reconfigures the accelerator 624 via the third virtual interface 632c. Thus, the virtual interfaces 632 enable FaaS architectures such as Figure 6A The use of Single Root IO Virtualization (SR-IOV) and / or Serial Rapid IO (sRIO) in the architecture illustrated in FIG.

[0181] Now turn Figure 6B , Figure 6B Illustrate the use of enhanced FaaS architecture (such as Figure 6A Method 640 for managing user-level capabilities using the architecture shown in FIG. 640 can be generally implemented in an enhanced FaaS system, such as, for example, the system 202 ( FIG. 2 ), the system 300 ( Figure 3 ), System 400( Figure 4 ), and / or system 500 ( Figure 5 ). More specifically, the method 640 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed function logic hardware using, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0182] Illustrated processing block 642 provides for detecting a security attestation token associated with a function executed within a container. For example, the orchestrator provides the requested security attestation token when sending the function to the function invoker. Furthermore, the invoker may maintain a registry of such security attestation tokens, which is framework-dependent. The security attestation token may be generated by the invoker or orchestrator, for example, before the function is registered with the orchestrator by a developer or consumer of the function.

[0183] A determination as to whether the security proof token is valid can be made in block 644 by verifying the signature or other applicable method. If so, block 644 permits the function to use (e.g., direct access) a user-level capability set, where the user-level capability set corresponds to one or more features outside the container. The capability set may include, for example, self-monitoring, control of a portion of the virtual space, control of read protection for a range of pages, control of write protection for a range of pages, establishment of a namespace for objects stored in persistent memory, memory management, system call management, device management, network configuration, access to network paths, virtual software reconfiguration, and the like. Additionally, the individual features may be host processor (e.g., CPU) features, OS features, virtual machine (e.g., VMM) features, accelerator features, and the like. Additionally, the individual features may include granular reservation or prioritization of components of the FaaS architecture (such as processor sharing, cache sharing, I / O throughput sharing, accelerator sharing, and the like) using specific techniques such as Resource Director Technology (RDT) or similar technology implemented in GPUs, ASICs, PCIe hubs, FPGAs, etc.

[0184] Additionally, a determination may be made at block 648 as to whether the extension of the user-level capability set would cause a correctness violation (e.g., a protocol and / or syntax error) or a security violation. If not, illustrated block 650 performs the extension in response to the determination that the extension would not cause a correctness or security violation. If it is determined at block 648 that the extension would cause a correctness or security violation, the method 640 terminates. If it is determined at block 644 that the security proof token is invalid, block 652 blocks the function from using the user-level capability set.

[0185] Now turn Figure 6C , Figure 6C Illustrate the use of enhanced FaaS architecture (such as Figure 6A Method 660 for managing user-level capabilities using the architecture shown in FIG. 660 can be generally implemented in an enhanced FaaS system, such as, for example, the system 202 ( FIG. 2 ), the system 300 ( FIG. 3 ), and the system 300 ( FIG. 4 ) already discussed. Figure 3 ), System 400( Figure 4 ), and / or system 500 ( Figure 5 ). More specifically, the method 660 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed function logic hardware using, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0186] The illustrated processing block 662 provides for determining that a security attestation token associated with a function is valid by verifying a signature or other applicable method. The function may be executed within a container. Block 664 determines whether the function uses and / or requires access to one or more unavailable resources and / or features. For example, block 664 may determine that a resource or capability is unavailable when the resource or capability is unavailable at the user level and / or unavailable to the container. Capability sources may include, for example, self-monitoring, control of a portion of virtual space, control of read protection of a range of pages, control of write protection of a range of pages, establishment of a namespace for objects stored in persistent memory, memory management, system call management, device management, network configuration, access to network paths, virtual software reconfiguration, and the like. Individual resources may be host processor (e.g., CPU) features, OS features, virtual machine (e.g., VMM) features, accelerator features, and the like. Additionally, individual capabilities and / or resources may include granular reservation or prioritization of components of the FaaS architecture (such as processor sharing, cache sharing, I / O throughput sharing, accelerator sharing, and the like) using techniques such as Resource Director Technology (RDT) or similar technology implemented in GPUs, ASICs, PCIe hubs, FPGAs, etc.

[0187] If so, block 666 enumerates the required resources and / or capabilities needed for user-level access. In some embodiments, block 666 may further include a check as to whether the expansion of the user-level resource and / or capability set would result in a correctness violation (e.g., a protocol and / or syntax error) or a security violation, and any resources and / or capabilities that would result in a violation are not enumerated. Block 668 adds the enumerated resources and / or capabilities. For example, block 668 temporarily adds the enumerated resources and / or capabilities to the container of the function. The temporary addition persists for the duration of the function. Block 670 enables user-level access to the enumerated resources and / or capabilities.

[0188] After frame 670 is executed or when the function does not use any unavailable resources and / or capabilities, illustrated frame 672 determines whether to enable any optional, additional resources and / or capabilities. Frame 672 consults a table describing optional, additional resources and / or capabilities that are beneficial to performance or enhance simplicity (e.g., the ability to pin down pages, the ability to associate threads, etc.). If so, frame 674 can enumerate the optional, additional resources and / or capabilities. As described above, frame 674 can further perform a check to determine whether correctness violations (e.g., protocol and / or syntax errors) or security violations will occur when specific optional, additional resources and / or capabilities are enumerated and allowed for user-level access. If security or correctness violations will occur for specific optional, additional resources and / or capabilities, then the specific optional, additional resources and / or capabilities are not enumerated. Thus, frame 674 will only enumerate safe and stable optional, additional resources and / or capabilities. That is, only optional, additional resources and / or capabilities that do not pose a security problem and do not pose a risk to the stable operation of the system (such as deadlock, livelock, starvation, etc.) will be enumerated. Block 676 grants user-level access to safe and stable additional resources and / or capabilities for the duration of function execution.

[0189] After completion of block 676, the method terminates. If block 672 determines that no further capabilities and resources are to be enabled, the method terminates.

[0190] It is worth noting that methods 640 and 660 can be modified and combined. For example, blocks 664, 666, 668, 670, 672, 674, 676 can be replaced with one or more of blocks 646, 648, 650.

[0191] Additional Notes and Examples

[0192] Example 601 includes at least one computer-readable storage medium comprising a set of computer-executable program instructions that, when executed by a computing device, cause the computing device to: detect a security attestation token associated with a function executing within a container, permit the function to use a user-level capability set if the security attestation token is valid, and prevent the function from using the user-level capability set if the security attestation token is invalid, wherein the user-level capability set corresponds to one or more features external to the container.

[0193] Example 602 includes at least one computer-readable storage medium of example 601, wherein the program instructions, when executed by the computing device, cause the computing device to: perform at least one extension of the user-level capability set while a security proof token is valid, wherein the security proof token is valid while the extension of the user-level capability set does not cause a correctness violation or a security violation.

[0194] Example 603 includes at least one computer-readable storage medium of example 601, wherein the set of user-level capabilities is selected from the group consisting of: self-monitoring, control of a portion of a virtual space, control of read protection for a range of pages, control of write protection for a range of pages, establishment of a namespace for objects stored in persistent memory, memory management, system call management, device management, network reconfiguration, access to network paths, and virtual software reconfiguration.

[0195] Example 604 includes the at least one computer-readable storage medium of example 601 , wherein one or more characteristics are selected from the group consisting of: host processor characteristics, operating system characteristics, virtual machine characteristics, and accelerator characteristics.

[0196] Figure 7A A FaaS compute node is shown in which various virtual power performance monitoring unit (vPMU) events 700 are monitored at the function or container level during execution of a function 714 within a container 702. Events 700 can be configured to be monitored via an architectural interface 704 (e.g., hardware and / or software) that enables nested monitoring. With respect to nesting, monitoring can be done within a container or even within a function in the same manner as it is done in a virtual machine guest, in a bare metal operating system, or in a bare metal container. In other words, the ability to measure a certain architectural event (e.g., the number of instructions executed, the number of cache misses counted, etc.) or the ability to sample a software event counter (such as the number of page faults, pages paged in or out, the number of packets received or transmitted, etc.) does not depend on where the counter is read from. Furthermore, measurement at each level is possible to obtain a reading that reflects the contribution of that particular level (e.g., function, container, guest OS, host OS, etc.) to the reading.

[0197] In the illustrated example, events 700 include time executed, instructions per cycle (IPC), memory bandwidth, cache usage, I / O operations per second (IOPs), and network bandwidth, although other events may also be monitored / collected. The programming of the events 700 to be monitored can be unified by system software capabilities. In one example, the unification of events 700 is performed by a vPMU driver provided by, for example, container 702 and / or the host. As will be discussed in more detail, events 700 can be tracked via various virtual PMU counters (e.g., "shadow" counters that mirror other counters).

[0198] For example, at time t0, a first snapshot 706 of the counter value associated with the execution of function 714 in container 702 is generated. Similarly, at time t n Generate a second snapshot 708 of the counter value, which can be done at time t n+1 A third snapshot 710 of the counter value is generated, and so on. The counter value may generally quantify one or more of the events 700, wherein snapshots 706, 708, 710 may be generated in response to a new instruction, a new interface call, etc. Implementing snapshots 706, 708, 710 in hardware may minimize software overhead. One or more of snapshots 706, 708, 710 may be shadow snapshots that mirror other portions of other pre-existing snapshots at a deeper level of software nesting. Snapshots 706, 708, 710 and / or differences (e.g., delta values) between snapshots 706, 708, 710 are stored in a vPMU buffer 712, which may be used as a staging area for function 714. In the illustrated example, vPMU buffer 712 is exposed to one or more of function 714, orchestrator 716, or node manager 718 via an API endpoint 720 (e.g., a Hypertext Transfer Protocol / HTTP endpoint).

[0199] The illustrated solution can therefore be used to prevent capacity or bandwidth starvation in FaaS infrastructures that provide limited memory resources for each function call. More specifically, and by way of example, a developer of function 714 can program a vPMU to obtain OS metrics, memory cache, and memory access statistics via snapshots 706, 708, and 710 to monitor whether and how much memory paging or cache thrashing is occurring for function 714. Furthermore, multiple node-level vPMUs can be combined into a distributed vPMU.

[0200] Now turn Figure 7B, a vPMU buffer 722 is shown, wherein first metric data 724 associated with the execution of a function 726 (e.g., telemetry snapshots and / or differences between snapshots) is provided to the function 726 and container software 728. In the illustrated example, the container software 728 performs aggregation of the first metric data 724 (e.g., cache usage over time) and generates second metric data 730 based on the aggregation. Additionally, the function 726 may collect and process the first metric data 724 to generate third metric data 732.

[0201] Now turn Figure 7C , showing a method 734 for monitoring the performance of a function. The method 734 can be generally implemented in an enhanced FaaS system, such as, for example, the system 202 ( FIG. 2 ), the system 300 ( Figure 3 ), System 400( Figure 4 ), and / or system 500 ( Figure 5 More specifically, the method 734 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed function logic hardware using, for example, ASIC, CMOS, or TTL technology, or any combination thereof.

[0202] The illustrated processing block 736 provides for generating a first snapshot of a counter value associated with the execution of a function in the container at a first point in time of execution, wherein a second snapshot of the counter value is generated at a second point in time of execution at block 738. In one example, block 740 stores one or more of the first snapshot, the second snapshot, or a difference between the first and second snapshots to a vPMU buffer. Additionally, at block 742, the vPMU buffer may be exposed to one or more of the function, the orchestrator, or the node manager via an API endpoint.

[0203] The techniques described herein thus enable function developers to work in conjunction with containers or any other execution engine to demarcate the boundaries of specific performance telemetry generated during function execution via background performance monitoring. Accordingly, event orientation, minimal management, high scalability, resource scale units, and granular billing can all be achieved. This further allows for privileged-appropriate views of the collected snapshot data. For example, a container software module that hosts function code can observe and / or identify some events that may be obscured by a developer-provided module. For example, due to different architectures, when function code is hosted in a developer-provided module, the function code may not exhibit certain trends, but when hosted in a container software module, it may exhibit these certain trends. Thus, the techniques herein allow for granular performance tracking on platforms that developers do not have direct access to (e.g., on which the developer does not develop and / or debug function code). In some embodiments, the FaaS system (e.g., an orchestrator or scheduler) can use the collected snapshot data to achieve more efficient scheduling and allocation.

[0204] Additional Notes and Examples

[0205] Example 701 includes at least one computer-readable storage medium comprising a set of executable program instructions that, when executed by a computing device, cause the computing device to: generate a first snapshot of a counter value associated with function execution in a container, generate a second snapshot of the counter value associated with function execution in the container, and store one or more of the first snapshot, the second snapshot, or a difference between the first snapshot and the second snapshot to a virtual performance monitoring unit (PMU) buffer.

[0206] Example 702 includes the at least one computer-readable storage medium of example 701, wherein the program instructions, when executed by the computing device, cause the computing device to: expose the virtual PMU buffer to one or more of a function, an orchestrator, or a node manager via an application programming interface (API) endpoint.

[0207] Shared Memory Example

[0208] Some embodiments may advantageously provide buffer expansion for intra-kernel message, intra-system message, intra-platform message, and / or machine message communication. Conventional inter-function communication via HTTP may involve more overhead than when cooperating functions are co-located within the same core, the same system, the same platform, and / or any execution unit with shared memory. Some embodiments may provide shared memory to allow two cooperating functions to share a memory segment for replicating data content exchanged between the two functions. Advantageously, some embodiments may avoid inter-function communication overhead by bypassing the OS kernel and other HTTP layers.

[0209] Now turn Figure 8A An embodiment of an electronic processing system 810 may include a processor 811, a memory 812 communicatively coupled to the processor 811, and logic 813 communicatively coupled to the processor 811 and the memory 812 to share a memory region of the memory 812 between a first transient function and a second transient function that collaborates with the first transient function. In some embodiments, the logic 813 may be further configured to provide buffer expansion for intra-core, intra-system, intra-platform, and / or machine message communication between collaborating transient functions of the FaaS platform (such as the first and second transient functions). In some embodiments, the logic 813 may be configured to exchange data between the first transient function and the second transient function via the shared memory region. For example, the logic 813 may be configured to synchronize the exchanged data with at least one of a call instruction and a return instruction. In some embodiments, the logic 813 may be located in or co-located with various components (e.g., on the same die), including the processor 811, the memory 812, and the like.

[0210] Embodiments of each of the processor 811, memory 812, logic 813, and other components of the system 810 described above can be implemented in hardware, software, or any suitable combination thereof. For example, a hardware implementation may include configurable logic such as a programmable logic array (PLA), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), or fixed-function logic hardware using circuit technology such as an application-specific integrated circuit (ASIC), complementary metal oxide semiconductor (CMOS), or transistor-transistor logic (TTL) technology, or any combination thereof.

[0211] Alternatively or additionally, all or part of these components may be implemented in one or more modules as a set of logical instructions stored in a machine or computer readable storage medium (such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the component can be written in any combination of programming languages applicable / appropriate to one or more operating systems (OS), including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as the "C" programming language or similar programming languages. For example, the memory 812, persistent storage medium, or other memory may store a set of instructions that, when executed by the processor 811, causes the system 810 to implement one or more components, features, or aspects of the system 810 (e.g., logic 813, shared memory areas, providing buffer expansion, exchanging data, etc.). Embodiments of suitable processors may include general-purpose processors, special-purpose processors, CPUs, GPUs, controllers, microcontrollers, cores, execution units, etc.

[0212] Now turn Figure 8B , an embodiment of the semiconductor package device 820 may include one or more substrates 821 and logic 822 coupled to the one or more substrates 821, wherein the logic 822 is at least partially implemented in one or more configurable logic and fixed-function hardware logic. The logic 822 coupled to the one or more substrates 821 can be configured to share a memory area between a first transient function and a second transient function that cooperates with the first transient function. In some embodiments, the logic 822 can be further configured to provide buffer expansion for intra-core, intra-system and / or intra-platform machine message communication between collaborative transient functions of the FaaS platform. In some embodiments, the logic 822 can be configured to exchange data between the first transient function and the second transient function through a shared memory area. For example, the logic 822 can be configured to synchronize the exchanged data with at least one of a call instruction and a return instruction. In some embodiments, the logic 822 coupled to the one or more substrates 821 may include a transistor channel region located within the one or more substrates 821.

[0213] Embodiments of logic 822, and other components of device 820, can be implemented in hardware, software, or any combination thereof (including at least partial hardware implementations). For example, a hardware implementation may include configurable logic such as, for example, PLA, FPGA, CPLD, or fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof. In addition, portions of these components may be implemented in one or more modules as a set of logical instructions stored in a machine or computer readable storage medium (such as, RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the components can be written in any combination of one or more OS-appropriate / suitable programming languages, including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as "C" programming language or similar programming languages.

[0214] The apparatus 820 may implement the method 830 ( Figure 8C ) or any of the embodiments discussed herein. In some embodiments, the illustrated device 820 may include one or more substrates 821 (e.g., silicon, sapphire, gallium arsenide) and logic 822 (e.g., transistor arrays and other integrated circuit / IC components) coupled to the substrate 821. The logic 822 may be implemented, at least in part, in configurable logic or fixed-function logic hardware. In one example, the logic 822 may include a transistor channel region located within (e.g., embedded in) the substrate(s) 821. Thus, the interface between the logic 822 and the substrate(s) 821 may not be an abrupt junction. The logic 822 may also be considered to include an epitaxial layer grown on an initial wafer of substrate(s) 821.

[0215] Now turn Figure 8C , an embodiment of a method 830 for sharing memory between multiple FaaS functions may include: allocating a memory region for a first transient function at block 831, and sharing the memory region between the first transient function and a second transient function that collaborates with the first transient function at block 832. Some embodiments of method 830 may further include providing, at block 833, buffer expansion for intra-core machine message communication between collaborating transient functions of the FaaS platform. Some embodiments of method 830 may further include providing, at block 833, buffer expansion for intra-node machine message communication between transient functions of the FaaS platform. Method 830 may also include exchanging data between the first transient function and the second transient function via the shared memory region at block 834. For example, method 830 may include synchronizing the exchanged data with at least one of a call instruction and a return instruction at block 835.

[0216] Embodiments of method 830 may be implemented in systems, devices, computers, equipment, etc. (e.g., such as those described herein). More specifically, hardware implementations of method 830 may include configurable logic (such as, for example, PLA, FPGA, CPLD), or fixed-function logic hardware using circuit technology (such as, for example, ASIC, CMOS, or TTL technology), or any combination thereof. Alternatively or additionally, method 830 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium (such as, RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the component can be written in any combination of one or more OS-appropriate / suitable programming languages, including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as "C" programming language or similar programming languages.

[0217] For example, method 830 may be implemented on a computer-readable medium as described in conjunction with examples 814 to 817 below. Embodiments or portions of method 830 may be implemented in firmware, an application (e.g., via an application programming interface (API)), or driver software running on an operating system (OS). Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for an integrated circuit, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0218] Now turn Figure 8D , an embodiment of a method 870 for sharing memory between multiple FaaS functions may include: allocating a memory region for a first transient function at block 872, and sharing the memory region between the first transient function and a second transient function that collaborates with the first transient function at block 874. Some embodiments of method 870 may further include providing buffer expansion for intra-system and / or intra-platform machine message communication between collaborating transient functions of the FaaS platform at block 876. Method 830 may also include exchanging data between the first transient function and the second transient function via the shared memory region at block 878. For example, method 870 may include synchronizing the exchanged data with at least one of a call instruction and a return instruction at block 880.

[0219] Embodiments of method 870 may be implemented in systems, devices, computers, equipment, etc. (e.g., such as those described herein). More specifically, hardware implementations of method 870 may include configurable logic (such as, for example, PLA, FPGA, CPLD), or fixed-function logic hardware using circuit technology (such as, for example, ASIC, CMOS, or TTL technology), or any combination thereof. Alternatively or additionally, method 870 may be implemented in one or more modules as a set of logical instructions stored in a machine or computer readable storage medium (such as, RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the component can be written in any combination of one or more OS-appropriate / suitable programming languages, including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as "C" programming language or similar programming languages.

[0220] For example, method 870 may be implemented on a computer-readable medium as described in conjunction with examples 814 to 817 below. Embodiments or portions of method 870 may be implemented in firmware, an application (e.g., via an application programming interface (API)), or driver software running on an operating system (OS). Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for an integrated circuit, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0221] In some embodiments, data transfer between two cooperating functions located on the same machine and using memory-based transfer as described in method 870 does not require that the processes or threads performing the communication must enter the kernel, and thus they can implement data exchange from user applications. In addition, the new instructions can safely perform the necessary data exchange at the user level. That is, because the instructions can be designed to operate with hardware privileges (even higher than kernel privileges), the execution of the functions has the right to copy data from the container of one function to the container of another function. Alternatively, the actual copying can be performed with firmware or with a helper thread in the OS, but the move can be set by a new instruction that does not require the send and receive functions in kernel mode to activate the firmware or helper thread.

[0222] Now turn Figure 8EAn embodiment of an enhanced FaaS system 840 may include two collaborating FaaS functions 841 and 842 (e.g., functions A and B) and a shared memory 843 that can be configured to facilitate collaboration between the two FaaS functions 841 and 842. Some embodiments may advantageously provide a buffer / ISA extension for machine message / JSON communication between FaaS functions 841 and 842. Certain FaaS architectures may use a standard data interchange format, such as JSON, for inter-function communication over HTTP. When the collaborating FaaS functions are scheduled by a scheduler to co-locate on the same machine (e.g., a core or host), inter-function communication does not have to traverse the OS kernel and other HTTP layers. In some embodiments, the shared memory 843 may provide a communication method by allowing the two functions 841 and 842 to share a memory segment for copying data content exchanged between the two functions 841 and 842. For example, just before the callee function is called, parameters passed by the caller function can be copied from the caller to the shared memory. When the callee function returns, the response / JSON object content can be copied from the shared memory to the caller. Synchronization can occur naturally via call and return instructions.

[0223] Now turn Figure 8F , an embodiment of a function lookaside buffer (FLB) 850 may include entries f and g for two collaborating FaaS functions. Whether communication should occur via shared memory (e.g., or internal buffers) or through regular HTTP / kernel routing can be determined by checking whether the caller is local / proximate to the callee (e.g., within the kernel, within the system, within the platform, and / or an execution unit with shared memory), and then branching to run the appropriate code for packaging / copying data based on the value of this check. To speed up this check, the FLB 850 may include a translation lookaside buffer (TLB) similar to a hardware structure that can be used to cache the process / lambda IDs of co-located functions. Locality information can be updated as functions move around similar to page table entries, and any resident entries in the cache are invalidated when the function is removed from the host.

[0224] Now turn Figure 8G, an embodiment of an enhanced FaaS system 860 may include an orchestrator 862 in communication with two or more servers 863, 864, and 865. Function code F1 to F9 may be distributed among servers 863, 864, and 865, with collaboration between F1, F4, F5, F6, and F8. Because F4, F5, and F6 are co-located on the same server 864, some embodiments may advantageously utilize shared memory on server 864 to facilitate collaboration between F4, F5, and F6. Some embodiments may utilize a graph-based representation of what functions are running on what servers to achieve better box packaging and / or locality optimizations. Some embodiments may also utilize a graph-based representation of cross-functional communication patterns. For example, call chains across nodes may be shared as a graph-based representation.

[0225] In some embodiments, the OS can expose a function call API to each function. For example, the API framework can use the OS API for function calls instead of network communication (if available). Advantageously, the API framework can provide more efficient transmission than the network for remote calls. Some embodiments can avoid the network by utilizing 128-bit addresses to make all functions global and enable remote direct memory access (RDMA).

[0226] System 810( Figure 8A ), device 820( Figure 8B ), Method 830( Figure 8C ), FaaS system 840( Figure 8D )、FLB 850( Figure 8E ) and / or FaaS system 860( Figure 8F ) can be replaced by the FaaS platform 102 ( Figure 1 ), enhanced FaaS system 202 ( FIG. 2 ), FaaS server architecture 300 ( Figure 3 ), enhanced FaaS system ( Figure 4 ), and / or enhanced FaaS systems ( Figure 5 ) or be incorporated into it. For example, the software components of various embodiments (e.g., functional code, aspects of logic, etc.) can be incorporated into the FaaS software subsystem 506 ( Figure 5 ), and the hardware components of various embodiments (e.g., shared memory, FLB, aspects of logic, etc.) can be incorporated into the FaaS hardware subsystem 508 ( Figure 5 )middle.

[0227] Additional Notes and Examples

[0228] Example 800 includes at least one computer-readable storage medium comprising a set of executable program instructions that, when executed by a computing device, cause the computing device to: share a memory region of a memory between a first transient function and a second transient function that collaborates with the first transient function, utilizing 128-bit addresses, exposing function call interfaces for the first and second transient functions, and creating a graph-based representation including the first and second transient functions.

[0229] Example 801 includes an electronic processing system comprising: a processor, a memory communicatively coupled to the processor, and logic communicatively coupled to the processor and the memory for sharing a memory region of the memory between a first transient function and a second transient function that cooperates with the first transient function.

[0230] Example 802 includes the system of example 801 , wherein the logic is further for providing buffer augmentation for machine message communication between cooperating transient functions of a function-as-a-service platform.

[0231] Example 803 includes the system of any of Examples 801-802, wherein the logic is further for exchanging data between the first transient function and the second transient function via the shared memory region.

[0232] Example 804 includes the system of example 803, wherein the logic is further for synchronizing the exchanged data with at least one of a call instruction and a return instruction.

[0233] Example 805 includes a semiconductor package device comprising: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic, the logic coupled to the one or more substrates being used to share a memory region of a memory between a first transient function and a second transient function that cooperates with the first transient function.

[0234] Example 806 includes the means of example 805, wherein the logic is further for providing buffer augmentation for machine message communication between cooperating transient functions of a function-as-a-service platform.

[0235] Example 807 includes the means of any of Examples 805 to 806, wherein the logic is further for exchanging data between the first transient function and the second transient function via a shared memory region.

[0236] Example 808 includes the means of Example 807, wherein the logic is further for synchronizing the exchanged data with at least one of a call instruction and a return instruction.

[0237] Example 809 includes the apparatus of any of Examples 805 to 808, wherein the logic coupled to the one or more substrates includes a transistor channel region within the one or more substrates.

[0238] Example 810 includes a method of sharing memory, comprising allocating a memory region for a first transient function, and sharing the memory region between the first transient function and a second transient function that cooperates with the first transient function.

[0239] Example 811 includes the method of example 810, further comprising providing buffer augmentation for machine message communication between collaborative transient functions of a function-as-a-service platform.

[0240] Example 812 includes the method of any one of Examples 810 to 811, further comprising exchanging data between the first transient function and the second transient function through the shared memory region.

[0241] Example 813 includes the method of Example 812, further comprising synchronizing the exchanged data with at least one of a call instruction and a return instruction.

[0242] Container speculative execution example

[0243] Some embodiments may advantageously provide for speculative execution of containers before they are executed. Some functions involve long latency / startup times, which may slow down execution. Some embodiments may provide a run-before-execution mechanism to obtain data / instruction streams at the processor / core level and also to reserve and / or reallocate resources. Advantageously, some embodiments may reduce latency / startup times for functions that can take advantage of the run-ahead capability.

[0244] By way of illustration and not limitation, runahead may refer to a technique that allows a processor to continue (e.g., speculatively) executing instructions during cache miss cycles rather than stalling. Speculative execution may be used to generate instruction and data stream prefetches by detecting instruction / cache misses before they would otherwise occur, using idle execution resources. Manageable costs may include providing speculative execution support to preserve register file state and prevent speculative stores from modifying memory.

[0245] Some FaaS functions may benefit from blocking to execute variable-length (e.g., some very long) latency operations, such as queries to a database, calls to other FaaS services, etc. Some embodiments may provide run-before-execution techniques to fetch data / instruction streams at the processor / core level, and may also reserve / allocate resources (e.g., such as reserving bandwidth for accessing a database, warming containers / FaaS functions that may be called, relocating containers / FaaS functions to locations near the current function, etc.).

[0246] To enable such capabilities in a FaaS environment, some embodiments may provide copy-on-write technology to support speculative execution at the processor level, as well as runtime routines to replace externally visible operations (e.g., such as external function calls, database updates, etc.) with appropriately matching operations for reserving / reallocating resources. Some embodiments may utilize Multi-Key Total Memory Encryption (MKTME) to tag keys to functions. For example, some embodiments may use MKTME to provide a free speculation side channel with RDMA. In some embodiments, the programmer may be given a hint as to whether to perform a run before speculative execution via a data structure.

[0247] In one embodiment, such as in combination Figure 8A The electronic processing system described is configured to operate in advance to obtain information related to FaaS and block one or more variable-length latency operations based on the obtained FaaS-related information. In some embodiments, the electronic processing system can be further configured to reserve and / or reallocate resources based on the obtained FaaS-related information. Additionally or alternatively, the electronic processing system can be further configured to replace one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources. In some embodiments, components, such as logic, processors, memory, etc., can be located within or co-located with each other (e.g., on the same die).

[0248] In another embodiment, Figure 8B A semiconductor package device that is the same as or similar to the semiconductor package device illustrated in , may include: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic. The logic coupled to the one or more substrates may be configured to run in advance to obtain information related to FaaS, and block one or more variable-length wait time operations based on the obtained FaaS-related information. In some embodiments, the logic may be further configured to reserve and / or reallocate resources based on the obtained FaaS-related information. Additionally or alternatively, the logic may be further configured to replace one or more externally visible operations with corresponding matching operations for reserving and reallocating resources. In some embodiments, the logic coupled to the one or more substrates may include a transistor channel region located within the one or more substrates.

[0249] Now turn Figure 9A, embodiments of method 930 for providing function as a service may include: executing ahead of time to obtain information related to FaaS at block 931, and blocking one or more variable-length latency operations based on the obtained FaaS information at block 932. Some embodiments of method 930 may further include reserving and / or reallocating resources based on the obtained FaaS information at block 933. Additionally or alternatively, method 930 may also include replacing one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources at block 934.

[0250] Embodiments of method 930 may be implemented in systems, devices, computers, equipment, etc. (e.g., such as those described herein). More specifically, hardware implementations of method 930 may include configurable logic (such as, for example, PLA, FPGA, CPLD), or fixed-function logic hardware using circuit technology (such as, for example, ASIC, CMOS, or TTL technology), or any combination thereof. Alternatively or additionally, method 930 may be implemented in one or more modules as a set of logical instructions stored in a machine or computer-readable storage medium (such as, RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the component can be written in any combination of one or more OS-applicable / appropriate programming languages, including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as "C" programming language or similar programming languages.

[0251] For example, method 930 may be implemented on a computer-readable medium as described in conjunction with examples 931 to 933 below. Embodiments or portions of method 930 may be implemented in firmware, an application (e.g., via an application programming interface (API)), or driver software running on an operating system (OS). Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for an integrated circuit, state information that personalizes electronic circuits and / or other structural components that are native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0252] Now turn Figure 9B, an embodiment of the enhanced FaaS system 910 may include a fetch module 942 communicatively coupled to a resource manager 944. The fetch module 942 may include technology for running ahead to obtain FaaS-related information for a function to be executed. The resource manager 944 may include technology for blocking one or more variable-length wait time operations based on the obtained FaaS-related information. In some embodiments, the resource manager 944 may be further configured to reserve and / or reallocate resources based on the obtained FaaS-related information. Additionally or alternatively, the resource manager 944 may be further configured to replace one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources. Some embodiments of the system 940 may advantageously provide container execution prior to speculative execution (e.g., in a FaaS environment).

[0253] Now turn Figure 9C , an embodiment of the enhanced FaaS system 950 may include speculative execution of some FaaS functions. User 951 can use a browser 952 to display a web page 954, which includes multiple images, for example, img1.jpg to imgN.jpg. User 951 can rotate one or more images of the web page 954 via the image rotation function 953 through the FaaS system 950. The FaaS system 950 can determine the possible user intention to rotate one or more images (for example, img1.jpg to imgN.jpg) and can speculatively rotate the images to one or more alternative orientations. For example, the FaaS system 950 can detect usage patterns / sequences of image orientations and start various rotation functions in advance to obtain a better user experience and / or performance metric. Speculative execution can greatly reduce the waiting time for rotating images from a user perspective.

[0254] System, device, method 930 ( Figure 9A ), FaaS system 940( Figure 9B ), and / or FaaS system 950 ( Figure 9C ) can be replaced by the FaaS platform 102 ( Figure 1 ), enhanced FaaS system 202 ( FIG. 2 ), FaaS server architecture 300 ( Figure 3 ), enhanced FaaS system ( Figure 4 ), and / or enhanced FaaS systems ( Figure 5 ) or be incorporated into it. For example, the software components of various embodiments (e.g., streamers, resource managers, functional code, aspects of logic, etc.) can be incorporated into the FaaS software subsystem 506 ( Figure 5), and the hardware components of various embodiments (e.g., various queues / buffers, aspects of logic, etc.) can be incorporated into the FaaS hardware subsystem 508 ( Figure 5 )middle.

[0255] Additional Notes and Examples

[0256] Example 900 includes at least one computer-readable storage medium comprising a set of executable program instructions that, when executed by a computing device, cause the computing device to: execute in advance to obtain FaaS-related information for at least one function, block one or more variable-length latency operations based on the obtained FaaS-related information, tag keys to functions via multi-key full memory encryption, detect image-related information, and initiate image operations in advance based on the detected image-related information.

[0257] Example 901 includes an electronic processing system comprising: a processor, a memory communicatively coupled to the processor, and logic communicatively coupled to the processor and the memory, the logic configured to execute ahead of time to obtain FaaS-related information for at least one function, and to block one or more variable-length wait time operations based on the obtained FaaS-related information.

[0258] Example 902 includes the system of example 901, wherein the logic is further for one or more of reserving and reallocating resources based on the obtained FaaS-related information.

[0259] Example 903 includes the system of any of Examples 901-902, wherein the logic is further for replacing one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources.

[0260] Example 904 includes a semiconductor package device comprising: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic, the logic coupled to the one or more substrates being used to: run in advance to obtain FaaS-related information of at least one function, and block one or more variable-length wait time operations based on the obtained FaaS-related information.

[0261] Example 905 includes the means of example 904, wherein the logic is further for one or more of reserving and reallocating resources based on the obtained FaaS-related information.

[0262] Example 906 includes the means of any of Examples 904 to 905, wherein the logic is further for replacing one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources.

[0263] Example 907 includes the apparatus of any of Examples 904 to 906, wherein the logic coupled to the one or more substrates includes a transistor channel region within the one or more substrates.

[0264] Example 908 includes a method of providing function as a service, comprising: running ahead to obtain FaaS-related information about at least one function, and blocking one or more variable-length wait time operations based on the obtained FaaS-related information.

[0265] Example 909 includes the method of example 908, further comprising one or more of reserving and reallocating resources based on the obtained FaaS-related information.

[0266] Example 910 includes the method of any one of Examples 908 to 909, further comprising replacing one or more externally visible operations with corresponding matching operations for one or more of reserving and reallocating resources.

[0267] Platform feedback, multi-version features, and asynchronous feature examples

[0268] A conventional FaaS call may involve a large number of triggering events, including a final triggering event that calls the function. After the function is called, the work is dispatched to the platform with some containers (e.g., newly started, warm, etc.). But there may be too few resources to support the new call. The accumulated wait time may delay the execution of the function. The conventional trigger mechanism flows in one direction from the antecedent to the function call, which may be difficult to optimize between the capabilities of the FaaS system itself and external resources. In some FaaS systems, speeding up the startup time of the function increases the wait time to execute the function. Another problem is that FaaS data / actions can be difficult to synchronize across decomposed systems. Fully synchronous actions require the calling function to wait for the task to complete before calling the subsequent function. The resources used to make the calling function are tied together and wait.

[0269] Some embodiments of the enhanced FaaS system may provide feedback from the platform executing the function to the trigger mechanism, which feedback indicates that the platform is ripe for the next function call. Some embodiments may also provide advance notification of required resources / conditions, where a notification is returned from the platform when such resources are available or expected to be available and when such conditions are met (e.g., pull versus push). Some embodiments may also hide the startup time of the acceleration function by providing an alternative form of the acceleration function, which can be used while the acceleration function is started. In some embodiments, some functions may be identified as asynchronous to support service chaining. Advantageously, some embodiments may avoid resource starvation, provide better utilization of resources, and / or experience less latency (e.g., or apparent latency) in function execution. For example, some embodiments may release the calling function after dispatching the chained function (e.g., making the functions more atomic and modular).

[0270] In one embodiment, in combination with Figure 8A The same or similar electronic processing system described may include: a processor, a memory communicatively coupled to the processor, and logic communicatively coupled to the processor and the memory to receive a request for a subsequent function call from a trigger agent, and to provide feedback to the trigger agent to indicate readiness for the subsequent function call. Alternatively or additionally, the logic may be configured to provide one or more alternative forms of acceleration functionality that can be used while the acceleration function is initiated. Alternatively or additionally, the logic may be configured to identify one or more functions as asynchronous to support service chaining. In some embodiments, the logic may be located in or co-located with various components, including (e.g., on the same die) a processor, memory, etc.

[0271] In another embodiment, in combination with Figure 8B The same or similar semiconductor package device described may include: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic. The logic coupled to the one or more substrates may be configured to receive a request for a subsequent function call from a trigger agent, and to provide feedback to the trigger agent to indicate readiness for the subsequent function call. Alternatively or additionally, the logic may be configured to provide one or more alternative forms of acceleration functions that can be used while the acceleration function is started. Alternatively or additionally, the logic may be configured to identify one or more functions as asynchronous to support service chaining. In some embodiments, the logic coupled to the one or more substrates may include a transistor channel region located within the one or more substrates 1021.

[0272] Now turn Figure 10A , an embodiment of the method 1030 for providing function as a service may include: receiving a request for a subsequent function call from a trigger agent at block 1031, and providing feedback to the trigger agent at block 1032 to indicate readiness for the subsequent function call. Alternatively or additionally, the method 1030 may include providing one or more alternative forms of acceleration functions at block 1033, which may be used while the acceleration function is being initiated. Alternatively or additionally, the method 1030 may include identifying one or more functions as asynchronous to support service chaining at block 1034.

[0273] Embodiments of method 1030 may be implemented in systems, devices, computers, equipment, etc. (e.g., such as those described herein). More specifically, hardware implementations of method 1030 may include configurable logic (such as, for example, PLA, FPGA, CPLD), or fixed-function logic hardware using circuit technology (such as, for example, ASIC, CMOS, or TTL technology), or any combination thereof. Alternatively or additionally, method 1030 may be implemented in one or more modules as a set of logic instructions stored in a machine or computer readable storage medium (such as, RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the component can be written in any combination of one or more OS-appropriate / suitable programming languages, including object-oriented programming languages such as PYTHON, PERL, JAVA, SMALLTALK, C++, C#, and conventional procedural programming languages such as "C" programming language or similar programming languages.

[0274] For example, method 1030 may be implemented on a computer-readable medium as described in conjunction with examples 1011 to 1013 below. Embodiments or portions of method 1030 may be implemented in firmware, an application (e.g., via an application programming interface (API)), or driver software running on an operating system (OS). Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for an integrated circuit, state information that personalizes electronic circuits and / or other structural components that are native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0275] Platform Feedback Example

[0276] Now turn Figure 10B, embodiments of an enhanced FaaS system 1040 may include a FaaS platform 1042 communicatively coupled (e.g., wired or wirelessly) to a client 1044. Some embodiments may advantageously provide architectural support for asserting FaaS and / or hardware-accelerated FaaS (AFaaS) actions on platform events and callbacks. Hardware-accelerated FaaS or AFaaS actions are actions that can be executed on a Figure 3 318, accelerators 314, GPUs, smart NICs 302, etc., in the FaaS server 300 shown in FIG. For example, the FaaS platform 1042 may include a platform manager 1046 that includes technology for receiving requests for subsequent function calls from clients 1044 and providing feedback to clients 1044 indicating readiness for subsequent function calls. When a function is triggered (e.g., invoked), the daemon may cause delays due to, for example, premature dispatch conditions (e.g., waiting longer for better conditions, QoS considerations, etc.). Premature conditions may also be related to limited resource scenarios (e.g., storage, accelerators, etc.) that may be in high demand, and therefore there may be queues for access to these resources. Due to various constraints, a function may fail to execute (e.g., time out after 5 minutes) because it takes too long to start. Some embodiments may provide platform monitoring (e.g., performed by a platform invoker) and / or segment functions into individual segments / slices. The platform manager 1046 can then run the individual function pieces that do not require access to limited resources, pre-queue the data, start virtual functions to monitor the resources, and execute the remaining function pieces when ready. By dividing the function into segments, the client 1044 (e.g., a consumer) can execute callbacks at the appropriate time and / or pause the function until resources become available.

[0277] Some embodiments may provide two potentially independent but complementary capabilities, including 1) platform- and capability-aware submission and execution of FaaS actions, and 2) ad-hoc execution of FaaS actions based on conditions or event feedback. Some embodiments may provide flexibility in allowing consumers (e.g., clients 1044) to guide platform providers, and for platform providers to provide more collaborative and informed execution of actions based on individual platform, network, and power conditions.

[0278] Now turn Figure 10C, an embodiment of the enhanced FaaS system 1050 may include a function partitioner 1051, a platform monitor 1052, a function callback module 1053, and a function pause module 1054. For example, various platform conditions may be monitored by the platform monitor 1052, and when resources for performing the action are available and the platform meets various other fine conditions as described below, the actual triggering of the action may then be performed. An example of a fine platform condition is distance to data. For example, if a dataset resides on a machine or storage location, and a function that requires the dataset runs on a different machine, the function's access to the dataset must go through one or more network or storage links, making access much slower than if the dataset and function were on the same machine (e.g., they require a significant amount of disk or network IO operations), then such a fine platform condition is not met. The meta-scheduler may initiate data migration operations to meet such a "distance to data" condition, and once sufficient data has been migrated, the action may proceed. Alternatively, if the function needs to be pushed closer to where the data resides (e.g., on a remote node), the fine condition changes to "sufficient computing resources are available near the data." Another example of a subtle condition is cost. For example, it may be desirable for a function to be triggered only if the function can be executed at a lower cost (e.g., assuming all other conditions for triggering the function are met). This may be true, for example, if the platform has sufficient spare capacity in terms of power, CPU cycles, memory bandwidth, etc., so that the function can be given a best effort treatment and expected to be completed within a certain free but limited waiting time. In other words, this approach contemplates the triggering and / or actual dispatching of functions upon asserting that various direct and indirect measures of effectiveness, efficiency, and even security are met (e.g., an example of a security criterion could be that no sensitive services are running on the platform in order to allow execution of actions that are gray-listed for security).

[0279] For AFaaS, resource availability can also be such a platform condition, especially when AFaaS is used for low-density background training of artificial intelligence (AI) models. By doing so, the goal can be to achieve more deterministic execution of functions, especially when the condition under which the function is executed is itself a second triggering event (for example, when the first triggering event can simply be to the effect of "all logical state or time-based triggering precursors have been satisfied"). In fact, this technique can explicitly and systematically link the execution of functions to the availability of the right resources and the requirements projected on the function.

[0280] Function callback module 1053 can allow consumers (e.g., or software agents acting as consumer agents) to bundle the actual delivery of FaaS activities while receiving and processing the information requested by the module through feedback from the platform infrastructure. For example, callback module 1053 can utilize certain meta-functions that are always ready to be processed (e.g., "pre-triggered") so that callbacks naturally activate the consumer agent at the controller or invoker. This agent provides the necessary triggering for the functions that the consumer expects to be dispatched immediately and under the platform conditions that are just met.

[0281] Advantageously, some embodiments of monitoring and / or callback techniques in FaaS systems can support a "bring your own capabilities" (BYOC) approach to achieve richer collaboration, where consumers can reserve some resources with the tacit consent of the CSP (e.g., by explicitly acquiring some resources by individually requesting them in advance) while waiting for other, more scarce resources at the CSP to be allocated judiciously. For example, some transformation services can prepare everything on their own resources before calling FaaS to perform the transformation. Deep neural networks (DNNs) may be too large for a FaaS platform, so consumers can execute the DNN portion on their own prepaid resources at the CSP and then use the CSP's FaaS platform to perform the remainder of the desired functionality.

[0282] System, device, method 1030 ( Figure 10A )、FaaS system 1040( Figure 10B ), and / or FaaS system 1050 ( Figure 10C ) can be replaced by the FaaS platform 102 ( Figure 1 ), enhanced FaaS system 202 ( FIG. 2 ), FaaS server architecture 300 ( Figure 3 ), enhanced FaaS system ( Figure 4 ), and / or enhanced FaaS systems ( Figure 5 ) or be incorporated into it. For example, the software components of various embodiments (e.g., platform manager, functional partitioner, platform monitor, callback features, pause features, functional code, aspects of logic, etc.) can be incorporated into the FaaS software subsystem 506 ( Figure 5 ), and the hardware components of various embodiments (e.g., various queues / buffers, aspects of logic, etc.) can be incorporated into the FaaS hardware subsystem 508 ( Figure 5 )middle.

[0283] Multi-version feature example

[0284] Now turn Figure 11A, embodiments of function 1110 may include multiple options for instantiation (e.g., options 1 through N). Some embodiments may advantageously provide multiple versions of the same function 1110, and further provide multiple accelerated versions of the same function 1110. Multiple versions of the same function 1110, including multiple accelerated versions of the same function 1110, may differ from one another in terms of the time or resources required for their instantiation, as well as the time it takes to execute or run the instantiation when function 1110 needs to be executed. Some embodiments provide these multiple versions as multiple tenants launched in a progressive manner. Some containers (e.g., containers using accelerators or utilizing cores, where the cores vary depending on the machine) may take a while to start. Software functions may provide low-latency startup, but longer-latency execution. Hardware functions may provide long-latency startup, but short-latency execution. In some embodiments, function 1110 may include multiple options (e.g., instantiation), one of which may be selected for execution depending on various factors (e.g., caller requirements, FaaS platform conditions, etc.).

[0285] Now turn Figure 11B , embodiments of the enhanced FaaS system 1120 can support a code block A with multiple versions of function B. For example, a caller of code block A can identify different versions of function B, including a minimum list accelerator version Bm, a high accelerator version Bh, and a no accelerator version Bs. If a container is started for an accelerator (e.g., minimum list accelerator Bm), some embodiments can transfer each version of the B function to that container. Some embodiments can downgrade the acceleration container based on a gradually decreasing number of new arrivals of function B (e.g., releasing the high accelerator Bh and warming the minimum list version Bm, or transferring to the software / non-accelerator version Bs). Some embodiments can combine multiple function versions with prefetching of data near the physical location of function B.

[0286] Some embodiments can advantageously execute accelerated functions in a multi-tenant scenario in a manner that hides the expected non-trivial startup latency when starting a new accelerated function. For a caller running on a general-purpose CPU, the target of the call may correspond to a function, the initializer for that function or the host CPU's code is code block A, and the accelerated function may correspond to function B. By the time A receives control, B may already be initialized on the accelerator, or B may need to be mapped to the accelerator.

[0287] Typically, initialization of a Bh may take from a few milliseconds to tens of milliseconds. For example, FPGA resources may be arbitrated between multiple active, idle, and requested (e.g., queued) functions. If the Bh is not active, it may take some time to allocate resources for the Bh, retrieve the bitstream for the Bh from near or far memory, and then activate (e.g., start) the bitstream. Once activated, the bitstream for the Bh may be kept active for a certain period of time before the resources for the Bh are reduced and the Bh is recycled, to amortize the cost of starting the Bh over the many duty cycles that the Bh is kept around. To hide this latency, some embodiments may support multiple versions of Bh, including, for example, two additional alternatives Bm and Bs, where Bm may correspond to a minimally accelerated version of Bh that takes very little time to start but may take a long time to execute (e.g., because it executes loops in time rather than space), and where Bs may correspond to a software (e.g., CPU) version of B (e.g., a non-accelerated version of B that may be much less power and performance efficient than Bh but can start almost instantly if Bs is warm).

[0288] Now turn Figure 11C , an embodiment of the method 1130 for providing function as a service may include: determining whether a fully accelerated version of the function (e.g., a function executing in a container operating on high accelerator hardware) is active in box 1131, and if so, using the fully accelerated version in box 1132. Otherwise, the method 1130 may determine whether a partially accelerated version of the function (e.g., a function executing in a container operating on minimum slate accelerator hardware) is active in box 1133, and if so, using the partially accelerated version in box 1134. Otherwise, the method 1130 may use a non-accelerated version of the function in box 1135. Depending on customer needs / platform conditions, the method 1130 may further include speculatively launching the fully accelerated version or the partially accelerated version of the function in box 1136 (e.g., and switching to it when it becomes active).

[0289] In other words, code block A may use Bh where Bh is already active (e.g., the fastest and most efficient execution of B) and there is no setup time for Bh; otherwise, A may use Bm if Bm is already active (e.g., has no setup time but has a longer duration of execution); or A may use Bs if neither Bh nor Bm is active, after which A may start Bm or start Bh, and then use the started Bm or Bs based on the desired latency-performance-area-power tradeoff.

[0290] Some embodiments may first pre-launch the Bm version for those acceleration functions that are known to be frequent (e.g., through historical profiling). For less frequent acceleration functions, some embodiments may launch these minimal versions on demand, but initially use software version Bs if Bm or Bs has not yet been launched. If the demand for Bs or Bm exceeds a certain threshold in the recent time window, some embodiments may launch the acceleration function Bh and continue to use Bs or Bm for requests for B until Bh is fully activated.

[0291] Some embodiments may also collect moving window utilization and cost metrics for each activated B. If the collected information falls below a threshold, or if demand for other functionality increases, some embodiments may initiate the reclamation of Bh and, if Bm meets the requirements, reclaim the resources used for Bh in response to a new request to execute B, but activate Bm in its place. Conversely, if demand for B increases from below a threshold to above a threshold, some embodiments may initiate the activation of Bh and, after Bh has been activated, reclaim Bm if BM is active.

[0292] In selecting the thresholds for initiating Bm or Bh (e.g., when the utilization of Bs or Bm, respectively, increases) and in selecting the thresholds for reclaiming Bh or Bm (e.g., when the utilization of Bh and BM, respectively, decreases), some embodiments may take into account the service level agreement (SLA) input dynamically provided by the AFaaS control service. If no SLA input is provided, these thresholds may be heuristically and dynamically set based on the arrival rate of requests for B (e.g., or a moving window average of the arrival rate).

[0293] Some embodiments can utilize continuous monitoring of the duty cycle (utilization) normalized to the hardware cost of Bh to achieve two goals: 1) determine how long to keep hB active, and 2) accumulate historical information so that future activations of Bh can be accelerated by using historical frequency and duration of active utilization as a guide. A greedy box packing heuristic can be used to determine the threshold for switching from the Bs to the Bm to the Bh version of a function. These statistics can be updated using epilogue for Bh and can be used to update in-memory statistics that future activations of code block A use to select between Bs, Bm, and BH.

[0294] Function 1110( Figure 11A )、FaaS system 1120( Figure 11B ), and / or method 1130 ( Figure 11C ) can be replaced by the FaaS platform 102 ( Figure 1 ), enhanced FaaS system 202 ( FIG. 2 ), FaaS server architecture 300 ( Figure 3), enhanced FaaS system ( Figure 4 ), and / or enhanced FaaS systems ( Figure 5 ) or be incorporated into it. For example, the software components of various embodiments (e.g., multiple versions of functional code, aspects of logic, etc.) can be incorporated into the FaaS software subsystem 506 ( Figure 5 ), and the hardware components of various embodiments (e.g., monitoring, aspects of logic, etc.) can be incorporated into the FaaS hardware subsystem 508 ( Figure 5 )middle.

[0295] Async Function Example

[0296] Now turn Figure 12 , an embodiment of the enhanced FaaS system 1210 may include: a scheduler 1212 having queues 1214, the queues 1214 including an event queue 1218, a function queue 1216, a trigger event queue 1220, and the like. Some embodiments may advantageously provide efficient synchronization between distributed actions. Chained functions that are linked through intermediary actions (for example, function X needs to call an action to store data to be used by function Y) may result in several calls. Consider action X, which needs to perform a network or storage operation P before it can consider itself completed, and triggers another action Y to process the data passed or filled in the storage area by the above operation P. In a conventional FaaS system, for example, action X must wait for the network or storage operation P to complete the filling and then call or trigger action Y. The scheduler 1212 may need to wait for a while, and during the waiting time, resources are being used by action X. Some embodiments of the enhanced FaaS solution can advantageously enable action X to trigger network or storage operation P and modify network or storage operation P so that the completion of network or storage operation P is used to trigger action Y, so that action X can be discarded before network or storage operation P completes to free up resources consumed by action X. Some embodiments can also allow for more interesting parallelization (e.g., modular design) because the root (e.g., parent function) does not have to wait so that other functions triggered by the completion can begin execution. Some embodiments can be implemented by a hardware queue manager (HQM).

[0297] Many distributed actions need to be synchronized, even if only to satisfy the requirement that state committed for durability or passed for visibility be updated via a serializable sequence of operations. Thus, for example, time should not appear to have moved backwards in the way that the results of these operations are recorded on durable media or observed from a third party. However, in some cases, the approach taken in a distribution system of processes may be too conservative. For example, even when the processes being employed are already data-parallel (e.g., they operate on disjoint partitions of the data), all agents may proceed across a barrier, such as in a two-phase lock. For low-latency FaaS actions that need to be executed across a set of coordinating nodes, such overhead is prohibitively expensive and they can introduce unnecessary delays.

[0298] Because some or most of the actual coordination may need to occur in the context of updating state for persistence or communication, some embodiments of enhanced FaaS solutions can extend the fabric and / or storage interfaces to support asynchronous submission of payloads for communication and storage (e.g., as a storage function or as a network function) so that updates to any given range of data are performed in a globally consistent temporal order. Accordingly, if the dependencies of two operations X and Y are chained such that X performs an asynchronous update to a block of data on a shared disk and X's completion triggers Y (e.g., where Y can consume updates made by X (a read-after-write or RAW dependency) or overwrite updates made by X (a write-after-write or WAW dependency)), some embodiments of chaining "storage FaaS actions" can avoid violating required ordering dependencies. Some embodiments can abstract the storage actions themselves into chained FaaS actions.

[0299] By way of explanation and not limitation, distributed functions may differ from distributed processes in that functions may follow a close-to-open or capture-to-release consistency model for shared data (e.g., because functions may not have any concept of session state beyond their finite duration or a run-until-completion model). Instead of action X requiring a network or storage operation P to be performed synchronously with action X, before signaling the completion of action X and triggering action Y (e.g., represented as [X, P] => Y), in some embodiments, one function X may complete, then execute (e.g., trigger) a linked action P (e.g., storage, network, etc.), and when P is scheduled and executed, specify that the completion of P may trigger function Y (e.g., represented as X => [P => Y]).

[0300] For example, function queue 1216 may include a queue of functions to be executed. Function Y (or some indication of function Y) may be stored in function queue 1216, for example, as a first function. Event queue 1218 may include different events for executing trigger events, such as "upon a first event (e.g., completion of a network or storage operation P) execute a first trigger event from trigger event queue 1220." Trigger event queue 1220 may store various trigger events for causing one or more functions in function queue 1216 to execute. The first trigger event in trigger event queue 1220 may include triggering a first function (e.g., function Y) from function queue 1216. Thus, upon completion of network or storage operation P, the system may determine from event queue 1218 that the first event (e.g., completion of network or storage operation P) has occurred to cause execution of the first trigger event from trigger event queue 1220. The first trigger event may then cause execution of one or more functions (e.g., function Y) from function queue 1216.

[0301] The conventional sequence [X, P] => Y requires function X to synchronously execute update action P (e.g., wait for P) before retiring, thereby increasing latency and potentially causing scaling bottlenecks (e.g., if X needs to contend for various locks and resources to execute P). Y itself may be related to X. For example, Y may be a continuation of X and require some credentials to be available to it as a result of executing P. As an example, X may be an image processing task that processes a certain camera feed, P may be a task that updates an image database, and Y may be a task that needs to process the most recent update to see if it contains a particular target object or pattern. In some embodiments, task P may be considered a "fire and forget" task related to X, so that resources allocated to X can be released after P is launched.

[0302] In some embodiments, such synchronization and coordination of capturing and releasing locks and executing two-phase transactions in a distributed system can be replaced by asynchronous command queuing techniques for high-performance fabric and storage operations (e.g., smart NICs, smart cards, etc.). This may be particularly useful for "rack scale design" or RSD architectures, where pooled storage and pooled memory can be used by different execution engines in a shared, flat access paradigm so that hardware-based queuing mechanisms can avoid the need for locks, conditions, etc., and at the same time avoid the need for more advanced software protocols (e.g., reader-writer lock management, deadlock detection and resolution, etc.). Some embodiments may utilize techniques similar to HQM. For example, tasks may be delegated to HQM to manage the execution of tasks.

[0303] System 1210( Figure 12) can be replaced by the FaaS platform 102 ( Figure 1 ), enhanced FaaS system 202 ( FIG. 2 ), FaaS server architecture 300 ( Figure 3 ), enhanced FaaS system ( Figure 4 ) and / or enhanced FaaS systems ( Figure 5 ) or be incorporated into it. For example, the software components of various embodiments (e.g., scheduler, function code, aspects of logic, etc.) can be incorporated into the FaaS software subsystem 506 ( Figure 5 ), and the hardware components of various embodiments (e.g., HQM, aspects of logic, etc.) can be incorporated into the FaaS hardware subsystem 508 ( Figure 5 )middle.

[0304] Additional Notes and Examples

[0305] Example 1200 includes at least one computer-readable storage medium comprising a set of executable program instructions that, when executed by a computing device, cause the computing device to: receive a request for a subsequent function call from a trigger agent, provide feedback to the trigger agent to indicate readiness for the subsequent function call, provide progressive multi-version initiation of a multi-tenant acceleration function, provide synchronization between distributed actions, and trigger the subsequent function call when criteria are met.

[0306] Example 1201 includes an electronic processing system comprising: a processor, a memory communicatively coupled to the processor, and logic communicatively coupled to the processor and the memory, the logic for receiving a request for a subsequent function call from a trigger agent and providing feedback to the trigger agent to indicate readiness for the subsequent function call.

[0307] Example 1202 includes the system of example 1201, wherein the logic is further for providing one or more alternative forms of the acceleration function that can be used while the acceleration function is enabled.

[0308] Example 1203 includes the system of any of Examples 1201 - 1202 , wherein the logic is further for identifying one or more functions as being asynchronous to support service chaining.

[0309] Example 1204 includes a semiconductor package device comprising: one or more substrates, and logic coupled to the one or more substrates, wherein the logic is at least partially implemented in one or more of configurable logic and fixed-function hardware logic, the logic coupled to the one or more substrates being used to receive a request for a subsequent function call from a trigger agent, and to provide feedback to the trigger agent to indicate readiness for the subsequent function call.

[0310] Example 1205 includes the apparatus of example 1204, wherein the logic is further for providing one or more alternative forms of the acceleration function that can be used while the acceleration function is enabled.

[0311] Example 1206 includes the means of any of Examples 1204 to 1205, wherein the logic is further for identifying one or more functions as being asynchronous to support service chaining.

[0312] Example 1207 includes the apparatus of any of Examples 1204 to 1206, wherein the logic coupled to the one or more substrates includes a transistor channel region within the one or more substrates.

[0313] Example 1208 includes a method of providing function as a service, comprising receiving a request for a subsequent function call from a trigger agent, and providing feedback to the trigger agent to indicate readiness for the subsequent function call.

[0314] Example 1209 includes the method of Example 1208, further comprising providing one or more alternative forms of the acceleration function that can be used while the acceleration function is enabled.

[0315] Example 1210 includes the method of any one of Examples 1208 to 1209, further comprising identifying one or more functions as being asynchronous to support service chaining.

[0316] Performance-enhanced computing architecture for FaaS

[0317] Serverless computing can allow FaaS and non-FaaS functions to execute on the same compute node. For example, similar to hyperthreading, it is possible to merge a large number of independent FaaS functions into the gaps (idle periods) between normal cloud applications (e.g., non-FaaS functions). Gaps can be gaps that cannot be filled by normal cloud applications in any case. Hyperthreading can use additional threads to utilize idle microarchitectural resources that are not fully exercised by only one thread per core.

[0318] Thus, FaaS and non-FaaS functions can be merged into multi-tenancy to operate concurrently on, for example, the same compute node (e.g., server or processor core). In such multi-tenancy scenarios, a noisy neighbor problem may exist, where cloud service provider (CSP) co-tenants monopolize resources (e.g., bandwidth, disk I / O, CPU), which may negatively impact the performance of other co-tenants. For example, non-FaaS functions may absorb resource bandwidth that is not fully subscribed by FaaS functions all the time, and vice versa. This leads to uneven cloud network performance across co-tenants and increases latency for FaaS function execution or degrades performance for non-FaaS co-tenants. The new noisy neighbor problem may be qualitatively unique in that it concerns efficient, adaptive, fine-grained, and fair sharing of resources within each compute node (e.g., at the processor core level), unlike other designs that assign or reserve resources (such as cache capacity, memory bandwidth, or processor scheduling priority) at the virtual machine or container level in current existing solutions.

[0319] The various embodiments described below can schedule each FaaS function and non-FaaS function to avoid noisy neighbor issues at the much higher multi-tenancy levels represented by these mergers. Doing so can reduce or stabilize latency caused by waiting, reduce function failures due to timeout errors, and control variability in execution performance. In addition, some embodiments can enhance resource distribution. For example, functions can be provisioned to nodes with dedicated hardware accelerators to enhance function execution.

[0320] Some embodiments may provide a ranged allocation of core execution resources among multiple hardware threads through prioritized scheduling. Prioritized scheduling may schedule functions with non-conflicting demand profiles for simultaneous multi-threaded (SMT) execution on a shared core (e.g., SMT4, SMT8, etc.) so that each function receives fair use of at least those resources for which the function is most sensitive. By doing so, inter-function resource conflicts or noisy neighbor problems may be avoided. Thus, some embodiments may include methods and tools and feedback mechanisms required for efficient and agile, short-duration, and finely delimited allocation of core execution resources during SMT (e.g., SMT2, SMT4).

[0321] Steering Figure 13A, illustrates a performance-enhancing computing architecture 1300 for FaaS that reduces resource contention. As described below, a server 1302 can reduce resource contention by distributing first through third functions 1312, 1314, 1316 to allow the first through third functions 1312, 1314, 1316 to have first through third resource allocations 1326, 1328, 1330 sufficient for execution. Furthermore, in some embodiments, fourth through sixth functions 1318, 1320, 1322 executing on first through third compute nodes 1304a-1304c can have allocations of the first through third resources that do not compromise access to the first through third resources by the first through third functions 1312, 1314, 1316.

[0322] Initially, the efficiency-enhanced server 1302 may receive requests for first to third functions 1312, 1314, 1316 (e.g., FaaS functions) from event-based triggering devices or simply triggering devices 1306, 1308, 1310 (e.g., individual computing devices). The requests for the first to third functions 1312, 1314, 1316 may be invoked by applications running on the triggering devices 1306, 1308, 1310. The server 1302 may determine sensitive resources for the first to third functions 1312, 1314, 1316. For example, the first function 1312 may require access to a first resource for execution, the second function 1314 may require access to a second resource for execution, and the third function 1316 may require access to a third resource for execution. The first to third resources may be any type of computing resources (eg, hardware resources, hardware accelerators, bandwidth, arithmetic logic units, power, frequency, etc.) required by the first to third functions 1312 , 1314 , 1316 during execution, and may be different from one another.

[0323] The server 1302 may direct hardware resource allocation so that each of the first to third functions 1312, 1314, 1316 receives fair use of at least the first to third resources that are sensitive to execution. Specifically, the server 1302 may schedule the first to third functions 1312, 1314, 1316 to execute on the respective computing nodes 1304a-1304c at different timings so that the first to third functions 1312, 1314, 1316 can access the first to third resources without resource contention. For example, to avoid resource contention, the server 1302 may distribute the first function 1312 to the first computing node 1304a so that the first function 1312 has a first resource allocation 1326 sufficient for execution. The server 1302 may distribute the second function 1314 to the third computing node 1304c so that the second function 1314 has a second resource allocation 1328 sufficient for execution. The server 1302 may distribute the third function 1316 to the second compute node 1304b such that the third function 1316 has a third resource allocation 1330 sufficient for execution.

[0324] Prior to execution, the server 1302 may speculatively determine the first to third resources required by the first to third functions 1312, 1314, 1316. For example, the server 1302 may determine the sensitive resources of the first to third functions 1312, 1314, 1316 by analyzing associated implementations (e.g., source code, transcoding, historical requirements, etc. of similar or identical functions). Thus, the server 1302 may identify the required resource allocation required by the function, and / or the type of resources required by the function, and mark such resources as sensitive resources. Resources may include one or more of power consumption, firmware requirements, hardware requirements (e.g., bandwidth requirements, accelerators, the number or fraction of available resources in CPU instruction fetches, TLBs, branch target buffers (BTBs), reservation stations, operation ports (such as arithmetic logic units (ALUs)), etc.), or clock frequency requirements. Resources may also be reserved for over-allocation, for example, by not allowing a CPU core to enter turbo execution to control performance variability under different multi-tenant conditions.

[0325] In some embodiments, server 1302 may determine whether one or more resource allocations required by each of first to third functions 1312, 1314, 1316 are above a threshold. If so, server 1302 may mark the one or more resources as sensitive resources. The threshold may correspond to the average historical resource availability at first to third computing nodes 1304a-1304c. The threshold may also be set to the current resource availability at each of one or more computing nodes 1304a-1304c.

[0326] The server 1302 may further schedule the first to third functions 1312, 1314, 1316 to be executed on different nodes among the computing nodes 1304a-1304c and / or at different timings based on the speculatively determined first to third resources to avoid resource conflicts. The server 1302 may thus reduce latency and increase the completion rate of the first to third functions 1312, 1314, 1316.

[0327] For example, the first function 1312 may perform a memory bandwidth intensive function, and thus the first resource (e.g., a sensitive resource) may be a high bandwidth resource. The second function 1314 may perform an ALU computation intensive function, and thus the second resource (e.g., a sensitive resource) may be an ALU resource. The third function 1316 may include power intensive operations, and thus the third resource (e.g., a sensitive resource) may be a high power resource.

[0328] As mentioned above, server 1302 can receive requests from trigger devices 1306, 1308, 1310 to execute first to third functions 1312, 1314, 1316 and distribute the first to third functions 1312, 1314, 1316 to the respective computing nodes 1304a-1304c to avoid resource contention. In some embodiments, server 1302 may have already called fourth to sixth functions 1318, 1320, 1322 (e.g., non-FaaS functions) executed at computing nodes 1304a-1304c. Server 1302 can identify that the first to third resources are being widely utilized by the fourth to sixth functions 1318, 1320, 1322 at the respective computing nodes 1304a-1304c by referring to the first to third resource allocations 1332, 1334, 1336 of the fourth to sixth functions 1318, 1320, 1322.

[0329] Server 1302 can distribute first function 1312 to first computing node 1304a to avoid resource contention. Specifically, the first resource required by first function 1312 is different from the third resource required by sixth function 1322, thereby avoiding resource contention between first function 1312 and sixth function 1322. In contrast, if first function 1312 were provided to third computing node 1304c, resource contention might occur because both first function 1312 and fourth function 1318 require the first resource, which is a sensitive resource.

[0330] As explained above, the first function 1312 and the fourth function 1318 require a high allocation of the first resource, and the high allocation may exceed the availability of the first resource on the third computing node 1304c. For example, although the first function 1312 may be able to access the first resource at the first node 1304c in a limited amount, the first function 1312 may not have sufficient access to the first resource to complete execution because the first resource has been allocated in large quantities to the fourth function 1318 via the first resource allocation 1332. Similarly, the third function 1316 is provided to the second computing node 1304b, and the second function 1314 may be provided to the third computing node 1304c.

[0331] In some embodiments, server 1302 may determine that sixth function 1322 is utilizing a first resource at first compute node 1304a. In such a scenario, server 1302 may determine whether first function 1312 will have sufficient access to the first resource to complete execution. For example, if sixth function 1322 is allocated only a small amount of the first resource and / or has only a small amount of access to the first resource, first function 1312 may still receive an allocation of the first resource that will facilitate execution.

[0332] Thus, the server 1302 can determine whether a function can access a sensitive resource based on the availability of the resource. For example, the server 1302 can follow the following equation 1300 to determine the total available allocation at the computing node:

[0333] Total available allocations = total potential allocations - existing allocations

[0334] Total available allocation=Total potentlal allocation-Existingallocatlon(s)

[0335] Equation 1300

[0336] In the above equation 1300, the total potential allocation is the total potential allocation of the sensitive resource at the computing node, and the existing allocation is the current allocation of the sensitive resource at the computing node, such as allocation to other functions. The server 1302 can follow the following equation 1301 to determine whether there is sufficient allocation of the sensitive resource:

[0337] Sensitive resource requirement ≤ total available allocation

[0338] Sensitive resource requirement≤Total avallable allocatlon

[0339] Equation 1301

[0340] Sensitive Resource Requirement is the sensitive resource requirement of the function. If the above equation is true, then sufficient allocation exists. That is, if the sensitive resource requirement is less than or equal to the total available allocation of sensitive resources at the compute node, then sufficient allocation exists to complete the execution of the function at the compute node.

[0341] In some embodiments, server 1302 may reduce first to third resource allocations 1332, 1334, 1336 for fourth to sixth functions 1318, 1320, 1322 and allocate those first to third resources to first to third functions 1312, 1314, 1316, so that sensitive resources are evenly distributed. For example, in some embodiments, server 1302 may distribute first function 1312 to third computing node 1304c instead of first computing node 1304a. Both first function 1312 and fourth function 1318 require the first resource as a sensitive resource, and therefore there may be resource contention. To reduce resource contention, server 1302 may reduce first resource allocation 1332 for fourth function 1318 and increase first resource allocation 1326 for first function 1312. Doing so may create an equal distribution of first resources between fourth function 1318 and first function 1312.

[0342] In some embodiments, server 1302 can schedule the timing of the first to third functions 1312, 1314, 1316 to avoid resource contention. For example, in some embodiments, server 1302 can provide first function 1312 to third computing node 1304c. As described above, both first function 1312 and fourth function 1318 require the first resource as a sensitive resource, and therefore there may be resource contention. To avoid resource contention, server 1302 can schedule first function 1312 to execute on third computing node 1304c after fourth function 1318 has completed execution. Doing so can avoid resource contention between the first function 1312 and fourth function 1318.

[0343] In some embodiments, the server 1302 may determine that a hardware accelerator and / or an FPGA may enhance the execution of one of the functions 1312, 1314, 1316. The server 1302 may accordingly schedule one of the functions 1312, 1314, 1316 to access the hardware accelerator.

[0344] In some embodiments, the computing nodes 1304a-1304c may be processor cores, computing devices, or servers. In some embodiments, the triggering devices 1306, 1308, 1310 may call an application running on a remote server (not illustrated), which in turn provides a request for functions 1312, 1314, 1316 to the server 1302. In some embodiments, the server 1302 may call functions 1312, 1314, 1316. The triggering devices 1306, 1308, 1310 may include, for example, a laptop computer, a mobile device, a server, a desktop computer, etc. In some embodiments, the server 1302 may include first to third computing nodes 1304a-1304c. Furthermore, in some embodiments, the server 1302 may include a processor core, a computing device, or a server. Figure 4 An enhanced FaaS system 400 , such as an orchestrator 404 .

[0345] Figure 13B An enhanced scheduling process 1338 is illustrated for Figure 13A 1 . In the present example, band 1 and band 2 for each of function 0 and function 1 may be allocated hardware resources. For example, band 1 may represent hardware accelerator resources, while band 2 may represent allocated processor core resources. For example, in a scenario where function 0 and function 1 are running on the same compute node at the same time, server 1302 may speculatively analyze function 0 and function 1 to identify predicted instructions per cycle (IPC) resources 1340 , 1342 allocated to function 0 and function 1 . Function 1 may be, for example, a non-FaaS function, while function 0 may be a FaaS function. As described above, function 1 may be allocated an unfairly high amount of resources 1340 , 1342 (e.g., allocated hardware resources) relative to function 0. For example, if function 1 begins operating before function 0, function 1 may have an unfair resource allocation.

[0346] Thus, the scheduling of function 0 and function 1 can be modified, as illustrated by arrow 1344, to achieve fairer box packing and / or be scheduled at different times and / or be scheduled at different nodes and / or hardware resources can be redistributed to establish fairness (e.g., equal amounts) between hardware resources required for higher IPC. Thus, function 0 can be allocated an amount of resources 1346 (e.g., allocated hardware resources) to achieve an IPC equal to the IPC achieved by the amount of resources 1348 (e.g., allocated hardware resources) allocated to function 1. Although not illustrated, power and frequency allocation can also be modified to be evenly distributed between function 0 and function 1.

[0347] Figure 13CA method 1350 for scheduling functions in a CSP environment is shown and may be implemented by Figure 13A Server 1302 and / or Figure 4 Enhanced FaaS system 400 (such as orchestrator 404) performs the method 1350. Method 1350 can be implemented in one or more modules as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, PLA, FPGA, CPLD, etc., in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination thereof.

[0348] For example, computer program code for implementing the operations shown in method 1350 may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, SMALLTALK, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0349] Illustrated processing block 1352 may include determining, for each respective function in a plurality of functions, one or more sensitive resources required to execute the respective function. Illustrated processing block 1354 may include determining a resource conflict between one or more sensitive resources of a first function and one or more sensitive resources of a second function. Illustrated processing block 1356 may include scheduling one or more of the first or second functions to avoid resource conflicts. For example, illustrated processing block 1356 may include scheduling one or more of the first or second functions to execute on the same node at completely non-overlapping times, to execute on different nodes, and / or to redistribute resource allocations from one function to another to establish fairness and maintain quality of service. Illustrated processing block 1358 may include scheduling one or more sensitive resources to the first and second functions to achieve target performance gains for the first and second functions, respectively. For example, illustrated processing block 1358 may include allocating sensitive resources to the first and second functions. Although not illustrated, method 1350 may include determining a lack of resource conflicts between one or more sensitive resources of the first function and one or more sensitive resources of the third function. The method 1350 may include scheduling the first and third functions to execute on the same node at overlapping times based on the lack of resource conflict.

[0350] The method 1350 can enhance the efficiency and operation of the CSP environment. For example, the method 1350 can reduce latency and enhance function completion rates.

[0351] Additional Notes and Examples

[0352] Example 1300 includes at least one computer-readable storage medium comprising a set of instructions that, when executed by a computing device, cause the computing device to: determine, for each respective function of a plurality of functions, one or more sensitive resources required to perform the respective function, determine a resource conflict between one or more sensitive resources of a first function of the plurality of functions and one or more sensitive resources of a second function of the plurality of functions, schedule one or more of the first function or the second function to avoid resource conflicts, schedule the first function and the second function to execute at different times, schedule the first function and the second function to execute on different computing nodes, determine a lack of resource conflicts between one or more sensitive resources of the first function and one or more sensitive resources of a third function of the plurality of functions, schedule the first function and the third function to execute on the same node at overlapping times, and schedule one or more of the resources sensitive to the first function and the second function to achieve target performance gains for the first function and the second function, respectively.

[0353] Example 1301 includes at least one computer-readable storage medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: determine, for each respective function among a plurality of functions, one or more sensitive resources required to perform the respective function, determine a resource conflict between one or more sensitive resources of a first function among the plurality of functions and one or more sensitive resources of a second function among the plurality of functions, and schedule one or more of the first function or the second function to avoid the resource conflict.

[0354] Example 1302 includes the at least one computer-readable storage medium of example 1301 , comprising a further set of instructions that, when executed by a computing device, cause the computing device to schedule the first function and the second function to be performed at different times.

[0355] Example 1303 includes the at least one computer-readable storage medium of example 1301, comprising a further set of instructions that, when executed by a computing device, cause the computing device to schedule the first function and the second function for execution on different computing nodes.

[0356] Example 1304 includes at least one computer-readable storage medium of example 1301, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: determine a lack of resource conflicts between one or more sensitive resources of a first function and one or more sensitive resources of a third function among a plurality of functions, and schedule the first function and the third function to be executed on the same node at overlapping times.

[0357] Example 1305 includes at least one computer-readable storage medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: determine, for each respective function among a plurality of functions, one or more sensitive resources required to perform the respective function, determine a resource conflict between one or more sensitive resources for a first function among the plurality of functions and one or more sensitive resources for a second function among the plurality of functions, and schedule one or more of the resources sensitive to the first function and the second function to achieve a target performance gain for the first function and the second function, respectively.

[0358] Enhanced function execution for FaaS

[0359] Figure 14A and 14B 1400 illustrates an enhanced function execution sequence according to some embodiments. Such an example may be represented by Figure 13A Server 1302 (eg, one or more of first to third computing nodes 1304a-1304c) and / or Figure 4 404). More specifically, function call 1402 may request function 1404 to execute. Function call 1402 may also be referred to as a "trigger event" in the FaaS context. Function 1404 may be a serverless function (e.g., a single FaaS function). Function 1404 may be viewed as a collection of internal operations that may be executed in a certain order. An operation may be the execution of a module within function 1404 that implements a portion of the overall function. For example, assume that function 1404 is to select a certain number of tuples from a database table based on a set of conditions, and then needs to sort the results to produce output in a desired order or further select from the sorted output. In this example, selection and sorting are two operations inside function 1404. The internal operations may be referred to as "small functions."

[0360] Rather than treating function 1404 as an atomic unit, decomposition process 1406 can decompose function 1404 into a series of small functions 1408a-1408e by, for example, transcoding or static analysis of which modules are called from the body of code for function 1404, and executing function 1404 as presented. Decomposition process 1406 can be specified by the programmer, by the cloud service provider, and / or implemented by an optimization toolchain authored by a third party. Decomposition process 1406 and its operation can be further controlled by options, prompts, instructions, etc. provided by the programmer or the cloud service provider. One or more of small functions 1408a-1408c can optionally be implemented by the user and assigned to process 1406. Unlike functions, small functions 1408a-1408e can share different levels of state with each other to achieve efficient computation and coordination. Individual small functions 1408a-1408e may undergo individual scheduling and execution processes 1410 to reduce latency and manage resources more efficiently. Specifically, some of the small functions 1408a-1408e may be executed concurrently to reduce latency. Furthermore, if a particular small function in the small functions 1408a-1408e requires resources such as dedicated hardware (e.g., an accelerator or FPGA) or firmware to execute, execution of that particular small function in the small functions 1408a-1408e may be delayed until the dedicated hardware becomes available to enhance resource utilization and scheduling.

[0361] As illustrated, function 1404 can be decomposed into a dependency graph 1414 consisting of small functions 1408a-1408e, wherein dependency graph 1414 indicates the execution order of small functions 1408a-1408e. In dependency graph 1414, small function 1408a can be executed first. Small functions 1408b and 1408c can be executed based on information from small function 1408a and executed concurrently. Small function 1408d can be executed after small functions 1408b and 1408c and based on information from both small functions 1408b and 1408c. Small function 1408e can be executed after small function 1408d based on information from small function 1408d, completing the execution of function 1404.

[0362] The scheduling and execution process 1410 can operate to schedule each of the small functions 1408a-1408e individually based on the dependency graph 1414. For example, the interconnections between the small functions 1408a-1408e are clear based on the dependency graph 1414. Thus, the scheduling and execution process 1410 can determine whether to schedule the small functions 1408a-1408e concurrently (in parallel) or serially (one after another) based on the dependent interconnections of the dependency graph 1414.

[0363] like Figure 14B As illustrated in the scheduling diagram 1416 of FIG. 1 , the small functions 1408 a - 1408 e may be scheduled and executed by the scheduling and execution process 1410 to be executed at a timing T 0- Small function 1408a may be scheduled to execute on compute node 1412a between times T0 and T1. Since other small functions 1408b-1408e require data from small function 1408a, none of the other small functions 1408 may execute during time T0-T1.

[0364] After functionlet 1408a completes execution, functionlets 1408b and 1408c can execute on compute nodes 1412a and 1412b between time T1 and T2. As illustrated in dependency graph 1414, functionlets 1408b and 1408c can execute concurrently because they only require information from functionlet 1408a and not from functionlets 1408e and 1408e. Thus, in schedule graph 1416, functionlets 1408b and 1408c execute concurrently on different compute nodes 1412a and 1412b. In some embodiments, if one of the compute nodes 1412a-1412c has sufficient resources to support both small functions 1408b, 1408c, both small functions 1408b, 1408c may be scheduled to execute on that one of the compute nodes 1412a-1412c.

[0365] After small functions 1408b and 1408c complete execution, small function 1408d may be executed on compute node 1412c during time T2-T3. The execution of small function 1408d may be enhanced by a hardware accelerator and / or FPGA found on compute node 1412c. Thus, the scheduling and execution process 1410 may further consider whether resources (such as hardware resources) can enhance the execution of one of small functions 1408a-1408e and schedule one of small functions 1408a-1408e accordingly. Furthermore, as illustrated in dependency graph 1414, small function 1408d may require data from small functions 1408b and 1408c but not from small function 1408e to execute, and is therefore scheduled to complete execution after small functions 1408b and 1408c.

[0366] After functionlet 1408d completes execution, functionlet 1408e can be executed on compute node 1412c during time T3-T4. Resource efficiency can be enhanced by having functionlet 1408e executed on compute node 1412c. That is, since functionlet 1408e may require data from functionlet 1408d, functionlet 1408e can be scheduled to execute on the same node as functionlet 1408d to minimize data transfer between compute nodes 1412a-1412c and enhance memory and cache usage. Thus, scheduling and execution process 1410 can further enhance resource usage by minimizing data transfer and enhancing cache usage through enhanced scheduling of functionlets 1408a-1408e.

[0367] In some embodiments, one or more of the small functions 1408a-1408e can be executed from a rebuilt state at a later time when the resources required by the small functions 1408a-1408e become available. For example, assume that the small function 1408d requires a hardware accelerator for execution. The small function 1408d can be scheduled to be executed in response to an indication that the hardware accelerator is available. The execution of the small function 1408d can be performed some time after the small functions 1408b and 1408c have completed execution. That is, the small function 1408d will not be automatically triggered to start execution as soon as the small functions 1408b and 1408c have completed execution. Instead, the small function 1408d can be scheduled to be executed when the hardware accelerator becomes available in addition to the completion of the execution of the small functions 1408b and 1408c.

[0368] Separating function 1404 into smaller functions 1408a-1408e can reduce the likelihood of failure of function 1404. For example, in some embodiments, some of the operations of function 1404 may require specific hardware, such as the aforementioned smaller function 1408d. If function 1404 must wait for this hardware, a timeout failure may occur, in which function 1404 fails to complete before a predetermined time limit and is therefore aborted.

[0369] By breaking function 1404 into a series of individual small functions 1408a-1408e, such timeout failures can be better avoided because the small functions 1408a-1408e can be independently scheduled to operate only when resources are available. In short, function 1404 can be "paused" between small functions 1408a-1408e by waiting until resources are available before launching the small functions 1408a-1408e. As mentioned, the small function 1408e can be scheduled to execute when the hardware accelerator becomes available, rather than starting execution and then waiting for the hardware accelerator to become available.

[0370] Furthermore, the decomposition process 1406 and the scheduling and execution process 1410 may provide opportunities for enhancing overall execution by capturing and releasing resources at the level of small functions 1408a-1408e, which is a finer granularity than that of function 1404. Furthermore, it may be possible to intermix small functions that can be accelerated on dedicated hardware accelerators (such as small function 1408d) with those that are better suited for conventional or CPU-based software execution (e.g., small functions 1408a-1408c and 1408e).

[0371] Furthermore, various embodiments allow for a simple flow of execution of function 1404, where function 1404 is broken down into smaller sub-functions 1408a-1408e along with their ordering, as illustrated in dependency graph 1414. The scheduling scheme implemented in scheduling and execution process 1414 can monitor the execution or priority of available resources / other contenders requiring those resources to achieve opportunistic scheduling while maintaining sub-function order. Furthermore, developers can be unaware of the enhanced function execution sequence 1400. This can simplify developer tasks, as developers do not have to identify sub-functions, as the process is performed opaquely on the CSP side.

[0372] As described above, functionlets 1408a-1408e can be provisioned to the execution platform based on the ordering dependencies described by the dependency graph 1414. Architectural support can be provided to ensure correct ordering dependencies in the dependency graph 1414, including support for efficient logical barriers and event coordination using techniques such as extended user-level interrupts (ULIs), hardware queue managers, and remote atomics (RAOs) for functionlets 1408a-1408e that can be scheduled as concurrent activities within or across address space boundaries. Functionlets 1408a-1408e can also be decomposed into smaller units called "mini-functionlets," which execute similarly as described above.

[0373] The enhanced function execution sequence 1400 can be coordinated by one or more hardware queue managers. Multiple hardware queue managers can schedule and queue small functions 1408a-1408e. The hardware queue managers can be on different nodes, but can maintain a queue of small functions 1408a-1408e according to the execution order of the dependency graph 1414 and schedule them accordingly.

[0374] The enhanced function execution queue 1400 can further be composed of the above-mentioned Figure 13A 1302, but may also be implemented in conjunction with the server 1302 Figure 4The enhanced FaaS system 400 (such as orchestrator 404) and / or one or more modules thereof may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0375] For example, computer program code for implementing the operations shown in the enhanced function execution sequence 1400 may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, SMALLTALK, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, central processing unit / CPU, microcontroller, etc.).

[0376] As an example, assume that the function includes analyzing a short speech document. The analysis of the short speech document can be broken down into various small functions (e.g., component operations), including converting (transcribing) the speech document into text, identifying and separating the words and / or sentences spoken by different participants, and categorizing the transcriptions according to the identity of the speaker. By breaking down the function as described above with reference to the enhanced function execution sequence 1400, partial execution of the function is achieved so that some small functions can be completed within the available time using a specific amount of resources. Thus, breaking down the function into various small functions can enhance execution, reduce resource usage, and reduce waiting time.

[0377] Figure 14C A method 1450 for scheduling a function with multiple operations in a CSP environment is shown and may be performed by Figure 13A However, it can also be implemented in conjunction with the server 1302. Figure 4 The enhanced FaaS system 400 (such as orchestrator 404) and / or one or more modules thereof may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0378] For example, computer program code for implementing the operations shown in method 1450 may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, SMALLTALK, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0379] The illustrated processing block 1452 can determine the operations (e.g., individual sub-functions) of the function. The illustrated processing block 1453 divides the function into individual sub-functions based on the identified operations. For example, each sub-function may include a single operation. In other embodiments, when the operations are related to each other (e.g., utilize similar data or require data from each other), each sub-function may include several operations. The illustrated processing block 1454 can generate a static interdependency graph to determine the order of the individual sub-functions. For example, and as mentioned above, the order may include determining whether the individual sub-functions (e.g., individual operations) can be executed concurrently or serially. The illustrated processing block 1456 can schedule the individual sub-functions separately based on the determined order. For example, a first sub-function (e.g., an operation) can be scheduled before other sub-functions (e.g., individual operations) if the other sub-functions depend on data from the first sub-function. The illustrated processing block 1456 can further schedule the first of the individual sub-functions to execute at a first computing node, and the second of the individual sub-functions to execute at a second computing node different from the first computing node. Thus, concurrent scheduling may be facilitated, or the use of different hardware (eg, dedicated and / or non-dedicated) to reduce latency of the first function and the second function.

[0380] The illustrated processing block 1456 may also schedule the first of the various small functions at a first time and the second of the various small functions at a second time different from the first time. Thus, serial execution of the first and second small functions can be achieved. In addition, the illustrated processing block 1456 may schedule the second small function to start execution in response to both an identification that resource allocation (e.g., a dedicated hardware accelerator and / or FPGA) is available and an identification that the first small function is completed. In such embodiments, method 1450 may further include intentionally delaying the execution of the second small function until resource allocation is available. In addition, the second small function may require output from the first small function.

[0381] The method 1450 can enhance the efficiency and operation of a CSP environment. For example, the method 1450 can reduce latency, enhance resource utilization, and enhance function completion rates.

[0382] Additional Notes and Examples

[0383] Example 1400 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: determine operations of a function, determine an order of the operations, schedule the operations separately based on the determined order, schedule a first operation of the operations to be performed at a first computing node and a second operation of the operations to be performed at a second computing node, schedule the first operation at a first time, schedule the second operation at a second time different from the first time, schedule the second operation to begin execution in response to both an indication that a resource allocation is available and an indication that the first operation is complete, and delay execution of the second operation until the resource allocation is available, wherein the second operation is to receive an output from the first operation.

[0384] Example 1401 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: determine operations of a function, determine an order of the operations, and schedule the operations individually based on the determined order.

[0385] Example 1402 includes at least one computer-readable medium of example 1401, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: schedule a first operation of each operation to be performed at a first computing node and a second operation of each operation to be performed at a second computing node.

[0386] Example 1403 includes at least one computer-readable medium of example 1401, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: schedule a first operation among the operations at a first time, and schedule a second operation among the operations at a second time different from the first time.

[0387] Example 1404 includes the at least one computer-readable medium of example 1403, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: schedule a second operation to begin execution in response to both an indication that a resource allocation is available and an indication that the first operation is complete.

[0388] Example 1405 includes the at least one computer readable medium of example 1404, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: delay performance of the second operation until resource allocation is available.

[0389] Example 1406 includes the at least one computer-readable medium of example 1403, wherein the second operation is to receive an output from the first operation.

[0390] Example 1407 includes the at least one computer-readable medium of example 1400 comprising a further set of instructions that, when executed by a computing device, cause the computing device to: determine operations based on user input.

[0391] Enhanced memory allocation for FaaS

[0392] Figure 15A 1500 illustrates a computing architecture 1500 with performance and memory storage enhancements according to some embodiments. Figure 13A Server 1302 (eg, one or more of first to third computing nodes 1304a-1304c) and / or Figure 4 This is achieved by an enhanced FaaS system 400 (such as orchestrator 404).

[0393] The computing architecture with enhanced performance and memory storage includes a container 1502 and a function 1504. The function 1504 can operate in conjunction with the container 1502. For example, the function 1504 can be loaded into the container 1502. In some embodiments, the function 1504 can be downloaded and instantiated into the container 1504. The platform operating system 1506 can host the container 1502 and the function 1504 and operate in conjunction with the memory 1510 to store data for the container 1502 and the function 1504.

[0394] As illustrated, the platform operating system 1506 may include a dedicated memory container list 1524 and a dedicated function memory list 1526. The dedicated container memory list 1524 and the dedicated function memory list 1526 may be data structures, arrays, or lookup tables that identify address ranges in memory 1510 that are dedicated to functions 1504 and / or containers 1502. For example, the dedicated container memory list 1524 identifies base container memory spaces 1520a-1520c. The dedicated function memory list 1526 identifies function memory spaces 1522a and 1522b. As illustrated, the memory 1510 may also include unallocated memory space 1530.

[0395] When platform operating system 1506 receives a memory allocation request, platform operating system 1506 may determine whether function 1504 or container 1502 is the source of the memory allocation request. If container 1502 is the source of the memory allocation request, memory allocation may be provided based on dedicated container memory list 1524. For example, the memory allocation may be an allocation of a memory range identified by dedicated container memory list 1524. Therefore, container 1502 may store and write data to base container memory spaces 1520a-1520c based on the memory allocation received from platform operating system 1506. Thus, container 1502 may be provisioned with a standard, trusted portion of memory via base container memory spaces 1520a-1520c.

[0396] In contrast, if function 1504 is the source of the memory allocation request, the memory allocation may be provided based on the dedicated function memory list 1526. For example, the memory allocation may be an allocation of a memory range identified by the dedicated function memory list 1526. Thus, function 1504 may store and write data to function memory spaces 1522a, 1522b.

[0397] Thus, container-specific data for container 1520 and function-specific data used only by function 1504 can be stored in different base container memory spaces 1520a-1520c and function memory spaces 1522a, 1522b. The container-specific data can be reused by many different functions and cannot be changed by function 1504 during its execution.

[0398] Once function 1504 terminates, for example, by completing execution or encountering an error, the function-specific data stored in function memory spaces 1522a, 1522b can be erased. When function 1504 completes execution, the container-specific data stored in base container memory spaces 1520a-1520c may not be erased. Thus, even when function 1504 terminates and is no longer executing, container 1502 remains in a semi-warm state, ready to receive another function to be executed. Thus, by limiting the data that is erased upon termination of function 1504, dismantling of the entire container 1502 can be avoided. By having function 1504 store data only in function memory spaces 1522a, 1522b, only changes affected by function 1504 can be removed from the state of container 1502, thereby allowing container 1502 to remain in a semi-warm state with trusted code stored in base container memory spaces 1520a-1520c. Thus, by partitioning data between function-specific data and container-specific data and storing accordingly, deletion of data associated with function 1504 may be facilitated to enhance security and reduce high-latency cold container boots.

[0399] In detail, for various container types, the overhead for terminating and restarting container 1502 may be too high and increase waiting time. For example, if container 1502 is used for a neural network (e.g., a convolutional neural network or a deep neural network), it may be too expensive to terminate container 1502 and then spin up again. This may be because the neural network includes a very large data load to operate (e.g., neural weights and / or constant values). Such data load can be part of container 1502 because the data load can be reused by different functions and cannot be changed by the functions. Thus, if function 1504 fails, container 1502 may be forced to terminate and spin up again, which increases waiting time and reduces efficiency.

[0400] The computing architecture 1500 can enhance the termination process of the function 1504. For example, if the function 1504 fails, the platform operating system 1506 can limit the scope of the data that is terminated. That is, only the data stored in the function memory space 1522a, 1522b can be erased or deallocated. Therefore, only the changes affected by the function 1504 are removed from the state of the container 1502, and the removal of the entire container 1502 is avoided. As illustrated, the platform operating system 1506 provides the container 1502 with the ability to maintain two different sets of resources, including the function memory space 1522a, 1522b allocated and used by the function 1504 to be run in the container 1502, and the base container memory space 1520a-1520c allocated and used by the container 1502 itself.

[0401] Additionally, security is enhanced because sensitive data of the function 1504 is erased after termination of the function 1504. For example, sensitive data specific to the function 1504 can be erased from the function memory spaces 1522a, 1522b.

[0402] The platform operating system 1506 may operate in conjunction with a virtual machine monitor (not shown) and libraries used by one or more of the containers 1502 or functions 1504. The platform operating system 1506, the virtual machine monitor, and one or more of the libraries may implement mechanisms to determine whether a memory allocation call (e.g., malloc, calloc, etc.) originates from a whitelisted code scope that is part of the container 1502 or a known and verified execution subject of the container 1502. If the memory allocation call originates from a whitelisted code scope or a known and verified execution subject of the container 1502, the memory allocation call is provided with a memory range from the base container memory space 1520a-1520c. Otherwise, by default, the memory allocation call is assumed to originate from the code of the function 1504 and is then given a memory range from the function memory space 1522a, 1522b. Private function memory ranges may be considered "temporary" ranges.

[0403] If a purge or teardown cycle needs to be performed, for example, as a result of a partition violation, instead of tearing down the base container memory spaces 1520a-1520c, only the function memory spaces 1522a, 1522b of the function 1502 are torn down. For example, the private container memory list 1524 may include a list of all allocated memory spaces utilized by the container 1502. Similarly, the private function memory list 1526 may include a list of all allocated memory spaces utilized by the function 1504. During purge, the memory space allocated to the function 1504 may be torn down and erased, while leaving the memory space allocated to the container 1502 unchanged.

[0404] In addition, any type of memory allocation call, for example, a Unix (POSIX) system call (e.g., mmap() call) or a dynamic memory allocation call (e.g., malloc, calloc, etc.) that complies with the Portable Operating System Interface can be identified and appropriately directed as described above. For example, if an mmap() call is received from function 1504, the virtual memory ranges that are provisioned for mapping can be provided based on the dedicated function memory list 1526. Thus, the dedicated function memory list 1526 can include identification of the virtual memory ranges. Therefore, only those virtual memory ranges and any associated physical memory ranges may need to be reclaimed.

[0405] Furthermore, file descriptors can be handled similarly. For example, for the "open()" command, the platform operating system 1506 can maintain two different groups of file descriptors in the dedicated container memory list 1524 and the dedicated function memory list 1526. File descriptors in the dedicated container memory list 1524 and the associated base container memory spaces 1520a-1520c do not need to be terminated or closed upon an abnormal condition or termination of the function 1504 (but may need to be reinitiated, rebuilt, reopened, or reinitialized). File descriptors in the dedicated function memory list 1526 and the associated function memory spaces 1522a, 1522b can always be closed upon an abnormal condition or termination of the function 1504.

[0406] When function 1504 terminates, for example, due to a trap caused by an illegal instruction or partition violation, container 1502 can perform cleanup of function 1504 without forcing container 1502 itself to exit. As mentioned above, one or more of the platform operating system 1506, software libraries, and the virtual machine monitor can facilitate the teardown of function 1504.

[0407] In some embodiments, the platform operating system 1506 and / or container 1502 may also remove the container 1502 in addition to the function 1504. That is, the underlying container memory space 1520a-1520c may be erased when the container 1502 is identified as causing an error. For example, if a termination condition (such as an illegal instruction error) occurs as a result of code executed from within the whitelisted body of the container 1502, the process or container 1502 should be removed in addition to the function 1504. In some embodiments, the container 1502 may be removed if it remains unused by the function for a predetermined amount of time.

[0408] In some embodiments, when a function 1504 terminates before completion (e.g., due to an error), a special entry point can be defined in the container 1502. The entry point is the location where execution is directed after the function 1504 is torn down and the function memory space 1522a, 1522b is deallocated. The re-entry point can be the location where the container 1502 executes the function startup during a warm boot. For example, the re-entry point can be the startup point where the container 1502 is fully initialized and ready to begin processing the function 1504. Directing to this re-entry point can be equivalent to a "long jump call" to the startup point.

[0409] Various adjustments can be made to each command or library interface to facilitate appropriate allocation and deallocation. For example, a "backtrace" command is usually required to determine the source of each allocation, but this "backtrace" command can be avoided by having two different entry points for each allocation and "open call". For example, one entry point is only linked to the whitelist code, while the other entry point is the default of the code of function 1504. Similar command bifurcations can be completed for other types of preset calls (such as "mmap()" (which requires providing a virtual range), "open()" (which allocates file descriptors, socket(), etc.). In each case, it can be assumed that the whitelist code is linked to the code that is allocated in a normal manner, while the non-whitelist code isolates resources in function memory space 1522a, 1522b via a dedicated function memory list 1526, or tracks the path of opened file descriptors, sockets, etc. as data associated with function 1504 that needs to be automatically released / automatically closed when an error occurs.

[0410] Thus, the enhanced computing architecture 1500 may include several enhancements, including limited teardown of containers and limited initialization of cold containers, thereby reducing costs. Furthermore, the enhanced computing architecture 1500 may enhance resource utilization efficiency because less bandwidth and power are used to tear down and rebuild containers. Furthermore, the enhanced computing architecture 1500 may have reduced latency due to faster startup of containers from a warm state, as well as reduced overhead from function terminations and errors. Security may also be enhanced by removing function-specific data from the function memory spaces 1522a, 1522b.

[0411] Figure 15B A method 1550 for memory allocation for functions and containers is shown and may be used by Figure 13A 1302, but may also be implemented in conjunction with the server 1302. Figure 4 The enhanced FaaS system 400 and / or one or more modules are implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0412] For example, computer program code for implementing the operations shown in method 1550 may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, SMALLTALK, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0413] The illustrated processing block 1552 may dedicate one or more container memory spaces to a container. The illustrated processing block 1554 may dedicate one or more function memory spaces to a function associated with the container. The illustrated processing block 1556 may receive a memory allocation request. The illustrated processing block 1558 may determine whether the memory allocation request originates from a container or a function. The illustrated processing block 1560 may provide memory allocation from one or more container memory spaces or one or more function memory spaces based on whether the memory allocation request originates from a container or a function. For example, when the memory allocation request originates from a container, the memory allocation may originate from one or more container memory spaces. In contrast, when the memory allocation request originates from a function, the memory allocation may be provided from one or more function memory spaces.

[0414] In addition, data used only for container operation can be stored in one or more container memory spaces. In addition, function-specific data can be stored only in one or more function memory spaces.

[0415] The illustrated processing block 1562 may deallocate memory from one or more function memory spaces but not deallocate memory from one or more container memory spaces in response to an indication that a function has terminated. Although not illustrated, the method may further include loading another function into the container for execution. Thus, method 1550 may include several enhancements, including limited removal of containers and limited initialization of cold containers, thereby reducing costs. In addition, method 1550 may enhance resource utilization efficiency because less bandwidth and power are used to remove and rebuild containers. In addition, method 1550 may have less latency due to faster startup of containers from a semi-warm state, and less overhead from function termination and errors.

[0416] Additional Notes and Examples

[0417] Example 1500 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: dedicate one or more container memory spaces to a container and dedicate one or more function memory spaces to a function associated with the container, receive a memory allocation request, determine whether the memory allocation request originates from the container or the function, provide a memory allocation from the one or more container memory spaces when the memory allocation request originates from the container, provide a memory allocation from the one or more function memory spaces when the memory allocation request originates from the function, and, in response to an identification of function termination, deallocate memory from the one or more function memory spaces but not deallocate memory from the one or more container memory spaces, wherein data only for container operations is stored in the one or more container memory spaces and data specific to the function is stored only in the one or more function memory spaces, wherein the one or more container memory spaces are used to store one or more of a file descriptor or a socket descriptor.

[0418] Example 1501 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: dedicate one or more container memory spaces to containers, and dedicate one or more function memory spaces to functions associated with the containers.

[0419] Example 1502 includes at least one computer-readable medium of example 1501, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: receive a memory allocation request, determine whether the memory allocation request originates from a container or a function, and provide a memory allocation from one or more container memory spaces or one or more function memory spaces based on whether the memory allocation request originates from a container or a function.

[0420] Example 1503 includes the at least one computer-readable medium of example 1502, comprising a further set of instructions that, when executed by a computing device, cause the computing device to provide a memory allocation from one or more container memory spaces when the memory allocation request originates from a container.

[0421] Example 1504 includes the at least one computer-readable medium of example 1502 comprising a further set of instructions that, when executed by a computing device, cause the computing device to provide a memory allocation from one or more functional memory spaces when the memory allocation request originates from a function.

[0422] Example 1505 includes the at least one computer-readable medium of example 1501 , wherein data for container operations only is stored in one or more container memory spaces, and function-specific data is stored only in one or more function memory spaces.

[0423] Example 1506 includes at least one computer-readable medium of example 1501, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: in response to identification of functional termination, deallocate memory allocations from one or more functional memory spaces but not deallocate memory from one or more container memory spaces.

[0424] Example 1507 includes the at least one computer-readable medium of example 1506, wherein the one or more container memory spaces are used to store one or more of a file descriptor or a socket descriptor.

[0425] Distribution of functions to warm and cold containers

[0426] Functions can be randomly distributed between warm and cold containers. As already described herein, the latency for initializing a cold container can be non-trivial. Consequently, functions dispatched to a cold container can have a higher latency because the cold container must be initialized, for example, by being pre-set, and then the function must be executed. In contrast, functions dispatched to a warm container can have a lower latency because cold container initialization can be avoided.

[0427] Cold container initialization latency can be caused by many stages of initialization and various factors. One set of factors can be the allocation and configuration of OS-level resources, such as namespaces for resource IDs (e.g., process IDs and file descriptors) and control groups to limit the amount of various resources (e.g., CPU and memory) allocated to the container. Another set of factors can be initializing the contents of the container, such as the language runtime for a managed language (such as Java). Starting the function itself also incurs additional overhead. A warm container can perform some number of these stages in advance to reduce the latency of calling functions hosted in the warm container.

[0428] Conventional schedulers may not consider the time required for cold container initialization when distributing functions, nor consider how to minimize cold container initialization. Furthermore, some schedules may have inefficient distribution of functions between warm and cold containers, resulting in higher resource usage and higher latency execution.

[0429] Now turn Figure 16A , shows an example 1600 where a function request 1604 is requested by a FaaS system (e.g. Figure 44. The FaaS system 400 of FIG. 4 is a FaaS system that schedules the execution of batched functions by the invoker and batch balancer 1606. The invoker and batch balancer 1606 can implement batch execution to reduce latency and enhance resource utilization. For example, some embodiments can identify the first N batchable function requests 1608a-1608n that each request the execution of a batchable function. The first N batchable function requests 1608a-1608n can be batched together and pre-scheduled so that the batchable functions are executed in the same container 1614a rather than in different containers. Thus, cold container initialization can be reduced by batching the execution of batchable functions to reduce latency and utilize fewer containers.

[0430] In addition, caller and batch balancer 1606 can identify the waiting time constraint of the specific function associated with function request 1604, and distribute function request 1604 to meet this waiting time constraint.Waiting time constraint can be corresponding with the time requested that function should complete execution.For example, when waiting time constraint allows, the function of shorter waiting time can be grouped together and sorted, so that the whole waiting time constraints of each function in this group are met.The above analysis especially can strengthen the specific scenario that the program code of wherein function changes greatly in length.A kind of such scenario can be the scenario that the long function with loose waiting time constraint (for example long waiting time constraint) and the short function with tight waiting time constraint (for example short waiting time constraint) are scheduled together.First dispatch short function so that the waiting time constraint of long function and short function can all be met.In contrast, first dispatching long function causes the waiting time requirement of short function to be met.

[0431] In some embodiments, an "earliest deadline first" scheduling strategy is used to generate a feasible schedule. Thus, functions with the shortest wait time constraints are scheduled before functions with longer wait time constraints. In some embodiments, alternative scheduling schemes (such as "shortest job first") and implementations (e.g., round robin) can be used. Thus, for example, the shortest functions are executed before longer functions, and can be executed in a round-robin fashion.

[0432] In some embodiments, a hybrid model of "earliest deadline first" and "shortest job first" can be employed. For example, a "shortest job first" strategy can schedule the shortest function to execute before a longer function. The system can then analyze all functions (e.g., predicting the timing required to complete each function and comparing the timing with the associated latency constraints) to predict whether all latency constraints will be met. If so, the system can schedule each function according to the determined schedule.

[0433] If the waiting time constraints of some functions will not be met, the system can modify the schedule based on the "earliest deadline first" policy so that those functions whose waiting time constraints will not be met will be executed before other functions whose waiting time constraints are met. As an example, assume that F1, F2, F3 and F4 are scheduled to be executed in this order by the "shortest job first" policy. If it is identified that F2 and F3 will not meet the waiting time constraints, the system can move F2 and F3 to be executed before F1 and F4. In order to determine whether F2 or F3 will be executed first, the system can implement the "earliest deadline first" policy and determine that F3 has the earliest deadline between F2 and F3 and should be scheduled before F2. Therefore, the final order will be similar to: F3, F2, F1 and F4.

[0434] In some embodiments, historical information, such as that collected from various telemetry sources (e.g., timers timing previous function executions), can be used to estimate the execution time of each function, which can help inform the scheduling of future invocations of the function. For more details, see Figure 24A and Figure 41A Some embodiments may also determine whether cold container initialization is necessary when processing batchable functions, and if so, consider the latency for cold container initialization when identifying whether the latency constraint will be satisfied.

[0435] Thus, some embodiments may limit cold container initialization while having a more efficient distribution of functions to reduce resource usage. For example, it may be more resource efficient to have several functions execute within the same container rather than initializing several new cold containers to execute the functions. It is apparent that, with the exception of the first function, functions batched in a serial execution model need to wait for preceding batched functions to complete, which may increase latency relative to an infinitely parallelized model in which functions in dedicated containers do not have to wait for any other functions to complete. Nevertheless, once the limitations of parallelization are considered and the reduction in initialization time is taken into account, the total latency for batched functions and even the latency for functions that are not the first function can actually be lower. Thus, some embodiments may have lower latency due to enhanced warm container usage, even when functions are batched together. Parallel execution of multiple batched functions in a single container can further reduce latency.

[0436] Additionally, as mentioned above, resource management can be enhanced. For example, after a certain amount of idle time, the warm container can be removed. Idle time can be time during which the warm container is not processing any functions. By enabling warm containers to be utilized more frequently through batching as described herein, embodiments can exhibit less frequent cold container initializations and less frequent warm container removals because the warm containers are used more frequently. Additionally, some embodiments can have fewer instances of the same container consuming resources. For example, instead of having one container per function, one container can serve several functions. Thus, embodiments enhance efficiency and reduce costs by utilizing fewer resources. Therefore, even if the first N batchable function requests 1608a-1608n occur in sudden and infrequent bursts, cold container losses can be avoided and the enhanced FaaS system can be allowed more control over the construction and removal of containers.

[0437] Furthermore, load balancing costs can be reduced. For example, scheduling each function individually by the orchestrator can require more computing resources, as opposed to scheduling a batch of functions together. Furthermore, messaging overhead can be reduced, since instead of sending a separate message for each function request, a batched function request message can be sent that includes all of the first N batched function requests 1608a-1608n.

[0438] As illustrated in example 1600, a function request 1604 is requested by a FaaS system (e.g. Figure 4 Invoker and batch balancer 1606 is scheduled by the FaaS system 400. Invoker and batch balancer 1606 can be part of orchestrator 404 or operate in conjunction with orchestrator 404.

[0439] like Figure 16A As illustrated in FIG16 , event handling API agent 1602 can route event calls to functions associated with function requests 1604. In the example of 1600, API agent 1602 can generate function requests 1604 via, for example, an API. Function requests 1604 can be provided to a caller and a batch balancer 1606. Function requests 1606 can each be a request to perform a corresponding function.

[0440] The caller and batch balancer 1606 can analyze the function request 1604 to identify whether the function request 1604 can be processed in batches. In detail, the caller and batch balancer 1606 can determine whether the function of the function request 1604 can be executed in the same container in the container 1614a-1614c. Each request can be allowed to be constructed unless there is a strong wait time constraint, and then they can be sent to an available warm container or a new cold container as a batch according to how much workload is contained in the batch. Thus, under this type of optional batch processing, when each action is triggered, the load balancing, calling and execution of each action are all performed in a decoupled manner (for example, a two-stage manner). Each function that can be executed in the same container in the container 1614a-1614c can be processed in batches together. In some embodiments, the caller and batch balancer 1606 can determine that the same function can be processed in batches together. Additionally, invoker and batch balancer 1606 may determine that different functions may be batched together if they may be executed in the same container in containers 1614a-1614c.

[0441] The invoker and batch balancer 1606 can identify non-batchable function requests 1610, 1612 from the function requests 1604. Non-batchable function requests 1610, 1612 can request non-batchable functions that cannot be grouped with other functions and are therefore considered non-batchable. Non-batchable function requests 1610, 1612 can be sent separately to containers 1614b, 1614c. That is, the non-batchable functions can be executed in separate containers 1614b, 1614c. Containers 1614b, 1614c can be warm or cold, with priority given to the non-batchable function requests 1614b, 1614c with the strongest latency constraints.

[0442] As illustrated, the invoker and batch balancer 1606 can identify the top N batchable function requests 1608a-1608n from the function requests 1604. As described, each of the top N batchable function requests 1608a-1608n can call a function configured to execute within the same container (which can be referred to as a "batchable function"). Thus, the top N batchable function requests 1608a-1608n can be sent to the container 1614a. The batchable functions can be executed within the container 1614a. The top N batchable function requests 1608a-1608n can include any number of function requests greater than one.

[0443] In some embodiments, the batchable functions associated with the first N batchable function requests 1608a-1608n can be executed simultaneously in container 1614a if container 1614a has sufficient resources to support such simultaneous execution. For example, the invoker and batch balancer 1606 can determine the resource requirements of each of the batchable functions and determine whether container 1614a has access to sufficient resources (e.g., CPU and memory) to meet all resource requirements simultaneously. If so, simultaneous execution can be scheduled. If not, simultaneous execution cannot be supported and the batchable functions can be executed serially (one after another).

[0444] In some embodiments, the caller and batch balancer 1606 can determine the security protocol of the batchable function. If the security protocol indicates that a particular batchable function has specific security requirements (e.g., high security), then the particular batchable function can be scheduled for non-concurrent execution in the container 1614a and a limited teardown of the container 1614a is performed, wherein only the data of the particular batchable function is removed while maintaining the data of the container 1614a, as described with reference to FIG. Figures 15A-15B As described in the description associated with the above.

[0445] In some embodiments, some of the batchable functions may be executed concurrently, while other batchable functions may be executed serially. In some embodiments, container 1614a may execute a first set of batchable functions, e.g., the maximum number of batchable functions that can be supported by the resources of container 1614a. After the first set completes execution, a second set of batchable functions may begin execution, e.g., the maximum number of batchable functions that can be supported by the resources of container 1614a. Thus, invoker and batch balancer 1606 may schedule a mix of serial and parallel execution of batchable functions based on security protocols and the availability of resources of container 1614a relative to the resource requirements of the functions.

[0446] In some embodiments, container 1614a may include a partitioned workspace to separate batch-processable functions that are executed in parallel. Each function may be executed in different partitions. All partitions may access shared data that may be utilized by any batch-processable function. A partition may have a separated memory space to maintain function-specific data. Therefore, each partition may store the data generated by a function in a separated memory space while accessing the shared data used by the function. After a function completes execution in a partition, the data from the corresponding separated memory space may be stored in another data storage (if necessary), and the separated memory space may be erased to prepare the partition for another function. However, shared data may not be changed by a function during execution, and shared data may be reused by each function executed in container 1614a.

[0447] In some embodiments, the caller and batch balancer 1606 can accumulate function requests 1604. For example, function requests 1604 can each be sent to the caller and batch balancer 1606 at different times. The caller and batch balancer 1606 can allow function requests 1604 to accumulate for a period of time. That is, the caller and batch balancer 1606 may not immediately dispatch function requests 1604 upon receiving them. Instead, the caller and batch balancer 1606 may wait before dispatching function requests 1604. Doing so can allow the caller and batch balancer 1606 to receive many potentially batchable function requests 1604 and then dispatch function requests 1604 based on whether the function requests 1604 can be batched together.

[0448] In some embodiments, the caller and batch balancer 1606 can determine the latency constraints of each of the function requests 1604 and schedule the associated functions accordingly. For example, a function request 1610 that cannot be batched may have a strong latency constraint. That is, the function request 1610 that cannot be batched may need to be scheduled to execute the corresponding function immediately due to the associated latency constraint. The latency constraint can be a numerical value and / or an absolute time. The latency constraint can indicate that the associated function of the function request 1610 that cannot be batched may need to be completed within a short time frame. Thus, the function request 1610 that cannot be batched can be scheduled to be executed within a container 1614b, which can be a warm container.

[0449] In some embodiments, if a latency constraint for one of the function requests 1604 satisfies a non-batchable threshold, the function request can be automatically classified as non-batchable to ensure timely execution. For example, if the latency constraint indicates that the function must be completed within a predetermined amount of time, the corresponding function request can be scheduled immediately without identifying other batchable functions to be batched with the corresponding function request.

[0450] In some embodiments, the invoker and batch balancer 1606 may further determine a wait time constraint for each of the top N batchable function requests 1608a-1608n. The invoker and batch balancer 1606 may wait for scheduling of the top N batchable function requests 1608a-1608n based on the strongest wait time constraint of the top N batchable function requests 1608a-1608n received so far.

[0451] For example, the caller and batch balancer 1606 can determine the time frame in which each corresponding function of the first N batch-processable function requests 1608a-1608n is to be completed according to the waiting time constraint. The time frame can correspond to the preferred time window in which the corresponding batch-processable function should be completed during this period. The shortest time frame in each time frame can be determined. The caller and batch balancer 1606 can send the first N batch-processable function requests 1608a-1608n to the container 1614a to meet the shortest time frame. For example, the caller and batch balancer 1606 can send the first N batch-processable function requests 1608a-1608n to the container 1614a at the first time to ensure that the function with the shortest time frame is completed within the shortest time frame. However, the caller and batch balancer 1606 can continue to receive and accumulate function requests 1604 until the first time is reached. Thus, the invoker and batch balancer 1606 may wait for a period of time and adhere to the wait time constraint before sending the first N batchable function requests 1608a-1608n to the container 1614a.

[0452] In some embodiments, the caller and batch balancer 1606 can determine the timing of starting the execution of a particular function in order to comply with the wait time constraint. The caller and batch balancer 1606 can determine the total expected wait time required to complete the execution of the particular function. The total expected wait time may include the time required to initialize the cold container, the time the caller and batch balancer 1606 has spent on accumulating function requests 1604, the wait time of functions scheduled to be executed in the container before the particular function, the communication wait time and / or the wait time for the particular function to complete execution in the container. The following equation 1600 can be used to determine whether the wait time constraint of the particular function will be met based on a comparison of the total expected wait time with the acceptable wait time:

[0453] L 等待 +LCCL+LF+LC≤L 可接受

[0454] Equation 1600

[0455] In the above equation 600, L 可接受 is the time period during which the caller and batch balancer 1606 have accumulated specific function requests for that specific function. 可接受 It can be the difference between the current time and the time when the specific function request was received. CCL is the time required to initialize the cold container. If the warm container is marked as available, then L CCL Can be set to 0. L F is the estimated execution wait time of each function scheduled to be executed before the specific function and the estimated wait time for the specific function to complete execution. F It can be an estimate of the total function execution wait time. C L is the communication latency for sending the first N batchable function requests 1608a-1608n to the container 1614a. 可接受 This can be determined based on the latency constraints and can be the total acceptable latency for that particular function. For example, L 可接受 This can be a threshold set by the service provider or client and can be the total time within which the function is expected to complete execution. 可接受 If the above is true or if L 可接受 Greater than or equal to L 等待 、L CCL 、L C and L F The wait time constraint can be considered to be satisfied if the sum of 可接受The waiting time constraint is considered to be satisfied only when the sum is greater than a predetermined amount.

[0456] Based on the above equation 1600, the caller and batch balancer 1606 can determine the starting timing to start the execution of the specific function. The caller and batch balancer 1606 can further determine the transmission time of the first N batchable function requests 1608a-1608n to meet the starting timing. For example, the caller and batch balancer 1606 can determine a static value such as L CCL and L C , and adjusts the dynamic waiting time by controlling the timing (transmission time) of the first N batchable function requests 1608a-1608n being sent to the container 1614a and the order of batchable function execution in the container 1614a, such as L 等待 and L F .

[0457] In some embodiments, the caller and batch balancer 1606 can send the first N batch-processable function requests 1608a-1608n to the container 1614a in response to the specific number of the first N batch-processable function requests 1608a-1608n being accumulated. For example, the container 1614a can support the concurrent execution of a supported number of functions. Thus, when the number of functions of the first N batch-processable function requests 1608a-1608n reaches the supported number, the first N batch-processable function requests 1608a-1608n can be sent to the container 1614a to begin execution. In some embodiments, each function can be concurrently run on an independent software thread within the container 1614a, and therefore each function can be executed in parallel. In some embodiments, the function can be executed on an available thread in a time-sharing mode.

[0458] Thus, in the above example, the first N batchable function requests 1608a-1608n can be received and accumulated by the invoker and batch balancer 1606 at different times. The invoker and batch balancer 1606 can send the first N batchable function requests 1608a-1608n as a batch to a container 1614a, which can be warm or cold. The invoker and batch balancer 1606 can schedule the batch based on how much work can be in the batch and in order to meet the associated latency constraints. Thus, under such a batch processing scheme, when each function is triggered, the load balancing, invocation, and execution of each function can be performed in a decoupled manner (a two-stage manner).

[0459] In some embodiments, the top N batchable function requests 1608a-1608n may be divided into two or more groups based on latency constraints by the invoker and batch balancer 1606. For example, the top N batchable function requests 1608a-1608n may be divided such that the first group has the strongest latency constraint, the second group has the second strongest latency constraint, and so on.

[0460] Concurrent execution of two or more groups by two or more containers can occur to reduce waiting time. In some embodiments, if there are not enough warm containers for two or more groups, warm containers can be assigned according to waiting time constraints. For example, the group with the strongest waiting time constraint (the shortest time window for execution) can be sent to the warm container, while the group with weaker waiting time constraint (the longer time window for execution) can be sent to the cold container. For example, the identification of the waiting time constraint from the first N batchable function requests 1608a-1608n can be made. The first N batchable function requests 1608a-1608n can be grouped according to waiting time constraints so that the function with the shortest time window for completing execution is executed in the warm container. In contrast, the function with the longer time window for completing execution can be grouped together to complete execution in the cold container.

[0461] In some embodiments, the groups may be scheduled to execute one after another in the same container 1614a. The group with the strongest latency constraint may be scheduled to execute before the group with weaker latency constraint.

[0462] Furthermore, in some embodiments, the invoker and batch balancer 1606 can operate as a coarse-grained dispatcher from the front-end to the back-end of the FaaS infrastructure to schedule batch processing of functions. For example, on the back-end, the first N batchable function requests 1608a-1608n can be given to a container 1614a in which the associated functions are executed normally. In contrast, commercial, high-volume usage of FaaS may require more resources to perform load balancing, container assignment, and transport operations based on the individual function request scheduling scheme (all of which may be pure overhead).

[0463] Furthermore, the batch scheduling scheme described herein can automatically reduce the ratio of cold containers to warm containers. Specifically, a cold container is cold only for the first action (e.g., function) in a batch, but is hot for the remaining actions in the batch. Furthermore, fewer cold containers are built to accommodate each function, thereby reducing the ratio of cold containers to warm containers. As mentioned above, a function cannot be delayed for more than a threshold duration based on a latency constraint, and any batches in which the function is located are pushed for execution.

[0464] Now go to Figure 16B , an embodiment of the semiconductor package device 1620 may include one or more substrates 1624 and logic 1622 coupled to the one or more substrates 1624, wherein the logic 1622 is at least partially implemented in one or more configurable logic and fixed-function hardware logic. The logic 1622 coupled to the one or more substrates 1624 may be configured to receive multiple function requests that each request the execution of a corresponding function, and to determine the function requests that can be executed in batches from the multiple function requests. In some embodiments, the logic 1622 may be configured to send the function requests that can be processed in batches to the same container. For example, the logic 1622 may be configured to determine that the same container is warm, and in response to determining that the container is warm, determine that the function requests that can be processed in batches are to be sent to the same container. In some embodiments, the logic 1622 coupled to the one or more substrates 1624 may include a transistor channel region located within the one or more substrates 1624.

[0465] Embodiments of logic 1622, and other components of device 1620, can be implemented in hardware, software, or any combination thereof (including at least partial hardware implementations). For example, a hardware implementation may include configurable logic such as, for example, PLA, FPGA, CPLD, or fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or any combination thereof. In addition, portions of these components may be implemented in one or more modules as sets of logic instructions stored in a machine or computer-readable storage medium (such as RAM, ROM, PROM, firmware, flash memory, etc.) to be executed by a processor or computing device. For example, the computer program code for implementing the operation of the components can be written in any combination of one or more OS-appropriate / suitable programming languages, including object-oriented programming languages such as Python, Perl, Java, SmallTalk, C++, C#, and conventional procedural programming languages such as the "C" programming language or similar programming languages.

[0466] Device 1620 can implement methods 1650, 1670, and 1690 ( Figure 16C 、 16Dand 16E), or any of the embodiments discussed herein. In some embodiments, the illustrated device 1620 may include one or more substrates 1624 (e.g., silicon, sapphire, gallium arsenide) and logic 1622 (e.g., transistor arrays and other integrated circuit / IC components) coupled to the substrate 1624. The logic 1622 may be implemented, at least in part, in configurable logic or fixed-function logic hardware. In one example, the logic 1622 may include a transistor channel region located within (e.g., embedded in) the substrate(s) 1624. Thus, the interface between the logic 1622 and the substrate(s) 1624 may not be an abrupt junction. The logic 1622 may also be considered to include an epitaxial layer grown on an initial wafer of substrate(s) 1624.

[0467] Figure 16C A method 1650 for batch processing of function requests is shown and may be performed by Figure 13A 1302, but may also be implemented in conjunction with the server 1302. Figure 4 The enhanced FaaS system 400 and / or one or more modules may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0468] For example, computer program code for implementing the operations shown in method 1650 may be written in any combination of one or more programming languages, including object-oriented programming languages such as JAVA, SMALLTALK, C++, and conventional procedural programming languages such as the "C" programming language or similar programming languages. Additionally, the logic instructions may include assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, state setting data, configuration data for integrated circuits, state information that personalizes electronic circuits and / or other structural components native to hardware (e.g., a host processor, a central processing unit / CPU, a microcontroller, etc.).

[0469] Illustrated processing block 1652 may receive a plurality of function requests, each requesting execution of a corresponding function. Illustrated processing block 1654 may determine batchable function requests from the plurality of function requests. Illustrated processing block 1656 may send the batchable function requests to the same container. Illustrated processing block 1656 may further determine that the same container is warm, and determine that the batchable function requests are to be sent to the same container based on the same container being warm.

[0470] Illustrated processing block 1658 may determine one or more non-batchable function requests from the plurality of function requests that are not batchable.Illustrated processing block 1660 may send each of the one or more non-batchable function requests to a different container.

[0471] Figure 16D A method 1670 for batch processing of two or more function requests is shown and may be performed by Figure 13A 1302, but may also be implemented in conjunction with the server 1302. Figure 4 The enhanced FaaS system 400 and / or one or more modules thereof may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0472] Illustrated processing block 1672 may receive a first batchable function request. Illustrated processing block 1674 may determine that the first batchable function request is to be batched. Illustrated processing block 1676 may wait until a second batchable function request is received. That is, the first batchable function request may not be immediately sent to a container for execution and / or scheduled immediately. Instead, method 1670 may wait for receipt of other function requests to effectively batch the function requests together before scheduling. Illustrated processing block 1678 may determine that the first and second batchable function requests can be batched together. Illustrated processing block 1680 may determine one or more latency constraints for the first and second batchable function requests. As described herein, the latency constraints may reflect a requested time within which the functions invoked by the first and second batchable function requests should complete execution. Illustrated processing block 1682 may determine a transmission time for sending the first and second batchable function requests to the same container based on the one or more latency constraints. Specifically, the transmission time can ensure that the first and second batchable function requests comply with one or more latency constraints or complete execution before the requested time. The illustrated processing block 1684 can send the first and second batchable function requests to the same container at the transmission time.

[0473] In some embodiments, the illustrated processing blocks 1680 and 1682 may occur simultaneously with or before one or more of the illustrated processing blocks 1674, 1676, and 1678. For example, the wait time constraint for the first batchable function request may be identified simultaneously with block 1674. Furthermore, the illustrated processing block 1674 may determine, based on the wait time constraint for the first batchable function request, that the first batchable function request is non-urgent and may wait for other function requests. Furthermore, the illustrated processing block 1676 may wait for the other functions for a period of time that complies with the wait time constraint for the first batchable function request.

[0474] Figure 16E A method 1690 for scheduling function requests to comply with wait time constraints is shown and may be performed by Figure 13A 1302, but may also be implemented in conjunction with the server 1302. Figure 4 The enhanced FaaS system 400 and / or one or more modules may be viewed as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., implemented in configurable logic such as, for example, a PLA, FPGA, CPLD, etc., implemented in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or implemented as any combination of the above.

[0475] The illustrated processing block 1692 receives a function request. For example, an orchestrator may receive the function request and schedule the function request. The illustrated processing block 1694 may determine a latency constraint for the function request. As described above, the latency constraint may reflect the total acceptable latency of the function called by the function request. The latency constraint may be a digital measurement (e.g., 5ms) and / or an absolute time (e.g., 2:48 EST) reflecting the preferred completion timing of the function. In some embodiments, the latency constraint may depend on another function request. For example, the function may operate on data from another function, and therefore the latency constraint may reflect that the function should complete execution within a predetermined amount of time for completion of the other function. In some embodiments, the latency constraint may reflect that the function of the function request is to be executed as quickly as possible without waiting for batch processing.

[0476] The illustrated processing block 1696 may send the function request to the container to comply with the latency constraint. The function request may be batched with other function requests. Thus, the function request and the other function requests may be sent to the container. In some embodiments, if the function requests cannot share a container or the latency constraint does not allow time for batching, the function requests may not be batched.

[0477] Additional Notes and Examples

[0478] Example 1600 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: receive a plurality of function requests each requesting performance of a respective function, determine a plurality of batchable function requests from the plurality of function requests, receive a first batchable function request from the plurality of batchable function requests, determine that the first batchable function request is to be batched, wait until a second batchable function request from the plurality of batchable function requests is received, determine that the first and second batchable function requests can be batched together, determine one or more wait times for the first and second batchable function requests constraints, determining a time for sending a first and a second batchable function request to the same container based on the one or more latency constraints, sending the multiple batchable function requests to the same container at the determined time, determining that the same container is warm, determining that the multiple batchable function requests are to be sent to the same container based on the same container being warm, determining one or more non-batchable function requests from the multiple function request calls that are not batchable, and sending each of the one or more non-batchable function requests to a different container, wherein the functions of the batchable function requests are to be performed serially, in parallel, or in a hybrid serial and parallel scheme.

[0479] Example 1601 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: receive a plurality of function requests each requesting execution of a corresponding function, and determine a plurality of batchable function requests from the plurality of function requests.

[0480] Example 1602 includes the at least one computer-readable medium of example 1601, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: send the plurality of batchable function requests to the same container.

[0481] Example 1603 includes at least one computer-readable medium of example 1602, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: determine that the same container is warm, and determine that the plurality of batchable function requests are to be sent to the same container based on the same container being warm.

[0482] Example 1604 includes the at least one computer-readable medium of example 1601, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: receive a first batchable function request from the plurality of batchable function requests, determine that the first batchable function request is to be batched, wait until a second batchable function request from the plurality of batchable function requests is received, and determine that the first and second batchable function requests can be batched together.

[0483] Example 1605 includes at least one computer-readable medium of example 1604, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: determine one or more wait time constraints for the first and second batchable function requests, determine a time for sending the first and second batchable function requests to the same container based on the one or more wait time constraints, and send the first and second batchable function requests to the same container at the determined time.

[0484] Example 1606 includes at least one computer-readable medium of example 1601, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: determine one or more non-batchable function requests that are non-batchable from a plurality of function request calls, and send each of the one or more non-batchable function requests to a different container.

[0485] Example 1607 includes the at least one computer readable medium of example 1601, wherein the functions of the batchable function requests are to be performed serially, in parallel, or in a hybrid serial and parallel scheme.

[0486] Redundancy function implementation

[0487] Now turn Figure 17A , shows an example 1700 of redundant function implementation, where a function request 1710 is received by a FaaS system (e.g. Figure 4 The FaaS system 400) is handled by the orchestrator 1704.

[0488] In some cases, a function may time out and never complete due to a number of factors (e.g., a crash of a compute node). For example, when triggered, original function 1712 may go through a series of operations (e.g., authentication, authorization, pre-launch resource availability, etc.), where orchestrator 1704 may identify compute node 1706c and assign compute node 1706c to original function 1712. However, different platforms may have unique characteristics (e.g., virtualization layers, etc.) and resources that are unknown and opaque to the client requesting the FaaS service of original function 1712. For example, original function 1712 may find itself running on a warm container of compute node 1706a that has sufficient unsubscribed resources to facilitate timely execution. In other cases, original function 1712 may execute on a cold container of compute node 1706a that is also running on a busy platform but does not have sufficient resources to facilitate execution. In addition, original function 1712 may further consume other services whose performance is also variable. As a result, there is limited certainty that the original function 1712 will actually continue to execute, complete execution, and further that the original function 1712 will complete in a timely manner.

[0489] For example, if the original function 1712 exceeds available resources (e.g., a dynamic limit on memory), the original function 1712 may not be dispatched. In addition, the original function 1712 may not be dispatched in a timely manner, or once dispatched, it may not be clear whether the original function 1712 will complete execution within the permitted time limit.

[0490] In some cases, the original function 1712 can be completed, but requires multiple serial retries. Thus, the original function 1712 may be generated several times, one after another, until the original function 1712 is finally completed. Completing such serial retries at the client is both expensive and difficult to ensure completion, because the client has no control over the factors that lead to success or failure. Testing and retrying for successful results also complicates programming at the requester.

[0491] Some functions may be considered to have a high requirement for success, and therefore the risk of non-completion or delayed completion may be unacceptable. As discussed below, orchestrator 1704 can mitigate the probability of non-completion or delayed completion by redundantly executing functions across multiple computing nodes 1706a-1706c. For example, redundant functions 1708a, 1708b of original function 1712 can be generated and executed on different nodes 1706a, 1706b.

[0492] like Figure 17AAs illustrated in FIG17 , event handling API agent 1702 can route event calls to functions. In the example of 1700 , API agent 1702 can generate function request 1710 via, for example, an API. Function request call 1710 can be provided to orchestrator 1704. Function request 1710 can be a request to execute an original function 1712. Orchestrator 1704 can analyze function request 1710. In detail, orchestrator 1704 can determine whether to provide redundant function execution. Such determination can be based on whether the original function 1712 meets a quality threshold. The quality threshold can be based on a user request (e.g., a token, a class, etc.), a quality of service (QoS) metric, or a service level agreement.

[0493] exist Figure 17A In the example of , the orchestrator 1704 may determine that the original function 1712 meets the quality threshold and, therefore, may mitigate the probability of the original function 1712 not being completed or not being completed in a timely manner through a redundant function execution scheme. The redundant function execution scheme may include generating redundant functions 1708a, 1708b. In one example, each of the redundant functions 1708a, 1708b may be an identical copy of the original function 1712, but each implementation of the original function 1712 may be optionally customized for slower but guaranteed execution at 1708a, 1708b, etc. For example, this may be achieved by using different iteration strategies, including predictive, proactive, and reactive measurements based on the available dynamic utilization level of resources that are critical for guaranteed function execution. For example, a 128MB function may be run on a container that is provisioned with 256MB of RAM. The function may be run on a container that is provisioned with 256MB of RAM. As another example, the function may be run with a longer timeout.

[0494] As illustrated, the original function 1712 and the redundant functions 1708a, 1708b can be provided to different computing nodes 1706a-1706c. Thus, the probability of a function timing out or not completing can be mitigated. Thus, there is enhanced reliability achieved by increasing the probability that one of the redundant functions 1708a, 1708b and the original function 1712 will complete within a predetermined amount of time.

[0495] Once the original function 1712 and one of the redundant functions 1708a, 1708b complete execution, the orchestrator 1704 or another mechanism can cancel the execution of the unfinished functions of the original function 1712 and the redundant functions 1708a-1708b. Thus, resources can be efficiently managed through cancellation.

[0496] In some embodiments, function request 1710 may include a field specifying whether the function should be guaranteed. Thus, users of the FaaS architecture (such as clients) may be able to specify when a function should be guaranteed. In some embodiments, orchestrator 1704 may include a whitelist of guaranteed functions and generate multiple copies of those functions during execution.

[0497] In some embodiments, redundant functions 1708a, 1708b may have non-overlapping execution with original function 1712. For example, compute node 1706c may provide a progress report of the execution of original function 1712. The progress report may indicate the number of completed operations of original function 1712, the currently executed line of code, etc. If the progress report indicates that original function 1712 is delayed or may time out, orchestrator 1704 may generate redundant functions 1708a, 1708b at some time after original function 1712 begins execution.

[0498] In some embodiments, orchestrator 1704 can determine the resource requirements that may be needed for the execution of primitive functions 1712. Orchestrator 1704 can determine the resource requirements based on available dynamic utilization measurements of the resources. In some embodiments, historical information such as that collected from various telemetry sources (e.g., from timers, function or debug logs, and performance monitors) can be used to estimate the execution time and / or resource requirements of each function, which can then be used to inform the scheduling of future invocations of the function. For more details, see Figure 24A and Figure 41A . Orchestrator 1704 may determine whether to generate redundant functions 1708a, 1708b based on resource requirements. For example, redundant functions 1708a, 1708b may be generated in response to an identification that compute node 1706c may lack sufficient resources to complete execution of original function 1712 or may be unable to meet resource requirements to complete execution of original function 1712. For example, when originally scheduled, compute node 1706c may have sufficient resources to execute original function 1712.

[0499] However, if compute node 1706c degrades in terms of resources, compute node 1706c may lack sufficient resources to execute original function 1712. For example, if compute node 1706 needs to be throttled due to overheating, or a newer, higher priority function is assigned to compute node 1706c (which forces a full allocation of resources), the available resources for execution of original function 1712 may be reduced. In some embodiments, an identification can be made that original function 1712 will require a certain amount of resources that exceeds a threshold and, therefore, may be difficult to maintain on a particular server (such as compute node 1706c). In some embodiments, original function 1712 and each of redundant functions 1708a-1708b can be scheduled by orchestrator 1704 to begin execution at the same time. For example, the original function 1712 and the redundant functions 1708a, 1708b can be scheduled for execution responsive to an identification that the original function 1712 meets a quality threshold so as to begin execution at approximately the same time or as quickly as possible for each of the original function 1712 and the redundant functions 1708a, 1708b. The redundant functions 1708a-1708b and the original function 1712 can have overlapping execution times.

[0500] In some embodiments, redundant functions 1708a, 1708b may be scheduled to execute only if the original function 1712 fails to complete execution. For example, orchestrator 1704 may determine that the original function 1712 has already failed to complete execution and meets a quality threshold. Therefore, orchestrator 1704 may determine that multiple redundant copies of the original function 1712 should be generated as redundant functions 1708a, 1708b to mitigate the occurrence of a second non-completion situation.

[0501] In addition, the API proxy 1702 or orchestrator 1704 can identify the importance of the original function 1712 based on specified parameters. The specified parameters can be set by an administrator or client, and the specified parameters specify different types of iteration strategies for different functions in order to obtain a certain degree of certainty. As described above, such strategies can include predictive, proactive, and reactive measurements based on available dynamic utilization measurements of resources required for the execution of the original function 1712. Once the original function 1712 runs to completion, the orchestrator 1704 can cancel, discard, or kill the redundant functions 1708a, 1708b regardless of what stage of execution they may be in, thereby saving unnecessary resource consumption. Therefore, the orchestrator 1704 can enhance FaaS function implementation by mitigating the probability of function failure and ensuring that the functions will be executed in a timely manner.

[0502] Figure 17B A method 1750 for implementing redundant FaaS functions is shown and may be implemented by Figure 13A1302, but may also be implemented in conjunction with the server 1302. Figure 4 Enhanced FaaS system 400 and / or Figure 17A The orchestrator 1704 and / or one or more modules thereof may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, etc., in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination thereof.

[0503] The illustrated processing block 1752 may receive a function request call for executing an original function. The function request call may originate, for example, from an application or a user device. The illustrated processing block 1754 may determine whether the original function meets a quality threshold. If not, only the original function may be executed. That is, no redundant function may be generated. However, if yes, the illustrated processing block 1756 may determine to execute one or more redundant functions along with the original function. Each of the one or more redundant copies may be an identical copy of the original function. The illustrated processing block 1756 may include executing the original function and the one or more redundant copies at different computing nodes at overlapping times. In addition, the illustrated processing block 1756 may further include starting execution of the original function at a first time and starting execution of the one or more redundant copies at a second time after the first time. In some embodiments, the original function and the one or more redundant copies may be executed at non-overlapping times. In some embodiments, the one or more redundant copies may begin execution in response to an indication that the original function fails to meet one or more of a progress threshold or resource requirements.

[0504] Additional Notes and Examples

[0505] Example 1700 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: receive a function request call for performing an original function, determine whether the original function satisfies a quality threshold, determine one or more redundant functions to be performed along with the original function in response to the original function satisfying the quality threshold, wherein each of the one or more redundant functions is a copy of the original function, execute the original function and the one or more redundant functions at overlapping times, execute the original function and the one or more redundant functions at different computing nodes, begin execution of the original function at a first time, and begin execution of the one or more redundant functions at a second time after the first time, and begin execution of the one or more redundant functions in response to an identification that the original function cannot meet a progress threshold or resource requirement.

[0506] Example 1701 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: receive a function request call for performing an original function, determine whether the original function satisfies a quality threshold, and, in response to the original function satisfying the quality threshold, determine to perform one or more redundant functions along with the original function, wherein each of the one or more redundant functions is a copy of the original function.

[0507] Example 1702 includes the at least one computer-readable medium of example 1701 comprising a further set of instructions that, when executed by a computing device, cause the computing device to: perform the original function and the one or more redundant functions at overlapping times.

[0508] Example 1703 includes the at least one computer-readable medium of example 1701 comprising a further set of instructions that, when executed by a computing device, cause the computing device to: perform the original function and the one or more redundant functions at different computing nodes.

[0509] Example 1704 includes at least one computer-readable medium of example 1701, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: begin execution of the original function at a first time, and begin execution of one or more redundant functions at a second time after the first time.

[0510] Example 1705 includes at least one computer-readable medium of example 1701, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: initiate execution of one or more redundant functions in response to an identification that the original function cannot meet a progress threshold or resource requirement.

[0511] Iterative execution of FaaS and / or AFaaS functions

[0512] Some tasks may trigger iterative execution of FaaS and / or AFaaS functions. Although reference is made below to FaaS, it will be understood that AFaaS functions may also be similarly executed.

[0513] FaaS functions can investigate different spaces or areas of interest. For example, a bid matching task can explore various bid combinations in an attempt to find the bid with the most favorable terms. Various combinations can be explored by iterating the FaaS functions. As an example, the FaaS functions may need to iterate repeatedly to perform the bid matching process. For example, if the terms of the bid include a requirement to change the bid or to be higher than other bids, one or more functions may need to operate iteratively. As another example, in an AI-enabled edge service, an inference may not be identified as a solution until the inference has an associated confidence metric that falls within an acceptable risk criterion.

[0514] In some cases, iterative execution can be supported by making iteration control dynamically scriptable or variable to account for the various dynamic conditions described above, as the conditions for completing an iterative task may be dynamically specified. Thus, FaaS functions in some contexts operate iteratively with dynamic control. However, iterative FaaS functions sometimes continue to operate unnecessarily when the dynamic conditions for completion cannot be met. For example, suppose there are two search spaces, each explored by a different FaaS function. After a certain amount of time, it may become apparent that one of the FaaS functions is exploring a search subspace that is unlikely to produce a viable solution. Causing such a FaaS function to operate unnecessarily is inefficient because it consumes resources that could be allocated to other FaaS functions that are more likely to produce a viable solution. Furthermore, a FaaS function may not reach a termination condition (e.g., a viable solution) and continue to operate until the time is up. As a result, waiting time is increased and resources are allocated inefficiently.

[0515] A search space can be an area of feasible solutions or answers in which a search scheme or process can be explored to seek possible solutions or answers that meet the search criteria. For example, a search space can be used to describe the entire set of probabilities, parameters, algorithms, schemes, values, or access or time constraints for which a search can be considered feasible. In a specific example, for a supercomputer playing chess, the search space may be very large, but for a program running on a laptop or tablet computer that is disconnected from the cloud, the search space may be constrained in terms of evaluation depth, evaluation time, or the number of replacement steps to be evaluated. Although reference is made to a search space above, reference will be made to a "solution space" below, where a search space is an example of a solution space. A solution space can be a space in which a solution is to be identified.

[0516] Steering Figure 18A and 18B An enhanced scheduler 1820 is provided to generate a function generation graph 1800 for a FaaS service. The enhanced scheduler 1820 can de-allocate or re-prioritize resource allocations from the solution space of the function generation graph 1800 based on conditions being met, such as the probability that the solution space (e.g., a particular set of explorations) is unlikely to yield a feasible solution. Doing so can enhance resource allocation and reduce the time required to find a feasible and near-optimal solution from the feasible solution space.

[0517] Figure 18A The function generation diagram 1800 is illustrated. The scheduler 1820 (e.g. Figure 18B1800 and implementing various hardware or software implementations for scheduling, canceling, and re-prioritizing meta-tasks for executing the functional graph 1800 in an attempt to find a sufficiently valuable solution. For example, the scheduler 1820 may be a branch and boundary operator built using thread building blocks; or it may be a SQL query optimizer, etc. Task 1812 may generate function A 1802, function B 1804, function C 1806, function D 1808, and function E 1810. In the following description, for simplicity, function A 1802, function B 1804, function C 1806, function D 1808, and function E 1810 may be collectively referred to as "generated functions."

[0518] Function generation graph 1800 is a data representation of a schedule for generated functions to operate in parallel or serially. The generated functions can be different but related functions. Function C 1806 may depend on data from function A 1802 for execution and is therefore dependent on function A 1802. Thus, function A 1802 and function C 1806 can be considered as the first branch 1814 of function generation graph 1800. Similarly, function D 1808 may depend on data from function B 1804 for execution and is therefore dependent on function B. Function B 1804 and function D 1808 can be considered as the third branch 1818 of function generation graph 1800. Function E 1810 may not require data from another function to operate. Function E 1810 can be considered as the second branch 1816 of function generation graph 1800. Each of the first, second, and third branches 1814, 1816, 1818 may include at least one iterative function and can be considered as separate solution spaces and / or exploration possibilities for potential solutions or outcomes. To be considered complete, the task 1812 may only take the best possible solution or result from one of the first, second, and third branches 1814, 1816, 1818.

[0519] As illustrated, the first, second, and third branches 1814, 1816, 1818 can be different from each other but related. That is, the first, second, and third branches 1814, 1816, 1818 can represent different search solution spaces or regions that are iteratively explored. The first, second, and third branches 1814, 1816, 1818 can operate independently, for example, by not sharing information, and therefore can be considered separate. However, the first, second, and third branches 1814, 1816, 1818 can be related because one of the first, second, and third branches 1814, 1816, 1818 can cause deprioritization or cancellation of itself or another of the first, second, and third branches 1814, 1816, 1818.

[0520] For example, if one of the first, second, and third branches 1814, 1816, 1818 reaches a cancellation condition (which can be considered a deprioritization condition), the scheduler 1820 can terminate the branch. In another example, the resource allocation for one of the first, second, and third branches 1814, 1816, 1818 can be modified by the scheduler 1820 based on the deprioritization condition being reached. For example, if the scheduler 1820 identifies that one of the first, second, and third branches 1814, 1816, 1818 has a significantly lower probability of success than another of the first, second, and third branches 1814, 1816, 1818, the deprioritization condition can be considered satisfied. Additionally, the scheduler 1820 can reduce the resource allocation for one branch and increase the resource allocation for the other branch.

[0521] In some embodiments, the first branch 1814, 1816, or 1818 is a branch that narrows the solution space (narrows the possible options of the solution) by iteration. In other ... For example, if one solution space of one of the first, second, and third branches 1814, 1816, 1818 has been reduced through iterations at a faster rate than another solution space of another of the first, second, and third branches 1814, 1816, 1818, then the other branch can be considered to meet the deprioritization condition.

[0522] Specifically, as described above, one of the functions generated may reach a deprioritization condition. The deprioritization condition can indicate that one or more of the first, second, and third branches 1814, 1816, 1818 can be deprioritized. Deprioritization can mean that one or more of the first, second, and third branches 1814, 1816, 1818 are considered to be exploring an area or space that is unlikely to produce a practical solution to the problem to be solved by the task, or may require more resources or time than a certain other generated function. If the scheduler 1820 deprioritizes one of the first, second, and third branches 1814, 1816, 1818, fewer resources can be allocated to the branch, or the branch can be suspended / terminated / released by suspending / terminating / releasing any generated function including the branch. The deprioritization condition can be when one of the generated functions identifies a termination condition, thereby causing the termination of one generated function or another generated function. Thus, efficient resource allocation can be achieved by avoiding the execution of less important or irrelevant functions for the task.

[0523] For example, assume that function D 1808 reaches a deprioritization condition. Third branch 1818 may be terminated, which may include ceasing execution of function D 1808 and / or function B 1804. In some embodiments, rather than terminating third branch 1818, resource allocation to third branch 1818 may be reduced. For example, resource allocation to function D 1808 and / or function B 1804 may be reduced. In some embodiments, a deprioritization condition may be identified by a value generated by function D 1808, a confidence interval for a solution generated by function D, a measure of the likelihood of success of function D, whether function D is approaching a potential solution, and the like.

[0524] like Figure 18B As illustrated in FIG, the enhanced FaaS system 1832 may include a scheduler 1820. As already described, the scheduler 1820 may generate Figure 18A The function diagram 1800 is generated, and the generated functions are scheduled according to the function diagram 1800 and canceled accordingly.

[0525] In detail, the scheduler 1820 can schedule and monitor the execution of the generated functions according to the function generation graph 1800. The iterative execution 1822 of function A, the iterative execution 1826 of function B, the iterative execution 1824 of function C, the iterative execution 1828 of function D and the iterative execution 1830 of function E correspond to Figure 18A Function A 1802, function B 1804, function C 1806, function D 1808 and function E 1810.

[0526] The scheduler 1820 can identify deprioritization conditions (e.g., termination conditions) during execution of the function generation graph 1800 and stop or suspend execution based on the deprioritization conditions. For example, the scheduler 1820 can terminate or suspend one or more of the generated functions and stop and / or pause any further function instantiations of these generated functions. The scheduler 1820 can do this using a hardware queue manager and multicast.

[0527] In detail, the iterative execution 1822 of function A includes FA1-FA n , which are iterations of function A 1802. Iterative execution 1822 of function A may complete, and then iterative execution 1824 of function C may begin. Similarly, iterative execution 1826 of function B may complete, and then iterative execution 1828 of function D may begin. In some embodiments, execution may be performed concurrently. For example, iterative execution 1822 of function A may proceed concurrently with iterative execution 1824 of function C. Furthermore, iterative execution 1826 of function B may proceed concurrently with iterative execution 1828 of function D.

[0528] During iterative execution 1824 of function C, scheduler 1820 may identify a deprioritization condition that terminates iterative execution 1824 of function C. Such a deprioritization condition may include an identification that first branch 1814, and specifically, function C 1806, is unlikely to produce a viable solution. Consequently, iterative execution 1824 of function C executes only one iteration, FC1. Following the cancellation or suspension of iterative execution of function C, resources allocated to iterative execution 1824 of function C may be reallocated to other executions. For example, iterative execution 1830 of function E may have increased resource allocations and / or iterative execution 1828 of function D may have increased resource allocations.

[0529] In some embodiments, instead of canceling the iterative execution 1824 of function C, scheduler 1820 may reduce the resource allocation for the iterative execution 1824 of function C. In some embodiments, scheduler 1820 may reallocate the resource allocation from the iterative execution 1824 of function C to the iterative execution 1830 of function E and / or the iterative execution 1828 of function D. In some embodiments, resources may be reallocated to one or more of the generated functions based on the likelihood that the one or more generated functions have the greatest chance of success and / or a greater chance of success than other generated functions that continue to operate. Doing so may reduce the latency of the iterative execution 1830 of function E and / or the iterative execution 1828 of function D, and may also reduce the latency of identifying a solution to the task. However, the latency of the iterative execution 1824 of function C may increase, but this may be considered an acceptable tradeoff given the lower likelihood of success presented by the iterative execution 1824 of function C.

[0530] In some embodiments, the iterative execution 1824 of function C can be re-prioritized. For example, if a re-prioritization condition is met, the iterative execution 1824 of function C can be re-prioritized so that the iterative execution 1824 of function C is resumed and / or has increased resource allocation. For example, the re-prioritization condition can be an indication that the iterative execution 1824 of function C is more likely to produce a feasible result than other functions in the generated functions. In some embodiments, the re-prioritization condition can be an indication that the likelihood of success of another generated function (e.g., the iterative execution 1828 of function D) is reduced.

[0531] In some embodiments, the iterative execution of the generated functions is performed at different nodes and / or containers. For example, the iterative execution 1822 of function A can be performed at different nodes and / or different containers. FA1 can be executed at a first node in a first container, while FA N The process may be executed at the second container at the second node.

[0532] As already described, the scheduler 1820 can reduce latency and have enhanced resource management by terminating generated functions and / or reducing resource allocation to less successful solution spaces, as described above. For example, efficiently deallocating resources in response to identification of deprioritization conditions.

[0533] Iterative execution control can be dynamically scriptable or variable because the conditions for completing the iterative task can be dynamically specified. Thus, the scheduler 1820 can explore different strategies represented by the first, second, and third branches 1814, 1816, 1818 and, in particular, the resulting functions of the first, second, and third branches 1814, 1816, 1818. The first, second, and third branches 1814, 1816, 1818 can be different search strategies determined based on dynamic cues, recently observed patterns and pattern transitions, constraint changes, etc.

[0534] For example, the scheduler 1820 may identify the best inference within a finite amount of time and within the power budget at any given time. To this end, if the time window is small enough or the threshold cannot be met, and / or if the computational budget is small enough or the threshold cannot be met, the scheduler 1820 may use a low-precision strategy. Figure 18B After exploring the first, second, and third branches 1814, 1816, 1818 as illustrated in FIG, the scheduler 1820 can use higher precision for later iterations after eliminating the solution space such as the first branch 1814. Thus, the scheduler 1820 can initiate several parallel, low-cost searches through the first, second, and third branches 1814, 1816, 1818, and then cancel other branches of the first, second, and third branches 1814, 1816, 1818 in response to an identification that some of the branches will produce more productive results. As already described, the cancellation of one of the first, second, and third branches 1814, 1816, 1818 may mean the cancellation of the resulting functionality including the cancelled branch.

[0535] Thus, the scheduler 1820 can effectively cancel or reduce resource allocations for iterative executions of deprioritized functions among iterative executions of function A 1822, iterative executions of function B 1826, iterative executions of function C 1824, iterative executions of function D 1828, and iterative executions of function E 1830. Deprioritization can mean that the deprioritized iterative functions are deemed to be exploring an area or space that is unlikely to produce a viable solution to the problem being solved by the task. For example, some embodiments may include various metaprogramming support for such dynamic initiation and cancellation of iterative executions of function A 1822, iterative executions of function B 1826, iterative executions of function C 1824, iterative executions of function D 1828, and iterative executions of function E 1830. If a solution subspace becomes temporarily deprioritized, the solution space resources consumed by exploring that solution subspace can be dynamically de-allocated, for example, by moving data out of the cache layer until the solution subspace is re-prioritized. Furthermore, cancellation of generated functions may be automatically generated so that resources can be quickly de-allocated from less important functions or work and allocated to other higher priority work or generated functions.

[0536] In some examples, a flexible multicast topology is implemented in which data (and events) from one iteration at a generated function can be automatically moved to another iteration of that generated function (or another generated function) that may or may not be performed locally. Thus, the topology implements an efficient communication strategy for canceling or re-prioritizing generated functions.

[0537] In addition, some embodiments may have an enhanced multicast topology that generalizes various point-to-point capabilities to meet network function virtualization performance requirements. For example, some embodiments may implement a flexible multicast topology so that when generated functions and tasks are generated in one part of the infrastructure and canceled in another part of the infrastructure, the communication topology adapts with low communication overhead. For example, in edge computing, when a requesting mobile client or a mobile target (e.g., a base station or customer premises equipment) changes location, some embodiments may modify the multicast arrangement between different nodes, containers, etc. in an efficient manner to redistribute the generated functions for the corresponding iterations.

[0538] For example, in some embodiments, the scheduler 1820 can identify the location of the mobile client and / or mobile target. When the mobile client and / or mobile target moves location, the iterative execution of the generated function can be moved to different containers and / or nodes to remain within a predetermined distance of the mobile client and / or target. Doing so can reduce communication latency and can be achieved through the flexible multicast infrastructure described above. In addition, the scheduler 1820 can be aware of each container and / or node so that the scheduler 1820 can control the execution, resource allocation and cancellation of the generated function. In addition, the ability to run metaprograms can be further moved into the hardware (such as a hardware queue manager) and provided to the platform software so that the generated function can have faster execution and efficient cancellation when needed. An embodiment of a hardware queue manager is a platform hardware feature that can receive a control operation or a data movement request for an operation from a first entity among a plurality of entities (e.g., the entity can be a task, a process, a container, a virtual machine, etc.), and after authenticating that the requested operation can be performed without violating applicable security policies and optionally without forcing the first requesting entity to wait until the operation has been completed in a second entity among the plurality of entities (e.g., another task, another process, another container, another virtual machine, etc.), perform the requested operation in a second entity.

[0539] Figure 18C The method 1870 of FaaS function implementation is shown and can be implemented by Figure 13A 1302, but may also be implemented in conjunction with the server 1302. Figure 4 Enhanced FaaS system 400 and / or Figure 18B The scheduler 1820 or one or more modules thereof may be implemented as a set of logic instructions stored in a machine-readable or computer-readable storage medium such as RAM, ROM, PROM, firmware, flash memory, etc., in configurable logic such as, for example, a PLA, FPGA, CPLD, in fixed-function logic hardware using circuit technology such as, for example, ASIC, CMOS, or TTL technology, or as any combination of the above.

[0540] The illustrated processing block 1872 can identify multiple functions to be executed. These multiple functions can be executed iteratively. The illustrated processing block 1874 can be identified in response to a deprioritization condition to deprioritize one or more of the multiple functions. For example, the illustrated processing block 1874 is identified in response to a deprioritization condition to cancel the execution of one or more of the multiple functions. The illustrated processing block 1874 can also be identified in response to a deprioritization condition to reduce the resource allocation of one or more of the multiple functions. The illustrated processing block 1876 can be identified in response to a deprioritization condition to increase the resource allocation of one or more of the multiple functions that are not deprioritized.

[0541] Method 1870 can reduce latency and provide enhanced resource management by terminating and / or reducing resource allocation to less successful solution spaces (e.g., one or more of the plurality of functions), as described above, for example, by efficiently deallocating resources in response to identification of a deprioritization condition, as described with reference to illustrated processing block 1874.

[0542] Additional Notes and Examples

[0543] Example 1800 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: identify a plurality of functions to be executed, deprioritize one or more of the plurality of functions in response to a deprioritization condition being identified, reduce resource allocation for one or more of the plurality of functions or cancel execution of one or more of the plurality of functions in response to the deprioritization condition being identified, and increase resource allocation for one or more of the plurality of functions that are not deprioritized, wherein each function of the plurality of functions is to be executed iteratively.

[0544] Example 1801 includes at least one computer-readable medium comprising a set of instructions that, when executed by a computing device, causes the computing device to: identify a plurality of functions to be performed, and deprioritize one or more of the plurality of functions in response to a deprioritization condition being identified.

[0545] Example 1802 includes the at least one computer-readable medium of example 1801, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: cancel execution of one or more of the plurality of functions in response to a deprioritization condition being identified.

[0546] Example 1803 includes the at least one computer-readable medium of example 1802, comprising a further set of instructions that, when executed by a computing device, cause the computing device to: reduce resource allocation for one or more of the plurality of functions in response to a deprioritization condition being identified.

[0547] Example 1804 includes the at least one computer-readable medium of example 1801 , wherein each function of the plurality of functions is to be performed iteratively.

[0548] Example 1805 includes the at least one computer-readable medium of example 1801 comprising a further set of instructions that, when executed by a computing device, cause the computing device to increase resource allocation to one or more of the plurality of functions that are not deprioritized.

[0549] Enhanced FaaS architecture with shared data storage

[0550] Communication latency in a FaaS environment can present significant overhead. For example, some FaaS environments may include thousands of functions, with corresponding data being passed between nodes for long-term storage and / or execution. Furthermore, functions can be considered "stateless." Consequently, storing data that survives the execution of the functions that created the data can present particular challenges. Some embodiments enhance caching of such data for future reuse.

[0551] For example, consider a database server that can be a node. The node (database server) can be remote relative to the execution node that performs the function. The database server can store data that the function can use to execute. Data access from the execution node to the remote database server can be expensive, and thus the data can be moved to the execution node or moved to a neighboring execution node. Such data transfer can be high latency and resource intensive, depending specifically on the characteristics and size of the data. For example, not only will wasteful round-trip input / output be caused, but the natural format of the data will be modified from a static encoding format to an in-use encoding format, and vice versa. Thus, the marshaling cost and the cost of changing between the static encoding format and the in-use encoding format of the data are increased.

[0552] Now turn Figure 19A , illustrates an enhanced FaaS architecture 1900 with shared data storage. Example 1900 may include Figure 4Enhanced FaaS system 400. Some embodiments can enhance efficiency by utilizing a common data store 1910 (which can be a temporary data store) to store data for reuse by one or more functions. For example, function F1 can generate common data to be used by other functions. The common data can be stored in the common data store 1910. After function F1 completes execution, the common data can remain stored in the common data store 1910. Thereafter, other functions, such as function F2, can begin execution and access the common data. For example, if function F2 begins execution after the first function F1 completes, function F2 can be instantiated to a node that is adjacent to and / or includes the common data store 1910, such as execution node 1904.

[0553] Therefore, the shared data can be kept in the shared data storage 1910 instead of being evicted. In addition, the format of the shared data can be encrypted for security reasons, but cannot be changed in other ways. Thus, the above implementation can reduce IO transfers and data encoding format modifications, thereby reducing latency and enhancing resource utilization.

[0554] Control node 1902 may invoke function F1 at time T0, as indicated by process 1912. Function F1 may be executed on node 1904. Node 1904 may be in communication with a local data store 1906. In some embodiments, local data store 1906 may be part of execution node 1904. In some embodiments, local data store 1906 and execution node 1904 are part of the same computing device (such as a server). As illustrated, local data store 1906 includes at least two data stores or partitions, including a shared data store 1910 and a specific data store 1928.

[0555] Function F1 may generate data when function F1 is executed in the container of execution node 1904. At least some of the data may be referred to as common data. Common data may be reused by other functions. For example, control node 1902 may generate a function generation graph such as Figure 18A1900. In some embodiments, the operation of the control node 1902 can be optionally guided by instructions received from a programmer or a cloud service provider. The control node 1902 can analyze the function generation graph to determine the interdependencies of the functions and, for example, whether one function is to operate based on data generated by another function. The control node 1902 can control whether data is stored in the common data store 1910 or in a specific data store 1928. In some embodiments, the decision of whether data is stored in the common data store 1910 or in a specific data store 1928 can be subject to additional input (e.g., prompts, instructions, etc.) from the programmer or cloud service provider, and the programmer or cloud service provider can change these inputs (e.g., prompts or instructions) from time to time.

[0556] In some embodiments, the control node 1902 can determine that function F2 will consume data generated by function F1. Thus, the control node 1902 can pass a command or message to the execution node 1904 that at least some of the data generated by function F1 are shared data. For example, when function F1 is called at time T0, the control node 1902 can instruct the execution node 1904 to store the data generated by function F1 as shared data. In some embodiments, only a subset of the data of function F1 (e.g., final calculation or conclusion data) rather than all the data of function F1 can be considered as shared data. The data subset can be data identified as reusable by function F2. Other data identified as not reusable by function F2 can be discarded.

[0557] Execution node 1904 may instantiate function F1 and allocate data storage from shared data storage 1910. Function F1 may therefore store shared data in shared data storage 1910, as indicated by process 1914. After function F1 completes execution, instead of sending the shared data to control node 1902, the shared data may reside in shared data storage 1910. For example, instead of immediately removing the shared data from shared data storage 1910, the shared data may remain stored in shared data storage 1910.

[0558] Shared data can be given a survival time strategy, which describes the survival time of shared data. The survival time can be extended by other functions that access the shared data. The survival time can be extended by a fixed amount (or a variable amount, as determined based on past history from a strategy or heuristics) each time a new function accesses the shared data. In some embodiments, shared data can be subject to a maximum survival time, after which the shared data will be automatically expelled from at least shared data storage 1910, and further automatically expelled from local data storage 1906. If shared data remains unaccessed after function F1 completes execution, the shared data can be expelled to control node 1902 after the survival time expires, without any adjustment to the survival time. Local data storage 1906 and / or execution node 1904 can implement the survival time strategy.

[0559] In some embodiments, the time-to-live policy can be balanced against security requirements. For example, if the local data store 1906 is maintained in an edge device that can be easily compromised by physical intrusion, the time-to-live policy can be set to a low value. Additionally, if the shared data has high security requirements, the time-to-live policy can be set to a low value.

[0560] In addition, the function F1 and / or the local data storage 1906 can encrypt the shared data. By doing so, security can be enhanced so that the shared data can only be accessed by authorized functions.

[0561] Function F1, execution node 1904 and / or local data storage 1906 can generate a descriptor (unique identifier, etc.) that describes where the shared data is stored and whether the shared data is encrypted. In addition, if the shared data is encrypted for security purposes, the descriptor may include a decryption protocol for accessing the shared data. For example, the descriptor may include a decryption key for decrypting the shared data. The descriptor may be provided to control node 1902, which may pass the descriptor to other functions at the appropriate time.

[0562] After function F1 completes execution, control node 1902 can call function F2 at time T2, as indicated by process 1924. Execution node 1904 can execute function F2. Function F2 can receive the descriptor of the common data stored by function F1. The descriptor can accurately describe the location of the common data and any appropriate decryption protocol. Function F2 can access the common data stored in the common data storage 1910, as indicated by process 1916, and execute based on the common data. Thus, the survival time of the common data can be increased. Access can mean that function F2 can read the common data, add new common data to the common data and / or overwrite the common data. As described above, since the common data is accessed, the survival time of the common data is extended by a predefined value. In some embodiments, as a replacement or supplement for the survival time, a reference counter can be used to prevent the premature expulsion of common data when function F2 has not yet reached its completion.

[0563] Function F2 may also generate specific data that only function F2 will use. This specific data may be stored in a partition separate from the common data store 1910, such as specific data store 1928, as indicated by process 1922. This specific data may be automatically evicted from specific data store 1928 when function F2 completes execution, as illustrated by process 1930.

[0564] Function F2 may complete execution at a time after T1 (the time of T1's invocation) but before function F3 is invoked at T2 (as indicated by process 1932). When resources dedicated to F2 are reclaimed, certain data generated by function F2 may be automatically evicted to control node 1902, as indicated by process 1930. Control node 1902 may discard the certain data without storing it in long-term storage, or may store the certain data at database node 1908.

[0565] Thus, the shared data store 1910 can provide temporary storage for data transferred between functions F1 and F2. Doing so can limit database access (read / write) to the database node 1908 by storing the data generated by function F1 in the shared data store 1910. In addition, Figure 19A The enhanced FaaS architecture illustrated in FIG1 may have an enhanced resource management protocol because less data is transferred between the local data store 1906 and the control node 1902, further reducing the latency of the function F2. For example, due to the reduced data movement and input / output bandwidth, the function F2 can begin execution with less overhead and without waiting for data transfer of common data.

[0566] Common data storage 1910 may be a local page cache of execution node 1904 to allow function F2 to access it. Specifically, function F2 executes on execution node 1904 and is therefore able to conveniently access the local page cache of execution node 1904. In some embodiments, data storage 1910 may be a storage (file or block cache) cache, a transient key value store using some combination of page cache and block cache, or a file store, etc.

[0567] Control node 1902 may invoke function F3 at time T2,...

Claims

1. A method, the method being implemented at least in part using a cloud service provider system, the cloud service provider system including a resource management subsystem and server node resources, the server node resources being included in a plurality of server nodes, the method comprising: Dynamically allocating, by the resource management subsystem, a portion of the server node resources for execution of a plurality of containers and / or a plurality of virtual machines, the plurality of containers and / or the plurality of virtual machines being used in providing services of the cloud service provider system, the dynamic allocation being based on: (1) request data, (2) telemetry data, (3) predicted resource utilization data, (4) currently available resource data, and (5) other data indicating respective amounts of the server node resources required for execution of the plurality of containers and / or the plurality of virtual machines; in: The dynamic allocation includes one or more of the following: container placement and / or virtual machine placement; Container migration and / or virtual machine migration; Container generation and / or virtual machine generation; Container kill and / or VM kill; and / or Container load balancing and / or virtual machine load balancing; and The resource management subsystem is implemented at least in part using: (1) server central processing unit hardware, and (2) network interface controller hardware.

2. The method of claim 1, wherein: The telemetry data includes container-level power performance monitoring data and / or virtual machine-level power performance monitoring data.

3. The method of claim 2, wherein: The service is provided in association with a plurality of tenants of the cloud service provider system; and The dynamic allocation is also based on service level agreement data.

4. The method of claim 3, wherein: The telemetry data includes resource usage metric data associated with container execution and / or virtual machine execution by the server node resources.

5. The method of claim 4, wherein: The server node resources include accelerator hardware; and The accelerator hardware can be configured to implement machine learning operations.

6. At least one machine-readable storage medium storing instructions that, when executed by at least one computing device, cause performance of the method of any one of the preceding claims.

7. Cloud service provider systems, including: a plurality of server nodes, the plurality of server nodes comprising server node resources; as well as a resource management subsystem for dynamically allocating portions of the server node resources for execution of a plurality of containers and / or a plurality of virtual machines, the plurality of containers and / or the plurality of virtual machines being used in providing services of the cloud service provider system; in: Dynamic allocation of portions of the server node resources by the resource management subsystem: Based on: (1) request data, (2) telemetry data, (3) predicted resource utilization data, (4) currently available resource data, and (5) other data indicating respective amounts of the server node resources required for execution of the plurality of containers and / or the plurality of virtual machines other data; and Dynamic allocation of parts of the server node resources by the resource management subsystem The allocation includes one or more of the following: container placement and / or virtual machine placement; Container migration and / or virtual machine migration; Container generation and / or virtual machine generation; Container kill and / or VM kill; and / or Container load balancing and / or virtual machine load balancing; and The resource management subsystem is implemented at least in part using: (1) server central processing unit hardware, and (2) network interface controller hardware.

8. The cloud service provider system according to claim 7, wherein: The telemetry data includes container-level power performance monitoring data and / or virtual machine-level power performance monitoring data.

9. The cloud service provider system of claim 8, wherein: The service is provided in association with a plurality of tenants of the cloud service provider system; and The dynamic allocation is also based on service level agreement data.

10. The cloud service provider system according to claim 9, wherein: The telemetry data includes resource usage metric data associated with container execution and / or virtual machine execution by the server node resources.

11. The cloud service provider system according to claim 10, wherein: The server node resources include accelerator hardware; and The accelerator hardware can be configured to implement machine learning operations.

Images