A quality of service guaranteed service computing system and method

Abstract

The quality of service guaranteed function-free computing system comprises an HTTP gateway, a scheduler and a work pool; the HTTP gateway serializes the incoming DAG request through a serializer, and sequentially adds the serialized function-free function in the DAG request to a waiting queue; the scheduler retrieves the function-free function from the waiting queue and the chain running queue to the work pool for processing according to a scheduling strategy. The quality of service guaranteed function-free system and method of the application process the incoming request through the producer-consumer mode by reforming the scheduler in the platform. In addition, a strategy framework containing rich strategies is added. Different kinds of strategies are set for different kinds of application requests, so that the queuing time is reduced to a certain extent, and the running performance is improved.

Description

Technical Field

[0001] This invention relates to the field of serverless computing technology, and in particular to a serverless computing system and method that ensures quality of service. Background Technology

[0002] In serverless computing, Function as a Service (FaaS) is also involved, where application developers provide an event-driven function to a cloud provider. As the event triggers, the cloud provider responds with seamlessly scalable function calls to meet demand. Serverless computing is powerful and expressive, designed for applications in video processing, high-performance computing and scientific computing, machine learning, data analytics, chatbots, backends, IoT, and general applications. Indeed, recent research on serverless production applications shows applications ranging in size from a single function to hundreds of functions, with execution times ranging from less than a second to minutes. Therefore, a rapidly growing serverless-native ecosystem is expected in the future, featuring various serverless function chains where serverless functions call subsequent serverless functions to create components, and must be supported by a wide variety of common frameworks.

[0003] In the cloud, for cloud providers, serverless computing simplifies the design and deployment of microservices, presenting many new challenges regarding resources and scheduling. Specifically, the scheduling challenges, particularly in multi-tenant settings supporting microservices from different applications, include: (1) short function execution times, with 90% of functions executing in less than 1 second, but a few taking 10 seconds; (2) the need for appropriate compute units, or sandboxes, to be isolated, but which can be used to serve future function requests; and (3) the arrival patterns of application requests as a whole, and the significant variations within these microservices and functions, requiring the scheduler to handle a large number of dynamic changes in the workload.

[0004] In large clusters, existing architectures for scheduling and resource allocation are inadequate to handle the above requirements. Centralized schedulers cannot scale to handle low latency and high request-per-second throughput requirements; they are not designed to provide good performance under rapidly changing request arrival patterns. On the other hand, distributed methods (such as Sparrow or Ray; Sparrow is a distributed, low-latency scheduler, and Ray is a distributed framework for AI applications) are scalable for scheduling by multiple schedulers with a global view (e.g., through random machine exploration), but they cannot find sandboxed machines with accessible and reusable access, leading to the additional overhead of sandbox setup. Finally, existing frameworks cannot track the execution time of individual functions and therefore cannot properly optimize DAG requests to ensure end-to-end latency (which may include sandbox provisioning and setup), which is crucial for incoming application requests to be as close as possible to their execution time.

[0005] However, much work remains to be done in managing the current state of serverless function chains. The strategies for managing serverless functions and function chains are relatively simple: current typical scheduling strategies implement a basic first-come, first-served algorithm. When constraints are imposed on parallel-running serverless work, either from provider-imposed hardware concurrency limitations or software concurrency limitations observed due to ineffective resource allocation, serverless applications appear inflexible, suggesting they should be managed under challenging conditions. Existing advanced serverless platforms exhibit several significant problems. Client requests may not receive responses, or response times may be excessively long. Considering the underlying causes, the platform's scheduling strategy may have considered too few factors, resulting in some DAG requests (directed acyclic graphs of multiple serverless functions) not receiving the appropriate priority, meaning service quality is not guaranteed (i.e., processing time and queuing latency increase). Furthermore, existing scheduling frameworks in cloud platforms are not well-suited for handling large volumes of requests in multi-tenant cluster environments. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a serverless computing system and method that ensures service quality.

[0007] To address the aforementioned technical problems, the present invention proposes the following technical solution: a serviceless computing system that ensures service quality, comprising an HTTP gateway, a scheduler, and a worker pool; the HTTP gateway transmits incoming DAG requests to a serializer for serialization, and sequentially adds the serviceless functions in the serialized DAG requests to a waiting queue; the first function of the DAG is processed, and the remaining unprocessed functions in the DAG are added to a chain execution queue; the scheduler retrieves serviceless functions from the waiting queue and the chain execution queue according to a scheduling strategy and adds them to the worker pool for processing.

[0008] In the aforementioned service-free computing system that ensures service quality, preferably, the scheduling strategy is added, removed, modified, or replaced through a strategy framework.

[0009] In the aforementioned serviceless computing system that guarantees service quality, preferably, the scheduler is a semi-global scheduler; the semi-global scheduler divides the cluster into multiple work pools, each work pool corresponding to a semi-global scheduler; when the load balancing service receives the observed queuing delay, and the DAG requests in the waiting queue reach the expansion threshold, the load balancer expands the DAG requests to another semi-global scheduler with the current DAG requests.

[0010] In the aforementioned service-free computing system that ensures service quality, preferably, when the DAG requests in the waiting queue are below the shrinkage threshold, the load balancer separates a semi-global scheduler from the DAG requests.

[0011] In the aforementioned service-free computing system that ensures quality of service, preferably, the semi-global scheduler also includes an evaluator and a sandbox manager; the sandbox manager makes decisions on sandbox allocation and recycling; and the evaluator evaluates the execution cycle of each DAG request.

[0012] In the aforementioned service-free computing system that guarantees service quality, preferably, the scheduling strategy includes one or more of the following: function-level allocation, chain-level allocation, reactive concurrent allocation, chain priority, shortest job priority, explicit priority allocation, hybrid scheduling, and resource-aware scheduling.

[0013] A service-free computation method for ensuring service quality includes the following methods:

[0014] 1) Serialize the DAG request at the HTTP gateway using a serializer;

[0015] 2) Step 1) After serializing the DAG requests, add the unserviced functions to the waiting queue one by one;

[0016] 3) The scheduler will retrieve no-service functions from the waiting queue and the chained run queue and process them in the work pool according to the scheduling policy.

[0017] In the above-mentioned service-free calculation method for ensuring service quality, preferably, in step 1), the evaluator evaluates the execution cycle of the DAG request, and the sandbox manager allocates the sandbox for the current DAG request based on the evaluation results.

[0018] In the above-described service-free computing method for ensuring service quality, preferably, the scheduler in step 3 is a semi-global scheduler; the semi-global scheduler divides the cluster into multiple work pools, each work pool corresponding to a semi-global scheduler; each DAG request can be routed to any load balancer; the load balancer can route requests to any semi-global scheduler.

[0019] In the above-mentioned service-free computing method for ensuring service quality, preferably, in step 3), when the DAG requests in the waiting queue reach the expansion threshold, the load balancer expands the DAG requests to another semi-global scheduler with the current DAG requests; when the DAG requests in the waiting queue are below the shrinkage threshold, the load balancer separates a semi-global scheduler from the DAG requests.

[0020] Compared with existing technologies, the advantages of this invention are as follows: The service-free system and method for ensuring service quality of this invention modify the scheduler within the platform and processes incoming requests using a producer-consumer pattern. Furthermore, by adding a policy framework containing rich strategies, different policies are set for different types of application requests, thereby reducing queuing time and improving operational performance to a certain extent. Attached Figure Description

[0021] Figure 1 These are different types of DAG diagrams of this invention.

[0022] Figure 2 This is a diagram of the internal structure of the scheduler of this invention.

[0023] Figure 3 This is an architecture diagram of ArchiQo, the invention in which the core services include a load balancing service and a scheduling service.

[0024] Figure 4 This invention addresses the execution time and relaxation of various DAG types.

[0025] Figure 5 This refers to the number of cold starts under the extended threshold change of this invention.

[0026] Figure 6 This is the end-to-end delay under the extended threshold change of this invention.

[0027] Figure 7 This invention relates to a packaging and horizontal allocation strategy.

[0028] Figure 8 This invention involves the interaction between a semi-global scheduler and a load balancer during the expansion process. Detailed Implementation

[0029] To facilitate understanding of the present invention, the present invention will be described more fully and in detail below with reference to preferred embodiments, but the scope of protection of the present invention is not limited to the following specific embodiments.

[0030] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by those skilled in the art. The technical terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the scope of the invention.

[0031] This invention proposes a method for ensuring timely response to client requests in serverless computing. This method incorporates components such as an HTTP gateway, waiting queue, chained queue, coordinator, and strategy framework into the scheduler of the serverless computing platform. Specific functionalities are described below:

[0032] Waiting queue: After serializing the DAG, the unserviced functions are added to this queue one by one, waiting for the coordinator to retrieve data from it.

[0033] The resource manager acts as a consumer within the framework, pulling function requests from the queue and then scheduling the functions to the worker pool. The workers in the worker pool are the main drivers of the framework, corresponding to dequeued functions and scheduling them to the appropriate workers for processing. When a worker schedules a function, it blocks until the function completes, and afterwards becomes accessible to the worker pool. A simple example is provided. Figure 1 As shown, in an initial state, there is only one worker in the work pool (concurrency limit of 1), and two newly arrived fan-out 2 function chains are in the waiting queue. The coordinator pulls one of the fan-out 2 function chains from the waiting queue and schedules λ1 to the corresponding worker for processing. When λ1 completes, λ2 and λ3 enter the chain run queue. The task of the chain run queue is to hold unprocessed functions in the chain. After the worker completes, it returns to the work pool. Then, the worker is reactivated and another function is scheduled. In this case, λ1 can be scheduled from the second enqueued function chain, and λ2 or λ3 can be scheduled from the first chain. Such scheduling decisions are left to the policy, so the resource manager tightly integrates the policy framework. (λ1 can be taken from either the chain run queue or the waiting queue, and the same applies to λ2 or λ3.)

[0034] The policy framework serves as an entry point for adding, removing, or changing policies within the system. The coordinator in the resource manager can consult the policy framework when making scheduling decisions. Many different types of policies are expressed and applied.

[0035] Next, we'll analyze a given incoming request and briefly outline the processing flow. For example... Figure 2 As shown, the process is as follows:

[0036] Upon receiving a DAG request, the request first passes through the HTTP gateway, then the corresponding DAG is serialized, and the functions within the serialized DAG are added sequentially to the waiting queue. The first function in the DAG is processed, and the remaining unprocessed functions are added to the chain execution queue. The coordinator, according to a strategy, retrieves functions from both the waiting queue and the chain execution queue for execution each time. When selecting a function, a corresponding scheduling strategy is used to select a worker from the worker pool to process the function.

[0037] In addition, the scheduler includes evaluators and sandbox managers. The evaluators periodically evaluate the execution of each DAG. The sandbox managers are responsible for creating, allocating, and reclaiming sandboxes.

[0038] The key ideas of the service-free computing system for ensuring quality of service in this invention are mainly three:

[0039] 1. Decoupling Sandbox Allocation from Request Scheduling: This invention removes sandbox allocation overhead from the critical path of request execution by proactively allocating sandboxes in advance based on the expected future load for a function. Furthermore, this invention uses a novel event placement method to diffuse sandboxes across the cluster, thereby maximizing the likelihood that future requests will benefit from these provided sandboxes.

[0040] 2. Autonomous Scheduler and Service-Level Protocol Awareness Scheduling: To scale scheduling, this invention provides a semi-global scheduler. Each semi-global scheduler corresponds to a partition of the management cluster machines (called a work pool). This ensures that a scheduler does not become a scaling bottleneck and that the scheduler makes optimal decisions within the work pool. We have also developed a shortest job-first scheduling strategy, which leverages the flexibility of different slack requirements across different requests and multiplexing between app requests to ensure deadlines are met.

[0041] 3. Co-designed Load Balancing and Scheduling Layer: Dividing the cluster into numerous semi-global schedulers introduces the challenge of determining which DAG request is assigned to which semi-global scheduler. We address this issue using load balancers and co-designed the load balancing and scheduling layers so that the load balancing layer has the required visibility for (a) sandbox-aware request routing and (b) preventing individual semi-global schedulers from becoming hotspots. This maximizes future requests and promotes proactive allocation. Furthermore, we developed a low-overhead, gradual scaling mechanism that allows for the logical scaling of schedulers, with each scheduler associated with a DAG request to prevent hotspots without unduly impacting request processing.

[0042] This invention provides an end-to-end example that highlights the various features of the fast-responding serverless computing system of this invention.

[0043] Initialize DAG request upload

[0044] Users develop functions that constitute DAG computation requests and upload them to the platform of this invention. During the initial upload, as is customary, users also specify the function resource requirements, accompanied by a JSON-based language for the DAG request structure. Crucially, this invention also requires users to specify the maximum execution time for a DAG request with a new input trigger, which may result in a 99% latency, acceptable for an application. The objective of this invention is to maximize the number of requests completed within the deadline.

[0045] Request control flow

[0046] like Figure 3 When a request arrives at our platform, it is routed to one of many load balancers, which constitute the load balancing service. The load balancer routes the request to one of many semi-global schedulers, and these semi-global schedulers form a scheduling service based on their routing policies. Within the semi-global scheduler, the request first passes through a gateway, is serialized, and then enters the appropriate queue, awaiting scheduling. Requests are prioritized by the semi-global scheduler according to policies within the scheduling framework and run on workers accessible in the worker pool, in a job-protected manner.

[0047] Dispatch service

[0048] The scheduling service manages the sandbox and the DAG requests generated by scheduling. This invention provides an extensible architecture and scheduling method. This architecture and scheduling method are used in this invention to minimize the impact of sandbox setup time overhead.

[0049] Semi-global scheduler

[0050] To handle low-latency requirements and make optimal scheduling decisions, the system and method of this invention divides the cluster into numerous worker pools, each containing a subset of machines in the cluster. Each worker pool is assigned to a semi-global scheduler, and these semi-global schedulers form part of the scheduling service. This invention handles a small number of independent, latency-sensitive Directed Acyclic Graph (DAG) requests. The invention partitions DAG requests such that each semi-global scheduler corresponds to a subset of the DAG. This allocation can be changed at coarse-grained time intervals and is managed by a load balancing service.

[0051] Scheduling principles

[0052] First, requests from the load balancer service are sent to the HTTP gateway in the producer. Then, the corresponding DAG request is serialized, and the relevant functions in the DAG are added to the waiting queue. For DAG requests currently being processed, the functions to be processed are placed in the chain execution queue. Each time, the coordinator retrieves a function from the waiting queue and the chain execution queue, and schedules the corresponding function to the workers in the worker pool according to the strategy set in the strategy framework for processing. The specific details of the strategy framework are as follows:

[0053] The strategy framework provides: (1) a series of scheduling strategies, which are richer and more reasonable than those provided by the current service platform; (2) the strategies provided are not limited to a specific use case, but also include an intuitive programming interface to express new strategies. This invention provides eight strategies:

[0054] 1) Function-level allocation: Distribute the concurrency of the entire serverless platform function evenly among all functions. This strategy may be configured to be an unequal distribution (e.g., λa is half the concurrency limit of λb) and can be used to prevent lower-priority functions from being affected.

[0055] 2) Chain-level allocation: Similar to function-level allocation, but it tracks the concurrency of each chain rather than the concurrency of the functions that make up each chain. This strategy can be configured to address unequal decomposition. Chain-level allocation enables a significant single-chain function to obtain a suitable concurrent pool for sharing when the shared pool has lower-priority chains that contain multiple functions. Furthermore, developers can use chain-level allocation to ensure fairness among clients / users.

[0056] 3) Reactive Concurrent Allocation: During initialization, equal chain-level allocation is enforced. The arrival rate of each function chain is calculated, and the concurrency limit for each function chain is adjusted proportionally to the rate. Concurrent allocation is minimized for non-public chains to prevent starvation. This strategy promotes demand-based chain-level allocation, rather than a first-come, first-served function strategy, in today's serverless platforms.

[0057] 4) Implement chain priority: Prioritize completing already started chains over new chain calls. Functions are retrieved from the chain run queue before being queued in the waiting queue. This can be used to ensure a chain is completed with minimal overhead between the beginning and end of the chain.

[0058] 5) Shortest Job First: The running chain has the shortest remaining running time. Alternatives, such as lottery scheduling to mitigate unfairness or deadline-aware scheduling to maintain service-level protocols, can also be implemented.

[0059] 6) Explicit Priority Assignment: Assign priorities to functions and chains. At runtime, the highest priority function is selected first. This strategy also supports weighted priority execution (e.g., every n λa calls will result in n / 2 λb calls). Priority scheduling is beneficial when users are sharing a framework with a long-running backend application for latency-sensitive applications.

[0060] 7) Hybrid Scheduling: Hybrid scheduling allows developers to specify where a function should run, such as in an edge / cloud setup. This strategy reduces latency for a given task, ensures compliance with private or managed environments, or simply keeps costs low in a private cloud.

[0061] 8) Resource-Aware Scheduling: A resource-aware scheduler can include accessible containers / virtual machines, or other accessible resources such as CPU, concurrency, or memory during scheduling. For example, with knowledge that cannot tolerate sudden increases, a resource-aware scheduler might gradually increase demand. Alternatively, a resource-aware scheduler can consider concurrency limits and schedule corresponding chains. (For example, only allowing 333 fan-out 2-chain workers to run at a time with a concurrency limit of 1000, because 333*3 functions will run in a stable state).

[0062] Active sandbox allocation

[0063] Given that typical serverless jobs have execution times on the order of their execution and overhead on the same scale as built-in sandboxes, we need to ensure that requests are not exposed to this overhead. To achieve this, this invention decouples sandbox allocation from incoming request scheduling, allowing each semi-global scheduler to proactively build sandboxes across its pool of jobs based on anticipated future load. This contrasts with current platforms, which are not job-aware and reactively build sandboxes as requests arrive. By decoupling sandbox allocation from scheduling, this invention facilitates a pipelined sandbox allocation process, with scheduling decisions leading to reduced cold-start impact.

[0064] Aggressive sandbox allocation consumes memory but does not consume any other resources. Machines with high memory usage become the norm, and service-free functions have small memory spaces; we believe this is a viable trade-off in memory consumption, using aggressive sandbox allocation to ensure users are not exposed to the overhead of sandbox setup. To limit memory usage, platform administrators can configure the amount of memory on each machine, which will be used for aggressive sandbox setup. From this point onward, this memory is referred to as the aggressive memory pool. Finally, we note that aggressive sandbox allocation is a soft-state approach, which can potentially improve performance without compromising correctness.

[0065] Sandbox requirements assessment

[0066] For each DAG request processed by the semi-global scheduler, our goal is to determine the minimum number of sandboxes that need to be allocated to each of its constituent functions, in accordance with the service level agreement. Given a function's execution time and service level agreement, we simulated how requests to the function arrive to determine the minimum number of sandboxes required.

[0067] In our platform design, we assume that request intervals follow an exponential distribution, simulating the expected number of requests within a given time interval T as a Poisson distribution. Specifically, given the service level protocol, we use a function of inverse distribution to find the maximum number of requests that will arrive within time T. However, given that the execution time of a function can be longer than T, we extend the maximum number of requests to account for requests overflowing from the current time interval to the next.

[0068] The semi-global scheduler requires evaluating the arrival rates of functions to construct a Poisson assignment, which, using the method described above, can then be used to determine the number of sandboxes. In the backend, the semi-global scheduler (through its evaluator module) continuously records the arrival rates of functions (at 100ms intervals in our prototype) and tests the rates at the current interval against previous evaluations, using an exponentially weighted moving average to obtain a new evaluation. For all the functions it is processing, the semi-global scheduler tests and evaluates this.

[0069] Sandbox Placement

[0070] Now, given the number of sandboxes, these sandboxes need to be actively built for a given function. The semi-global scheduler needs to decide how to place these sandboxes using a variety of workers in a worker pool. Ideally, we want to deploy these sandboxes to maximize the number of future requests that will use them.

[0071] One approach is to package many sandboxes of the same function onto the same worker. While this reduces memory overhead, it doesn't increase the likelihood of benefiting from aggressively allocating future requests. For example, as... Figure 7 Consider a scenario where there are two worker machines, and the evaluation requirement for two functions is that each machine has two sandboxes. Using the above method, the sandboxes belong to the same function, and the function is built on the same worker. In such a case, when a core is accessible on one worker, and a request for the second function is scheduled, it incurs the overhead of building a new sandbox because there is no compatible sandbox accessible on the worker.

[0072] Sandbox Expulsion

[0073] However, sandbox demand assessments do not pre-allocate all required sandboxes. For these excess sandboxes, a decision needs to be made regarding what to do. One reasonable approach is to evict these containers from the work pool as they consume memory. However, in this invention, we do not need to evict containers from the work pool, thus avoiding unnecessary sandbox allocation overhead.

[0074] Load balancing service

[0075] The load balancing service routes requests to the semi-global scheduler.

[0076] Service responsibility

[0077] The load balancing service has two key responsibilities: (1) Load balancing across semi-global schedulers: Given the following semi-global schedulers to divide the cluster, the load balancing service should ensure that the load is spread across all semi-global schedulers so that no single semi-global scheduler becomes a bottleneck; (2) Sandbox-aware routing: The load balancing service should be able to properly handle routing requests with the goal of maximizing the number of requests, which is conducive to active allocation.

[0078] The semi-global scheduler used for each DAG request is expanded.

[0079] Given a cluster that is partitioned and managed by various semi-global schedulers, a key issue to be addressed is how many semi-global schedulers should receive an incoming DAG request. One approach is to use all accessible semi-global schedulers and distribute the incoming DAG request evenly.

[0080] Extended indicator

[0081] In cases with a large number of DAG requests, the current number of semi-global schedulers associated with a single DAG request may be too few, necessitating expansion.

[0082] First, when a DAG request arrives, the current semi-global scheduler cannot process the work due to resource inaccessibility. This is either because the incoming load is too high or because of contention among other DAG requests being processed by the same semi-global scheduler. Second, we also need to consider the scaling issue: when significant pressure accumulates on the active memory pool, this can lead to user experience sandbox allocation overhead.

[0083] Extension mechanism

[0084] Initialize the semi-global scheduler selection. For a specific DAG request, when a request first arrives at the load balancer, we use consistent hashing to determine which semi-global scheduler to route the request. Specifically, the load balancing service maintains a consistent hashing ring—all the subsequent semi-global schedulers are hashed to this ring (using their IDs). Now, when the first request arrives, the load balancing service hashes the DAG's ID to this ring and assigns it to the initialized semi-global scheduler. Consistent hashing ensures that no single semi-global scheduler is overwhelmed by responding to large shared DAG requests.

[0085] like Figure 8 The diagram illustrates the scaling process. The load balancing service receives the observed queuing latency through this DAG request across various semi-global schedulers. If the queuing latency exceeds the scaling threshold, the load balancing service scales by associating with another semi-global scheduler that also has this DAG. The load balancing service updates the mapping in the reliable storage system and notifies each semi-global scheduler associated with this DAG to reinitialize the queuing latency window, allowing us to observe the impact of our decision. Once the window is full, the load balancing service makes the next scaling decision to avoid the brief changes in response queuing latency.

[0086] The load balancing service follows a similar shrinking process, just as the above method determines whether we need to remove a semi-global scheduler from the DAG. The only difference is that we will shrink if the queuing latency falls below the shrinking threshold. We remove recently added semi-global schedulers from the associated semi-global scheduler pool. To avoid oscillations in the scaling process, we keep the shrinking threshold below the scaling threshold.

[0087] How to expand transparently

[0088] When a load balancing service dynamically scales a semi-global scheduler that is associated with a DAG request, it is necessary to ensure that this does not negatively impact the request. This invention achieves this by scaling gradually rather than rapidly.

[0089] When scaling, we associate an additional semi-global scheduler with this DAG request. However, rapidly sending requests to this new semi-global scheduler will cause these requests to experience the overhead of sandbox allocation.

[0090] The load balancing service circumvents this problem by gradually increasing the number of newly added semi-global schedulers in the following ways: (1) using a lottery scheduling method to represent sandbox-aware routing among various semi-global schedulers, where the number of sandboxes is actively allocated for each semi-global scheduler, and for this DAG request, it has already built a sandbox; (2) notifying the new semi-global scheduler to actively allocate an average number of sandboxes, which are present across the active semi-global schedulers (the calculation includes the new semi-global scheduler). We initialize the new semi-global scheduler so that requests will pass through it, and this will also be updated when the sandbox is built. When they are greedily backed up on the response, the load balancing service remembers the number of sandboxes. Once the required number of sandboxes has been allocated, the system will reach a stable state.

[0091] Similarly, we need to expand gradually. A rapid contraction could lead to an overwhelming reduction in the subset of semi-global schedulers. For a DAG request, we address this by maintaining two tables for semi-global schedulers—an active table and a removal table. During contraction, we remove a semi-global scheduler from the active table and place it in the removal table. During lottery scheduling, we continuously consider semi-global schedulers in the removal table, but reduce the number of lotteries allocated to such schedulers by a discount factor. This ensures that the subset is not overwhelmed and that semi-global schedulers are gradually removed.

[0092] Example 1

[0093] We evaluated the end-to-end benefits of our invention, deployed on a cluster of 74 machines in CloudLab, and compared it to a baseline (IBM's OpenWhisk), which reflects the current state-of-the-art serverless platforms. We also included several micro-benchmarks to delve deeper into the benefits of our invention.

[0094] In Example 1, we tested 38 machines with 20 cores each and 36 machines with 28 cores each. All machines had 256GB of memory and 10Gbps network cards. We partitioned the cluster with 8 semi-global schedulers, each with a worker pool containing 8 machines. Each semi-global scheduler ran on a separate machine. This setup used a single load balancer to construct the load balancing service. We chose a scaling threshold of 0.3 and simulated sandbox setup overhead for different DAGs, ranging from 125ms to 400ms. A conservative assessment of the test overhead involved downloading code packages from S3.

[0095] Baseline stack

[0096] Our baseline uses a centralized scheduler where requests are processed in FIFO order. We also reactively allocate sandboxes and keep them in memory with a fixed inactive timeout of 15 minutes.

[0097] Workload. We considered four different types of DAG requests: (1) C1 contains DAG requests with a single function, short execution time, and tight deadline. These DAG requests represent user-facing functions. (2) C2 contains DAG requests with a single function, short execution time, and less stringent deadline. These DAG requests represent non-critical user-facing functions (such as updating a dashboard). (3) C3 contains DAG requests with chained functions, intermediate execution time, and relatively strict deadlines compared to their execution time. These DAG requests represent more expensive user-facing functions. (4) C4 contains DAG requests with branching structures, high execution time, and loose deadlines. These DAG requests represent backend tasks, typically exhibiting batch execution. The execution time and slack details of our randomized samples are derived from... Figure 4 The range within.

[0098] We construct two load jobs to simulate request arrival rates belonging to different classes. For load job 1, we simulate request arrival patterns following a Poisson distribution. From class C1-C4, we periodically sample this average arrival rate, from the intervals 800-1200, 600-900, 600-800, and 50-150 (requests per second). For load job 2, we simulate request arrival patterns following a sinusoidal distribution. Details are captured in... Figure 4 Both workloads maintained cluster CPU load between 70% and 110%.

[0099] In this embodiment, the following metrics are used to evaluate the different components of the platform: (1) End-to-end latency—representing the round-trip time of a request. (2) Deadline percentage—the percentage of requests that are completed within their deadlines. (3) Queuing latency—the time a request spends in the queue before it is scheduled. (4) Cold start—the number of requests that experience sandbox allocation overhead.

[0100] Macro benchmarking

[0101] For workload 1, experiments were conducted on the system and baseline of this invention to demonstrate end-to-end latency. This invention achieves a 20.83x reduction in tail latency compared to the baseline at the 99.9th percentile. Furthermore, in steady state, this invention matches the performance of the baseline at the 50th percentile. Experiments show that these tail latencies cause the baseline to miss approximately 33% of deadlines, while this invention misses only 0.76% of deadlines.

[0102] When sandboxes are reactively allocated, we find that high tail latency is caused by baseline queuing requests. This invention minimizes the number of cold starts through proactive sandbox allocation and deadline awareness.

[0103] For workload 2, similar results were observed—the invention reduced tail latency by 35.97 times compared to the baseline and missed the deadline by 0.98% compared to 9.66% of the baseline.

[0104] Furthermore, in the context of the baseline, we see this typical DAG class with a slower arrival rate, but it misses more deadlines (C4 misses its deadline more than others, C2 misses its deadline more than C1). Further analysis shows that DAG requests with lower request rates tend to get stuck after requests in the queue of DAGs with higher request rates. This invention essentially mitigates this through a queue-aware extension indicator that triggers an extension to another semi-global scheduler.

[0105] Microbenchmark

[0106] To further explore the benefits of this invention, we ran micro-benchmarks on a smaller scale, with 1LB, one or more semi-global schedulers, and 10 workers for each semi-global scheduler.

[0107] We investigated the effectiveness of sandbox placement and used a semi-global scheduler to evict alternative strategies.

[0108] Uniformly Distributed Sandboxes. We compare methods of uniformly distributing sandboxes across a worker pool with methods of packing sandboxes onto the same worker using a semi-global scheduler. We select a workload worker with a single DAG request, where the request arrives following a sinusoidal distribution with an average RPS of 1200, an amplitude of 600, and a period of 20 seconds.

[0109] Load balancing strategy

[0110] We are now evaluating all aspects of adopting a scaling strategy, using a load balancing service with 5 semi-global schedulers, each with 10 workers.

[0111] Benefits of component scaling. Using lottery scheduling for a given DAG, this invention progressively scales the number of semi-global schedulers. We evaluate the benefits of violating a policy where scaling occurs immediately, causing the load balancing service to route requests in a round-robin manner across the semi-global schedulers. We select load jobs with a single DAG, where request arrivals follow a sinusoidal distribution with 800 RPS (requests per second), an amplitude of 600, and a 100-second period (extended period to capture a snapshot of the scaling benefits).

[0112] System overhead

[0113] Because the goal of this invention is to provide low-latency scheduling, we present some of the overhead that improves performance on the critical path of request execution. From our macro benchmarks, we observed that 50% (99th percentile) of the overhead added per request is determined by the load balancer service, at 190µs (212µs). Scheduling decisions made by the semi-global scheduler add an additional 50% (99th percentile) of overhead, at 241µs (342µs) per request. We also tested the time spent scaling the load balancer service and making an evaluation decision. Regardless of whether these occur on the critical path, they help determine system robustness. Scaling takes 50% (99th percentile) time, or 128µs (197µs). Evaluation by a semi-global scheduler takes 50% (99th percentile) time, or 879µs (1352µs).

[0114] Sensitivity Analysis

[0115] Scaling Threshold (SOT). A lower SOT value means the load balancing service will scale more aggressively. This will result in more frequent scaling and a larger number of cold starts, as... Figure 5 As shown. On the other hand, compared to a passive expansion strategy, aggressive expansion helps keep queuing latency low. Therefore, we observe a trade-off between managing queuing latency and the number of cold starts. From Figure 6 We observed that (1) at very low SOT values, a large number of cold starts negatively impact queuing latency. (2) at higher SOT values, higher queuing latency negatively impacts tail latency. A cluster operator can therefore configure SOT based on knowledge of the load and the overhead of sandbox setup. Based on the above observations, this invention selects an SOT of 0.3.

Claims

1. A quality-of-service (QoS) guaranteed service computing system, comprising: The HTTP gateway, the dispatcher and the work pool; the HTTP gateway serializes the incoming DAG request through the serializer, and sequentially adds the service-free function in the serialized DAG request to the waiting queue; the first function of the DAG is processed, and the remaining functions in the DAG that have not been processed are added to the chain running queue; the dispatcher will dispatch the service-free function to the work pool for processing according to the dispatching strategy from the waiting queue and the chain running queue; The dispatching strategy is added, removed, modified or replaced through the strategy framework; The dispatcher is a semi-global dispatcher; the semi-global dispatcher divides the cluster into multiple work pools, and each work pool corresponds to a semi-global dispatcher; When the DAG request in the waiting queue reaches the expansion threshold, the load balancer expands the DAG request to another semi-global dispatcher with the current DAG request; When the DAG request in the waiting queue is below the contraction threshold, the load balancer separates and removes a recently added semi-global dispatcher from the DAG request; The semi-global dispatcher further comprises an evaluator and a sandbox manager; the sandbox manager makes decisions on sandbox allocation and recycling; the evaluator performs periodic evaluation on the execution of each DAG request; The dispatching strategy comprises one or more of function-level allocation, chain-level allocation, reactive concurrent allocation, chain priority, shortest job first, display priority allocation, hybrid scheduling and resource-aware scheduling.

2. The computing method of the quality-of-service-enabled service-free computing system according to claim 1, wherein: The method comprises the following steps: 1) serializing the DAG request at the HTTP gateway through the serializer; 2) sequentially adding the service-free function to the waiting queue after serializing the DAG request in step 1); 3) the dispatcher will dispatch the service-free function to the work pool for processing according to the dispatching strategy from the waiting queue and the chain running queue.

3. The computing method of the quality of service assured service-less computing system according to claim 2, characterized in that: In step 1), the evaluator evaluates the DAG request execution period, and the sandbox manager allocates the sandbox for the current DAG request according to the evaluation result.

4. The computing method of the quality-of-service-guaranteed service-free computing system according to claim 2, wherein: In step 3), the dispatcher is a semi-global dispatcher; the semi-global dispatcher divides the cluster into multiple work pools, and each work pool corresponds to a semi-global dispatcher; each DAG request can be routed to any load balancer; The load balancer can route the request to any semi-global dispatcher.

5. The computing method of a quality of service guaranteed service free computing system according to claim 2, wherein: In step 3), when the DAG request in the waiting queue reaches the expansion threshold, the load balancer expands the DAG request to another semi-global dispatcher with the current DAG request; when the DAG request in the waiting queue is below the contraction threshold, the load balancer separates a semi-global dispatcher from the DAG request.

Images