Delay-sensing-based serverless request scheduling apparatus and system
By using a latency-aware request scheduling device and system, the request scheduling strategy of the serverless platform was optimized, the SLO violation problem of latency-sensitive requests was solved, and more efficient resource utilization and request processing were achieved.
Patent Information
- Application Number
- PCT/CN2024/092825
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-23
- Filing Date
- 2024-05-13
- Publication Date
- 2024-11-28
AI Technical Summary
Existing serverless platforms cannot effectively avoid service level objectives (SLOs) violations when handling latency-sensitive requests, and fail to fully utilize node resources, resulting in increased request scheduling latency and a higher rate of SLO violations.
A latency-aware request scheduling device and system are adopted. The first scheduling module and the second scheduling module are used to select target pods and target nodes respectively, optimize the node selection strategy, ensure that user requests are executed under the condition of SLO, and reduce request queuing latency and resource waste.
It effectively reduced request scheduling latency, optimized node resource utilization, significantly reduced SLO violation rate, and improved request execution efficiency and user request success rate.
Smart Images

Figure CN2024092825_28112024_PF_FP_ABST
Abstract
Description
Serverless request scheduling device and system based on latency awareness
[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on May 23, 2023, with application number 202310590809.3, and invention name “Serverless request scheduling device and system based on delay perception”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of computer technology, and in particular to a serverless request scheduling device and system based on latency perception. Background Art
[0003] With the development of the cloud computing industry in recent years, more and more front-line engineers are deploying their applications on the Serverless platform. This is because Serverless can save business teams a lot of operation and maintenance expenses of underlying resources. Developers can focus more on the business code logic without worrying about the complex configuration of underlying resources.
[0004] In a paper published in February 2019, Berkeley stated that serverless computing will be the development direction of cloud computing for the next decade. In recent years, a growing number of open-source serverless platforms have emerged, such as Knative and OpenWhisk. (Due to the popularity and unique features of Kubernetes, most open-source serverless platforms use Kubernetes as their cluster management platform.) As more and more businesses rely on serverless platforms, this poses new requirements and challenges for these platforms. This is especially true for short-running, latency-sensitive requests (synchronous user requests with SLO targets). Serverless platforms in related technologies cannot yet address SLO violations, making resolving this technical issue a top research priority.
[0005] Summary of the Invention
[0006] In view of this, a serverless request scheduling device, system and method based on latency awareness are proposed.
[0007] On the one hand, an embodiment of the present application provides a serverless request scheduling device based on latency awareness, the device comprising: a first scheduling module and a second scheduling module,
[0008] The first scheduling module is configured to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module if it is determined that there is no target pod that meets the execution conditions to execute the target user request;
[0009] The second scheduling module is configured to, upon receiving the pod creation request, select a target node from multiple nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information about the new pod to the first scheduling module;
[0010] The first scheduling module is further configured to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
[0011] This device can create a new pod by selecting a target node that meets the node creation requirements, even if no target pod meets the execution requirements. This newly created pod is then designated as the target pod for executing the target user's request. This optimizes the node selection strategy, reusing / creating pods on the target node with the highest processing power. This reduces platform request scheduling latency, more efficiently utilizes node resources, and reduces the rate of SLO violations.
[0012] In one possible implementation, the execution condition is that the sum of the estimated processing time for the target user request, the total network communication time, and the total time it takes for the pod to process all user requests in the current task queue is less than the remaining time until the SLO for the target user request is violated. This allows the most suitable target pod to execute the target user request to be selected based on the SLO and the target user request, reducing the queue time for user requests and the SLP violation rate.
[0013] In one possible implementation, the node creation condition is that the sum of the pod's cold start duration, the time required for the first scheduling module to manage the newly created pod, and the estimated time to process the target user's request is less than the remaining time until the target user's request's SLO is violated. This allows the system to perceive node processing capacity and select target nodes based on indicators such as historical request processing time, historical SLO violation rates, and the current remaining node hardware resources, thereby reducing the SLO violation rate.
[0014] In one possible implementation, the first scheduling module is further configured to receive a new user request and, if the user request satisfies the acceptance criteria, add the user request to the current request queue based on the SLO deadline of the user request. This allows the cluster to determine whether it can execute the user request without violating the SLO upon receiving a new user request, thereby reducing the rate of SLO violations.
[0015] In a possible implementation, the first scheduling module is further configured to, when it is determined that the user request does not meet a receiving condition, determine that the user request cannot be executed and return a request failure report.
[0016] In one possible implementation, the acceptance condition is that the sum of the average latency of the cluster processing the user request, the queuing latency of the user request in the current request queue, and the network transmission latency within the cluster is less than the remaining duration of the current distance violation for the user request. This allows the overall device and cluster's ability to process user requests to be comprehensively considered, thereby determining whether the user request can be executed without violating the SLO.
[0017] In one possible implementation, the first scheduling module is further configured to, upon determining that a target pod exists that satisfies the execution conditions and is intended to execute the target user request, identify the pod that satisfies the execution conditions as the target pod and send the target user request to the target pod. In this way, if a target pod that satisfies the execution conditions exists, the target user request can be directly sent to the target pod for execution, thereby improving the execution efficiency of user requests, reducing queuing delays, and lowering the rate of SLO violations.
[0018] In one possible implementation, a pod that meets the execution conditions is determined as a target pod, including:
[0019] Using the SLO model and each pod's task queue information, the pod with the lowest load is selected from the currently managed pods as the candidate pod.
[0020] If the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
[0021] In this way, the candidate pods can be selected based on the pre-set SLO model, improving the efficiency, speed, and accuracy of selecting the target pods.
[0022] In one possible implementation, the second scheduling module is further configured to, if it is determined based on the pod creation request that the newly created pod does not meet the node creation conditions, determine that the target user request cannot be executed and return a request failure report. This allows the user to be promptly notified of the execution status of the target user request, allowing the user to respond and process the request promptly.
[0023] In one possible implementation, selecting a target node from multiple nodes and creating a new pod in the target node includes:
[0024] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0025] Create a new pod on the target node;
[0026] The cluster status information includes resource usage of each node, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0027] In this way, using the model to select the target node can improve the speed and efficiency of target node selection.
[0028] On the other hand, an embodiment of the present application provides a serverless request scheduling system based on latency awareness, including: a serverless request scheduling device based on latency awareness and a serverless computing resource cluster, the device including a first scheduling module and a second scheduling module, the serverless computing resource cluster including multiple nodes, each node including at least one pod,
[0029] The first scheduling module is configured to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module if it is determined that there is no target pod that meets the execution conditions to execute the target user request;
[0030] The second scheduling module is configured to, upon receiving the pod creation request, select a target node from multiple nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information about the new pod to the first scheduling module;
[0031] The first scheduling module is further configured to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
[0032] In one possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request.
[0033] In one possible implementation, the node creation condition is: the sum of the cold start duration of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time until the SLO of the target user request is violated.
[0034] In a possible implementation, the first scheduling module is further configured to receive a new user request, and when it is determined that the user request meets a receiving condition, add the user request to a current request queue according to an SLO deadline of the user request.
[0035] In a possible implementation, the first scheduling module is further configured to, when it is determined that the user request does not meet a receiving condition, determine that the user request cannot be executed and return a request failure report.
[0036] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0037] In a possible implementation, the first scheduling module is further configured to, when it is determined that there is a target pod that meets the execution condition for executing the target user request, determine the pod that meets the execution condition as the target pod, and send the target user request to the target pod.
[0038] In one possible implementation, a pod that meets the execution conditions is determined as a target pod, including:
[0039] Using the SLO model and each pod's task queue information, the pod with the lowest load is selected from the currently managed pods as the candidate pod.
[0040] If the candidate pod meets the execution condition, determine the candidate pod as the target pod;
[0041] The system further includes a data warehouse, and the task queue information is sent by the data warehouse.
[0042] In a possible implementation, the second scheduling module is further configured to, when it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, determine that the target user request cannot be executed and return a request failure report.
[0043] In one possible implementation, selecting a target node from multiple nodes and creating a new pod in the target node includes:
[0044] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0045] Create a new pod on the target node;
[0046] Among them, the cluster status information includes the resource usage of each node, and the system also includes a data warehouse. The cluster status information is the node condition sent by the data warehouse, which is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0047] The beneficial effects of the latency-aware Serverless request scheduling system provided in the above-mentioned other aspect and various possible implementations of the other aspect are the same as those of the latency-aware Serverless request scheduling device provided in the above-mentioned one aspect and various possible implementations of the one aspect. To avoid redundancy, they are not elaborated here.
[0048] In another aspect, an embodiment of the present application provides a latency-aware Serverless request scheduling method, the method comprising:
[0049] The user request at the head of the current request queue is determined as the target user request. If it is determined that there is no target pod that meets the execution conditions to execute the target user request, a pod creation request is generated.
[0050] If it is determined based on the pod creation request that the newly created pod meets the node creation condition, select a target node from multiple nodes, create a new pod in the target node, and determine information about the newly created pod;
[0051] The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
[0052] The beneficial effects of the latency-aware Serverless request scheduling method provided in the above-mentioned further aspect and various possible implementations of the further aspect are the same as those of the latency-aware Serverless request scheduling device provided in the above-mentioned first aspect and various possible implementations of the first aspect. To avoid redundancy, they are not elaborated here.
[0053] In one possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request.
[0054] In one possible implementation, the node creation condition is: the sum of the cold start duration of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time until the SLO of the target user request is violated.
[0055] In a possible implementation, the method further includes: receiving a new user request, and when it is determined that the user request meets a receiving condition, adding the user request to a current request queue according to an SLO deadline of the user request.
[0056] In a possible implementation, the method further includes: if it is determined that the user request does not meet the receiving condition, determining that the user request cannot be executed, and returning a request failure report.
[0057] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0058] In a possible implementation, the method further includes:
[0059] In the case of determining that there is a target pod that meets the execution condition and executes the target user request, the pod that meets the execution condition is determined as the target pod, and the target user request is sent to the target pod.
[0060] In one possible implementation, a pod that meets the execution conditions is determined as a target pod, including:
[0061] Using the SLO model and each pod's task queue information, the pod with the lowest load is selected from the currently managed pods as the candidate pod.
[0062] If the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
[0063] In a possible implementation, the method further includes: if it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, determining that the target user request cannot be executed, and returning a request failure report.
[0064] In one possible implementation, selecting a target node from multiple nodes and creating a new pod in the target node includes:
[0065] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0066] Create a new pod on the target node;
[0067] The cluster status information includes resource usage of each node, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0068] On the other hand, an embodiment of the present application provides a latency-aware Serverless request scheduling device, including:
[0069] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.
[0070] On the other hand, an embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor.
[0071] On the other hand, an embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code runs in an electronic device, the processor in the electronic device executes the above method.
[0072] These and other aspects of the present application will become more readily apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS
[0073] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the application and, together with the description, serve to explain the principles of the application.
[0074] FIG1 shows a schematic flow chart of a related art 1. FIG.
[0075] FIG2 shows a block diagram of a serverless request scheduling device and system based on latency awareness according to an embodiment of the present application.
[0076] FIG3 shows a flowchart of a serverless request scheduling method based on latency awareness according to an embodiment of the present application.
[0077] FIG4 shows a schematic diagram of a parameter configuration interface according to an embodiment of the present application.
[0078] FIG5 shows a schematic diagram of a picture beautification process according to an embodiment of the present application.
[0079] FIG6 shows a schematic diagram of user request allocation according to an embodiment of the present application.
[0080] FIG7 shows a structural block diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION
[0081] Various exemplary embodiments, features, and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.
[0082] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0083] In addition, numerous specific details are provided in the detailed description below to better illustrate the present application. Those skilled in the art will appreciate that the present application can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present application.
[0084] In order to better understand this application, the terms appearing in this application are explained below.
[0085] Platform as a Service (PaaS): A business model that provides a server platform as a service. The service provided by programs through the Internet is called SaaS (Software as a Service). It is one of the three service models of cloud computing. In the era of cloud computing, the corresponding server platform or development environment provided as a service becomes PaaS.
[0086] Serverless is a cloud computing model. Based on Platform as a Service (PaaS), serverless computing provides a microarchitecture where end users do not need to deploy, configure, or manage server services. Instead, the cloud platform provides all the server services needed to run the code.
[0087] Function as a Service (FaaS): Also known as Function as a Service, this refers to a service or technology that modularizes individual functions and makes them shared between service providers and users. This refers to providing Function as a Service or Global Functions as a Service. For example, in a web application, a function might only require a single line of code. However, by repackaging it into a remote call service, it can be executed with minimal resources, significantly reducing development and maintenance time and costs.
[0088] Service-level objective (SLO): A quantitative measure of the service guarantee made by a service provider to its customers. For example, a cloud provider guarantees to its customers an SLO of 300ms or less for 95% of requests.
[0089] Multi-access Edge Computing (MEC): A network solution that provides user services and computing capabilities at edge nodes, bringing application services and content closer to users and collaborating with the network to provide users with a reliable and ultimate service experience.
[0090] A Pod (plain old data structure) is the smallest unit created or deployed in Kubernetes. A Pod encapsulates one or more containers, storage resources, a dedicated network IP address, and policy options to manage and control how the container runs. In serverless scenarios, one or more Pods are typically used to load developer code and deploy developer services.
[0091] Node: A worker node in Kubernetes. A node can be a virtual machine (VM) or a physical machine. Each node has the necessary services to run a pod.
[0092] Latency-aware queue model: When entering the queue (also known as the task queue), the task queue is sorted according to the deadline (urgency) calculated based on the SLO of the user request. When leaving the queue (also known as the task queue), the user request is distributed to the Pod with the smallest current load.
[0093] Request: A user-oriented service request. In the Serverless scenario, this is manifested as an HTTP request sent by the user through the terminal to the cloud, requesting the cloud cluster to process the payload carried by the request. After the cloud computing is completed, the calculation results are returned to the terminal.
[0094] Queue latency: The time required for a task request to enter the queue (also known as the task queue) and exit the queue (also known as the task queue).
[0095] There are several different technical solutions in the related art, and the solutions in the related art are described below.
[0096] Related technology 1:
[0097] As shown in Figure 1, after receiving a user request, Knative first determines whether the request concurrency has an upper limit. If it is determined that there is no upper limit, two pods are randomly selected, and then the pod with the smaller load is selected from these two pods to process the user request. If it is determined that an upper limit is specified, the judgment is continued to determine whether the request concurrency is less than 3 (assuming that the upper limit of the request concurrency is 3). If it is determined that the request concurrency is less than 3, all pods are traversed until a pod with a current request queue length less than the request concurrency is found. This pod is then determined to execute the user request. If it is determined that the request concurrency is greater than or equal to 3, a round-robin scheduling algorithm is used to select the pod that executes the user request.
[0098] As can be seen, Knative considers the user-configured request concurrency and the current pod task queue length to schedule subsequent user requests. However, there are the following issues: Knative's current request scheduling strategy is insufficient for latency-sensitive scenarios, as the processing latency of different requests can vary significantly. For example, with high-computing loads like image processing functions, the processing latency fluctuates greatly depending on the size of each request. If existing Serverless platforms only consider queue length or simply use round-robin, if a pod takes too long to process a single request (the request output is large or the underlying CPU and other hardware resources are not suitable for processing this type of request), it is likely to increase the latency of subsequent request queues, causing a large number of requests to violate the SLO. In worse cases, the pod will have to wait for all requests in the queue to time out.
[0099] Related technology 2:
[0100] Many Serverless platforms reuse the resource scheduling capabilities provided by K8s at the bottom layer. When the K8s scheduling component (generally called Scheduler) receives notification of a Pod creation request, K8s primarily uses Filter and Score strategies to place the Pod on the node with the highest score. The Filter stage filters nodes based on dimensions such as remaining resources and labels. The Score stage scores existing nodes based on dimensions such as resource usage, image location, and Pod affinity, and the node with the highest score is bound to the Pod. However, there are the following problems: the current Pod binding strategy does not take into account the processing power of the node's underlying hardware. For example, in MEC data centers, due to hardware resource limitations, there may be heterogeneous nodes. Different node specifications, CPUs, and memory models have certain differences, and the resulting request processing capabilities will also be different. If K8s cannot perceive node differences, it may also schedule user requests to slower nodes, causing a large number of requests to violate SLOs.
[0101] The relevant technologies mainly have the following problems:
[0102] SLO target unawareness: If the Serverless platform is unaware of the SLO target associated with a request, the resource scheduling component will typically distribute the request load evenly across nodes based on the number of requests. This may cause excessive queuing delays in local request queues for latency-sensitive functions, resulting in SLO violations.
[0103] Lack of awareness of node processing capacity: The Serverless platform is unaware of the usage of the underlying hardware resources of edge computing nodes, historical SLO violation rates, and historical request processing delays. This can cause requests to be assigned to slow-executing computing resources, thereby violating the SLO targets set by users.
[0104] The embodiments of the present application provide a method, device, and system for Serverless request scheduling based on latency awareness, which can solve the following problems in the request load balancing of Serverless platforms in related technologies: 1) The SLO carried by user requests is not taken into account, resulting in a large number of requests violating the SLO target in certain scenarios. 2) The heterogeneity of the platform's underlying resources is not taken into account, resulting in requests being scheduled to certain nodes with lower processing capabilities, causing request accumulation. As a result, the method, device, and system provided by the embodiments of the present application can optimize the request queue load balancing strategy, determine the request processing delay based on the request body size, and select the Pod with the shortest queue delay to process the request to be assigned, without the need to additionally set the request queue concurrency. The node selection strategy can also be optimized, and based on indicators such as historical request processing time, historical SLO violation rate, and the current node hardware resource remaining amount, Pods can be reused / created on the node with the strongest processing capability. In this way, the platform request scheduling delay (queuing delay + processing delay) is reduced, node resources are more effectively utilized, and the SLO violation ratio is reduced.
[0105] As shown in Figure 2, the latency-aware Serverless request scheduling system provided in this application (also referred to herein as the request scheduling system) includes a latency-aware Serverless request scheduling device (also referred to herein as the request scheduling device) 100, a Serverless computing resource cluster (also referred to herein as the cluster) 200, and a data warehouse (Telemetry Repository) 300. The Serverless computing resource cluster 200 includes multiple nodes 201. The request scheduling device 100 includes a first scheduling module 101 and a second scheduling module 102. Each node includes one or more pods. In some embodiments, the request scheduling device 100 may also include a data warehouse (different from that shown in Figure 2). Those skilled in the art may configure the request scheduling device according to actual needs, and this disclosure does not limit this.
[0106] In some embodiments, the first scheduling module 101 may be a request-level load balancing component (Load Balancer, LB, also known as a load scheduler) of a request scheduling system. In some embodiments, the second scheduling module 102 may be a computing resource-level load balancing scheduler of the system.
[0107] The working process and principle of the request scheduling system and device shown in FIG2 will be described below in conjunction with FIG3 .
[0108] The first scheduling module 101 is configured to receive a new user request (S401). It then determines whether the newly received user request meets the reception conditions (S402). If the user request meets the reception conditions, the user request is added to the request queue and the new user request is added to the current request queue according to the SLO deadline. If the user request does not meet the reception conditions, it is determined that the user request cannot be executed, the user request is discarded, and a request failure report is returned (S409). In this way, when a new user request is received, it can be determined whether the cluster can execute the user request without violating the SLO, thereby reducing the proportion of SLO violations.
[0109] In some embodiments, the acceptance condition can be: the sum of the average latency of the cluster processing the user request, the queuing latency of the user request in the current request queue, and the network transmission latency within the cluster is less than the remaining duration of the current distance violation of the user request. In this way, the ability of the entire device and the cluster to process user requests can be comprehensively considered to determine whether the user request can be executed without violating the SLO. In other words, the acceptance condition can be expressed as:
[0110] T(input, cluster)+T(queuing latency)+T(network) <T(current deadline)
[0111] Where T(input, cluster) is the average latency for the cluster to process the user request. T(queuing latency) is the queuing latency of the user request in the current request queue. T(network) is the network transmission latency within the cluster. T(current deadline) is the remaining duration of the current distance violation for the user request.
[0112] In this way, the first scheduling module 101 makes a judgment based on the receiving condition when receiving the user request, so that all the user requests queued in the request queue can be completed without violating the SLO.
[0113] The first scheduling module 101 is further configured to determine the user request at the head of the current request queue as the target user request based on the current request queue (S404). Then, it determines whether there is a target pod that meets the execution condition for the target user request (S405). If there is a target pod that meets the execution condition, the target user request is directly routed to the target pod (S407) so that the target pod can execute the received target user request. If there is no target pod that meets the execution condition, a pod creation request is generated and sent to the second scheduling module 102. In this way, when there is a target pod that meets the execution condition, the target user request can be directly sent to the target pod for execution, improving the execution efficiency of user requests, reducing the queuing delay, and reducing the proportion of SLO violations. When there is no target pod that meets the execution condition, a target node that meets the node creation condition can be selected to create a new pod, and the newly created pod is determined as the target pod for executing the target user request. The node selection strategy can be optimized to reuse / create a Pod on the target node with the strongest current processing capacity, thereby reducing the platform request scheduling delay, making more effective use of node resources, and reducing the SLO violation ratio.
[0114] In some embodiments, the execution condition may be that the sum of the estimated duration for processing the target user request, the total network communication duration, and the total duration for the pod to process all user requests in the current task queue is less than the remaining duration until violating the SLO of the target user request. That is, the execution condition can be expressed as: T(input,Pod)+T(queuing latency,Pod)+T0<T(current deadline). Where T(input,Pod) is the estimated duration for the corresponding pod to process the target user request, and the estimated duration can be determined according to the input size of the target user request. T(queuing latency,Pod) is the total duration for the pod to process all user requests in the current task queue. T0 is the network communication duration for the system scheduling and the like for the target user request, where it can be set as T0 = constant + Δt, and Δt is a random perturbation (with a dynamic distribution law). In this way, the most suitable target pod for executing the target user request can be selected according to the SLO and the target user request, reducing the queuing duration of user requests and the SLP violation ratio.
[0115] In some embodiments, the first scheduling module 101 can determine the target pod for the execution condition by using an SLO model and each pod's task queue information to select the pod with the lowest load from among the currently managed pods as a candidate pod. If the candidate pod meets the execution condition, the candidate pod is determined as the target pod. In this case, the task queue information of each pod can be used as input to the SLO model to measure the load of each pod and calculate the pod with the lowest current load. Selecting candidate pods based on a pre-set SLO model improves the efficiency, speed, and accuracy of target pod selection.
[0116] Among them, the queue task information may include the processing latency of the node, the total SLO time window SLO time window, the number of user requests that violate the SLO SLO violation Request Counte, the number of user requests that have been processed Request Counter, the resource utilization of the current node, etc. This application does not impose any restrictions on this.
[0117] In some embodiments, the load cost of each pod can be calculated using the following formula:
[0118] in, It is the ratio of processing latency to the total SLO time window. The ratio of the SLO violation request count to the total number of processed request counts. The utility cost is a fraction calculated based on the current node's resource utilization. For example, if a cluster administrator wants to maintain a node's CPU resource utilization at X%, they can set a formula around this X%. This application does not impose any restrictions on this.
[0119] The second scheduling module 102, after receiving the pod creation request, determines the newly created pod based on the pod creation request, and then determines whether the newly created pod meets the node creation conditions (S406). If the newly created pod does not meet the node creation conditions, it is determined that the target user request cannot be executed, the target user request is discarded, and a request failure report is returned (S409). If the newly created pod meets the node creation conditions, the Utility model is used based on the cluster status information sent by the data warehouse 300 to determine a target node from the nodes that meet the node conditions among the multiple nodes 201 of the cluster 200, and then a new pod is created on the target node, and the information of the newly created pod is sent to the first scheduling module 101 so that the first scheduling module 101 can manage it based on the information of the constructed pod (S408). Among them, the second scheduling module 102 can apply to the data warehouse 300 for cluster status information before selecting the target node using the cluster status information, so that the data warehouse 300 can timely return the current latest cluster status information to the second scheduling module 102. In some embodiments, the cluster status information may include the resource usage of each node. Then the second scheduling module 102 can use the Utility model and the resource usage of each node to determine a target node that can create a new pod from multiple nodes, and then expand the number of pods on the target node to create a new pod.
[0120] In some embodiments, the node creation condition may be that the sum of the cold start duration of the pod, the duration required for the first scheduling module to manage the newly created pod, and the estimated duration for processing the target user request is less than the remaining duration until the current SLO for the target user request is violated. That is, the node creation condition can be expressed as: T(cold start)+T(acquire new Pod)+T(input)<T(current deadline). Where T(cold start) is the cold start duration. T(acquire new Pod) is the duration required for the first scheduling module to manage the newly created pod. T(input) is the estimated duration for the newly created pod to process the target user request. The estimated duration can be determined according to the input size of the target user request. In this way, the node processing ability can be sensed, and the target node can be selected based on indicators such as historical request processing time, historical SLO violation rate, and remaining amount of current node hardware resources, reducing the SLP violation ratio.
[0121] In some embodiments, the node condition may be that the P95 latency for processing the target user request at the node level is less than the remaining duration of the current distance violating the SLO of the target user request. That is, the node condition can be expressed as: T(input, P95) < T(current deadline). T(input, P95) is the P95 latency for processing the target user request at the node level. The P95 latency can represent the latency of each node processing the target user request ranked at the 95% position in ascending order.
[0122] In some embodiments, the data warehouse 300 is used to monitor the status of each pod, record the current task queue status of each surviving (usable) pod to form task queue information, monitor the resource usage of the nodes 201 where each pod is located, and send the task queue information of each pod to the first scheduling module 101 and send the resource usage of each node 201 to the second scheduling module 102 as cluster status information in response to the application of the second scheduling module 102.
[0123] In some embodiments, during the execution of the target user request by the pods in each node 201, the node 201 can monitor and record the execution latency of the target user request, task queue status such as the current task queue latency of each pod within the node 201, the parameter size of the target user request, the resource usage of the node 201 and other node-related information, and then send the node-related information to the data warehouse 300 in real time.
[0124] In a possible implementation manner, the apparatus may further include an SLO determination module. The target determination module is used to determine the SLO indicated by the user according to the user operation. In some embodiments, the SLO determination module may display a parameter configuration interface as shown in FIG. 4 for the user. A switch control M may be displayed on the parameter configuration interface. The SLO determination module may determine the SLO indicated by the user based on the user's operation on the switch control M.
[0125] The working process and principle of the request scheduling system according to the embodiments of the present application will be schematically described below with reference to the example shown in FIG. 5. As shown in FIG. 5, taking the beautification of mobile phone photos in the end-cloud collaboration scenario as an example, the user needs to upload pictures to the cloud. After receiving the user request for processing photos, the request scheduling system in the cloud needs to allocate target Pods of processing functions to each picture to complete a series of image processing operations. The photos need to pass through multiple function Pods in the form of a pipeline and finally complete the beautification of the pictures. In this scenario, the processing latency of the photos is a very critical service quality measurement index. Therefore, using the request scheduling system provided by the embodiments of the present application in this scenario can effectively reduce the number of user requests violating the SLO and maximize the computing power of a cluster.
[0126] At the same time, in input-sensitive applications such as cloud-based photo beautification (where photo size and processing speed show a certain correlation), it is also very necessary to estimate the processing latency of each user request. Therefore, this application uses a segmented sliding window strategy for regression prediction to achieve processing latency estimation for input-sensitive applications. For example, the photo input included in the target user request is divided into five levels, and regression prediction is performed on each level using a sliding window at the cluster, node, and pod levels, as shown in Table 1:
[0127] Table 1 Estimated request processing delay using the segmented sliding window strategy
[0128] From the three levels of cluster, node, and Pod, regression prediction is performed on the input of each level in the form of a sliding window. That is, when the first scheduling module 101 receives a user request, it determines whether the user request meets the reception conditions from the perspective of the cluster. Then, after the user request reaches the head of the request queue and is determined as the target user request, the first scheduling module 101 will select the candidate pod with the lowest load and meeting the execution conditions as the target pod when selecting a pod for the target user request. Alternatively, after the user request reaches the head of the request queue and is determined as the target user request, because the existing pod does not meet the execution conditions and a new pod needs to be created, the first scheduling module 101 will select the target node and in the target node if the node creation conditions are met. Selecting the target node can take into account the processing speed of the user request in the node dimension, create the pod on a better node as much as possible, and speed up the processing delay of subsequent target user requests.
[0129] The latency-aware Serverless request scheduling device and system provided in this application have the following advantages:
[0130] After receiving the user request, the first scheduling module can preliminarily estimate the queuing delay and processing time (that is, based on the reception condition judgment), and perform fast failure processing for user requests that cannot be processed to avoid the request being blocked in the first scheduling module.
[0131] Using a segmented sliding window strategy to predict request processing latency can more accurately perceive the current Pod load. This data can be used to route user requests to the Pod with the least load, reducing the occurrence of user request SLO violations.
[0132] It can sense the Pod load situation and, by predicting the queuing delay of each Pod, allocate user requests to the Pod with the lowest load, thus avoiding SLO violations due to improper matching between user requests and Pods.
[0133] Perceive the node load and resource situation, and prioritize capacity expansion on nodes with strong processing capabilities (that is, select target nodes) to avoid violations of SLO caused by long request processing time due to differences in request processing capabilities between nodes.
[0134] In some embodiments, for function service products in public clouds, such as Huawei Cloud FunctionGraph, AWS Lambda, and Azure Function, in public cloud scenarios, some user service requests are also latency-sensitive, such as in user-oriented scenarios such as finance and commodity trading. Most serverless products in related technologies still use concurrency as a decision metric for elastic scaling, and users are generally unable to set this metric correctly. This means that if the metric is set too low, each Pod cannot maintain a relatively saturated task queue. Although this can reduce processing latency, it will result in resource waste. If the metric is set too high, each Pod will be overloaded, and the tail task cannot be scheduled in time. Some requests will violate the SLO, affecting product revenue. The request scheduling device and system provided in the embodiment of the present application, as shown in Figure 6, uses SLO and request queuing latency as indicators to measure a Pod load, which can save user costs while reducing the proportion of requests that violate the SLO compared to the above-mentioned concurrency method.
[0135] An embodiment of the present application provides a latency-aware Serverless request scheduling method, which is applied to the latency-aware Serverless request scheduling device described above. The method includes:
[0136] The user request at the head of the current request queue is determined as the target user request. If it is determined that there is no target pod that meets the execution conditions to execute the target user request, a pod creation request is generated.
[0137] If it is determined based on the pod creation request that the newly created pod meets the node creation condition, select a target node from multiple nodes, create a new pod in the target node, and determine information about the newly created pod;
[0138] The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
[0139] The beneficial effects of the latency-aware Serverless request scheduling method provided in the above-mentioned further aspect and various possible implementations of the further aspect are the same as those of the latency-aware Serverless request scheduling device provided in the above-mentioned first aspect and various possible implementations of the first aspect. To avoid redundancy, they are not elaborated here.
[0140] In one possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request.
[0141] In one possible implementation, the node creation condition is: the sum of the cold start duration of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time until the SLO of the target user request is violated.
[0142] In a possible implementation, the method further includes: receiving a new user request, and when it is determined that the user request meets a receiving condition, adding the user request to a current request queue according to an SLO deadline of the user request.
[0143] In a possible implementation, the method further includes: if it is determined that the user request does not meet the receiving condition, determining that the user request cannot be executed, and returning a request failure report.
[0144] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0145] In a possible implementation, the method further includes: if it is determined that there is a target pod that meets the execution condition and executes the target user request, determining the pod that meets the execution condition as the target pod, and sending the target user request to the target pod.
[0146] In one possible implementation, determining a pod that meets the execution condition as a target pod includes: using an SLO model and task queue information of each pod to select a pod with the lowest load from multiple currently managed pods as a candidate pod; and determining the candidate pod as the target pod if the candidate pod meets the execution condition.
[0147] In a possible implementation, the method further includes: if it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, determining that the target user request cannot be executed, and returning a request failure report.
[0148] In one possible implementation, selecting a target node from multiple nodes and creating a new pod in the target node includes: using a utility model and cluster status information to determine the target node from multiple nodes in the cluster that meet node conditions; and creating a new pod on the target node; wherein the cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 latency of processing the target user request at the node level is less than the remaining duration of the current distance violating the SLO of the target user request.
[0149] The implementation methods and beneficial effects of each step of the latency-aware Serverless request scheduling method of this application can be referred to the relevant descriptions of the above-mentioned latency-aware Serverless request scheduling device and corresponding components in the system. To avoid redundancy, they will not be repeated here.
[0150] In some embodiments, the first scheduling module 101, the second scheduling module 102, and the data warehouse 300 can all be implemented in software or hardware. For example, the implementation of the first scheduling module 101 will be described below using the first scheduling module 101 as an example. Similarly, the implementation of the second scheduling module 102 and the data warehouse 300 can refer to the implementation of the first scheduling module 101.
[0151] As an example of a software functional unit, the first scheduling module 101 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the first scheduling module 101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.
[0152] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.
[0153] As an example of a hardware functional unit, the first scheduling module 101 may include at least one computing device, such as a server. Alternatively, the first scheduling module 101 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.
[0154] The multiple computing devices included in the first scheduling module 101 can be distributed in the same region or in different regions. The multiple computing devices included in the first scheduling module 101 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first scheduling module 101 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0155] It should be noted that, in other embodiments, the first scheduling module 101 can be used to execute any step in the Serverless request scheduling method based on latency awareness in the embodiment of the present application, and the second scheduling module 102 and other modules can also be used to execute any step in the Serverless request scheduling method based on latency awareness. The steps that each module is responsible for implementing can be specified as needed. By implementing different steps in the Serverless request scheduling method based on latency awareness respectively, all functions of the Serverless request scheduling device and system based on latency awareness can be realized.
[0156] This application also provides a computing device 500. As shown in Figure 7, computing device 500 includes a bus 502, a processor 504, a memory 506, and a communication interface 508. Processor 504, memory 506, and communication interface 508 communicate with each other via bus 502. Computing device 500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 100.
[0157] Bus 502 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG7 shows a single line, but this does not imply a single bus or type of bus. Bus 504 may include a path for transmitting information between various components of computing device 500 (e.g., memory 506, processor 504, and communication interface 508).
[0158] The processor 504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0159] The memory 506 may include volatile memory, such as random access memory (RAM). The processor 504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).
[0160] Memory 506 stores executable program code, which processor 504 executes to implement the functions of the aforementioned first scheduling module, second scheduling module, data warehouse, and other modules, thereby implementing a latency-aware serverless request scheduling method. Specifically, memory 506 stores instructions for executing the latency-aware serverless request scheduling method.
[0161] Alternatively, the memory 506 stores executable code, and the processor 504 executes the executable code to implement the functions of the aforementioned latency-aware serverless request scheduling device, thereby implementing the latency-aware serverless request scheduling method. In other words, the memory 506 stores instructions for executing the latency-aware serverless request scheduling method.
[0162] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.
[0163] An embodiment of the present application provides a latency-aware Serverless request scheduling device, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.
[0164] An embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor.
[0165] An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0166] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof.
[0167] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.
[0168] The computer program instructions for performing the operations of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present application.
[0169] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.
[0170] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0171] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.
[0172] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, systems, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a part for a module, program segment or instruction, and the part for the module, program segment or instruction comprises one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous boxes can actually be performed substantially in parallel, and they can sometimes also be performed in the opposite order, depending on the function involved.
[0173] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.
[0174] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other variations of the disclosed embodiments by reviewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple situations. A single processor or other unit can implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0175] The embodiments of the present application have been described above. The above description is illustrative and not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or improvements to the technology in the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein.
Claims
1. A serverless request scheduling device based on latency perception, characterized in that: The device comprises: a first scheduling module and a second scheduling module, The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request; The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module; The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
2. The device according to claim 1, characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
3. The device according to claim 1, characterized in that The first scheduling module is further configured to receive a new user request, and if it is determined that the user request meets the receiving condition, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving condition, determine that the user request cannot be executed and return a request failure report. The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
4. The device according to claim 2, characterized in that The first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod.
5. The device according to claim 4, characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
6. The device according to claim 1, characterized in that The second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined that the newly created pod cannot meet the node creation condition based on the pod creation request.
7. The device according to claim 1, characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
8. A latency-aware Serverless request scheduling system, characterized in that: The invention comprises a serverless request scheduling device based on latency perception and a serverless computing resource cluster, wherein the device comprises a first scheduling module and a second scheduling module, and the serverless computing resource cluster comprises a plurality of nodes, each node comprises at least one pod, The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request; The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module; The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
9. The system according to claim 8, characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
10. The system according to claim 8, characterized in that The first scheduling module is further configured to receive a new user request, and if it is determined that the user request meets the receiving condition, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving condition, determine that the user request cannot be executed and return a request failure report; The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
11. The system according to claim 9, characterized in that The first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod.
12. The system according to claim 11, characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod; Wherein, the system further includes a data warehouse, and the task queue information is sent by the data warehouse.
13. The system according to claim 8, characterized in that The second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined that the newly created pod cannot meet the node creation condition based on the pod creation request.
14. The system according to claim 8, characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; Among them, the cluster status information includes the resource usage of each of the nodes, and the system also includes a data warehouse. The cluster status information is the node condition sent by the data warehouse, which is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
15. A serverless request scheduling method based on latency awareness, characterized in that: The method comprises: The user request at the head of the current request queue is determined as the target user request, and if it is determined that there is no target pod that satisfies the execution condition and executes the target user request, a pod creation request is generated; When it is determined based on the pod creation request that the newly created pod meets the node creation condition, a target node is selected from multiple nodes, a new pod is created in the target node, and information of the newly created pod is determined; The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
16. The method according to claim 15, characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining distance to violate the SLO of the target user request. Duration.
17. The method according to claim 15, characterized in that The method further comprises: Receive a new user request, and if it is determined that the user request meets the receiving conditions, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving conditions, determine that the user request cannot be executed and return a request failure report, The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
18. The method according to claim 16, characterized in that The method further comprises: In the case where it is determined that there is a target pod that satisfies the execution condition and executes the target user request, the pod that satisfies the execution condition is determined as the target pod, and the target user request is sent to the target pod.
19. The method according to claim 18, characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
20. The method according to claim 15, characterized in that The method further comprises: In the case where it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, it is determined that the target user request cannot be executed, and a request failure report is returned.
21. The method according to claim 14, characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
22. A serverless request scheduling device based on latency perception, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 15-21 when executing the instructions.
23. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 15 to 21 is implemented.
24. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, characterized in that: When the computer-readable code runs in an electronic device, a processor in the electronic device executes the method according to any one of claims 15 to 21.
Citation Information
Patent Citations
Serverless request scheduling device and system based on time delay perception
CN119030901A
Resource scheduling method and device
CN110795234A
Load balancing method and device in pod, equipment and medium
CN112000477A
Serverless efficient resource allocation method and system based on reinforcement learning
CN116069512A
SLO-aware artificial intelligence inference scheduler for heterogeneous processors in edge platforms
US20220414503A1
