Serverless request scheduling device and system based on time delay perception
By using a latency-aware request scheduling device and system, and optimizing node selection and Pod creation strategies, the SLO violation problem in latency-sensitive requests on the Serverless platform is solved, achieving more efficient resource utilization and request processing.
Patent Information
- Application Number
- CN202310590809.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
AI Technical Summary
Existing serverless platforms cannot effectively prevent SLO (Service Level Objective) violations when handling latency-sensitive requests, and fail to fully utilize node resources, resulting in increased request scheduling latency and resource waste.
A latency-aware request scheduling device and system are adopted. Through the collaborative work of the first scheduling module and the second scheduling module, the node selection strategy is optimized to select the most suitable node to create or reuse Pods, so as to meet the SLO requirements of user requests and reduce the violation rate.
It effectively reduced request scheduling latency, improved resource utilization, reduced the proportion of SLO violations, and improved the execution efficiency of user requests.
Smart Images

Figure CN119030901A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a latency-aware Serverless request scheduling device and system. Background Art
[0002] With the development of the cloud computing industry in recent years, more and more front-line engineers have deployed their applications on the Serverless platform. This is because Serverless can save business teams a lot of operation and maintenance costs of underlying resources. Developers can focus more on the code logic of the business without worrying about the complex configuration of underlying resources.
[0003] In a paper published in February 2019, Berkeley mentioned that Serverless will be the development direction of cloud computing in the next decade. In recent years, more and more open source Serverless platforms have begun to appear, such as Knative, OpenWhisk, etc. (Due to the popularity and characteristics of K8s, most open source Serverless platforms use K8s (Kubernetes) as the cluster management base). With more and more businesses relying on Serverless platforms, new requirements and challenges have been put forward for Serverless platforms, especially for short-running latency-sensitive requests (user requests are synchronous requests with SLO targets). Serverless platforms in related technologies cannot solve the problem of SLO violations. How to solve this technical problem is the top priority of research. Summary of the invention
[0004] In view of this, a Serverless request scheduling device, system and method based on latency awareness are proposed.
[0005] On the one hand, an embodiment of the present application provides a latency-aware Serverless request scheduling device, the device comprising: a first scheduling module and a second scheduling module,
[0006] The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request;
[0007] The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module;
[0008] The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
[0009] Through the above device, when there is no target pod that meets the execution conditions, a target node that meets the node creation conditions can be selected to create a new pod, and the newly created pod can be determined as the target pod for executing the target user's request. The node selection strategy can be optimized to reuse / create pods on the target node with the strongest current processing capability, thereby reducing the platform request scheduling delay, more effectively utilizing node resources, and reducing the SLO violation ratio.
[0010] In a possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request. In this way, the target pod that is most suitable for executing the target user request can be selected according to the SLO and the target user request, thereby reducing the queuing time of the user request and the SLP violation ratio.
[0011] In a possible implementation, the node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time from violating the SLO of the target user request. In this way, the node processing capacity can be perceived, and the target node can be selected based on indicators such as historical request processing time, historical SLO violation rate, and the remaining amount of hardware resources of the current node, thereby reducing the SLP violation ratio.
[0012] In a possible implementation, the first scheduling module is further configured to receive a new user request, and when determining that the user request meets the receiving condition, add the user request to the current request queue according to the SLO deadline of the user request. In this way, when receiving a new user request, it can be determined whether the cluster can execute the user request without violating the SLO, thereby reducing the proportion of SLO violations.
[0013] In a possible implementation, the first scheduling module is further configured to, when it is determined that the user request does not satisfy a receiving condition, determine that the user request cannot be executed and return a request failure report.
[0014] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violating the user request. In this way, the ability of the entire device and the cluster to process user requests can be comprehensively considered to determine whether the user request can be executed without violating the SLO.
[0015] In a possible implementation, the first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod. In this way, when there is a target pod that satisfies the execution condition, the target user request can be directly sent to the target pod for execution, thereby improving the execution efficiency of user requests, reducing queuing delays, and reducing the proportion of SLO violations.
[0016] In a possible implementation, determining a pod that meets the execution condition as a target pod includes:
[0017] Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod;
[0018] When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
[0019] In this way, the candidate pods can be selected based on the preset SLO model, which improves the efficiency, speed, and accuracy of selecting the target pods.
[0020] In a possible implementation, the second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition. In this way, the user can be notified of the execution status of the target user request in a timely manner so that the user can respond and process in a timely manner.
[0021] In a possible implementation, selecting a target node from multiple nodes and creating a pod in the target node includes:
[0022] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0023] Create a new pod on the target node;
[0024] The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0025] In this way, using the model to select the target node can improve the speed and efficiency of selecting the target node.
[0026] On the other hand, an embodiment of the present application provides a serverless request scheduling system based on latency perception, including: a serverless request scheduling device based on latency perception and a serverless computing resource cluster, the device including a first scheduling module and a second scheduling module, the serverless computing resource cluster including multiple nodes, each node including at least one pod,
[0027] The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request;
[0028] The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module;
[0029] The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
[0030] In a possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time of the current distance violating the SLO of the target user request.
[0031] In a possible implementation, the node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
[0032] In a possible implementation, the first scheduling module is further used to receive a new user request, and when it is determined that the user request meets a receiving condition, add the user request to a current request queue according to an SLO deadline of the user request.
[0033] In a possible implementation, the first scheduling module is further configured to, when it is determined that the user request does not satisfy a receiving condition, determine that the user request cannot be executed and return a request failure report.
[0034] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0035] In a possible implementation, the first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod.
[0036] In a possible implementation, determining a pod that meets the execution condition as a target pod includes:
[0037] Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod;
[0038] When the candidate pod meets the execution condition, the candidate pod is determined as the target pod;
[0039] Wherein, the system further includes a data warehouse, and the task queue information is sent by the data warehouse.
[0040] In a possible implementation, the second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition.
[0041] In a possible implementation, selecting a target node from multiple nodes and creating a pod in the target node includes:
[0042] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0043] Create a new pod on the target node;
[0044] Among them, the cluster status information includes the resource usage of each of the nodes, and the system also includes a data warehouse. The cluster status information is the node condition sent by the data warehouse, which is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0045] The beneficial effects of the latency-aware Serverless request scheduling system provided in the above-mentioned other aspect and various possible implementations of the other aspect are the same as the corresponding latency-aware Serverless request scheduling device provided in the above-mentioned one aspect and various possible implementations of the one aspect, and are not repeated here to avoid redundancy.
[0046] On the other hand, an embodiment of the present application provides a latency-aware Serverless request scheduling method, the method comprising:
[0047] The user request at the head of the current request queue is determined as the target user request, and if it is determined that there is no target pod that satisfies the execution condition and executes the target user request, a pod creation request is generated;
[0048] When it is determined based on the pod creation request that the newly created pod meets the node creation condition, a target node is selected from multiple nodes, a new pod is created in the target node, and information of the newly created pod is determined;
[0049] The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
[0050] The beneficial effects of the latency-aware Serverless request scheduling method provided in the above-mentioned further aspect and various possible implementations of the further aspect are the same as those of the latency-aware Serverless request scheduling device provided in the above-mentioned first aspect and various possible implementations of the first aspect, and are not repeated here to avoid redundancy.
[0051] In a possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time of the current distance violating the SLO of the target user request.
[0052] In a possible implementation, the node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
[0053] In a possible implementation, the method further includes: receiving a new user request, and when it is determined that the user request meets a receiving condition, adding the user request to a current request queue according to an SLO deadline of the user request.
[0054] In a possible implementation, the method further includes: in a case where it is determined that the user request does not satisfy a receiving condition, determining that the user request cannot be executed, and returning a request failure report.
[0055] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0056] In a possible implementation, the method further includes:
[0057] In the case where it is determined that there is a target pod that satisfies the execution condition and executes the target user request, the pod that satisfies the execution condition is determined as the target pod, and the target user request is sent to the target pod.
[0058] In a possible implementation, determining a pod that meets the execution condition as a target pod includes:
[0059] Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod;
[0060] When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
[0061] In a possible implementation, the method further includes: when it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, determining that the target user request cannot be executed, and returning a request failure report.
[0062] In a possible implementation, selecting a target node from multiple nodes and creating a pod in the target node includes:
[0063] Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster;
[0064] Create a new pod on the target node;
[0065] The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
[0066] On the other hand, an embodiment of the present application provides a latency-aware Serverless request scheduling device, including:
[0067] A processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.
[0068] On the other hand, an embodiment of the present application provides a non-volatile computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions implement the above method when executed by a processor.
[0069] On the other hand, an embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code, and when the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.
[0070] These and other aspects of the present application will become more apparent from the following description of the embodiment(s). BRIEF DESCRIPTION OF THE DRAWINGS
[0071] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the present application and, together with the description, serve to explain the principles of the present application.
[0072] Figure 1 A flow chart of related technology 1 is shown.
[0073] Figure 2 A block diagram of a latency-aware Serverless request scheduling device and system according to an embodiment of the present application is shown.
[0074] Figure 3 A flowchart of a latency-aware Serverless request scheduling method according to an embodiment of the present application is shown.
[0075] Figure 4 A schematic diagram of a parameter configuration interface according to an embodiment of the present application is shown.
[0076] Figure 5 A schematic diagram of a picture beautification process according to an embodiment of the present application is shown.
[0077] Figure 6 A schematic diagram of user request allocation according to an embodiment of the present application is shown.
[0078] Figure 7 A structural block diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0079] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.
[0080] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
[0081] In addition, in order to better illustrate the present application, numerous specific details are provided in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.
[0082] In order to better understand the present application, the terms appearing in the present application are explained below.
[0083] Platform as a Service (PaaS): A business model that provides a server platform as a service. The service provided by programs through the Internet is called SaaS (Software as a Service), which is one of the three service models of cloud computing. In the era of cloud computing, the corresponding server platform or development environment provided as a service becomes PaaS.
[0084] Serverless: It is a model of cloud computing. Based on Platform as a Service (PaaS), serverless computing provides a micro-architecture where end customers do not need to deploy, configure or manage server services. The server services required for code execution are all provided by the cloud platform.
[0085] Function as a service (FaaS): also known as function as a service, refers to a service or technology that modularizes individual functions so that they can be shared by service providers and users, that is, function as a service or global function as a service. For example, in a network application, a function may only require one line of code, but as long as it is repackaged as a remote call service, a small amount of resources can be used to perform the task, thereby greatly saving development and maintenance time and costs.
[0086] Service-level objective (SLO): refers to the quantitative indicator of the service guarantee made by the service provider to the customer. For example, the cloud provider guarantees to the customer that the SLO is 95% of the request processing latency does not exceed 300ms.
[0087] Multi-access Edge Computing (MEC): A network solution that provides the services and computing functions required by users at edge nodes, bringing application services and content closer to users and collaborating with the network to provide users with a reliable and ultimate business experience.
[0088] Pod (Plain old data structure): is the smallest unit created or deployed by Kubernetes. A Pod encapsulates one or more containers, storage resources, an independent network IP, and policy options for managing and controlling how the container runs. In the Serverless scenario, one or more Pods are usually used to load developer code and complete the deployment of developer services.
[0089] Node: A working node in Kubernetes. A Node can be a virtual machine (VM) or a physical machine. Each Node has some necessary services to run a Pod.
[0090] Latency-aware queue model: When entering the queue (also known as the task queue), the task queue is sorted according to the deadline (urgency) calculated by the SLO of the user request. When leaving the queue (also known as the task queue), the user request is distributed to the Pod with the smallest current load.
[0091] Request: A user-oriented service request. In the Serverless scenario, it is expressed as an http request sent by the user to the cloud through the terminal, requesting the cloud cluster to process the load (payload) carried by the request. After the cloud computing is completed, the calculation result is returned to the terminal.
[0092] Queue latency: The time required for a task request to enter the queue (also known as the task queue) and exit the queue (also known as the task queue).
[0093] There are several different technical solutions in the related art, and the solutions in the related art are described below.
[0094] Related technology 1:
[0095] like Figure 1 As shown in the figure, in Knative, after receiving a user request, it first determines whether the request concurrency has an upper limit. If it is determined that there is no upper limit, two pods are randomly selected, and then the pod with a smaller load is further selected from the two pods to process the user request. If it is determined that an upper limit is specified, the judgment of whether the request concurrency is less than 3 (assuming that the upper limit of the request concurrency is 3) is continued. If it is determined that the request concurrency is less than 3, all pods are traversed until a pod with a current request queue length less than the request concurrency is traversed, and the pod is determined to execute the user request. If it is determined that the request concurrency is greater than or equal to 3, the round robin polling scheduling algorithm is used to select the pod that executes the user request.
[0096] It can be seen that Knative considers the user-configured request concurrency and the current pod task queue length to schedule subsequent user requests, but there are the following problems: Knative's current request scheduling strategy is not enough to meet latency-sensitive scenarios, because the processing delays of different requests can vary greatly. For example, for large computing loads like image processing functions, the processing delay fluctuation range of each request image size is large. If the existing Serverless platform only considers the queue length or simply uses round robin, when a Pod processes a single request (the request output data volume is large or the underlying CPU and other hardware resources are not suitable for processing such requests) for too long, it is very likely to increase the delay of the subsequent request queue and cause a large number of requests to violate the SLO. In the worst case, the Pod will have to wait for all requests in the queue to time out.
[0097] Related technology 2:
[0098] Many Serverless platforms reuse the resource scheduling capabilities provided by K8s at the bottom layer. When the K8s scheduling component (generally called Scheduler) receives notification of a Pod creation request, K8s mainly uses the Filter and Score strategies to place the Pod on the node with the highest score. In the Filter stage, nodes are filtered based on dimensions such as remaining resources and labels. In the Score stage, existing nodes are scored based on dimensions such as resource usage, image location, and Pod affinity, and the node with the highest score is bound to the Pod. However, there are the following problems: the current Pod binding strategy does not take into account the processing power of the underlying hardware of the node. For example, in the MEC data center, due to hardware resource limitations, there may be heterogeneous nodes. There are certain differences in node specifications and CPU and memory models, and the represented request processing capabilities will also be different. If K8s cannot perceive the differences in nodes, it is also possible to schedule user requests to slower running nodes, causing a large number of requests to violate the SLO.
[0099] The relevant technologies mainly have the following problems:
[0100] No awareness of SLO targets: If the Serverless platform cannot perceive the SLO targets of requests, the resource scheduling component generally distributes the request load evenly to each node based on the number of requests. This may cause the local request queue of the latency-sensitive function to have too long a queuing delay, resulting in a violation of the SLO.
[0101] No perception of node processing capacity: The Serverless platform cannot perceive the usage of the underlying hardware resources of the edge computing nodes, the historical SLO violation rate, and the historical request processing latency. This will also cause requests to be assigned to slow-executing computing resources, thereby violating the SLO goals set by users.
[0102] The serverless request scheduling method, device and system based on latency perception provided in the embodiment of the present application can solve the following problems in the request load balancing of the serverless platform in the related technology: 1) The SLO carried by the user request is not considered, resulting in a large number of requests violating the SLO target in some scenarios. 2) The heterogeneity of the underlying resources of the platform is not considered, resulting in the request being scheduled to some nodes with lower processing capabilities, causing request accumulation. Therefore, the method, device and system provided in the embodiment of the present application can optimize the request queue load balancing strategy, judge the request processing delay according to the request body size, and select the Pod with the shortest queue delay to process the request to be allocated, without the need to set the request queue concurrency additionally. The node selection strategy can also be optimized, based on indicators such as historical request processing time, historical SLO violation rate, and the remaining amount of hardware resources of the current node, the Pod is reused / created on the node with the strongest processing capability at present. In this way, the platform request scheduling delay (queue delay + processing delay) is reduced, node resources are more effectively utilized, and the SLO violation ratio is reduced.
[0103] like Figure 2 As shown, the latency-aware Serverless request scheduling system (also referred to as the request scheduling system herein) provided in the present application includes a latency-aware Serverless request scheduling device (also referred to as the request scheduling device herein) 100, a Serverless computing resource cluster (also referred to as the cluster herein) 200, and a data warehouse (Telemetry Repository) 300. Among them, the Serverless computing resource cluster 200 includes multiple nodes (Node) 201. The request scheduling device 100 includes a first scheduling module 101 and a second scheduling module 102. Each node includes one or more pods. In some embodiments, the request scheduling device 100 may also include a data warehouse (with Figure 2 Those skilled in the art may configure the request scheduling device according to actual needs, and the present disclosure does not limit this.
[0104] In some embodiments, the first scheduling module 101 may be a request-level load balancing component (Load Balancer, LB, also referred to as a load scheduler) of a request scheduling system. In some embodiments, the second scheduling module 102 may be a load balancing scheduler (Scheduler) at the computing resource level of the system.
[0105] The following combination Figure 3 right Figure 2 The working process and principle of the request scheduling system and device shown are explained.
[0106] The first scheduling module 101 is used to receive a new user request (user request) (S401). It is then determined whether the newly received user request meets the receiving condition (S402). If it is determined that the user request meets the receiving condition, the user request is added to the request queue, and the new user request is added to the current request queue according to the SLO deadline. If it is determined that the user request does not meet the receiving condition, it is determined that the user request cannot be executed, the user request is discarded, and a request failure report is returned (S409). In this way, when receiving a new user request, it can be determined whether the cluster can execute the user request without violating the SLO, thereby reducing the proportion of SLO violations.
[0107] In some embodiments, the receiving condition may be: the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the sum of the network transmission delay within the cluster is less than the remaining time of the current distance violating the user request. In this way, the ability of the entire device and the cluster to process user requests can be comprehensively considered to determine whether the user request can be executed without violating the SLO. That is, the receiving condition can be expressed as:
[0108] T(input, cluster)+T(queuing latency)+T(network) <T(current deadline)
[0109] Wherein, T(input, cluster) is the average latency of the cluster processing the user request. T(queuing latency) is the queuing latency of the user request in the current request queue. T(network) is the network transmission latency within the cluster. T(current deadline) is the remaining duration of the current distance violation of the user request.
[0110] In this way, the first scheduling module 101 makes a judgment based on the receiving condition when receiving the user request, so that all the user requests queued in the request queue can be completed without violating the SLO.
[0111] The first scheduling module 101 is further configured to determine the user request at the head of the current request queue as the target user request based on the current request queue (S404). Then, it determines whether there is a target pod that meets the execution condition for the target user request (S405). If there is a target pod that meets the execution condition, the target user request is directly routed to the target pod (S407) so that the target pod can execute the received target user request. If there is no target pod that meets the execution condition, a pod creation request is generated and sent to the second scheduling module 102. In this way, when there is a target pod that meets the execution condition, the target user request can be directly sent to the target pod for execution, improving the execution efficiency of the user request, reducing the queuing delay, and reducing the proportion of SLO violations. When there is no target pod that meets the execution condition, a target node that meets the node creation condition can be selected to create a new pod, and the newly created pod is determined as the target pod for executing the target user request. The node selection strategy can be optimized to reuse / create a Pod on the target node with the strongest current processing capacity, thereby reducing the platform request scheduling delay, making more effective use of node resources, and reducing the SLO violation ratio.
[0112] In some embodiments, the execution condition may be that the sum of the estimated duration for processing the target user request, the total network communication duration, and the total duration for the pod to process all user requests in the current task queue is less than the remaining duration until the SLO of the target user request is violated. That is, the execution condition can be expressed as: T(input,Pod) + T(queuinglatency,Pod) + T0 < T(current deadline). Wherein, T(input,Pod) is the estimated duration for the corresponding pod to process the target user request, and the estimated duration can be determined according to the input size of the target user request. T(queuinglatency,Pod) is the total duration for the pod to process all user requests in the current task queue. T0 is the network communication duration regarding the target user request such as system scheduling for the target user request, and it can be set as T0 = constant + Δt, where Δt is a random perturbation (with a dynamic distribution rule). In this way, the most suitable target pod for executing the target user request can be selected according to the SLO and the target user request, reducing the queuing duration of the user request and the SLP violation ratio.
[0113] In some embodiments, the first scheduling module 101 determines the target pod of the execution condition by the following method: using the SLO model and the task queue information of each pod, select the pod with the lowest load from the multiple pods currently managed as the candidate pod; if the candidate pod meets the execution condition, determine the candidate pod as the target pod. In the process, the task queue information of each pod can be used as the SLO model input to measure the load of each pod, and the pod with the lowest current load is calculated and output. The candidate pod is selected based on the preset SLO model to improve the efficiency, speed and accuracy of selecting the target pod.
[0114] Among them, the queue task information may include the processing latency of the node, the total SLO time window SLO time window, the number of user requests that violate the SLO SLO violation Request Counte, the number of user requests that have been processed Request Counter, the resource utilization of the current node, etc., and this application does not impose any restrictions on this.
[0115] In some embodiments, the load cost of each pod can be calculated using the following formula:
[0116]
[0117] in, It is the ratio of processing latency to the total SLO time window (SLO time window). The ratio of the number of user requests that violate the SLO (SLO violationRequest Counte) to the number of all processed user requests (Request Counter). Util cost is a score calculated based on the resource utilization of the current node. For example, the cluster administrator expects to maintain the node CPU resource utilization at X%, and can set the formula of the utility cost item around X%. This application does not limit this.
[0118] The second scheduling module 102, after receiving the pod creation request, determines the newly created pod based on the pod creation request, and then determines whether the newly created pod meets the node creation conditions (S406). If the newly created pod does not meet the node creation conditions, it is determined that the target user request cannot be executed, the target user request is discarded, and a request failure report is returned (S409). If the newly created pod meets the node creation conditions, the Utility model is used based on the cluster status information sent by the data warehouse 300 to determine a target node from the nodes that meet the node conditions among the multiple nodes 201 of the cluster 200, and then a new pod is created on the target node, and the information of the newly created pod is sent to the first scheduling module 101 so that the first scheduling module 101 can manage it based on the information of the constructed pod (S408). Among them, the second scheduling module 102 can apply to the data warehouse 300 for the cluster status information before selecting the target node using the cluster status information, so that the data warehouse 300 can return the current latest cluster status information to the second scheduling module 102 in a timely manner. In some embodiments, the cluster status information may include the resource usage of each node. Then the second scheduling module 102 can use the Utility model and the resource usage of each node to determine a target node on which a new pod can be created from multiple nodes, and then expand the number of pods on the target node to create a new pod.
[0119] In some embodiments, the node creation condition may be that the sum of the cold start duration of the pod, the duration required for the first scheduling module to manage the newly created pod, and the estimated duration for processing the target user request is less than the remaining duration from the current time to the violation of the SLO of the target user request. That is, the node creation condition can be expressed as: T(cold start)+T(acquirenew Pod)+T(input)<T(current deadline). Where T(cold start) is the cold start duration. T(acquire new Pod) is the duration required for the first scheduling module to manage the newly created pod. T(input) is the estimated duration for the newly created pod to process the target user request. The estimated duration can be determined according to the input size of the target user request. In this way, the node processing ability can be sensed, and the target node can be selected based on indicators such as the historical request processing time, the historical SLO violation rate, and the remaining amount of the current node's hardware resources, reducing the SLP violation ratio.
[0120] In some embodiments, the node condition may be that the P95 latency for processing the target user request at the node level is less than the remaining duration of the current distance violating the SLO of the target user request. That is, the node condition can be expressed as: T(input, P95) < T(current deadline). T(input, P95) is the P95 latency for processing the target user request at the node level. The P95 latency can represent the latency at the 95% position when the latencies of each node processing the target user request are sorted in ascending order.
[0121] In some embodiments, the data warehouse 300 is used to monitor the status of each pod, record the current task queue status of each surviving (usable) pod to form task queue information, monitor the resource usage of each node 201 where the pod is located, and send the task queue information of each pod to the first scheduling module 101 and send the resource usage of each node 201 to the second scheduling module 102 as cluster status information in response to an application from the second scheduling module 102.
[0122] In some embodiments, during the process of each pod in each node 201 executing the target user request, the node 201 can monitor and record the execution latency of the target user request, task queue status such as the current task queue latency of each pod within the node 201, the parameter size of the target user request, the resource usage of the node 201 and other node-related information, and then send the node-related information to the data warehouse 300 in real time.
[0123] In a possible implementation manner, the apparatus may further include an SLO determination module. The target determination module is used to determine the SLO indicated by the user according to the user operation. In some embodiments, the SLO determination module may display a parameter configuration interface as Figure 4 shown. The switch control M can be displayed on the parameter configuration interface, and the SLO determination module can determine the SLO indicated by the user based on the user's operation on the switch control M.
[0124] The following Figure 5 gives examples to schematically illustrate the working process and principle of the request scheduling system according to the embodiments of the present application. As Figure 5As shown, taking the beautification of mobile phone photos in the end-cloud collaboration scenario as an example, users need to upload pictures to the cloud. After receiving the user request to process the photos, the request scheduling system in the cloud needs to assign the target Pod of the processing function to each picture to complete a series of image processing operations. The photos need to be in the form of a pipeline, through multiple function Pods, and finally complete the beautification of the pictures. In this scenario, the processing delay of the photos is a very critical indicator to measure the service quality. Therefore, using the request scheduling system provided by the embodiment of the present application in this scenario can effectively reduce the number of user requests that violate the SLO and maximize the computing power of a cluster.
[0125] At the same time, in the input-sensitive application scenario of cloud photo beautification (photo size and photo processing speed show a certain correlation), it is also very necessary to estimate the processing delay of each user request, so this application uses a segmented sliding window strategy regression prediction to achieve the processing delay estimation of input-sensitive applications. For example, the photo input with the target user request is divided into five levels, and the input of each level is regressed and predicted in the form of a sliding window from the cluster, node, and Pod levels, as shown in Table 1:
[0126] Table 1. Estimated request processing delay using segmented sliding window strategy
[0127] Photo size range Current estimated latency (0K–500K] 50ms (500K,1M] 80ms (1M,5M] 200ms (5M,10M] 350ms (10M,-) 500ms
[0128] From the three levels of cluster, node, and Pod, the input of each level is subjected to regression prediction in the form of a sliding window. That is, when the first scheduling module 101 receives a user request, it determines whether the user request meets the receiving conditions from the perspective of the cluster. Then, after the user request reaches the head of the request queue and is determined as the target user request, the first scheduling module 101 will select the candidate pod with the lowest load and meeting the execution conditions as the target pod when selecting a pod for the target user request. Alternatively, after the user request reaches the head of the request queue and is determined as the target user request, the first scheduling module 101, because the existing pod does not meet the execution conditions and a new pod needs to be created, will select the target node and select the target node in the target node if the node creation conditions are met. Selecting the target node can consider the processing speed of the user request in the node dimension, and create the pod on a better node as much as possible, thereby accelerating the processing delay of subsequent target user requests.
[0129] The latency-aware Serverless request scheduling device and system provided in this application have the following advantages:
[0130] After receiving the user request, the first scheduling module can preliminarily estimate the queuing delay and processing time (that is, based on the receiving condition judgment), and perform fast failure processing for user requests that cannot be processed to avoid the request being blocked in the first scheduling module.
[0131] Using a segmented sliding window strategy to predict request processing latency can more accurately perceive the current Pod load. This data can be used to route user requests to the Pod with the smallest load, reducing the occurrence of user requests violating SLOs.
[0132] It can sense the Pod load situation and allocate user requests to the Pod with the lowest load by predicting the queuing delay of each Pod, thus avoiding SLO violations due to improper matching between user requests and Pods.
[0133] Perceive the node load and resource situation, and prioritize capacity expansion on nodes with strong processing capabilities (that is, select target nodes) to avoid violation of SLO caused by long request processing time due to differences in request processing capabilities between nodes.
[0134] In some embodiments, for function service products in the public cloud, such as Huawei Cloud FunctionGraph, AWSlambda and Azure Function, etc., in the public cloud scenario, some users' service requests are also latency-sensitive, such as finance, commodity trading and other user-oriented scenarios. Most Serverless products in the related technology still use concurrency as the decision indicator for elastic scaling, and users usually cannot set this indicator correctly. This means that if the indicator is set too small, each Pod cannot maintain a relatively saturated task queue. Although it can reduce processing latency, it will cause waste of resources; if the indicator is set too large, each Pod will be overloaded, and the tail task cannot be scheduled in time. Some requests will violate the SLO, affecting product revenue. The request scheduling device and system provided in the embodiments of the present application, such as Figure 6 As shown below the user request, SLO and request queuing delay as indicators of a Pod load can reduce the proportion of requests violating SLO while saving user costs compared to the above concurrency method.
[0135] The embodiment of the present application provides a serverless request scheduling method based on latency perception, which is applied to the serverless request scheduling device based on latency perception as described above. The method includes:
[0136] The user request at the head of the current request queue is determined as the target user request, and if it is determined that there is no target pod that satisfies the execution condition and executes the target user request, a pod creation request is generated;
[0137] When it is determined based on the pod creation request that the newly created pod meets the node creation condition, a target node is selected from multiple nodes, a new pod is created in the target node, and information of the newly created pod is determined;
[0138] The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
[0139] The beneficial effects of the latency-aware Serverless request scheduling method provided in the above-mentioned further aspect and various possible implementations of the further aspect are the same as those of the latency-aware Serverless request scheduling device provided in the above-mentioned first aspect and various possible implementations of the first aspect, and are not repeated here to avoid redundancy.
[0140] In a possible implementation, the execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time of the current distance violating the SLO of the target user request.
[0141] In a possible implementation, the node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
[0142] In a possible implementation, the method further includes: receiving a new user request, and when it is determined that the user request meets a receiving condition, adding the user request to a current request queue according to an SLO deadline of the user request.
[0143] In a possible implementation, the method further includes: in a case where it is determined that the user request does not satisfy a receiving condition, determining that the user request cannot be executed, and returning a request failure report.
[0144] In a possible implementation, the receiving condition is: the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining duration of the current distance violation of the user request.
[0145] In a possible implementation, the method further includes: when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determining the pod that satisfies the execution condition as the target pod, and sending the target user request to the target pod.
[0146] In a possible implementation, determining a pod that meets the execution condition as a target pod includes: selecting a pod with the lowest load from multiple pods currently managed as a candidate pod by using an SLO model and task queue information of each pod; and determining the candidate pod as the target pod if the candidate pod meets the execution condition.
[0147] In a possible implementation, the method further includes: when it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, determining that the target user request cannot be executed, and returning a request failure report.
[0148] In a possible implementation, a target node is selected from a plurality of nodes and a new pod is created in the target node, including: using a utility model and cluster status information to determine the target node from nodes in the cluster that meet node conditions; and creating a new pod on the target node; wherein the cluster status information includes resource usage of each of the nodes, and the node condition is that a P95 delay of processing the target user request at the node level is less than a remaining duration of a current distance violation of an SLO of the target user request.
[0149] The implementation methods and beneficial effects of each step of the latency-aware Serverless request scheduling method of the present application can refer to the relevant descriptions of the above-mentioned latency-aware Serverless request scheduling device and corresponding components in the system. To avoid redundancy, they will not be repeated here.
[0150] In some embodiments, the first scheduling module 101, the second scheduling module 102, and the data warehouse 300 and other modules can be implemented by software, or can be implemented by hardware. Exemplarily, the implementation of the first scheduling module 101 is described below by taking the first scheduling module 101 as an example. Similarly, the implementation of the second scheduling module 102 and the data warehouse 300 and other modules can refer to the implementation of the first scheduling module 101.
[0151] As an example of a software functional unit, the first scheduling module 101 may include code running on a computing instance. Among them, the computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the above-mentioned computing instance may be one or more. For example, the first scheduling module 101 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed in the same region (region) or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code can be distributed in the same availability zone (AZ) or in different AZs, each AZ including a data center or multiple data centers with similar geographical locations. Among them, usually a region can include multiple AZs.
[0152] Similarly, multiple hosts / virtual machines / containers used to run the code can be distributed in the same virtual private cloud (VPC) or in multiple VPCs. Usually, a VPC is set up in a region. For cross-region communication between two VPCs in the same region and between VPCs in different regions, a communication gateway needs to be set up in each VPC to achieve interconnection between VPCs through the communication gateway.
[0153] As an example of a hardware functional unit, the first scheduling module 101 may include at least one computing device, such as a server, etc. Alternatively, the first scheduling module 101 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.
[0154] The multiple computing devices included in the first scheduling module 101 can be distributed in the same region or in different regions. The multiple computing devices included in the first scheduling module 101 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the first scheduling module 101 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.
[0155] It should be noted that, in other embodiments, the first scheduling module 101 can be used to execute any step in the Serverless request scheduling method based on latency awareness in the embodiment of the present application, and the second scheduling module 102 and other modules can also be used to execute any step in the Serverless request scheduling method based on latency awareness. The steps that each module is responsible for implementing can be specified as needed. By implementing different steps in the Serverless request scheduling method based on latency awareness respectively, all functions of the Serverless request scheduling device and system based on latency awareness can be realized.
[0156] The present application also provides a computing device 500. Figure 7 As shown, computing device 500 includes: bus 502, processor 504, memory 506 and communication interface 508. Processor 504, memory 506 and communication interface 508 communicate through bus 502. Computing device 500 can be a server or a terminal device. It should be understood that the present application does not limit the number of processors and memories in computing device 100.
[0157] The bus 502 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus 504 is represented by only one line, but it does not mean that there is only one bus or one type of bus. The bus 504 may include a path for transmitting information between various components of the computing device 500 (eg, the memory 506, the processor 504, and the communication interface 508).
[0158] The processor 504 may include any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).
[0159] The memory 506 may include a volatile memory, such as a random access memory (RAM). The processor 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).
[0160] The memory 506 stores executable program codes, and the processor 504 executes the executable program codes to respectively implement the functions of the aforementioned first scheduling module, the second scheduling module, and the data warehouse, thereby realizing a serverless request scheduling method based on latency perception. That is, the memory 506 stores instructions for executing a serverless request scheduling method based on latency perception.
[0161] Alternatively, the memory 506 stores executable codes, and the processor 504 executes the executable codes to respectively implement the functions of the aforementioned delay-aware Serverless request scheduling device, thereby implementing the delay-aware Serverless request scheduling method. That is, the memory 506 stores instructions for executing the delay-aware Serverless request scheduling method.
[0162] The communication interface 503 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 500 and other devices or a communication network.
[0163] An embodiment of the present application provides a latency-aware Serverless request scheduling device, comprising: a processor and a memory for storing processor executable instructions; wherein the processor is configured to implement the above method when executing the instructions.
[0164] An embodiment of the present application provides a non-volatile computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented.
[0165] An embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.
[0166] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing.
[0167] The computer-readable program instructions or codes described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.
[0168] The computer program instructions for performing the operation of the present application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, by utilizing the state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA), the electronic circuit can execute the computer-readable program instructions to implement various aspects of the present application.
[0169] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.
[0170] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0171] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.
[0172] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the device, system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some alternative implementations, the functions marked in the square frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can also be executed in reverse order sometimes, depending on the functions involved.
[0173] It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by hardware (such as a circuit or ASIC (Application Specific Integrated Circuit)) that performs the corresponding function or action, or can be implemented by a combination of hardware and software, such as firmware.
[0174] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.
[0175] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.
Claims
1. A serverless request scheduling device based on latency awareness, It is characterized in that The device comprises: a first scheduling module and a second scheduling module, The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request; The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module; The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
2. The device according to claim 1, It is characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
3. The device according to claim 1, It is characterized in that The first scheduling module is further configured to receive a new user request, and if it is determined that the user request meets the receiving condition, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving condition, determine that the user request cannot be executed and return a request failure report. The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
4. The device according to claim 2, It is characterized in that The first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod.
5. The device according to claim 4, It is characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
6. The device according to claim 1, It is characterized in that The second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined that the newly created pod cannot meet the node creation condition based on the pod creation request.
7. The device according to claim 1, It is characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
8. A serverless request scheduling system based on latency awareness, It is characterized in that The invention comprises a serverless request scheduling device based on latency perception and a serverless computing resource cluster, wherein the device comprises a first scheduling module and a second scheduling module, and the serverless computing resource cluster comprises a plurality of nodes, each node comprises at least one pod, The first scheduling module is used to determine the user request at the head of the current request queue as the target user request, and send a pod creation request to the second scheduling module when it is determined that there is no target pod that satisfies the execution condition and executes the target user request; The second scheduling module is configured to, upon receiving the pod creation request, select a target node from a plurality of nodes and create a new pod in the target node if it is determined based on the pod creation request that the new pod meets the node creation condition, and send information of the new pod to the first scheduling module; The first scheduling module is further used to manage the newly created pod according to the received information of the newly created pod, determine the newly created pod as a target pod, and send the target user request to the target pod so that the target pod executes the received target user request.
9. The system according to claim 8, It is characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
10. The system according to claim 8, It is characterized in that The first scheduling module is further configured to receive a new user request, and if it is determined that the user request meets the receiving condition, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving condition, determine that the user request cannot be executed and return a request failure report; The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
11. The system according to claim 9, It is characterized in that The first scheduling module is further configured to, when it is determined that there is a target pod that satisfies the execution condition and executes the target user request, determine the pod that satisfies the execution condition as the target pod, and send the target user request to the target pod.
12. The system according to claim 11, It is characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod; Wherein, the system further includes a data warehouse, and the task queue information is sent by the data warehouse.
13. The system according to claim 8, It is characterized in that The second scheduling module is further configured to determine that the target user request cannot be executed and return a request failure report when it is determined that the newly created pod cannot meet the node creation condition based on the pod creation request.
14. The system according to claim 8, It is characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; Among them, the cluster status information includes the resource usage of each of the nodes, and the system also includes a data warehouse. The cluster status information is the node condition sent by the data warehouse, which is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
15. A serverless request scheduling method based on latency awareness, It is characterized in that The method comprises: The user request at the head of the current request queue is determined as the target user request, and if it is determined that there is no target pod that satisfies the execution condition and executes the target user request, a pod creation request is generated; When it is determined based on the pod creation request that the newly created pod meets the node creation condition, a target node is selected from multiple nodes, a new pod is created in the target node, and information of the newly created pod is determined; The newly created pod is managed according to the information of the newly created pod, the newly created pod is determined as a target pod, and the target user request is sent to the target pod, so that the target pod executes the received target user request.
16. The method according to claim 15, It is characterized in that The execution condition is: the sum of the estimated time for processing the target user request, the total network communication time, and the total time for the pod to process all user requests in the current task queue is less than the remaining time for the current distance to violate the SLO of the target user request; and / or The node creation condition is: the sum of the cold start time of the pod, the time required for the first scheduling module to manage the newly created pod, and the estimated time for processing the target user request is less than the remaining time of the current distance from violating the SLO of the target user request.
17. The method according to claim 15, It is characterized in that The method further comprises: Receive a new user request, and if it is determined that the user request meets the receiving conditions, add the user request to the current request queue according to the SLO deadline of the user request; or if it is determined that the user request does not meet the receiving conditions, determine that the user request cannot be executed and return a request failure report, The receiving condition is that the sum of the average delay of the cluster processing the user request, the queuing delay of the user request in the current request queue, and the network transmission delay within the cluster is less than the remaining time of the current distance violation of the user request.
18. The method according to claim 16, It is characterized in that The method further comprises: In the case where it is determined that there is a target pod that satisfies the execution condition and executes the target user request, the pod that satisfies the execution condition is determined as the target pod, and the target user request is sent to the target pod.
19. The method according to claim 18, It is characterized in that The pods that meet the execution conditions are determined as target pods, including: Using the SLO model and the task queue information of each pod, the pod with the lowest load is selected from the multiple pods currently managed as the candidate pod; When the candidate pod meets the execution condition, the candidate pod is determined as the target pod.
20. The method according to claim 15, It is characterized in that The method further comprises: In the case where it is determined based on the pod creation request that the newly created pod cannot meet the node creation condition, it is determined that the target user request cannot be executed, and a request failure report is returned.
21. The method according to claim 14, It is characterized in that Selecting a target node from multiple nodes and creating a pod in the target node includes: Using the utility model and cluster status information, determine the target node from the nodes that meet the node conditions in the multiple nodes of the cluster; Create a new pod on the target node; The cluster status information includes resource usage of each of the nodes, and the node condition is that the P95 delay of processing the target user request at the node level is less than the remaining time of the current distance violating the SLO of the target user request.
22. A serverless request scheduling device based on latency perception, It is characterized in that include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 15-21 when executing the instructions.
23. A non-volatile computer readable storage medium having computer program instructions stored thereon, It is characterized in that When the computer program instructions are executed by a processor, the method according to any one of claims 15 to 21 is implemented.
24. A computer program product comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, It is characterized in that When the computer readable code is executed in an electronic device, a processor in the electronic device executes the method according to any one of claims 15 to 21.
Citation Information
Cited By
GPU hot plug method based on Kubernetes architecture and readable storage medium
CN119718993A
Delay-sensing-based serverless request scheduling apparatus and system
EP4708048A1
Delay-sensing-based serverless request scheduling apparatus and system
WO2024240003A1
