Serverless function scheduling method based on deep reinforcement learning

Abstract

The invention relates to the field of Serverless function scheduling, and particularly provides a Serverless function scheduling method based on deep reinforcement learning. According to the method, a Serverless operation environment is simulated, a near-end strategy optimization (PPO) reinforcement learning framework is adopted, and a PPO algorithm utilizes a strategy constraint mechanism to maintain the model updating stability and improve the training data utilization rate. In a model training stage, an intelligent agent generates a scheduling action according to local observation information, inputs a global system state into a Critic network for joint value evaluation, and outputs a reward signal with weight correction. During actual deployment, the agent autonomously executes instance scheduling based on the trained strategy network, and rapidly responds to a scheduling request. According to the method, a reward function is set by taking minimization of response time and energy consumption as optimization objectives, and intelligent function scheduling of Serverless under dynamic resources and workloads is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of Serverless function scheduling, and specifically proposes a Serverless function scheduling method based on deep reinforcement learning. Background Art

[0002] With the rapid development of information technology, cloud computing has become an important pillar of modern IT infrastructure. Through on-demand allocation and highly scalable resource management methods, cloud computing significantly reduces infrastructure investment and operation and maintenance costs, enabling users to dynamically adjust resource allocation according to actual needs, optimize costs and improve efficiency. However, traditional cloud computing architectures still require users to manage underlying servers and operating systems, including tasks such as resource configuration, load balancing, and elastic scaling. In high-concurrency or business load fluctuation scenarios, this model increases operation and maintenance complexity and limits development efficiency. To simplify resource management and improve flexibility, serverless computing (Serverless) has emerged. Serverless delegates the responsibility of underlying resource scheduling, scaling, and maintenance to cloud service providers, and developers only need to focus on implementing business logic, greatly improving resource management efficiency and development convenience.

[0003] Serverless represents the latest stage in the abstraction of cloud computing resource management, marking the evolution from infrastructure as a service (IaaS) to platform as a service (PaaS), and then to software as a service (SaaS). At the IaaS level, users need to manage infrastructure resources such as virtual machines and storage; PaaS provides a platform on which users can develop, run, and manage applications without managing the underlying hardware; SaaS further abstracts, and users can directly use software services without caring about the underlying infrastructure or platform. Serverless further simplifies the cloud computing model through function as a service (FaaS) and backend as a service (BaaS), and users only need to upload code, and the cloud platform is responsible for running, scaling, and downsizing.

[0004] According to research predictions, the global Serverless market size will reach $14 billion by 2024, with a compound annual growth rate of 23.4%. With the popularization and application of Serverless technology, its role in accelerating enterprise digital transformation and optimizing IT efficiency has become increasingly important. Currently, major technology companies have launched iconic Serverless platforms, such as AWS Lambda, Google Cloud Functions, Azure Functions, and Alibaba Cloud Function Compute. These platforms fully meet the modern application requirements of high concurrency, low latency, and rapid development through automated resource scheduling and flexible task execution. For example, AWS Lambda provides a function-centric programming model and excels in tasks such as real-time monitoring and data analysis in the Internet of Things; Google Cloud Functions relies on a powerful infrastructure to support streaming data processing and background automation; Alibaba Cloud Function Compute is deeply integrated into the local ecosystem and efficiently handles massive traffic in e-commerce scenarios. During the Double Eleven promotion, for example, it optimizes resource utilization through dynamic resource scaling.

[0005] As an emerging research field, Serverless function scheduling focuses on how to effectively allocate and manage functions in a Serverless architecture. In Serverless, cloud service providers are responsible for maintaining server infrastructure, while functions run in stateless computing containers triggered by events, and the entire process is fully controlled by cloud service providers. The core of Serverless function scheduling lies in determining the execution timing and location of functions, and while ensuring security and privacy, reasonably allocating computing resources to achieve performance improvement, cost control, and shortening of task completion time. Therefore, researching how to design efficient Serverless function scheduling algorithms, combined with the resource characteristics and constraints of Serverless, can further optimize resource utilization, improve performance, and reduce operating costs, which is of great significance for modern computing applications.

[0006] Deep Reinforcement Learning (DRL) is a cutting-edge branch of artificial intelligence that combines the powerful data processing capabilities of deep learning with the decision-making capabilities of reinforcement learning. Deep learning captures deep features in data by simulating the multi-layer neural network of the human brain, while reinforcement learning enables an agent to learn how to obtain the maximum cumulative reward through actions by interacting with the environment. Through this combination, DRL enables the agent to process complex data inputs, understand the sequential nature of the data, and continuously learn and improve from interactions with the environment. In DRL, the goal of the agent is to maximize the total reward it obtains in the long term, which is usually achieved by evaluating the expected value in different states and the value of specific actions. The decision-making strategy of the agent is usually represented by a deep neural network, enabling it to make precise decisions in complex and changing environments. In short, DRL is a technology that enables computers to learn autonomously and optimize decisions, showing great potential in dealing with complex tasks.

[0007] Applying DRL to Serverless function scheduling brings innovative solutions to the field of cloud computing. The Serverless architecture, with its flexibility of automatic scaling, pay-per-use, and no need to manage servers, has changed the traditional application development model. In such an environment, the DRL agent learns and optimizes its decision-making strategy by continuously interacting with the environment, which perfectly matches the event-driven characteristics of Serverless. The stateless nature, horizontal scalability, and high availability of Serverless functions enable the DRL agent to quickly respond to changes in the workload, intelligently schedule resources, effectively reduce costs, and accelerate business iteration and innovation. This combination not only improves the system's response speed and processing capacity but also brings revolutionary progress to resource management and cost control in cloud computing. Summary of the Invention

[0008] To achieve the above technical objectives, a Serverless function scheduling method based on deep reinforcement learning includes:

[0009] S1: Based on the Serverless computing framework, establish a model of the function scheduling problem, considering the total response time and energy consumption of function scheduling;

[0010] S2: Deploy an agent on the master node to execute the scheduling strategy of the function to be scheduled. The agent dynamically learns how to optimize the function execution time and cluster energy consumption by interacting with the environment;

[0011] S3: Construct a multi-objective reward function to adjust the balance between the response time and energy consumption metrics to achieve a globally optimal effect;

[0012] S4: Combine with the PPO algorithm to train the agent under the Critic network. The agent makes actions based on the Actor network and the current state, and updates the policy according to the environmental feedback. In actual scheduling, use the trained model for real-time decision-making;

[0013] S5: Deploy the trained model to the Serverless custom scheduler, conduct experiments in a real cluster, and evaluate the performance of the proposed method.

[0014] Furthermore, the Serverless function scheduling system based on deep reinforcement learning adopts a "monitoring-training-scheduling" closed-loop architecture: The monitoring module collects metrics such as node resources, function queues, and response latency in real time through integration with Prometheus, and encodes them into a standardized state tensor; The training module builds a dynamic simulation environment relying on OpenAI Gym to simulate heterogeneous clusters and function workflows, and the PPO agent learns scheduling strategies through a composite reward mechanism; The scheduling module generates a node selection probability distribution based on the policy network, combines with Kubernetes to achieve elastic scaling and progressive deployment, and at the same time triggers a policy rollback mechanism through Prometheus SLI monitoring. The system realizes online incremental training through the continuous iteration of "data collection → policy decision → execution feedback" to ensure the dynamic adaptability of the scheduling strategy.

[0015] Furthermore, the available resources of a node are calculated as follows:

[0016]

[0017] where N ,

[0017] , ,

[0018] , kji , , i , , , i ,

[0020] , , i ,

[0019] , i ,

[0021] , , i , , , i , , , , (t) represents the total available resources of node N at time t i of, represents the available CPU resources of node N at time t i of, represents the available memory resources of node N at time t i of. and are calculated as follows:

[0018]

[0019] where, and are the total available CPU and memory of node N respectively, and u i (t) represents whether pod kji is scheduled to node N at time t, and its calculation is as follows: i at time t, and its calculation is as follows:

[0020]

[0021] Furthermore, the resource constraints are as follows:

[0022]

[0023] Among them, is the minimum CPU requirement for the pod, is the minimum memory requirement for the pod.

[0024] Furthermore, the concurrent request count of the pod is calculated as follows:

[0025]

[0026] Among them, k r (t) is the total number of requests arriving at the function within time t, and k n (t) is the number of function instances deployed within time t.

[0027] Furthermore, the total response time of the function is calculated as follows:

[0028]

[0029] Among them, w b is the priority weight of application b, is the total response time of the q-th request of application b, and L b is the number of requests received by application b.

[0030] Furthermore, the total energy consumption of the cluster is calculated as follows:

[0031]

[0032] Among them, S is the number of nodes in the cluster, and P i (t) is the power consumption of node i at time t, which is calculated as follows:

[0033] P i (t) = P idle +(P max -P idle )·U i,t

[0034] Among them, P idle is the power consumption of node i when idle, P max is the power consumption of node i when fully loaded, and U i,t is the CPU utilization rate of node i at time t.

[0035] Furthermore, the reward function of the PPO algorithm is calculated as follows:

[0036] Reward = λ·ΔWTRT+(1 - λ)·ΔE total

[0037] Among them, λ is the weight of the response time ratio, reflecting the degree of emphasis of the model on the response time.

[0038] Furthermore, the objective function of the PPO algorithm is calculated as follows:

[0039]

[0040] Among them, θ is the policy network parameter, ∈ is the hyperparameter that controls the policy update amplitude, r t (θ) and A t are the action probability ratios of the old and new policies and the advantage function respectively, and the calculations are as follows:

[0041]

[0042] Among them, when r t > 1 indicates that the new policy is more inclined to select this action, and r t < 1 indicates that the new policy reduces the selection probability of this action.

[0043] A t = Q(s t , a t ) - V(s t )

[0044] Among them, the advantage function A t is used to quantify the advantages and disadvantages of a specific scheduling action (such as deploying a function instance to a high-load node) compared to the average policy. Q(s t , a t ) represents the state-action value function (predicting the long-term return of executing a t in state s t ), and V(s t ) represents the state value function (predicting the average return level of state s t ).

[0045] The present invention has at least the following beneficial effects

[0046] The present invention considers the function scheduling scenario under multi-tenant and multi-application, and designs a reward function with the total response time of function execution and the total energy consumption consumed by the cluster as the optimization objectives; when performing function scheduling, the model will perform function scheduling in the direction of obtaining the maximum reward; at the same time, the trained model is deployed to the cluster for real-time function scheduling to adapt to the dynamic changes of the function flow in the Serverless environment, optimize resource allocation, reduce execution latency and energy consumption, and improve resource utilization efficiency. Description of the Drawings

[0047] Figure 1 It is a system structure diagram under the Serverless computing architecture;

[0048] Figure 2 It is the algorithm flowchart of the deep reinforcement learning model of the present invention; Specific implementation manners

[0049] Next, the technical solutions of the present invention will be clearly and comprehensively described in conjunction with the accompanying drawings in the embodiments of the present invention. It should be noted that the described embodiments only represent a part of the implementation manners of the present invention, not all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without making creative efforts fall within the scope of protection of the present invention.

[0050] A Serverless function scheduling method based on deep reinforcement learning includes:

[0051] S1: Based on the Serverless computing framework, establish a model for the function scheduling problem, considering the total response time and energy consumption of function scheduling;

[0052] S2: Deploy an agent on the master node to execute the scheduling policy of the function to be scheduled. The agent dynamically learns how to optimize the function execution time and cluster energy consumption by interacting with the environment;

[0053] S3: Construct a multi-objective reward function to adjust the balance between the response time and energy consumption metrics to achieve the global optimal effect;

[0054] S4: Combine the PPO algorithm to train the agent under the Critic network. The agent makes actions based on the Actor network and the current state, and updates the policy according to the environmental feedback. In actual scheduling, use the trained model for real-time decision-making;

[0055] S5: Deploy the trained model to the Serverless custom scheduler, conduct experiments in a real cluster, and evaluate the performance of the proposed method.

[0056] Such as Figure 1As shown in the figure, the Serverless function scheduling system based on deep reinforcement learning adopts a "monitoring - training - scheduling" closed - loop architecture: The monitoring module collects metrics such as node resources, function queues, and response latency in real - time by integrating Prometheus, and encodes them into a standardized state tensor; The training module builds a dynamic simulation environment relying on OpenAI Gym to simulate heterogeneous clusters and function workflows. The PPO agent learns scheduling strategies through a composite reward mechanism; The scheduling module generates a probability distribution of node selection based on the policy network, and combines with Kubernetes to achieve elastic scaling and progressive deployment. At the same time, it monitors and triggers the policy rollback mechanism through Prometheus SLI. The system realizes online incremental training through continuous iteration of "data collection → policy decision → execution feedback" to ensure the dynamic adaptability of the scheduling strategy.

[0057] The process of Serverless function scheduling based on deep reinforcement learning in this embodiment is as Figure 2 shown, including the following steps:

[0058] The Serverless function scheduling training process based on the PPO algorithm starts with environment and network initialization: First, configure the Serverless cluster and initialize the policy network (Actor) and value function network (Critic); Then enter the policy iteration loop. The Agent selects scheduling actions according to the current policy, calculates the advantage function to evaluate the action value through the GAE method, uses the Clip or KL - divergence algorithm to constrain the policy update amplitude to stabilize training, and at the same time optimizes the Critic network to improve the state value estimation; If the training reaches the convergence condition (stable reward or iteration limit), then save the model and deploy it to the cluster for dynamic scheduling; When the conditions are not met, continuously monitor real - time state features such as cluster resources and function queues, and use them as the input of the new round of policy for further optimization; Finally, form a closed - loop of "training - deployment - monitoring - retraining", and achieve the adaptive evolution of the scheduling strategy through online incremental learning.

[0059] The calculation of the reward function includes the following steps:

[0060] The available resources of the node are calculated as follows:

[0061]

[0062] where N i (t) represents the total available resources of node N at time t i , represents the available CPU resources of node N at time t i , represents the available memory resources of node N at time t i . and The calculation is as follows:

[0063]

[0064] in, and Node N i The total available CPU and memory, u kji (t) indicates pod Whether to schedule to node N i The calculation is as follows:

[0065]

[0066] Resource restriction requirements are as follows:

[0067]

[0068] in, is the minimum CPU requirement required by the pod, is the minimum memory requirement required by the pod.

[0069] The number of concurrent requests for a pod is calculated as follows:

[0070]

[0071] Among them, k r (t) is the total number of requests that arrive at the function within time t, k n (t) is the function instance deployed at time t.

[0072] The total response time of a function is calculated as follows:

[0073]

[0074] Among them, w b is the priority weight of application b, is the total response time of the qth request of application b, L b is the number of requests received by application b.

[0075] The total energy consumption of the cluster is calculated as follows:

[0076]

[0077] Where S is the number of nodes in the cluster, P i (t) is the power consumption of node i at time t, which is calculated as follows:

[0078] P i (t) = P idle +(P max -P idle )·Ui,t

[0079] Among them, P idle is the power consumption when node i is idle, and P max is the power consumption when node i is fully loaded, and U i,t is the CPU utilization rate of node i at time t.

[0080] The reward function of the PPO algorithm is calculated as follows:

[0081] Reward = λ·ΔWTRT+(1 - λ)·ΔE total

[0082] Among them, λ is the weight of the response time ratio, reflecting the degree of emphasis of the model on the response time.

[0083] The objective function of the PPO algorithm is calculated as follows:

[0084]

[0085] Among them, θ is the parameter of the policy network, ∈ is the hyperparameter that controls the update amplitude of the policy, and r t (θ) and A t are the action probability ratios of the new and old policies and the advantage function respectively, and the calculations are as follows:

[0086]

[0087] Among them, when r t > 1, it means that the new policy is more inclined to select this action, and when r t < 1, it means that the new policy reduces the selection probability of this action.

[0088] A t = Q(s t , a t ) - V(s t )

[0089] Among them, the advantage function A t is used to quantify the advantages and disadvantages of a specific scheduling action (such as deploying a function instance to a high-load node) compared to the average policy. Q(s t , a t ) represents the state-action value function (predicting the long-term return of executing a t in state s t ), and V(s t ) represents the state value function (predicting the average return level of state s t ).

[0090] Through a deep reinforcement learning model, the present invention automatically generates an efficient task allocation scheme according to function characteristics (such as required computing resources and startup time), the real-time status of the server (CPU and memory load), and the dynamic operating environment (network latency and task backlog). The system is set with multiple objective optimizations to achieve a dynamic balance between response time and energy consumption, so as to implement an optimal scheduling strategy.

[0091] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A Serverless function scheduling method based on deep reinforcement learning, characterized in that, Including: S1: Based on the Serverless computing framework, establish a model for the function scheduling problem, considering the total response time and energy consumption of function scheduling; S2: Deploy an agent on the master node to execute the scheduling strategy of the function to be scheduled. The agent dynamically learns how to optimize the function execution time and cluster energy consumption by interacting with the environment; S3: Construct a multi-objective reward function to adjust the balance between the response time and energy consumption metrics to achieve the global optimal effect; S4: Combine the PPO algorithm to train the agent under the Critic network. The agent makes actions based on the Actor network and the current state, and updates the policy according to the environmental feedback. In actual scheduling, use the trained model for real-time decision-making; S5: Deploy the trained model to the Serverless custom scheduler, conduct experiments in a real cluster, and evaluate the performance of the proposed method.

2. A Serverless function scheduling system based on deep reinforcement learning according to claim 1 adopts a "monitoring-training-scheduling" closed-loop architecture: The monitoring module collects metrics such as node resources, function queues, and response latency in real time through integration with Prometheus and encodes them into a standardized state tensor; The training module relies on OpenAI Gym to build a dynamic simulation environment, simulating heterogeneous clusters and function workflows. The PPO agent learns the scheduling strategy through a composite reward mechanism; The scheduling module generates a node selection probability distribution based on the policy network, combines with Kubernetes to achieve elastic scaling and progressive deployment, and at the same time triggers a policy rollback mechanism through Prometheus SLI monitoring. The system realizes online incremental training through the continuous iteration of "data collection → policy decision → execution feedback" to ensure the dynamic adaptability of the scheduling strategy.

3. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that The available resources of the node are calculated as follows: Among them, N i (t) represents the total available resources of node N at time t i . represents the CPU available resources of node N at time t i . represents the memory available resources of node N at time t i . and are calculated as follows: Among them, and are the total available CPU and memory of node N i respectively, and u kji (t) indicates whether the pod is scheduled to node N i as follows:

4. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The resource limit requirements are as follows: Among them, is the minimum CPU requirement for the pod, is the minimum memory requirement for the pod.

5. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that, The requested concurrency of the pod is calculated as follows: where k r (t) is the total number of requests that reach the function within time t, and k n (t) is the function instances deployed within time t.

6. The Serverless function scheduling method based on deep reinforcement learning according to claim 1, wherein, The total response time of the function is calculated as follows: where, w b is the priority weight of application b, is the total response time of the q-th request of application b, and L b is the number of requests received by application b.

7. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that The total energy consumption of the cluster is calculated as follows: where S is the number of nodes in the cluster, and P i (t) is the power consumption of node i at time t, which is calculated as follows: P i P(t) = P idle + (P max - P idle ) · U i,t Among them, P idle is the power consumption when node i is idle, and P max is the power consumption when node i is fully loaded. U i,t is the CPU utilization rate of node i at time t.

8. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that The reward function of the PPO algorithm is calculated as follows: Reward=λ·ΔWTRT+(1-λ)·ΔE total Among them, λ is the weight of the response time ratio, reflecting the degree of importance of the model to the response time.

9. A Serverless function scheduling method based on deep reinforcement learning according to claim 1, characterized in that The objective function of the PPO algorithm is calculated as follows: where θ are the policy network parameters, ∈ is the hyperparameter that controls the magnitude of policy update, r t (θ) and A t are the action probability ratio and the advantage function of the old and new policies respectively, and the calculations are as follows: Among them, when r t > 1 indicates that the new policy is more inclined to select this action, and r t < 1 indicates that the new policy reduces the selection probability of this action. A t = Q(s t , a t ) - V(s t ) Among them, the advantage function A t is used to quantify the advantages and disadvantages of a specific scheduling action (such as deploying a function instance to a high-load node) compared to the average policy. Q(s t , a t ) represents the state-action value function (predicting the long-term return of executing a t in state s t ), and V(s t ) represents the state value function (predicting the average return level of state s t ).

Images