Cloud cluster resource scheduling method based on deep reinforcement learning

Abstract

A cloud cluster resource scheduling method based on deep reinforcement learning comprises the steps of constructing a job resource demand vector, constructing a reward function, constructing a resource scheduling agent network and training the resource scheduling agent network. The present application aims at the problems of poor real-time performance, low adaptability and low scheduling performance of the existing resource scheduling method of the cloud computing cluster, proposes a method with the target of minimizing the average weighted turnaround time of the job, inputs the system state s of the cluster into the resource scheduling agent network by representing the system state s as an image to generate a scheduling decision, models the resource scheduling problem of the cloud cluster as a deep reinforcement learning problem, and uses the synchronous actor-critic deep reinforcement learning method to make the training process of the agent more stable. The trained agent has the advantages of strong real-time performance and high adaptability and can be used for cloud cluster resource scheduling.

Description

Technical Field

[0001] This invention belongs to the technical field of cloud computing resource scheduling, specifically involving a cloud cluster resource scheduling method based on deep reinforcement learning. Background Technology

[0002] In the digital age, cloud computing has become an indispensable part of daily life for businesses and individuals. With the development of internet and artificial intelligence technologies, the demand for cloud computing resources is growing rapidly. This necessitates cloud computing vendors and large internet companies maintaining extremely large-scale physical server clusters to meet task demands. How to efficiently schedule tasks and rationally manage server cluster resources has become a pressing technical problem. Optimizing resource scheduling can further improve the performance and efficiency of server clusters, while also providing users with more stable and reliable cloud computing services.

[0003] Traditional cluster resource scheduling is generally implemented using heuristic methods. Although this method has achieved certain results in specific situations, cluster resource scheduling implemented using heuristic methods still has the following drawbacks:

[0004] Because resource requirements in the cluster often change, including different types of tasks and job loads, dynamic allocation of resources in the cluster is required, which increases the difficulty of scheduling and makes the method still susceptible to limitations of low real-time performance and low adaptability.

[0005] When the cluster structure changes, the coding that relies on expert knowledge heuristics needs to be redesigned, wasting a lot of the designer's time.

[0006] Heuristic methods are difficult to converge to the optimal solution and are prone to getting stuck in local optima, resulting in low resource scheduling performance, low cluster resource utilization during scheduling, and long average waiting time for user jobs. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a highly efficient cloud cluster resource scheduling method based on deep reinforcement learning.

[0008] The technical solution adopted to solve the above technical problems consists of the following steps:

[0009] (1) Constructing the task resource demand vector

[0010] The task resource demand vector r is determined by the following formula. j

[0011] r j =(r j,1 ,r j,2 ,…,r j,d )

[0012] Where, r j,d Let d represent the amount of resource type d required by task j, where j and d are finite positive integers.

[0013] (2) Constructing the reward function

[0014] Determine the weighted turnover time W according to formula (1):

[0015] W = C / T (1)

[0016] Where C is the turnaround time of the task, and T represents the duration of the task.

[0017] Determine the average weighted turnover time according to formula (2).

[0018]

[0019] Among them, W i Let N represent the weighted turnaround time of job i, and N be the number of jobs, where N is a finite positive integer.

[0020] The reward function r for each time step t is determined according to equation (3). t :

[0021]

[0022] Among them, T j This indicates the required service time for task j;

[0023] (3) Constructing a resource scheduling intelligent agent network

[0024] A resource scheduling agent network is constructed using a synchronous actor-critic deep reinforcement learning method. This network includes an actor network for performing scheduling actions and a critic network for estimating state values. The actor network consists of input layer 1, hidden layer 1, and output layer 1 connected in series. The critic network consists of input layer 2, hidden layer 2, and output layer 2 connected in series.

[0025] (4) Training resource scheduling agent network

[0026] 1) Construct the gradient of the objective function

[0027] Construct the gradient of the objective function according to equation (5).

[0028]

[0029]

[0030] Where, π θ (at |s t ) represents the neural network π of the resource scheduling agent. θ At time t, the system state s t Input and output action a t The probability, γ k R(s) represents the discount factor. t+k ,a t+k ) represents the system state s at time t+k. t+k Next, execute action a t+k The reward obtained, V(s) t ) represents the commentator network's opinion on the system state s at time t. t Value estimate.

[0031] 2) Training resource scheduling agent network

[0032] Using the system state s during the scheduling process, the resource scheduling agent outputs action a, A(s,a), to train the resource scheduling agent network.

[0033] Different job sequences are used in different cloud cluster simulation environments. By using 20 resource scheduling agents to interact with the simulation environment at the same time, the gradient of the objective function of each agent is determined according to Equation (5) and summarized into the global gradient.

[0034] Using the root mean square propagation method as the optimizer, the parameters of the neural network are updated according to equation (6):

[0035] g=α×g+(1-α)×Δθ 2 (6)

[0036]

[0037] Where α is the decay rate, with a value of 0.99, g is the exponentially weighted average, η is the learning rate, with a value of 0.001, θ is the parameter of the neural network, and Δθ is the gradient of the parameter. After 10,000 iterations, the cloud cluster resource scheduling agent is obtained.

[0038] In step (3) of the present invention, the number of neurons in the resource scheduling intelligent agent network is 4480, the number of neurons in the hidden layer 1 is 20, and the number of neurons in the output layer 1 is 11; the number of neurons in the input layer 2 is 4480, the number of neurons in the hidden layer 2 is 20, and the number of neurons in the output layer 2 is 1.

[0039] In step (4) of the present invention, step 2) of training the resource scheduling agent network, the system state s is represented by an image, which contains three parts:

[0040] ① Cluster image: Represents the jobs that have been assigned in the cluster and their running status.

[0041] ② Job Image: Represents the resource requirements of jobs in the waiting queue.

[0042] ② The backlog image represents the number of jobs waiting to be scheduled that exceed the capacity of the job slot.

[0043] In step (3) of the present invention, the construction of the resource scheduling agent network, the actor network takes the system state s as input and outputs the probability of executing each scheduling action, and the critic network takes the system state s as input and outputs the value estimate V(s) of the current state.

[0044] This invention addresses the problems of poor real-time performance, low adaptability, and low scheduling performance in existing resource scheduling methods for cloud computing clusters. It proposes a method that aims to minimize the average weighted turnaround time of jobs. By representing the system state s of the cluster as an image and inputting it into a resource scheduling agent network to generate scheduling decisions, the resource scheduling problem of cloud clusters is modeled as a deep reinforcement learning problem. By using the synchronous actor-critic deep reinforcement learning method, the training process of the agent is made more stable, and the trained agent has the advantages of strong real-time performance and high adaptability. Attached Figure Description

[0045] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0046] Figure 2 This is the cumulative reward curve obtained during the training process in Example 1.

[0047] Figure 3 This is a comparison curve of Example 1 and the comparative method. Detailed Implementation

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0049] Example 1

[0050] The cloud cluster resource scheduling method based on deep reinforcement learning in this embodiment consists of the following steps (see...). Figure 1 ):

[0051] (1) Constructing the task resource demand vector

[0052] The task resource demand vector r is determined by the following formula. j :

[0053] r j =(r j,1 ,r j,2 ,…,r j,d)

[0054] Where, r j,d Let d represent the amount of resource type d required by task j, where j and d are finite positive integers.

[0055] (2) Constructing the reward function

[0056] Determine the weighted turnover time W according to formula (1):

[0057] W = C / T (1)

[0058] Where C is the turnaround time of the task, and T represents the duration of the task.

[0059] Determine the average weighted turnover time according to formula (2).

[0060]

[0061] Among them, W i Let N represent the weighted turnaround time of job i, and N be the number of jobs, where N is a finite positive integer.

[0062] The reward function r for each time step t is determined according to equation (3). t :

[0063]

[0064] Among them, T j This indicates the required service time for task j.

[0065] (3) Constructing a resource scheduling intelligent agent network

[0066] A resource scheduling agent network is constructed using a synchronous actor-critic deep reinforcement learning method. This network comprises an actor network for executing scheduling actions and a critic network for estimating state values. The actor network takes the system state `s` as input and outputs the probability of executing each scheduling action. The critic network takes the system state `s` as input and outputs a value estimate `V(s)` for the current state. In this embodiment, the actor network consists of input layer 1, hidden layer 1, and output layer 1 connected in series. Input layer 1 has 4480 neurons, hidden layer 1 has 20 neurons, and output layer 1 has 11 neurons. Similarly, the critic network consists of input layer 2, hidden layer 2, and output layer 2 connected in series. Input layer 2 has 4480 neurons, hidden layer 2 has 20 neurons, and output layer 2 has 1 neuron.

[0067] (4) Training resource scheduling agent network

[0068] 1) Construct the gradient of the objective function

[0069] Construct the gradient of the objective function according to equation (5).

[0070]

[0071]

[0072] Where, π θ (a t |s t ) represents the neural network π of the resource scheduling agent. θ At time t, the system state s t Input and output action a t The probability, γ k R(s) represents the discount factor. t+k ,a t+k ) represents the system state s at time t+k. t+k Next, execute action a t+k The reward obtained, V(s) t ) represents the commentator network's opinion on the system state s at time t. t Value estimate.

[0073] 2) Training resource scheduling agent network

[0074] Using the system state s during the scheduling process, the resource scheduling agent outputs action a, A(s,a), to train the resource scheduling agent network.

[0075] In this embodiment, the system state s is represented by an image, which consists of three parts:

[0076] ① Cluster image: Represents the jobs that have been assigned in the cluster and their running status.

[0077] ② Job Image: Represents the resource requirements of jobs in the waiting queue.

[0078] ③ The backlog image represents the number of jobs waiting to be scheduled that exceed the capacity of the job slot.

[0079] Different job sequences are used in different cloud cluster simulation environments. By using 20 resource scheduling agents to interact with the simulation environment at the same time, the gradient of the objective function of each agent is calculated according to Equation (5) and summarized into the global gradient.

[0080] Using the root mean square propagation method as the optimizer, the parameters of the neural network are updated according to equation (6):

[0081] g=α×g+(1-α)×Δθ 2 (6)

[0082]

[0083] Where α is the decay rate, with a value of 0.99, g is the exponentially weighted average, η is the learning rate, with a value of 0.001, θ is the parameter of the neural network, and Δθ is the gradient of the parameter. After 10,000 iterations, the cloud cluster resource scheduling agent is obtained.

[0084] To verify the beneficial effects of the present invention, an experiment was conducted using the method described in Example 1 of the present invention. The experimental results are shown below. Figure 2 .exist Figure 2 In the graph, the horizontal axis represents the number of iterations, and the vertical axis represents the cumulative reward obtained by the resource scheduling agent in each iteration. Figure 2 It is evident that as the number of training iterations increases, the cumulative reward obtained by the resource scheduling agent exhibits a good convergence effect.

[0085] A comparative experiment was conducted using the method of Embodiment 1 of this invention, along with the Shortest Job First scheduling method, the Random scheduling method, the Tetris scheduling method, and the Packer scheduling method. The experimental results are shown in [Figure 1]. Figure 3 .exist Figure 3 In the diagram, the horizontal axis represents the training time, and the vertical axis represents the average weighted turnaround time of the task. Figure 3 It is evident that the method of this invention is superior to the comparative experimental method in terms of scheduling target for average weighted turnaround time of operations.

Claims

1. A cloud cluster resource scheduling method based on deep reinforcement learning, characterized in that, It consists of the following steps: (1) Constructing the task resource demand vector The task resource demand vector r is determined by the following formula. j : r j ＝(r j,1 ,r j,2 ,…,r j,d ) Where, r j,d Let represent the amount of resource type d required by task j, where j and d are finite positive integers; (2) Constructing the reward function Determine the weighted turnover time W according to formula (1): W = C / T (1) Where C is the turnaround time of the task, and T represents the duration of the task; Determine the average weighted turnover time according to formula (2). Among them, W i Let N represent the weighted turnaround time of task i, and N be the number of tasks, where N is a finite positive integer. The reward function r for each time step t is determined according to equation (3). t : Among them, T j This indicates the required service time for task j; (3) Constructing a resource scheduling intelligent agent network A resource scheduling agent network is constructed using a synchronous actor-critic deep reinforcement learning method. This network consists of an actor network for performing scheduling actions and a critic network for estimating state values. The actor network is composed of input layer 1, hidden layer 1, and output layer 1 connected in series. The critic network is composed of input layer 2, hidden layer 2, and output layer 2 connected in series. (4) Training resource scheduling agent network 1) Construct the gradient of the objective function Construct the gradient of the objective function according to equation (5). Where, π θ (a t |s t ) represents the neural network π of the resource scheduling agent. θ At time t, the system state s t Input and output action a t The probability, γ k R(s) represents the discount factor. t+k ,a t+k ) represents the system state s at time t+k. t+k Next, execute action a t+k The reward obtained, V(s) t ) represents the commentator network's opinion on the system state s at time t. t Value estimation; 2) Training resource scheduling agent network Using the system state s during the scheduling process, the resource scheduling agent outputs action a, A(s,a), to train the resource scheduling agent network; Different job sequences are used in different cloud cluster simulation environments. By simultaneously using 20 resource scheduling agents to interact with the simulation environment, the gradient of the objective function of each agent is determined according to Equation (5) and summarized into a global gradient: Using the root mean square propagation method as the optimizer, the parameters of the neural network are updated according to equation (6): g=α×g+(1-α)×Δθ 2 (6) Where α is the decay rate, with a value of 0.99, g is the exponentially weighted average, η is the learning rate, with a value of 0.001, θ is the parameter of the neural network, and Δθ is the gradient of the parameter. After 10,000 iterations, the cloud cluster resource scheduling agent is obtained.

2. The cloud cluster resource scheduling method based on deep reinforcement learning according to claim 1, characterized in that: In step (3) of constructing the resource scheduling agent network, the number of neurons in the input layer 1 is 4480, the number of neurons in the hidden layer 1 is 20, and the number of neurons in the output layer 1 is 11; the number of neurons in the input layer 2 is 4480, the number of neurons in the hidden layer 2 is 20, and the number of neurons in the output layer 2 is 1.

3. The cloud computing cluster resource scheduling method based on deep reinforcement learning according to claim 1, characterized in that: In step (4), specifically step 2) of training the resource scheduling agent network, the system state s is represented using an image, which comprises three parts: ① Cluster diagram: Represents the jobs that have been assigned in the cluster and their running status; ② Job graph: Represents the resource requirements of jobs in the waiting queue; ③ The backlog image represents the number of jobs waiting to be scheduled that exceed the capacity of the job slot.

4. The cloud cluster resource scheduling method based on deep reinforcement learning according to claim 1, characterized in that: In step (3), the resource scheduling agent network is constructed. The actor network takes the system state s as input and outputs the probability of executing each scheduling action. The critic network takes the system state s as input and outputs the value estimate V(s) of the current state.

Images