Cloud task deep reinforcement learning scheduling method based on correlation perception

Abstract

The invention relates to a cloud task deep reinforcement learning scheduling method based on correlation perception, which belongs to the technical field of deep reinforcement learning, is realized by a DDPG deep reinforcement learning network based on an Actor-Critic framework, and comprises the following steps: S1, initializing the network; s2, uploading a calculation task; s3, monitoring attributes of the virtual machine; s4, counting task attributes; s5, calculating the sensing state of the virtual machine; s6, selecting a state-action transfer pair from the memory playback buffer area; s7, calculating a priori reward; s8, calculating a posterior reward according to the response time and the operation cost; s9, updating and iterating the parameters of the network by utilizing the reward; and S10, executing scheduling by the virtual machine migration scheduler, and returning to the step S3 to repeat task scheduling at the next moment until the calculation task is completed. According to the method, the Pearson correlation coefficient and the standard deviation are used for evaluating the characteristics of the workload, accurate representation of the state perceived by the virtual machine is achieved, a cost and performance reward function is constructed, and optimization of the performance and the total cost of the virtual machine cluster is achieved.

Description

Technical Field

[0001] The present invention relates to a deep reinforcement learning scheduling method for cloud tasks based on correlation perception, belongs to the technical field of deep reinforcement learning, and in particular to a deep reinforcement learning scheduling method for cloud tasks based on correlation perception. Background Art

[0002] Currently, more and more traditional applications are migrated to the cloud. Load balancing without sacrificing the performance of cloud-based applications is a very challenging problem in cloud computing. Due to the highly dynamic nature of cloud workloads and virtual machines (VMs) with heterogeneous hardware configurations, existing technologies mainly focus on solving the problem of inefficient task scheduling.

[0003] Reinforcement learning-based methods can learn directly from their experience without prior knowledge of the workload, which makes it generalizable to various cloud environments and has fairly good performance. Therefore, most studies focus on using reinforcement learning methods to solve cloud scheduling problems. Most of these methods only focus on optimizing performance or cost-effectiveness. Among them, the literature SLAs-Aware Online Task Scheduling Based on Deep Reinforcement Learning Method in Cloud Environment (Longyu Ran, Xiaoyu Shi, Mingsheng Shang; 10.1109 / HPCC / SmartCity / DSS.2019.00209) relies on the powerful perception and decision-making capabilities of deep reinforcement learning (DRL) in automatic control problems, and proposes a task scheduling method based on deep reinforcement learning, which can dynamically assign submitted tasks to different VMs to achieve performance optimization. However, the electricity cost is not considered, so in actual scenarios, VMs need to consider the cost price. However, when only optimizing cost-effectiveness, these algorithms will tend to assign most tasks to those low-cost VMs, resulting in too long task queues and too long response times. Therefore, in practical applications, it is necessary to take into account both cost-effectiveness and task execution speed.

[0004] At the same time, most existing methods can only schedule a single task at a time, and many existing studies are only applicable to specific clusters and workloads, so it is necessary to consider the scheduling of multiple tasks and the universality of application scenarios. Summary of the invention

[0005] In view of this, the purpose of the present invention is to schedule multiple cloud computing tasks at one time and thereby optimize the task response time and cost efficiency. Therefore, a deep reinforcement learning scheduling method for cloud tasks based on correlation perception is provided. First, by designing the state representation of reinforcement learning and constructing a reward function based on cost and performance, the strategy of the reinforcement learning network is updated, thereby realizing the perceptual scheduling of optimized virtual machine tasks.

[0006] In order to achieve the above objectives, the symbols involved in the present invention are first defined as follows:

[0007] J is the total number of virtual machines in the cloud cluster; I is the total number of tasks in the current batch; speed is the data transmission speed; vm j =(mips j ,ram j , price j ) is the attribute of the jth virtual machine in the cluster; mips j For the jth virtual machine vm j The speed at which instructions are executed; RAM j For the jth virtual machine vm j Memory size; price j For the jth virtual machine vm j Running cost per second; coreNum j is the number of cores of the jth virtual machine; task i =(mi i , cpu i , data i ) is the attribute of the i-th task in the current task batch; mi i is the current i-th task task i Length; cpu i is the current i-th task task i Required cpu utilization; data i is the current i-th task task i The data size; The execution time of the current i-th task for the j-th virtual machine in the cluster; is the task queue waiting time of the jth virtual machine in the cluster; The response time of the j-th virtual machine to the current i-th task; where 1≤j≤J, 1≤i≤I, and C is the total number of tasks in the task queue of the j-th virtual machine.

[0008] The present invention provides the following technical solutions:

[0009] A deep reinforcement learning scheduling method for cloud tasks based on correlation perception is characterized by being implemented by a DDPG (Deep Deterministic Policy Gradient) deep reinforcement learning network based on an Actor-Critic framework; the DDPG deep reinforcement learning network based on the Actor-Critic framework adopts a Markov decision process (MDP for short) as a gradient strategy, and includes a triple (S, A, R), wherein S is the perception state space of the virtual machine, A is the virtual machine task scheduling action space, and R is a reward.

[0010] The DDPG deep reinforcement learning network based on the Actor-Critic framework is composed of an actor network μ(s|θ μ ), critic network Q(s, a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s, a|θ Q′ ); where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s∈S is the perception state of the virtual machine; a∈A is the virtual machine task scheduling action, and according to the current perception state s of the virtual machine, the batch processing tasks are assigned to the virtual machines of the J cloud clusters.

[0011] The steps include:

[0012] S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the memory replay buffer (Replay Buffer);

[0013] S2: Obtain computing tasks uploaded by users in real time and perform batch processing;

[0014] S3: Use performance monitors to monitor the properties of each virtual machine in the cloud server cluster;

[0015] S4: Count the attributes of each task in the task batch at the current moment;

[0016] S5: Calculate the perception state of the virtual machine at the current moment based on the attributes of each virtual machine and each task;

[0017] S6: Selecting a number of state-action transition pairs (transitions) from the memory playback buffer according to the perception state of the virtual machine at the current moment;

[0018] S7: Calculate the prior reward according to the virtual machine task scheduling action of the state-action transfer pair and the perception state of the virtual machine at the current moment;

[0019] S8: Calculate the response time and running cost of the current batch of tasks, and calculate the posterior reward using the normalization method;

[0020] S9: Generate rewards using prior rewards and posterior rewards to iterate the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework;

[0021] S10: The DDPG deep reinforcement learning network based on the Actor-Critic framework generates virtual machine task scheduling actions, the virtual machine migration scheduler executes the scheduling, and returns to step S3 to repeat the task scheduling at the next moment until the computing task is completed.

[0022] Further, the step S1 is specifically as follows:

[0023] S101: Initialize the network parameters θ of the DDPG deep reinforcement learning network based on the Actor-Critic framework μ ,θ Q ,θ μ′ ,θ Q′ , and the exploration probability ε of the Markov decision process;

[0024] S202: Based on experience or historical data, establish a state-action transition pair (s t , a t ,s t+1 ) constitutes a memory playback buffer; where the subscript t represents the current moment and t+1 represents the next moment.

[0025] Further, the step S5 is specifically as follows:

[0026] S501: Calculate the Pearson correlation coefficient ρ(mi, data) between all task lengths and data sizes in the current task batch; where, mi=[mi i ] 1≤i≤I ,data=[data i ] 1≤i≤I , cov(·) is the covariance, σ(·) is the standard deviation;

[0027] S502: Calculate the Pearson correlation coefficient ρ(mi, cpu) between the lengths of all tasks in the current task batch and the required CPU utilization; where, cpu=[cpu i ] 1≤i≤I ;

[0028] S503: Calculate the Pearson correlation coefficient ρ(data, cpu) between all data sizes in the current task batch and the required cpu utilization; where,

[0029] S504: Calculate the task length, required CPU utilization, and standard deviation of data size σ(mi), σ(data), and σ(cpu) in the current task batch;

[0030] S505: Construct the perception state s of the virtual machine corresponding to the correlation part cor ;

[0031] Among them, s cor =[ρ(mi,data),ρ(mi,cpu),ρ(data,cpu),σ(mi),σ(data),σ(cpu)];

[0032] S506: Construct the perception state s of the virtual machine corresponding to the computing task part Task ;

[0033] Among them, s Task =[mi1,cpu1,data1,...,mi I , cpu I , data I ];

[0034] S507: Constructing the perception state s of the virtual machine corresponding to the virtual machine part VM ;

[0035] in,

[0036] S508: Constructing the perception state s=s of the virtual machine cor ∪s Task ∪s VM .

[0037] Further, the prior reward described in step S7 is in, reward is a constant used to control the maximum prior reward.

[0038] Furthermore, the posterior reward in step S8 is: in, is the average response time of the jth virtual machine in the cluster executing the current i-th task, The execution time of the current i-th task on the j-th virtual machine in the cluster, is the operating cost of the j-th virtual machine for the current i-th task, β is a hyperparameter set artificially.

[0039] Preferably, in the calculation process of the posterior reward described in step S8, the following constraints need to be satisfied for any j-th virtual machine and any i-th task:

[0040] Furthermore, at the current time t, the reward of the DDPG deep reinforcement learning network based on the Actor-Critic framework described in step S9 is r t =reward prior +reward poster .

[0041] Preferably, the reward of the DDPG deep reinforcement learning network based on the Actor-Critic framework described in step S9 at the current time t is r t =reward prior +η·reward poster , where η is a hyperparameter.

[0042] Further, the step S9 is specifically as follows:

[0043] S901: Using loss function Update the critic network Q(s, a|θ Q ) parameter θ Q ; Among them, y i =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θ Q' ), N is the number of state-action transition pairs, λ is a hyperparameter, and t is the corresponding current time s of the memory playback buffer t The subscript of the state-action transition pair;

[0044] S902: Update the actor network μ(s|θ) using the gradient strategy μ )’s network parameters θ μ ;

[0045] S903: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s, a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0046] The step S10 generates a virtual machine task scheduling action according to the probability of ε, and the remaining virtual machine tasks are scheduled according to the action μ(s) of the DDPG deep reinforcement learning network based on the Actor-Critic framework. t |θ μ ).

[0047] Preferably, in order to promote load balancing of the cluster, when the batch processing task size of a virtual machine exceeds the memory size, the virtual machine migration scheduler in step S10 randomly schedules the remaining tasks.

[0048] The beneficial effects of the present invention are as follows: a deep reinforcement learning scheduling method for cloud tasks based on correlation perception is provided, which is implemented using a DDPG deep reinforcement learning network based on an Actor-Critic framework; firstly, batch tasks are perceived as a whole, and the characteristics of the workload are evaluated using the Pearson correlation coefficient and the standard deviation of resource requirements to achieve virtual machine-aware state representation; then a composite reward function is constructed using the optimization objective to optimize the network update strategy; finally, task scheduling is implemented using a reinforcement learning network to achieve optimization of performance and the total cost of a virtual machine cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to make the purpose and technical solution of the present invention more clear, the present invention provides the following drawings for explanation:

[0050] Figure 1 It is a flow chart of the deep reinforcement learning scheduling method for cloud tasks based on correlation perception of the present invention;

[0051] Figure 2 Embodiment 1 of the present invention is a schematic diagram of the total number of tasks per minute and the total task length of Alibaba-Cluster-trace-v2018;

[0052] Figure 3 The embodiment 1 of the present invention is a schematic diagram of the total number of tasks per second and the total task length of Google ClusterData2011;

[0053] Figure 4 It is the task scheduling framework on the cloud in Embodiment 1 of the present invention;

[0054] Figure 5 This is a diagram of the DDPG deep reinforcement learning network architecture based on the Actor-Critic framework of Example 1 of the present invention;

[0055] Figure 6This is a comparison experiment result diagram of the scheduling results of Alibaba-Cluster-trace-v2018 of Example 1 of the present invention; wherein the horizontal axis corresponds to the number of the virtual machine, 0 to 4 corresponds to mode 1, 5 to 9 corresponds to mode 2, 10 to 14 corresponds to mode 3, 15 to 19 corresponds to mode 4, and the vertical axis is the number of tasks;

[0056] Figure 7 The graph is a comparison experiment result of the scheduling result of Google ClusterData2011 of Example 1 of the present invention; wherein the horizontal axis corresponds to the number of the virtual machine, 0 to 4 corresponds to mode 1, 5 to 9 corresponds to mode 2, 10 to 14 corresponds to mode 3, 15 to 19 corresponds to mode 4, and the vertical axis is the number of tasks;

[0057] Figure 8 This is a graph showing the operating cost comparison experiment results of Alibaba-Cluster-trace-v2018 of Example 1 of the present invention;

[0058] Fig. 9 This is a graph showing the results of a comparative experiment on the operating costs of Google ClusterData2011 of Example 1 of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose and technical solution of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0060] Example 1: Consider that 20 virtual machines are deployed on the enterprise-level computing type (c7) server provided by Alibaba Cloud to provide services to users. These virtual machines have four different pricing models, as shown in Table 1. Assume that users use the real dataset Alibaba-Cluster-trace-v2018 (such as Figure 2 ) and GoogleClusterData2011 (as shown Figure 3 As shown in the figure, the computing task is used as the task. To make the model stable, it is assumed that the user can submit a maximum of 100 computing tasks in one second. The agent in the virtual machine needs to interact with the environment within 1 second to make a scheduling decision for the upcoming computing task. In order to achieve this function, combined with Figure 4 The present invention provides a “cloud-based task deep reinforcement learning scheduling method based on correlation perception”, and completes the implementation and calculation of the relevant method on a local tower server. The server contains a 2.1GHz Intel Xeon E5 CPU, 250GB RAM, and is installed with Ubuntu 18.04LTS operating system, JDK1.8, Python3.6, PyTorch 1.0, CloudSim Plus 4.0 and other software.

[0061] This method is implemented by the DDPG deep reinforcement learning network based on the Actor-Critic framework; Figure 5 As shown, the DDPG deep reinforcement learning network based on the Actor-Critic framework adopts the Markov decision process as the gradient strategy, which includes a triple (S, A, R), where S is the perception state space of the virtual machine, A is the task scheduling action space of the virtual machine, and R is the reward.

[0062] The DDPG deep reinforcement learning network based on the Actor-Critic framework is composed of an actor network μ(s|θ μ ), critic network Q(s, a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s, a|θ Q′ ), each network consists of four fully connected layers. This embodiment uses the Adam optimizer to optimize all networks. The actor network and the critic network are updated in each round of training, with learning rates of 0.0006 and 0.001 respectively; and the two target networks are updated later according to τ = 0.01; where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s∈S is the perception state of the virtual machine; a∈A is the virtual machine task scheduling action, and according to the current perception state s of the virtual machine, the batch processing tasks are assigned to the virtual machines of the J cloud clusters.

[0063] Table 1 Some parameters of cloud server

[0064]

[0065] Combination Figure 1 , the method comprises the following steps:

[0066] Step 1: Build and initialize the DDPG deep reinforcement learning network and memory playback buffer based on the Actor-Critic framework on the local tower server.

[0067] Specifically:

[0068] S101: Build and initialize the network parameters θ of the DDPG deep reinforcement learning network based on the Actor-Critic framework μ ,θ Q ,θ μ′ ,θ Q′ , and the exploration probability ε of the Markov decision process;

[0069] S202: Based on experience or historical data, establish a state-action transition pair (s t , a t ,s t+1 ) constitutes a memory playback buffer; where the subscript t represents the current moment and t+1 represents the next moment.

[0070] Among them, the capacity of the memory playback buffer is set to 10000.

[0071] Step 2: Obtain computing tasks uploaded by users in real time and perform batch processing, where the size of the mini-batch is 16.

[0072] Step 3: Use the cloud server's performance monitor to monitor the properties of each virtual machine in the cloud server cluster.

[0073] The total number of virtual machines involved in the cloud cluster is J = 20; the data transmission speed is speed; the attribute of the jth virtual machine in the cluster is vm j =(mips j ,ram j , price j ); mips j For the jth virtual machine vm j The speed at which instructions are executed; RAM j For the jth virtual machine vm j The memory size is shown in Table 1; price j For the jth virtual machine vm j The running cost per second is shown in Table 1; coreNum j is the number of cores of the jth virtual machine, as shown in Table 1.

[0074] Step 4: Count the attributes of each task in the task batch at the current moment.

[0075] Involves the total number of tasks in the current batch within one second, I, and the i-th task attribute task in the current task batch i =(mi i , cpu i , data i );mi i is the current i-th task task i Length; cpu i is the current i-th task task i Required cpu utilization; data i is the current i-th task task i The data size.

[0076] Step 5: Based on the attributes of each virtual machine and each task, calculate the perception state of the virtual machine at the current moment. Specifically:

[0077] S501: Calculate the Pearson correlation coefficient ρ(mi, data) between all task lengths and data sizes in the current task batch; where, mi=[mi1] 1≤i≤I ,data=[data i ] 1≤i≤I , cov(·) is the covariance, σ(·) is the standard deviation;

[0078] S502: Calculate the Pearson correlation coefficient ρ(mi, cpu) between the lengths of all tasks in the current task batch and the required CPU utilization; where, cpu=[cpu i ] 1≤i≤I ;

[0079] S503: Calculate the Pearson correlation coefficient ρ(data, cpu) between all data sizes in the current task batch and the required cpu utilization; where,

[0080] S504: Calculate the task length, required CPU utilization, and standard deviation of data size σ(mi), σ(data), and σ(cpu) in the current task batch;

[0081] S505: Construct the perception state s of the virtual machine corresponding to the correlation part cor ;

[0082] Among them, s cor =[ρ(mi,data),ρ(mi,cpu),ρ(data,cpu),σ(mi),σ(data),σ(cpu)];

[0083] S506: Construct the perception state s of the virtual machine corresponding to the computing task part Task ;

[0084] Among them, s Task =[mi1,cpu1,data1,...,mi I , cpu I , data I ];

[0085] S507: Constructing the perception state s of the virtual machine corresponding to the virtual machine part VM ;

[0086] in, is the waiting time of the task queue of the j-th virtual machine in the cluster, and C is the total number of tasks in the task queue of the j-th virtual machine;

[0087] S508: Constructing the perception state s=s of the virtual machine cor ∪s Task ∪s VM .

[0088] Step 6: Select 16 state-action transition pairs from the memory playback buffer according to the perception state of the virtual machine at the current moment.

[0089] Step 7: Calculate the prior reward according to the virtual machine task scheduling action of the state-action transfer pair and the perception state of the virtual machine at the current moment.

[0090] The prior reward is in, reward is a constant used to control the maximum prior reward.

[0091] Step 8: Calculate the response time and running cost of the current batch of tasks, and calculate the posterior reward using the normalization method.

[0092] The posterior reward is: in, is the average response time of the jth virtual machine in the cluster executing the current i-th task, is the operating cost of the j-th virtual machine for the current i-th task, β is a hyperparameter set artificially. The response time of the jth virtual machine to the current i-th task, The execution time of the current i-th task on the j-th virtual machine in the cluster.

[0093] Step 9: Generate reward r using prior rewards and posterior rewards t =reward prior +reward poster , update and iterate the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework to generate virtual machine task scheduling actions. Specifically:

[0094] S901: Using loss function Update the critic network Q(s, a|θ Q ) parameter θ Q ; Among them, y t =r t +λQ′(s t+1 ,μ′(s t+1 |θ μ′ )|θQ′ ), N is the number of state-action transition pairs, λ is a hyperparameter, and t is the corresponding current time s of the memory playback buffer t The subscript of the state-action transition pair;

[0095] S902: Update the actor network μ(s|θ) using the gradient strategy μ )’s network parameters θ μ ;

[0096] S903: Update actor target network μ′(s|θ μ′ ), critic target network Q′(s, a|θ Q′ )’s network parameters θ μ′ ←τθ μ +(1-τ)θ μ′ ,θ Q′ ←τθ Q +(1-τ)θ Q′ ; where τ is a hyperparameter.

[0097] Step 10: The DDPG deep reinforcement learning network based on the Actor-Critic framework generates virtual machine task scheduling actions. The virtual machine migration scheduler executes the scheduling and returns to step 3 to repeat the task scheduling for the next second until all computing tasks are completed.

[0098] The virtual machine task scheduling action is generated according to the probability of ε, and the remaining virtual machine tasks are scheduled according to the action μ(s) of the DDPG deep reinforcement learning network based on the Actor-Critic framework. t |θ μ ).

[0099] In order to better demonstrate the beneficial effects of the method of the present invention, this embodiment is compared with four prior arts, which are: (1) random: each task is randomly assigned to a virtual machine; (2) Round-Robin (rr): tasks are assigned to each virtual machine in turn; (3) earliest: tasks are assigned to the first idle virtual machine according to the arrival time; (4) DQN: a DRL-based task scheduling method [1]. The method of the present invention is denoted as (DDPG) here.

[0100] [1]MTIslam, S.Karunasekera, and R.Buyya, "Performance and cost-effcient spark job scheduling based on deep reinforcement learning in cloudcomputing environments," IEEE Transactions on Parallel and DistributedSystems, vol.33, no.7, pp.1695-1710, 2021.

[0101] In the experiment, the virtual machine migration scheduler makes dynamic decisions based on the current state of the cluster and workload, and has no prior knowledge of the entire workload. Each technical method is repeated 10 times and the average result is recorded. Figure 6 As shown. It can be seen that the virtual machine has better performance and can process more computing tasks per unit time. The computing tasks scheduled to it should be more. The method of the present invention can better implement task scheduling. The operating cost comparison results are shown in Figure 7 As shown, it can be seen that the operating cost of the method of the present invention on both data sets is the lowest. Therefore, it can be said that the method of the present invention can achieve virtual machine task scheduling with better performance and cost.

[0102] Example 2: For the scenario of Example 1, considering that task scheduling must meet the constraints of the actual scenario and improve load balancing, this example provides a "cloud-based task deep reinforcement learning optimization scheduling method based on correlation perception". It is implemented based on the DDPG deep reinforcement learning network of the Actor-Critic framework. The specific network parameter settings and specific steps are almost exactly the same as those of Example 1, and will not be repeated here. The difference is that the method includes the following steps:

[0103] Step 1: Build and initialize the DDPG deep reinforcement learning network and memory playback buffer based on the Actor-Critic framework on the local tower server.

[0104] Step 2: Obtain computing tasks uploaded by users in real time and perform batch processing, where the size of the mini-batch is 16.

[0105] Step 3: Use the cloud server's performance monitor to monitor the properties of each virtual machine in the cloud server cluster.

[0106] Step 4: Count the attributes of each task in the task batch at the current moment.

[0107] Step 5: Based on the attributes of each virtual machine and each task, calculate the perceived state of the virtual machine at the current moment.

[0108] Step 6: Select 16 state-action transition pairs from the memory playback buffer according to the perception state of the virtual machine at the current moment.

[0109] Step 7: Calculate the prior reward according to the virtual machine task scheduling action of the state-action transfer pair and the perception state of the virtual machine at the current moment.

[0110] Step 8: Calculate the response time and running cost of the current batch of tasks, and calculate the posterior reward using the normalization method.

[0111] In the calculation process of the posterior reward, the following constraints must be strictly met for any j-th virtual machine and any i-th task:

[0112] Step 9: Generate reward r using prior rewards and posterior rewards t =reward prior +η·reward poster , update and iterate the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework. Where η is a hyperparameter.

[0113] Step 10: The DDPG deep reinforcement learning network based on the Actor-Critic framework generates virtual machine task scheduling actions. The virtual machine migration scheduler executes the scheduling and returns to step 3 to repeat the task scheduling for the next second until all computing tasks are completed.

[0114] In order to promote load balancing of the cluster, when the batch processing task size of a virtual machine exceeds the memory size, the virtual machine migration scheduler randomly schedules the remaining tasks.

[0115] Finally, it should be noted that the above preferred embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A deep reinforcement learning scheduling method for cloud tasks based on correlation perception, characterized in that: It is implemented by a DDPG (Deep Deterministic Policy Gradient) deep reinforcement learning network based on the Actor-Critic framework; the DDPG deep reinforcement learning network based on the Actor-Critic framework adopts Markov Decision Process (MDP) as a gradient strategy, including a triple (S, A, R), where S is the perception state space of the virtual machine, A is the virtual machine task scheduling action space, and R is the reward; The DDPG deep reinforcement learning network based on the Actor-Critic framework is composed of an actor network μ(s|θ μ ), critic network Q(s, a|θ Q ), actor target network μ′(s|θ μ′ ), critic target network Q′(s, a|θ Q′ ); where: θ μ ,θ Q ,θ μ′ ,θ Q′ are the weight coefficients of the four networks respectively; s∈S is the perception state of the virtual machine; a∈A is the virtual machine task scheduling action, which assigns the batch processing task to the J virtual machines in the cloud cluster according to the perception state s of the current virtual machine; J is the total number of virtual machines in the cloud cluster; The method comprises the following steps: S1: Initialize the DDPG deep reinforcement learning network based on the Actor-Critic framework and the memory replay buffer (Replay Buffer); S2: Obtain computing tasks uploaded by users in real time and perform batch processing; S3: Use performance monitors to monitor the properties of each virtual machine in the cloud server cluster; S4: Count the attributes of each task in the task batch at the current moment; S5: Calculate the perception state of the virtual machine at the current moment based on the attributes of each virtual machine and each task; S6: Selecting a number of state-action transition pairs (transitions) from the memory playback buffer according to the perception state of the virtual machine at the current moment; S7: Calculate the prior reward according to the virtual machine task scheduling action of the state-action transfer pair and the perception state of the virtual machine at the current moment; S8: Calculate the response time and running cost of the current batch of tasks, and calculate the posterior reward using the normalization method; S9: Generate rewards using prior rewards and posterior rewards to iterate the parameters of the DDPG deep reinforcement learning network based on the Actor-Critic framework; S10: The DDPG deep reinforcement learning network based on the Actor-Critic framework generates a virtual machine task scheduling action, the virtual machine migration scheduler performs the scheduling, and returns to step S3 to repeat the task scheduling at the next moment until the computing task is completed; The data transmission speed is speed; the attribute of the jth virtual machine in the cluster is vm j =(mips j ,ram j , price j ); mips j For the jth virtual machine vm j The speed at which instructions are executed; RAM j For the jth virtual machine vm j The memory size is shown in Table 1; price j For the jth virtual machine vm j The running cost per second is shown in Table 1; coreNum j is the number of cores of the jth virtual machine, the total number of tasks in the current batch is I, and the attribute task of the i-th task in the current task batch is i =(mi i , cpu i , data i );mi i is the current i-th task task i Length; cpu i is the current i-th task task i Required cpu utilization; data i is the current i-th task task i The data size; 1≤j≤J, 1≤i≤I; The step S5 is specifically as follows: S501: Calculate the Pearson correlation coefficient ρ(mi, data) between all task lengths and data sizes in the current task batch; where, mi=[mi i ] 1≤i≤I ,data=[data i ] 1≤i≤I , cov(·) is the covariance, σ(·) is the standard deviation; S502: Calculate the Pearson correlation coefficient ρ(mi, cpu) between the lengths of all tasks in the current task batch and the required CPU utilization; where, S503: Calculate the Pearson correlation coefficient ρ(data, cpu) between all data sizes in the current task batch and the required cpu utilization; where, S504: Calculate the task length, required CPU utilization, and standard deviation of data size σ(mi), σ(data), and σ(cpu) in the current task batch; S505: Construct the perception state s of the virtual machine corresponding to the correlation part cor ; among them, s cor =[ρ(mi, data),ρ(mi, cpu),ρ(data, cpu),σ(mi),σ(data),σ(cpu)]; S506: Construct the perception state s of the virtual machine corresponding to the computing task part Task ; Among them, s Task =[mi1,cpu1,data1,...,mi1,cpu I , data I ]; S507: Constructing the perception state s of the virtual machine corresponding to the virtual machine part VM ; in, is the waiting time of the task queue of the j-th virtual machine in the cluster, and C is the total number of tasks in the task queue of the j-th virtual machine; S508: Constructing the perception state s=s of the virtual machine cor ∪s Task ∪s VM ; The prior reward is in, reward is a constant used to control the maximum prior reward; The posterior reward is: in, is the average response time of the jth virtual machine in the cluster executing the current i-th task, is the operating cost of the j-th virtual machine for the current i-th task, β is a hyperparameter set artificially. The response time of the jth virtual machine to the current i-th task, The execution time of the current i-th task for the j-th virtual machine in the cluster; The reward of the DDPG deep reinforcement learning network based on the Actor-Critic framework is r t =reward prior +reward poster .

2. The method for scheduling cloud-based tasks based on deep reinforcement learning based on correlation perception according to claim 1, characterized in that: The step S1 is specifically as follows: S101: Initialize the network parameters θ of the DDPG deep reinforcement learning network based on the Actor-Critic framework μ ,θ Q ,θ μ′ ,θ Q′ , and the exploration probability ε of the Markov decision process; S202: Based on experience or historical data, establish a state-action transition pair (s t , a t ,s t+1 ) constitutes a memory playback buffer; where the subscript t represents the current moment and t+1 represents the next moment.

3. The method for scheduling cloud-based tasks based on deep reinforcement learning based on correlation perception according to claim 1, characterized in that: In the calculation process of the posterior reward described in step S8, the following constraints need to be met for any j-th virtual machine and any i-th task:

4. The method for scheduling cloud-based tasks based on deep reinforcement learning based on correlation perception according to claim 1, characterized in that: At the current time t, the reward of the DDPG deep reinforcement learning network based on the Actor-Critic framework described in step S9 is r t =reward prior +η·reward poster , where η is a hyperparameter.

5. The method for scheduling cloud-based tasks based on deep reinforcement learning based on correlation perception according to claim 1, characterized in that: The step S10 generates a virtual machine task scheduling action according to the probability of ε, and the remaining virtual machine tasks are scheduled according to the action μ(s) of the DDPG deep reinforcement learning network based on the Actor-Critic framework. t |θ μ ).

6. The method for scheduling cloud-based tasks based on deep reinforcement learning based on correlation perception according to claim 1, characterized in that: In order to promote load balancing of the cluster, when the batch processing task size of a virtual machine exceeds the memory size, the virtual machine migration scheduler randomly schedules the remaining tasks in step S10.

Images