Cloud computing task scheduling method, device and system based on deep reinforcement learning

Abstract

The invention discloses a cloud computing task scheduling method, device and system based on deep reinforcement learning, and the method comprises the steps: carrying out the preprocessing of a task, and obtaining the priority of the task; the task with the highest priority is selected according to the priorities of the tasks, partition selection is converted into a Markov decision process, the Markov decision process is executed through a first D3QN agent, and the tasks are distributed to corresponding virtual machine partitions; and converting task scheduling into a Markov decision process, and executing the Markov decision process through the second D3QN intelligent agent to distribute the tasks to the corresponding virtual machines so as to realize task scheduling. A task scheduling model is designed, task scheduling is regarded as a packing problem by the model, and resource requirements of tasks and execution capability of heterogeneous virtual machines are considered at the same time. Meanwhile, reduction of task completion time and maintenance of load balance of a cluster environment are taken as optimization objectives, so that multi-objective optimization of task scheduling is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of cloud computing task scheduling, and more particularly, to a cloud computing task scheduling method, device and system based on deep reinforcement learning. BACKGROUND

[0002] With the rapid development of information technology and Internet economy, cloud computing gradually replaces traditional computing technology and becomes one of the research directions in the current Internet field. Cloud computing is a new IT computing mode developed on the basis of parallel distributed computing and grid computing. It is based on virtualization technology to aggregate network, server, storage, application and other computing resources together to form a shared resource pool, and the resources in the resource pool can be quickly configured and released to effectively meet the needs of different users.

[0003] At present, many well-known enterprises have carried out in-depth research on cloud computing, and they have built cloud platforms to provide cloud computing services for users. However, due to the popularity of cloud computing technology, cloud traffic is increasing, the number of users is rapidly increasing, the types of tasks submitted by users are becoming more diverse, and the number of tasks is showing a trend of rapid growth, which leads to the problem of resource shortage in the cloud computing resource pool, and also affects the performance of the cloud platform. As one of the core technologies of cloud computing, task scheduling is a key component to achieve high performance in cloud environment, and plays a decisive role in the change of resources in the cloud computing resource pool, directly affecting the actual problems such as resource utilization, user service quality and load balancing of cloud computing. Task scheduling is the process of allocating tasks to be executed to computing resources, which is usually automated and completed by a task scheduler. In the process of task scheduling, the task scheduler needs to consider the resource requirements, priority and execution time of each task, as well as the availability and performance of computing resources, and then reasonably allocate computing resources to tasks according to these information to ensure that the resource requirements of the task can be met and completed within the expected time. Reasonable task scheduling can improve the utilization of computing resources, improve user service quality and help users better adapt to changing business needs. Unreasonable task scheduling may cause the response time of the task to be too long, affecting the user service quality. It may also cause the load imbalance between computing nodes, resulting in some nodes being overloaded and some nodes being underloaded, affecting the overall performance of the cloud environment. Therefore, it is crucial to choose a good task scheduling algorithm.

[0004] The goal of task scheduling in cloud environment is to reasonably allocate tasks to a heterogeneous virtual machine cluster, so as to meet the requirements of resources and performance. Optimizing task scheduling can improve the overall performance of the cloud environment, maintain the load balance of the cluster environment, reduce the response time of the task, and improve the service quality of the user. In recent years, in order to pursue better task scheduling effect, many scholars have invested a lot of manpower and material resources to study the task scheduling algorithm. The common task scheduling algorithms at present are mainly divided into four categories, including traditional scheduling algorithm, meta-heuristic scheduling algorithm, machine learning-based scheduling algorithm and deep reinforcement learning-based scheduling algorithm.

[0005] Deep reinforcement learning is one of the research directions in the field of artificial intelligence, which ingeniously combines the perception ability of deep learning and the decision-making ability of reinforcement learning. Through the end-to-end learning method, it realizes the seamless connection from perception to decision-making, and provides an effective solution for many complex tasks. However, the method of applying deep reinforcement learning to task scheduling in cloud environment in the prior art still has the problem of poor scheduling effect. SUMMARY

[0006] In order to solve the above problems, the present application provides a cloud computing task scheduling method, device and system based on deep reinforcement learning, a task scheduling model is designed, which regards task scheduling as a packing problem and considers the resource requirements of tasks and the execution ability of heterogeneous virtual machines at the same time. At the same time, reducing the task completion time and maintaining the load balance of the cluster environment are also taken as optimization goals to realize the multi-objective optimization of task scheduling. The present application proposes a two-stage task scheduling algorithm based on D3QN, which abstracts the task scheduling problem into a Markov decision process, designs the state space, action space and reward function of the algorithm, and improves the action exploration strategy, finally realizes the reduction of the task completion time. The experiment is performed on Alibaba Cluster-trace-v2018, a real cluster tracking data published by Alibaba, and compared with the traditional method to verify the effectiveness of the method.

[0007] Specifically, the first aspect of the present application provides a cloud computing task scheduling method based on deep reinforcement learning, comprising: preprocessing the task to obtain the priority of the task; selecting the task with the highest priority according to the priority of the task, converting the partition selection into a Markov decision process, and executing the Markov decision process through a first D3QN intelligent agent to allocate the task to the corresponding virtual machine partition, which is obtained by pre-dividing the virtual machine cluster; The task scheduling is converted into a Markov decision process, and the tasks are allocated to corresponding virtual machines by executing the Markov decision process through a second D3QN agent, so as to realize the task scheduling.

[0008] In an implementation, the tasks are preprocessed to obtain the priorities of the tasks, including: The three indexes of resource requirement, length and waiting time of the tasks are positively processed and normalized to obtain the priorities of the tasks.

[0009] In an implementation, the Markov decision process for partition selection includes a state space, an action space and a reward function, wherein the state space includes CPU resource requirement and memory resource requirement of a current task to be scheduled, and CPU available resource utilization and memory available resource utilization of each partition; the size of the action space is the number of partitions in the entire virtual machine cluster environment, the composition of the action space is a set of output actions of the first deep reinforcement learning agent, and the reward function is obtained according to the change degree of the standard deviation of CPU and memory resource utilization in each partition.

[0010] In an implementation, the reward function is calculated in the following manner:

[0011] wherein, ，

[0012] wherein, is a reward function for partition selection, is a reward function for CPU load, is a reward function for memory load, is used to measure the change degree of the standard deviation of CPU resource utilization, is used to measure the change degree of the standard deviation of memory resource utilization.

[0013] In an implementation, the tasks are allocated to corresponding virtual machine partitions by executing the Markov decision process through the first D3QN agent, including: selecting a task with the highest priority; obtaining a current state according to the selected task and the state of the partitions in the virtual machine cluster; generating an action by the first D3QN agent according to the current state; executing the action to allocate the task to a corresponding virtual machine partition.

[0014] In one implementation, the Markov process for task scheduling includes a state space, an action space, and a reward function. The state space includes the number of virtual machines in the partition, the CPU and memory resources required for task scheduling, and the available CPU and memory resource utilization of the virtual machines. The action space is the set of actions output by the second deep reinforcement learning agent. The reward function is obtained by dividing the minimum possible execution time of the task by the actual completion time of the task.

[0015] In one implementation, the reward function for task scheduling... The calculation method is as follows:

[0016] in,

[0017]

[0018] In the formula, The reward function for load balancing, A reward function for task completion time. The degree of CPU resource load imbalance. This refers to the uneven load distribution of memory resources. This represents the minimum execution time of the task. This refers to the actual completion time of the task.

[0019] In one implementation, a Markov decision process is executed by a second D3QN agent to assign tasks to corresponding virtual machines, including: Get the current status based on the currently selected task and the status of the selected virtual machine partition; Generate an action based on the current state; The action is executed to assign tasks to the corresponding virtual machines in order to achieve task scheduling.

[0020] Based on the same inventive concept, a second aspect of the present invention provides a cloud computing task scheduling device based on deep reinforcement learning, comprising: Preprocessor; preprocesses tasks to obtain task priorities; The task allocation module is used to select the highest priority task based on the task priority, and transform the partition selection into a Markov decision process. The Markov decision process is executed by the first D3QN agent to allocate the task to the corresponding virtual machine partition. The virtual machine partition is obtained by pre-dividing the virtual machine cluster. The task scheduling module is used to transform task scheduling into a Markov decision process. The second D3QN agent executes the Markov decision process to allocate tasks to the corresponding virtual machines in order to achieve task scheduling.

[0021] Based on the same inventive concept, a third aspect of the present invention provides a cloud computing task scheduling system based on deep reinforcement learning, including the cloud computing task scheduling device based on deep reinforcement learning of the second aspect, and further including a user layer and an execution layer. The user layer is used by users to submit tasks on a cloud platform via physical devices; the execution layer includes a virtual machine cluster, in which the virtual machines are pre-divided into several virtual machine partitions of equal size, and the virtual machines in the execution layer are used to execute the assigned tasks. Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: This invention proposes a deep reinforcement learning-based method for task scheduling in cloud computing, aiming at load balancing. The method partitions the virtual machine cluster into several equally sized virtual machine partitions and divides task scheduling into two phases: task allocation and task scheduling. In the allocation phase, partition selection is transformed into a Markov decision process, executed by a first D3QN agent to allocate tasks to the corresponding virtual machine partitions. In the scheduling phase, task scheduling is transformed into a Markov decision process, executed by a second D3QN agent to allocate tasks to the corresponding virtual machines, thus achieving task scheduling. By using a two-stage task scheduling algorithm based on D3QN, the task scheduling problem is abstracted into a Markov decision process. The algorithm's state space, action space, and reward function are designed, and the action exploration strategy is improved, ultimately reducing task completion time and thus improving task scheduling performance. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the task scheduling model in an embodiment of the present invention.

[0024] Figure 2 This is a framework diagram of the task scheduling method in an embodiment of the present invention.

[0025] Figure 3 This is a comparative experimental result diagram of task sorting preprocessing in an embodiment of the present invention.

[0026] Figure 4 This is a graph showing the experimental results of CPU load balancing comparison of the partition selection algorithm in this embodiment of the invention.

[0027] Figure 5This is a graph showing the comparison results of memory load balancing experiments using the partition selection algorithm in this embodiment of the invention.

[0028] Figure 6 This is a training effect diagram of the task scheduling algorithm in an embodiment of the present invention.

[0029] Figure 7 This is a graph showing the experimental results of comparing the completion times of the task scheduling algorithm in this embodiment of the invention.

[0030] Figure 8 The figure shows the experimental results comparing the completion times of the two-stage task scheduling algorithm in this embodiment of the invention.

[0031] Figure 9 The figure shows the experimental results of CPU load balancing comparison of the two-stage task scheduling algorithm in this embodiment of the invention.

[0032] Figure 10 The figure shows the comparison results of memory load balancing experiments for the two-stage task scheduling algorithm in this embodiment of the invention. Detailed Implementation

[0033] This embodiment provides a cloud computing task scheduling method based on deep reinforcement learning, including: S1: Preprocess the task to obtain its priority; S2: Select the highest priority task based on the task priority, transform the partition selection into a Markov decision process, execute the Markov decision process through the first D3QN agent, and assign the task to the corresponding virtual machine partition. The virtual machine partition is obtained by pre-dividing the virtual machine cluster. S3: Transform task scheduling into a Markov decision process. The second D3QN agent executes the Markov decision process to allocate tasks to the corresponding virtual machines to achieve task scheduling.

[0034] Specifically, current task scheduling research lacks a unified scheduling model. Some models treat tasks and virtual machines as a one-to-one relationship, reducing virtual machine resource utilization. Others assume unlimited resources, ignoring task resource requirements. Still others consider task resource requirements but fail to account for the varying execution capabilities of heterogeneous virtual machines. Therefore, based on previous research, this invention designs a task scheduling model that comprehensively considers task resource requirements and the execution capabilities of heterogeneous virtual machines during cloud task scheduling, treating task scheduling as a bin packing problem to improve resource utilization. Task scheduling in a cloud environment is also a multi-objective optimization problem. Regarding completion time, reducing task completion time improves resource utilization. Virtual machine resources in a cluster are limited; shortening task completion time allows for faster resource release to process the next task, thus improving resource utilization. Furthermore, reducing task completion time also increases user satisfaction. Shorter task completion times mean users quickly receive the desired results, increasing user satisfaction. For load balancing, maintaining load balance in the cluster environment allows the load to be distributed across different resources, preventing overuse of certain resources that could lead to system performance degradation, thereby improving the overall system efficiency and stability. Therefore, the task scheduling model designed in this invention aims to reduce task completion time and maintain load balance in the cluster environment as optimization objectives, thus achieving multi-objective optimization of task scheduling. The essence of task scheduling is to allocate tasks to suitable virtual machines for processing. However, large cluster sizes lead to a huge solution space for task scheduling. Faced with a large solution space, task scheduling algorithms struggle to make reasonable scheduling decisions, thus impacting performance. Therefore, to simplify the task scheduling problem, this invention's task scheduling model partitions the virtual machine cluster into several equally sized virtual machine partitions and divides task scheduling into two phases: task allocation and task scheduling. See the overall framework diagram of this invention's task scheduling algorithm. Figure 2 .

[0035] Specifically, S1 includes: The task's resource requirements, length, and waiting time are normalized and positively evaluated to obtain the task's priority.

[0036] Specifically, the waiting time of tasks is affected by their scheduling order, which in turn affects the entire task scheduling process. The waiting time of a task is influenced by factors such as its resource requirements and length. Therefore, this invention addresses this by determining the resource requirements of tasks before scheduling. ,length and waiting time The method of comprehensive evaluation involves positiveizing and normalizing the three indicators to obtain the task priority.

[0037] In one instance, the priority is calculated as follows: ， By adjusting the weight parameters , and The size of the variable can change the impact of three indicators—task resource requirements, length, and waiting time—on task scheduling priority, thereby selecting the highest priority task for scheduling.

[0038] In the task scheduling model, the virtual machine cluster is divided into several virtual machine partitions of the same size. Task scheduling is divided into two phases: task allocation and task scheduling. These two phases work together to achieve the overall optimization goal. Therefore, the optimization goal of the task allocation phase is to reduce the load imbalance between partitions, thereby further maintaining the load balance of the entire cluster environment.

[0039] In one implementation, the Markov decision process for partition selection in S2 includes a state space, an action space, and a reward function. The state space includes the CPU and memory resource requirements of the currently scheduled task, as well as the CPU and memory resource utilization rates of each partition. The action space is the number of partitions in the entire virtual machine cluster environment and consists of a set of actions output by the first deep reinforcement learning agent. The reward function is obtained based on the degree of change in the standard deviation of CPU and memory resource utilization rates in each partition.

[0040] Specifically, when selecting a partition, the actions of a deep reinforcement learning agent correspond to partitions in the cluster, thus its action space is discrete, typically solved using value-function-based deep reinforcement learning algorithms. Currently, the most commonly used value-function-based deep reinforcement learning algorithm is DQN. However, in DQN, both the selection of the target action and the evaluation of the target Q-value are performed in the target value network, leading to overestimation of the Q-value. To address this, Double DQN decouples action selection from Q-value evaluation, selecting actions in the current value network and evaluating them in the target value network, improving the stability of the network model. Simultaneously, when selecting a partition, if the resources in all partitions do not meet the task's resource requirements, the task will fail to schedule regardless of which partition the agent chooses. In this case, the Q-value depends only on the current state of the environment, not on the actions taken by the agent. To solve this problem, Dueling DQN divides the network output into two branches: one representing the state-value function and the other representing the action advantage-value function. These two branches form a dual network, which helps the agent select more advantageous actions during decision evaluation. Therefore, this invention will use the D3QN algorithm, which combines Double DQN and Dueling DQN, to implement partition selection.

[0041] The following model is defined, and the task set is defined as shown below, where N represents the number of tasks in the task queue. This represents the i-th task to be scheduled in the task queue.

[0042]

[0043] The definition of a virtual machine cluster is shown below.

[0044]

[0045] The specific constraints on the virtual machine's CPU and memory resources are shown below. Represents virtual machine Total CPU count Represents virtual machine Total memory, Represents virtual machine The amount of CPU resources currently available, Represents virtual machine The amount of memory resources currently available.

[0046] To apply the D3QN algorithm to the partitioning problem, it needs to be transformed into a Markov decision process. The main components of a Markov decision process include a state space, an action space, and a reward function.

[0047] The state space needs to include the CPU and memory resource requirements of the currently scheduled task, as well as the CPU and memory utilization rates of each partition. The specific state space content is represented as follows. This represents the CPU resources required for the task. The memory resources required for this task and This represents the CPU and memory utilization rates for each partition.

[0048] The size of the action space is equal to the number of partitions in the entire cluster environment, and the action space is composed of the set of actions output by the deep reinforcement learning agent. The specific action space is shown below.

[0049] For the reward function, it is necessary to first calculate the degree of change in the standard deviation of CPU and memory resource utilization, as shown in the following formula. and The standard deviation of CPU resource utilization before and after the change. Used to measure the degree of change in the standard deviation of CPU resource utilization; and The standard deviation of memory resource utilization before and after the change. It is used to measure the degree of change in the standard deviation of memory resource utilization.

[0050]

[0051]

[0052] The reward function is designed using the degree of change in the standard deviation of this resource utilization, where the reward function for CPU load is shown below.

[0053]

[0054] The reward function for memory load is shown below.

[0055]

[0056] Finally, the reward functions for CPU load and memory load are combined, with each weighted at 1, as shown in the following formula. The reward function for a partition selection algorithm that takes into account both CPU load and memory load.

[0057]

[0058] In one implementation, a Markov decision process is executed by a first D3QN agent to assign tasks to corresponding virtual machine partitions, including: Select the task with the highest priority; Get the current status based on the currently selected task and the partition status in the virtual machine cluster; The first D3QN agent generates an action based on the current state; Execute the action and assign the task to the corresponding virtual machine partition.

[0059] In practice, the overall process of the partition selection algorithm is as follows.

[0060]

[0061] In one implementation, the Markov process for task scheduling in S3 includes a state space, an action space, and a reward function. The state space includes the number of virtual machines in the partition, the CPU and memory resources required for task scheduling, and the available CPU and memory resource utilization of the virtual machines. The action space is the set of actions output by the second deep reinforcement learning agent. The reward function is obtained by dividing the minimum possible execution time of the task by the actual completion time of the task.

[0062] Specifically, after task allocation, the task scheduling phase begins. To schedule tasks to suitable virtual machines within a partition, this invention proposes a task scheduling algorithm based on D3QN. This algorithm makes scheduling decisions based on the status information of the task and all virtual machines in the partition, thereby achieving the optimization goal of reducing task completion time and maintaining load balancing within the partition.

[0063] The optimization objective during the task scheduling phase is to reduce task completion time and maintain load balance within the virtual machine partition. For multi-objective optimization problems, a weight value is typically assigned to each optimization objective. This invention sets the weight of task completion time as follows: The load balancer's weight is Therefore, the overall optimization objective of the task scheduling phase is as follows.

[0064]

[0065] in, , represent The completion time of (the i-th task). represent The waiting time represent Execution time, and The standard deviations of CPU utilization and memory utilization of the virtual machine are represented by the following formulas:

[0066]

[0067] in, and Represents virtual machine CPU utilization and average CPU utilization of virtual machines within the partition, and Represents virtual machine The memory utilization rate and the average memory utilization rate of virtual machines within the partition.

[0068] To apply the D3QN algorithm to the task scheduling phase of this invention, the task scheduling phase needs to be transformed into a Markov decision process. The main components of a Markov decision process include a state space, an action space, and a reward function.

[0069] The state space is represented as follows, where k represents task scheduling in the k-th partition, and M represents the number of virtual machines in the partition. and Representative task Required CPU and memory resources and Represents virtual machine Available CPU resource utilization and available memory resource utilization. Because task execution speed is related to CPU resource utilization and MIPS, the virtual machine's MIPS divided by the total CPU resources is calculated as follows: ,…, It also needs to be passed into the state space.

[0070]

[0071] The action space is a set of actions output by the deep reinforcement learning agent. The specific action space is shown below.

[0072]

[0073] Regarding task completion time, the longer the completion time, the smaller the reward value should be. This is typically achieved by designing the reward function using the reciprocal of the completion time. However, different tasks vary significantly in length. Therefore, to eliminate the differences in completion time between tasks, the reward function is designed by dividing the minimum possible execution time of the task by its actual completion time. The specific formula for calculating task execution time is shown below.

[0074]

[0075] Because of the length of the task CPU resource requirements Since this is known, the CPU resources of the scheduled virtual machine determine the task execution time. and execution speed By obtaining all virtual machines Divide by The minimum value of the ratio indicates the minimum execution time of the task. Therefore, the reward function for task completion time is designed as follows, where This indicates the task's completion time, which equals the task's execution time plus the task's waiting time.

[0076]

[0077] To amplify the load imbalance of virtual machine partitions, the average difference between the resource utilization of virtual machines in a partition and the average resource utilization is used as the evaluation criterion for load imbalance in the reward function. The specific formula is shown below, where M represents the number of virtual machines in the partition. and Represents virtual machine CPU resource utilization and memory resource utilization and This represents the average CPU resource utilization and average memory resource utilization within the partition. This indicates the degree of load imbalance of CPU resources in different regions. This indicates the degree of load imbalance of memory resources in different regions.

[0078]

[0079]

[0080] The smaller the load imbalance, the larger the reward value. Considering that the load imbalance value is always less than 1, the reward function is designed by subtracting the load imbalance value from 1, as shown in the formula below. This is the reward function for load balancing.

[0081]

[0082] Finally, the task completion time and load balancing reward functions are combined, with both weighted at 1. The load balancing reward function addresses the question of "which partition the task is assigned to," focusing on global load balancing within a partition. The task completion time reward function addresses the question of "which virtual machine the task is assigned to within a partition," focusing on task completion efficiency and load balancing within the partition. The specific formula is shown below, where... The reward function for load balancing, A reward function for task completion time.

[0083]

[0084] Action exploration strategies can be optimized to some extent; traditional deep reinforcement learning algorithms generally employ... A strategy is used to balance exploration and exploitation, and the specific formula is as follows.

[0085]

[0086] However, the strategies learned by the agent in the early stages of training are not powerful, and they are directly applied... Choosing the action with the highest Q-value based on probability can mislead the agent and reduce exploration efficiency. Meanwhile, in the task scheduling phase of this invention, because the virtual machines in the partition are heterogeneous, the agent, in order to obtain a higher reward value, often tends to choose the faster virtual machine in the early stages of training, neglecting to explore the slower virtual machines. Therefore, to improve the agent's exploration efficiency in the early stages of training, the task scheduling algorithm of this invention considers combining the Upper Bound Belief Algorithm (UCB) with... Strategy combination. The formula for the upper bound confidence algorithm is shown below, where, The Q-value represents the value calculated by the neural network, and N represents the number of actions that have been performed. This indicates the number of times action a is executed.

[0087]

[0088] As the formula shows, the upper bound confidence algorithm considers not only the Q-value of the current state but also a series of historical data when selecting actions. By taking into account the number of times an action has been selected, the probability of selecting actions with a high selection frequency is reduced, while the probability of selecting actions with a low selection frequency is increased. In this way, the agent can explore slower virtual machines more extensively in the early stages of training, effectively balancing exploration and exploitation, and exhibiting good robustness.

[0089] In one implementation, a Markov decision process is executed by a second D3QN agent to assign tasks to corresponding virtual machines, including: Get the current status based on the currently selected task and the status of the selected virtual machine partition; Generate an action based on the current state; The action is executed to assign tasks to the corresponding virtual machines in order to achieve task scheduling.

[0090] Specifically, during task scheduling, the D3QN agent makes scheduling decisions based on the task and the state of the selected virtual machine partition, scheduling the task to the corresponding virtual machine. If the virtual machine meets the task's resource requirements, the task executes successfully; otherwise, scheduling fails. A reward value is obtained based on the scheduling result and fed back to the D3QN agent for training. In summary, the specific implementation flow of this task scheduling algorithm is shown below.

[0091]

[0092] In another aspect, the present invention also provides a cloud computing task scheduling device based on deep reinforcement learning, comprising: Preprocessor; preprocesses tasks to obtain task priorities; The task allocation module is used to select the highest priority task based on the task priority, and transform the partition selection into a Markov decision process. The Markov decision process is executed by the first D3QN agent to allocate the task to the corresponding virtual machine partition. The virtual machine partition is obtained by pre-dividing the virtual machine cluster. The task scheduling module is used to transform task scheduling into a Markov decision process. The second D3QN agent executes the Markov decision process to allocate tasks to the corresponding virtual machines in order to achieve task scheduling.

[0093] Specifically, the cloud computing task scheduling device based on deep reinforcement learning is the scheduling layer.

[0094] On the other hand, the present invention also provides a cloud computing task scheduling system based on deep reinforcement learning, including a cloud computing task scheduling device based on deep reinforcement learning, and further including a user layer and an execution layer. The user layer is used by users to submit tasks on the cloud platform through physical devices. The execution layer includes a virtual machine cluster, in which the virtual machines are pre-divided into several virtual machine partitions of the same size, and the virtual machines in the execution layer are used to execute the assigned tasks.

[0095] Specifically, a cloud computing task scheduling system and task scheduling model based on deep reinforcement learning. The task scheduling model designed in this embodiment is described in [reference]. Figure 1As shown in the figure, the model is mainly divided into a user layer, a scheduling layer, and an execution layer.

[0096] At the user layer, different users submit tasks on the cloud platform through various physical devices. These tasks are then sent to the task queue of the scheduling layer according to their submission order. The scheduling layer is the core of the entire task scheduling model, primarily responsible for scheduling tasks to virtual machines. The execution layer mainly consists of clusters. The virtual machines in the cluster are divided into several virtual machine partitions of the same size. Each virtual machine partition contains multiple heterogeneous virtual machines, each with different execution capabilities, CPU and memory resources.

[0097] To verify the effectiveness of the task scheduling algorithm proposed in this invention, the method of this invention will be further explained below with reference to specific experimental data.

[0098] This experiment uses Alibaba Cluster-trace-v2018, a publicly available real-world cluster tracing dataset from Alibaba Cloud, as the experimental dataset. Alibaba Cluster-trace-v2018 records detailed information about tasks running on 4000 servers in a real Alibaba Cloud production environment over eight days, primarily including server information, container information, and task information. Since this invention studies the task scheduling problem, the batch_task data table describing task information is selected as the experimental dataset, and its specific information is shown in the table below.

[0099]

[0100] This experiment was mainly conducted on a personal computer on a Python-based cloud task scheduling experimental platform. The specific experimental environment is shown in the table below.

[0101]

[0102] This experiment filtered the task data based on the task length and the number of tasks submitted simultaneously, selecting a total of 3 task datasets, the specific dataset configurations of which are shown in the table below.

[0103]

[0104] Considering that the task scheduling order affects the task scheduling result, this invention performs sorting preprocessing on the tasks in the task queue before scheduling them to the virtual machine. The scheduling priority of tasks is obtained by comprehensively evaluating their resource requirements, length, and waiting time, thereby determining the task scheduling order and reducing task waiting time. To verify the effectiveness of the sorting preprocessing in this invention, this experiment preprocesses tasks using the sorting rules proposed in this paper and the first-come, first-served (FFS) approach before scheduling. Then, the Round Robin algorithm is used uniformly for task scheduling, and a comparative experiment is conducted. In the sorting rules, the values ​​of parameters w1, w2, and w3 are 0.5, 0.25, and 0.25, respectively. Experimental results are shown below. Figure 3 .

[0105] The optimization goal of the task allocation phase is to reduce the load imbalance between partitions, thereby maintaining the overall load balance of the cluster environment. To verify the effectiveness of the algorithm designed in this invention, this experiment compared the algorithm with the Random and RoundRobin algorithms. The experimental results are shown in [link to experimental results]. Figure 4 and Figure 5 .

[0106] This invention proposes a task scheduling algorithm, D3QNTS, based on D3QN, for the task scheduling phase. Because the task allocation phase affects the task scheduling result, this experiment compares the algorithm's training performance and average task completion time to verify its effectiveness. The experimental results are shown below. Figure 6 and Figure 7 .

[0107] To simplify the task scheduling problem, the task scheduling model designed in this invention decomposes task scheduling into two stages: task allocation and task scheduling. Previous experiments have verified the effectiveness of the algorithms for each stage. Therefore, this experiment mainly compares the two-stage task scheduling algorithm 2stage-D3QNTS with single-stage algorithms such as D3QNTS, Random, Round Robin, NextFit, and MinMin to verify the superiority of the two-stage task scheduling algorithm over the single-stage algorithm. Experimental results are shown in [link to experimental results]. Figure 8 , Figure 9 and Figure 10 .

[0108] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0109] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0110] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A cloud computing task scheduling method based on deep reinforcement learning, characterized in that, include: Preprocess the tasks to obtain their priorities; The task with the highest priority is selected based on the task priority. The partition selection is transformed into a Markov decision process. The Markov decision process is executed by the first D3QN agent to allocate the task to the corresponding virtual machine partition. The virtual machine partition is obtained by pre-dividing the virtual machine cluster. Task scheduling is transformed into a Markov decision process, and the second D3QN agent executes the Markov decision process to allocate tasks to the corresponding virtual machines in order to achieve task scheduling.

2. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Preprocess the tasks to obtain their priorities, including: The task's resource requirements, length, and waiting time are normalized and positively evaluated to obtain the task's priority.

3. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The Markov decision process for partition selection includes a state space, an action space, and a reward function. The state space includes the CPU and memory resource requirements of the currently scheduled task, as well as the CPU and memory resource utilization rates of each partition. The action space is the number of partitions in the entire virtual machine cluster environment and consists of the set of output actions of the first deep reinforcement learning agent. The reward function is obtained based on the degree of change in the standard deviation of CPU and memory resource utilization in each partition.

4. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 3, characterized in that, The reward function is calculated as follows: in, ， In the formula, The reward function chosen for the partition. The reward function for CPU load. The reward function for memory load. Used to measure the degree of change in the standard deviation of CPU resource utilization. It is used to measure the degree of change in the standard deviation of memory resource utilization.

5. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 3, characterized in that, The first D3QN agent executes a Markov decision process to assign tasks to the corresponding virtual machine partitions, including: Select the task with the highest priority; Get the current status based on the currently selected task and the partition status in the virtual machine cluster; The first D3QN agent generates an action based on the current state; Execute the action and assign the task to the corresponding virtual machine partition.

6. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The Markov process for task scheduling includes a state space, an action space, and a reward function. The state space includes the number of virtual machines in the partition, the CPU and memory resources required for task scheduling, and the available CPU and memory resource utilization of the virtual machines. The action space is the set of output actions of the second deep reinforcement learning agent. The reward function is obtained by dividing the minimum possible execution time of the task by the actual completion time of the task.

7. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Reward function for task scheduling The calculation method is as follows: in, In the formula, The reward function for load balancing, A reward function for task completion time. The degree of CPU resource load imbalance. This refers to the uneven load distribution of memory resources. The minimum execution time for the task. This refers to the actual completion time of the task.

8. The cloud computing task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The task is assigned to the corresponding virtual machine by executing a Markov decision process through the second D3QN agent, including: Get the current status based on the currently selected task and the status of the selected virtual machine partition; Generate an action based on the current state; The action is executed to assign tasks to the corresponding virtual machines in order to achieve task scheduling.

9. A cloud computing task scheduling device based on deep reinforcement learning, characterized in that, include: Preprocessor; Preprocess the tasks to obtain their priorities; The task allocation module is used to select the highest priority task based on the task priority, and transform the partition selection into a Markov decision process. The Markov decision process is executed by the first D3QN agent to allocate the task to the corresponding virtual machine partition. The virtual machine partition is obtained by pre-dividing the virtual machine cluster. The task scheduling module is used to transform task scheduling into a Markov decision process. The second D3QN agent executes the Markov decision process to allocate tasks to the corresponding virtual machines in order to achieve task scheduling.

10. A cloud computing task scheduling system based on deep reinforcement learning, characterized in that, The cloud computing task scheduling device based on deep reinforcement learning as described in claim 9 further includes a user layer and an execution layer. The user layer is used by users to submit tasks on the cloud platform through physical devices. The execution layer includes a virtual machine cluster, in which the virtual machines are pre-divided into several virtual machine partitions of the same size. The virtual machines in the execution layer are used to execute the assigned tasks.

Images