Intelligent cloud data center task scheduling method based on double-agent deep reinforcement learning, electronic equipment and medium

Abstract

The invention discloses a cloud data center task scheduling intelligent method based on double-agent deep reinforcement learning, electronic equipment and a medium, belongs to the field of cloud data center task scheduling, and aims at solving the problems that a traditional method is tedious in design and difficult to adapt to a dynamic cloud computing environment, and the dimension of an existing deep reinforcement learning model is limited by data center configuration. The method is characterized in that the agent scheduling model is updated; wherein the agent scheduling model comprises a scheduler, and the scheduler comprises a host selection agent and a task selection agent; the updated agent scheduling model responds to a host list and a task list and outputs a target host and a target task in the host list and the task list; the target host executes the target task, the effect is that configuration attributes and a training method of the data center are met, a better effect can be achieved, and after model training is completed, the method can be applied to data centers with different configurations without modification. According to the method, the advantages of TD3 are utilized, so that tasks can be efficiently scheduled, and the training overhead can be saved.

Description

Technical Field

[0001] The present invention belongs to the field of cloud data center task scheduling, and relates to an intelligent method for cloud data center task scheduling based on dual-agent deep reinforcement learning. Background Art

[0002] With the development of the information industry, many computing-intensive tasks have become an indispensable part of people's daily lives, and users' demand for computing power is increasing day by day. Cloud computing has become the choice of more and more users to solve the problem of insufficient computing power. As an important part of cloud computing, the abundant computing resources of cloud data centers must be used efficiently to meet the needs of users. Task scheduling strategies are the key to the rational use of data center computing resources. Different scheduling strategies will significantly affect the experience of users and operators. For users, a good strategy can shorten the completion time and waiting time of tasks; for operators, a good strategy can increase the efficiency of resource use and reduce energy waste. Usually, there are many heterogeneous hosts in cloud data centers, and each host contains limited computing resources. The task scheduling strategy selects the appropriate host-task pair based on the resource requirements of the task and the operating status of the host, and assigns the host to the task. In theory, the task scheduling problem is NP-hard.

[0003] To address this problem, many studies have designed many different cloud data center scheduling strategies to optimize various objectives. Traditionally, researchers design scheduling strategies based on heuristic methods such as First Come First Serve (FCFS), Shortest Job First (SJF), and Highest Response Ratio First (HRRF). In addition, many researchers design scheduling strategies based on meta-heuristic methods such as Particle Swarm Optimization (PSO), Genetic Algorithm (GA), Ant Colony Optimization (ACO), Lion Optimization Algorithm (LOA), and Firefly Algorithm (FA). However, these methods are usually very complex and require manual adjustments based on prior knowledge, which brings a huge burden to researchers; moreover, these methods cannot adapt to the dynamic changes of the environment and load in cloud computing, and usually obtain suboptimal results.

[0004] In recent years, the development of technologies such as deep reinforcement learning has attracted a lot of attention. In reinforcement learning, the intelligent agent in the algorithm interacts with the environment to maximize the cumulative reward, thereby improving its strategy and optimizing the actions performed. Many deep reinforcement learning methods have solved the dimensionality explosion problem and the problem of not being able to handle continuous space in traditional reinforcement learning to a certain extent by combining deep neural networks. Common deep reinforcement learning algorithms include Deep Q-Learning (DQN), Deep Deterministic Policy Gradient (DDPG), Twin Delayed Deep Deterministic Policy Gradient (TD3), etc. The emergence of these algorithms has enabled deep reinforcement learning to be applied in more fields, including video games, computer go, data center scheduling, etc.

[0005] As one of the typical algorithms, DDPG has the ability to process continuous value action space that is lacking in traditional reinforcement learning, and is widely used in fields such as continuous control. However, the algorithm still has the problem of overestimating Q value, and DDPG is considered to be sensitive to parameter settings and initialization conditions, so there is still room for optimization. TD3, as an improved version of DDPG, solves this problem through truncated double Q learning, delayed policy update, target policy smoothing, etc. The TD3 algorithm is a different policy algorithm. In the algorithm, there are actors, critics, and replay-buffers. Among them, the actor contains an evaluation network and a target network for selecting actions; the critic contains two evaluation networks and two target networks for evaluating the quality of actions; the replay buffer stores the agent's past actions, states, rewards, and other experiences, and uses experience replay to train the model parameters. In addition, by combining the model update method of soft-update, this method has good stability and convergence performance.

[0006] However, many current scheduling strategies based on deep reinforcement learning are designed for specific data center configurations. For example, the state or action space of the model is constructed based on the length of the waiting queue or the number of hosts in the data center to meet the requirements of deep neural networks for fixed input dimensions. When these configuration parameters change dramatically, these models may face severe performance degradation or even become difficult to use, and must be redesigned and retrained, a process that consumes huge resources. Summary of the invention

[0007] In order to overcome the problems that the traditional method is cumbersome in design, difficult to adapt to the dynamic cloud computing environment, and the existing deep reinforcement learning model dimension is limited by the data center configuration, in a first aspect, according to some embodiments of the present application, the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning includes:

[0008] Updating the agent scheduling model; wherein the agent scheduling model includes a scheduler, and the scheduler includes a host selection agent and a task selection agent;

[0009] The updated agent scheduling model responds to the host list and the task list and outputs the target host and the target task therein;

[0010] The target task is executed by the target host.

[0011] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the input features of the host selection agent include:

[0012] Host characteristics F of all hosts in the host list ts (H), where any host h j Host characteristics F ts (h j );

[0013] Task characteristics of all tasks in the task list F ts (Q), where any task t k,i Task characteristics F ts (t k,i );

[0014] Among them, the input features of the task selection agent include:

[0015] Target host h T The target host characteristics F t ts (h T );

[0016] Task Characteristics

[0017] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, wherein:

[0018] In the formula, c CPU (h j ) indicates that any host h j CPU capacity, c MEM (h j ) indicates that any host h jMemory capacity, Indicates that any host h j The current CPU utilization, Indicates that any host h j The current memory utilization; h M Indicates the Mth host;

[0019] in:

[0020] In the formula, r CPU (t k,i ) represents any task t k,i CPU requirements, r MEM (t k,i ) represents any task t k,i Memory requirements, CP k,i Represents any task t k,i The current completion percentage of Represents any task t k,i The completion time of the current instance; Indicates tasks;

[0021] in:

[0022] In the formula, Indicates the target host h T The current CPU utilization, Indicates the target host h T The current memory utilization;

[0023] in:

[0024] Where: Represents any task t k,i The CPU demand of the target host h T The ratio of CPU capacity, Represents any task t k,i The memory requirement of the target host is h T The ratio of memory capacity, CP k,i Represents any task t k,i The current completion percentage of Represents any task t k,i The completion time of the current instance.

[0025] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the step of updating the agent scheduling model includes:

[0026] The host selection agent responds to the input features and outputs any host h j Host characteristics Task Characteristics

[0027] The host characteristics Task Characteristics The average value of the task characteristics Splicing is performed to obtain any host h j Host status

[0028] According to any host h j Host status For any of the hosts h j Scoring, the host selects the action of the agent Among them, any host h with the highest score j The target host h T ;

[0029] The task selection agent responds to the input feature and outputs the target host feature of the target host Any task k,i Task characteristics

[0030] The target host characteristics Task Characteristics Splice and get the target host h T Any of the tasks t k,i Task status

[0031] According to the target host h T Any of the tasks t k,i Task status For any of the tasks t k,i Scoring, the task selects the action of the agent Among them, any task t with the highest score k,i The target task t T ;

[0032] According to the target host h T Create the target task t T After that, the target host h T CPU utilization With the target host h T Memory utilization Calculate the target host h T With the target task t T The matching coefficient f;

[0033] According to the matching coefficient f, the host selection agent calculates and selects the target host h T Rewards Select the target task t for the task selection agent computation T Rewards

[0034] According to the above steps, get the target host h for the next time step T Host status The experience of selecting the agent Stored to host experience replay buffer pool B H Among them, Indicates the target host h at the current time step T The host status, Indicates the target host h at the current time step T Actions, Indicates the target host h at the current time step T Rewards, represents the target host h in the next time step T Host status;

[0035] According to the above steps, the target task t of the next time step is obtained. T Task status The experience of the agent selected for the task Stored in task experience replay buffer pool B T Among them, represents the target task t at the current time step T The task status, represents the target task t at the current time step T Actions, represents the target task t at the current time step T Rewards, represents the target task t at the next time step T The task status;

[0036] Replay buffer pool B based on the host experience of the host selection agent H Extracting the experience Update the host selection agent; and replay the buffer pool B according to the task experience of the task selection agent T Extracting the experience Update the task selection agent.

[0037] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the step of Task Characteristics The average value of the task characteristics Splicing is performed to obtain any host h j Host status It is expressed as:

[0038]

[0039]

[0040] Wherein, the steps are based on any host h j Host status For any of the hosts h j Scoring, the host selects the action of the agent It is expressed as:

[0041]

[0042]

[0043] Score represents the scoring module;

[0044] Among them, the step is to set the target host characteristics Task Characteristics Splice and get the target host h T Any of the tasks t k,i Task status It is expressed as:

[0045]

[0046]

[0047] Among them, the steps are based on the target host h T Any of the tasks t k,i Task status For any of the tasks t k,i Scoring, the task selects the action of the agent It is expressed as:

[0048]

[0049] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the step is based on the target host h T Create the target task t T After that, the target host h T CPU utilization With the target host h T Memory utilization Calculate the target host hT With the target task t T The matching coefficient f is expressed as:

[0050]

[0051] Among them, the step is to select the target host h for the host selection agent calculation based on the matching coefficient f. T Rewards Select the target task t for the task selection agent computation T Rewards It is expressed as:

[0052]

[0053]

[0054] In the formula, Represents the target task t T The completion time of the current instance;

[0055] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the step replays the buffer pool B of the host experience of the host selection agent according to the host H Extracting the experience Update the host selection agent; and replay the buffer pool B according to the task experience of the task selection agent T Extracting the experience Updating the task selection agent, including:

[0056] Alternately update the parameters of the host selection agent and the task selection agent; wherein the update includes updating the evaluation network in the critic and and the target network and Update the evaluation network π in the actor φ and the target network π φ′ ;

[0057] Combined with the learning rate α, the evaluation network is updated by minimizing the TD-error and The loss function is expressed as:

[0058]

[0059] In the formula, Represents the evaluation network Or evaluate the network MSE represents the loss function based on mean square error; Indicates the target network or s ts is the state of the current time step, indicating the host state of the target host at the current time step or the task state of the target task at the current time step; a ts is the action of the current time step, indicating the action of the target host in the previous time step or the action of the target task in the current time step; ts Represents the updated evaluation network and The target value of the parameter;

[0060]

[0061] In the formula, r ts is the reward of the current time step, which indicates the reward of the target host at the current time step or the reward of the target task at the current time step; γ indicates the discount factor; π φ Represents the evaluation network π in the actor φ ;s ts+1 is the state of the next time step, indicating the host state of the target host in the next time step or the task state of the target task in the next time step;

[0062] In Update and After K times, the evaluation network is used Combined with the learning rate, the policy gradient method is used to update the evaluation network π in the actor. φ Once, it is expressed as:

[0063]

[0064] represents the gradient;

[0065] In updating the evaluation network π φ After K times, the target network π is updated using soft update φ′ , and Once, the weight adjustment parameter τ is used in the soft update, expressed as:

[0066] θ i ′←——τθ i +(1-τ)θ i ′,i=1,2

[0067] φ′←——τφ+(1-τ)φ′

[0068] In the formula, θ i Represents the evaluation network The parameter θ i ′ represents the target network Parameters of , φ represents the evaluation network π φ , φ′ represents the target network π φ′ Parameters.

[0069] According to the cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning in some embodiments of the present application, the above steps of updating the agent scheduling model are repeated until all tasks in the current time step are completed or a suitable host-task pair cannot be selected.

[0070] In the second aspect, according to the electronic device in the embodiment of the present application, the electronic device includes: one or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, and when the instructions are executed by the electronic device, the electronic device executes any one of the methods described in the first aspect.

[0071] In the second aspect, according to the computer-readable storage medium in the embodiment of the present application, the computer-readable storage medium includes a computer program, and when the computer program is run on an electronic device, the electronic device executes any one of the methods described in the first aspect.

[0072] The beneficial effects of the present invention are as follows: The present invention schedules data center tasks and designs a dual-agent host task selection method based on the TD3 algorithm in view of the highly dynamic characteristics of data center configuration and load tasks. This method can efficiently train a model with good performance and efficiently schedule task instances in the data center, providing a theoretical basis and practical experience for the application of deep reinforcement learning technology in the field of cloud data center task scheduling.

[0073] Compared with other scheduling strategies, the design of the present invention is in line with the configuration attributes and training methods of the data center, and can achieve better results. After the model training is completed, it can be applied in data centers with different configurations without modification. The present invention utilizes the advantages of TD3 to efficiently schedule tasks and save training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 It is a training flow chart provided by the present invention.

[0075] Figure 2 It is a structural diagram of the present invention.

[0076] Figure 3 It is a flow chart of the agent updating algorithm of the present invention.

[0077] Figure 4 It is a convergence performance diagram of the method of the present invention. DETAILED DESCRIPTION

[0078] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the specific implementation modes of the present invention will be described in detail below with reference to the accompanying drawings.

[0079] This paper proposes a cloud data center task scheduling intelligent method (DAHTS) based on dual-agent deep reinforcement learning. The training flow chart is as follows: Figure 1 As shown in the figure, the task scheduling problem is decomposed into two sub-problems: the host selection problem and the task selection problem. An agent is designed for each sub-problem, namely the host selection agent HSA and the task selection agent TSA, so as to select the appropriate host-task pair by combining deep reinforcement learning and other technologies. The states of the host and the task are used as input, and the output score is used as the action of the agent to select the host and task with the highest score respectively. The overall framework is as follows: Figure 2 As shown in the figure, all agents are trained using TD3. In each agent, there is a target network corresponding to an evaluation network in the actor, two evaluation networks and corresponding target networks in the critic, and an experience replay buffer pool. This method provides a theoretical basis and practical experience for the application of deep reinforcement learning technology in the field of cloud data center scheduling.

[0080] Specifically, in one embodiment, the present invention uses the selected cluster data center dataset AlibabaCluster Data V2017 to construct a task sequence and build a simulated data center environment. The task sequence is submitted to the simulated data center in sequence according to the submission time. The simulated data center schedules tasks according to a scheduler composed of different strategies. After all instances of all tasks are executed, the experimental results are calculated. Figure 1 As shown, the present invention provides an intelligent method for cloud data center task scheduling based on dual-agent deep reinforcement learning, which specifically includes the following steps:

[0081] Step 1: Set the parameters related to the TD3 algorithm; use random numbers to initialize the evaluation network of the actor in the host selection agent TSA With the target network Critics’ evaluation network and With the target network and Set up experience replay buffer pool B H The size is 10000; use random numbers to initialize the task to select the evaluation network and target network of the actor in the agent TSA Critics’ evaluation network and With the target network and Set up experience replay buffer pool BT The size is 10000; all networks are set to contain two hidden layers, each containing 64 neurons; the preset number of iterations EPS is set max is 150.

[0082] Step 2: Extract a continuous job sequence of length N = 15 from the selected data set, sort it by submission time from small to large, divide it into several time steps, and construct a host list H = {h1,h2,…,h j ,…,h M} simulated training environment, setting M=10.

[0083] Step 3: For any time step ts, the waiting queue contains Task, collect the characteristics F of all hosts in the data center in the current time step ts (H), the characteristics of all tasks in the task waiting queue F ts (Q).

[0084] Step 4: Based on the host feature F collected at the current time step ts (H) and task characteristics F ts (Q) Calculate the current status of the host queue and task waiting queue using the host selection agent HSA and the task selection agent TSA and Generate scores for hosts and tasks as actions of agents and Select the target host h with the highest score T 、Target task T Generate host-task pairs; compute host selection agent selects h T Action Rewards And get the h T Next state Store it in the experience replay buffer pool; the computing task selection agent selects the target task t T Action Rewards And get t T The next state Store it in the experience replay buffer pool; where h T is a host in H, t T is a task in Q; update the parameters of the host selection agent HSA and the task selection agent TSA alternately; repeat the current step until all tasks in the current time step are completed, or no suitable host-task pair can be selected.

[0085] Step 5: Repeat steps 3 and 4 until all tasks in step 2 are completed.

[0086] Step 6: Repeat steps 2 to 5 until the preset number of iterations is reached.

[0087] Step seven: Save the iterated model parameters as the final model parameters.

[0088] In the step 2, for the job list J={j1,…,j k ,…,j N Any job j in} k , which contains n k tasks; for j k Any task t in k,i , which requires the creation of Instances are executed, belonging to the same task t k,i The instances have the same resource requirements and duration d k,i ;

[0089] In step three, the specific steps are as follows:

[0090] 3.1 Collect all host features in the current time step ts; for any host h in H j ,The collected features include the host's CPU and memory capacity,c CPU (h j ) and c MEM (h j ), the host's current CPU and memory utilization, and Host characteristics are:

[0091]

[0092] F ts (H) = {F ts (h1),…,F ts (h j ),…,F ts (h M )}

[0093] 3.2 Collect all task features in the current time step ts; for any task in the task waiting queue, the collected features include the CPU and memory requirements of the task, and Since the task needs to create multiple identical instances, the number of completed instances can be calculated based on the current time. The total number of instances Calculate the current completion percentage of the task Similarly, we can use the duration d of the task k,i , and the submission time of the task Calculates the completion time of the current instance of the task Task characteristics are:

[0094]

[0095]

[0096] In the step 4, the specific steps are as follows:

[0097] 4.1 Host Selection Agent HSA calculates the host state F based on the collected features ts (H); For any host h in the host list j First, the feature extraction module is used to further extract host features. The features used include the host feature F in step 3.1. ts (h j ), the task feature F in step 3.2 ts (Q); Then, for each host and each task, the feature extraction module is used to further extract their features to obtain the extracted host features And calculate the average value of all extracted task features Finally, the extracted host features and the average value of all task features are concatenated as the host status, that is:

[0098]

[0099]

[0100] 4.2 According to the host status obtained in step 4.1, score each host as the action of the host selection agent HSA, and combine the noise added by TD3 to the action to select the host with the highest score as the target host; specifically, the action of the host selection agent HSA is recorded as:

[0101]

[0102]

[0103] 4.3 Task Selection Agent TSA calculates the task status based on the target host; for any task t in the task list k,i First, use the target host h T Some of the characteristics of The task characteristics calculated based on the target host include: the ratio of the task's CPU demand to the host's CPU capacity, the ratio of the task's memory demand to the host's memory capacity, the current completion percentage of the task, and the completion time of the current instance of the task, i.e. The two parts of features are concatenated and the task status is calculated using the feature extraction module, namely:

[0104]

[0105]

[0106] 4.4 Score each task based on the task status obtained in step 4.3, and use the noise added by TD3 as the action of the task selection agent TSA, and select the task with the highest score as the target task t T ; Thus, according to step 4.2, the target host h T The target task t obtained in this step T , generating a target host-target task pair; specifically, the action of the task selection agent TSA is recorded as:

[0107]

[0108]

[0109] 4.5 Utilize the CPU utilization after creating a target task on the target host and memory utilization Calculate the matching degree between the target host and the target task, and obtain the matching coefficient f, that is:

[0110]

[0111] 4.6 Based on the matching coefficient f, the HSA agent is selected for the host to calculate the target host h T Rewards Among them, the reward of the host selecting the intelligent agent HSA is recorded as

[0112] 4.7 Based on the matching coefficient f, the task selection agent TSA calculates and selects the target task t T Rewards Among them, the reward of the task selection agent TSA is recorded as

[0113] 4.8 Get the next state And the host selects the experience of intelligent agent HSA Stored to host experience replay buffer pool B H middle.

[0114] 4.9 Get the next state And the experience of the task selection agent TSA Stored in task experience replay buffer pool B T middle.

[0115] 4.10 Use the TD3 algorithm to alternately update the parameters of the host selection agent and the task selection agent. The TD3 algorithm update process is mainly divided into three parts, namely: updating the evaluation network in the critic, updating the evaluation network in the actor, and updating the target network. In TD3, there are two evaluation networks in the critic. and Based on their parameters, the state and action evaluation strategies are calculated with a discount factor γ = 0.9 to update the target value y for their own parameters. ts , written as:

[0116]

[0117] TD3 combined with learning rate α = 1e-5 is updated by minimizing TD-error and Using MSE as the loss function, it is written as:

[0118]

[0119] TD3 is updating and After K=2 times, update the evaluation network π in the actor φ Once. TD3 exploit To update π φ , combined with the learning rate α = 1e-5 using the policy gradient method, denoted as:

[0120]

[0121] TD3 is updating and After K times, the target network is updated using soft-update, π φ′ , and Once, a weight adjustment parameter τ = 0.005 was used in the soft-update update to enhance the stability of learning, denoted as

[0122] θ i′ ←——τθ i +(1-τ)θ i ′,i=1,2

[0123] φ′←——τφ+(1-τ)φ′

[0124] The update process of TD3 algorithm is as follows: Figure 3 shown.

[0125] 4.11 Repeat steps 4.1-4.10 until all tasks in the current time step are completed, or no suitable host-task pair can be selected.

[0126] Figure 4 To evaluate the convergence of the model, a model environment with 15 jobs and 10 hosts is used for training. The reward values ​​of the agents trained with TD3 and those trained with DDPG can be compared. Figure 4 As can be seen in Figure 2, after 150 cycles, the reward values ​​of both methods converge.

[0127] The performance indicators evaluated include average waiting time AW, average completion time AC and average speed reduction ratio AS. k,i , its waiting time Refers to submission time The time when the first instance was created The average waiting time AW refers to the sum of the waiting times of all tasks divided by the total number of tasks and is defined as:

[0128]

[0129]

[0130] For any task t k,i , its completion time Refers to submission time The completion time of the last instance The average completion time AC refers to the sum of the completion time of all tasks divided by the total number of tasks, which is defined as:

[0131]

[0132]

[0133] For any task t k,i , its reduction ratio SL k,i refer to With d k,i The average reduction ratio AS refers to the sum of the reduction ratios of all tasks divided by the total number of tasks, and is defined as:

[0134]

[0135]

[0136] Through the above steps, the trained model is obtained, and the performance of the model is evaluated through experiments. The comparison methods include the DAHTS method trained with TD3, first come first served FCFS, short job priority SJF, high response ratio priority HRRN, and the method DAHTS-DDPG trained with the DDPG algorithm. In addition, the test set is constructed using the selected data set. The simulation experiment constructs a data center with 40 hosts. Five continuous job lists with a length of 30 are randomly selected from the selected data set, named JS-1, JS-2, JS-3, JS-4, and JS-5 respectively. The experimental results are shown in Table 1. It can be seen from Table 1 that DAHTS is slightly inferior to DAHTS-DDPG in terms of average waiting time AW, but it is better than other methods in terms of average completion time AC and average speed reduction ratio AS, which verifies the effectiveness of this method.

[0137] Table 1 Comparative experimental results

[0138]

[0139] The present invention schedules data center tasks and designs a dual-agent host task selection method based on the TD3 algorithm in view of the highly dynamic characteristics of data center configuration and load tasks. This method can efficiently train a model with good performance and efficiently schedule task instances in the data center, providing a theoretical basis and practical experience for the application of deep reinforcement learning technology in the field of cloud data center task scheduling.

[0140] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning, characterized in that: include: Updating the agent scheduling model; wherein the agent scheduling model includes a scheduler, and the scheduler includes a host selection agent and a task selection agent; The updated agent scheduling model responds to the host list and the task list and outputs the target host and the target task therein; The target task is executed by the target host.

2. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 1 is characterized in that: in, The input features of the host selection agent include: Host characteristics F of all hosts in the host list ts (H), where any host h j Host characteristics F ts (h j ); Task characteristics of all tasks in the task list F ts (Q), where any task t k,i Task characteristics F ts (t k,i ); Among them, the input features of the task selection agent include: Target host h T Target host characteristics Task Characteristics 3. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 2 is characterized in that: in: In the formula, c CPU (h j ) indicates that any host h j CPU capacity, c MEM (h j ) indicates that any host h j Memory capacity, Indicates that any host h j The current CPU utilization, Indicates that any host h j The current memory utilization; h M Indicates the Mth host; in: In the formula, r CPU (t k,i ) represents any task t k,i CPU requirements, r MEM (t k,i ) represents any task t k,i Memory requirements, CP k,i Represents any task t k,i The current completion percentage of Represents any task t k,i The completion time of the current instance; Indicates tasks; in: In the formula, Indicates the target host h T The current CPU utilization, Indicates the target host h T The current memory utilization; in: Where: Represents any task t k,i The CPU demand of the target host h T The ratio of CPU capacity, Represents any task t k,i The memory requirement of the target host is h T The ratio of memory capacity, CP k,i Represents any task t k,i The current completion percentage of Represents any task t k,i The completion time of the current instance.

4. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 2 or 3 is characterized in that: The step of updating the agent scheduling model comprises: The host selection agent responds to the input features and outputs any host h j Host characteristics Task Characteristics The host characteristics Task Characteristics The average value of the task characteristics Splicing is performed to obtain any host h j Host status According to any host h j Host status For any of the hosts h j Scoring, the host selects the action of the agent Among them, any host h with the highest score j The target host h T ; The task selection agent responds to the input feature and outputs the target host feature of the target host Any task k,i Task characteristics The target host characteristics Task Characteristics Splice and get the target host h T Any of the tasks t k,i Task status According to the target host h T Any of the tasks t k,i Task status For any of the tasks t k,i Scoring, the task selects the action of the agent Among them, any task t with the highest score k,i The target task t T ; According to the target host h T Create the target task t T After that, the target host h T CPU utilization With the target host h T Memory utilization Calculate the target host h T With the target task t T The matching coefficient f; According to the matching coefficient f, the host selection agent calculates and selects the target host h T Rewards Select the target task t for the task selection agent computation T Rewards According to the above steps, get the target host h for the next time step T Host status The experience of selecting the agent Stored to host experience replay buffer pool B H Among them, Indicates the target host h at the current time step T The host status, Indicates the target host h at the current time step T Actions, Indicates the target host h at the current time step T Rewards, represents the target host h in the next time step T Host status; According to the above steps, the target task t of the next time step is obtained. T Task status The experience of the agent selected for the task Stored in task experience replay buffer pool B T Among them, represents the target task t at the current time step T The task status, represents the target task t at the current time step T Actions, represents the target task t at the current time step T Rewards, represents the target task t at the next time step T The task status; Replay buffer pool B based on the host experience of the host selection agent H Extracting the experience Update the host selection agent; and replay the buffer pool B according to the task experience of the task selection agent T Extracting the experience Update the task selection agent.

5. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 4 is characterized in that: in, Step 1: Set the host characteristics Task Characteristics The average value of the task characteristics Splicing is performed to obtain any host h j Host status It is expressed as: Wherein, the steps are based on any host h j Host status For any of the hosts h j Scoring, the host selects the action of the agent It is expressed as: Score represents the scoring module; Among them, the step is to set the target host characteristics Task Characteristics Splice and get the target host h T Any of the tasks t k,i Task status It is expressed as: Among them, the steps are based on the target host h T Any of the tasks t k,i Task status For any of the tasks t k,i Scoring, the task selects the action of the agent It is expressed as:

6. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 4 is characterized in that: in, Steps according to the target host h T Create the target task t T After that, the target host h T CPU utilization With the target host h T Memory utilization Calculate the target host h T With the target task t T The matching coefficient f is expressed as: Among them, the step is to select the target host h for the host selection agent calculation based on the matching coefficient f. T Rewards Select the target task t for the task selection agent computation T Rewards It is expressed as: In the formula, Represents the target task t T The completion time of the current instance.

7. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to any one of claims 4 to 6, characterized in that: in, Step 1: Select the agent based on the experience of the host Updating the host selection agent; and selecting the agent based on the experience Updating the task selection agent, including: Alternately update the parameters of the host selection agent and the task selection agent; wherein the update includes updating the evaluation network in the critic and and the target network and Update the evaluation network π in the actor φ and the target network π φ′ ; Combined with the learning rate α, the evaluation network is updated by minimizing the TD-error and , the loss function is expressed as: In the formula, Represents the evaluation network Or evaluate the network MSE represents the loss function based on mean square error; Indicates the target network or s ts is the state of the current time step, indicating the host state of the target host at the current time step or the task state of the target task at the current time step; a ts is the action of the current time step, indicating the action of the target host in the previous time step or the action of the target task in the current time step; ts Represents the updated evaluation network and The target value of the parameter; In the formula, r ts is the reward of the current time step, which indicates the reward of the target host at the current time step or the reward of the target task at the current time step; γ indicates the discount factor; π φ Represents the evaluation network π in the actor φ ;s ts+1 is the state of the next time step, indicating the host state of the target host in the next time step or the task state of the target task in the next time step; In Update and After K times, the evaluation network is used Combined with the learning rate, the policy gradient method is used to update the evaluation network π in the actor. φ Once, it is expressed as: represents the gradient; In updating the evaluation network π φ After K times, the target network π is updated using soft update φ′ , and Once, the weight adjustment parameter τ is used in the soft update, expressed as: i i ′←——τθ i +(1-τ)θ i ′,i=1,2 φ′←——τφ+(1-τ)φ′ In the formula, θ i Represents the evaluation network The parameter θ i ′ represents the target network Parameters of , φ represents the evaluation network π φ , φ′ represents the target network π φ′ Parameters.

8. The cloud data center intelligent task scheduling method based on dual-agent deep reinforcement learning according to claim 7 is characterized in that: The above steps of updating the agent scheduling model are repeated until all tasks in the current time step are completed or a suitable host-task pair cannot be selected.

9. An electronic device, comprising: One or more processors, a memory, and one or more programs; wherein the one or more programs are stored in the memory, and the one or more programs include instructions, which, when executed by the electronic device, enable the electronic device to execute the method described in any one of claims 1-8.

10. A computer-readable storage medium, comprising a computer program, which enables an electronic device to execute the method according to any one of claims 1 to 8 when the computer program is executed on the electronic device.

Images