Multi-objective optimization cloud task scheduling method and system based on MADDPG algorithm

Abstract

The invention discloses a multi-objective optimization cloud task scheduling method and system based on an MADDPG algorithm. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm comprises the following steps: obtaining load information; obtaining a trained multi-agent reinforcement learning model; inputting the load information into the multi-agent reinforcement learning model so as to obtain an action set obtained after decision making of all agents; and sending an action set obtained after decision making of all the agents to a cloud task scheduling center module, so that the cloud task scheduling center module completes cloud task scheduling. According to the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm, multiple task scheduling actions can be generated at a time through learning and cooperation among multiple agents, and the problems that a single agent can only generate one task scheduling action at a time, and the scheduling efficiency is low are solved.

Description

Technical Field

[0001] The present application relates to the technical field of cloud computing, and in particular to a multi-objective optimization cloud task scheduling method based on a MADDPG algorithm and a multi-objective optimization cloud task scheduling system based on a MADDPG algorithm. Background Art

[0002] The rise of cloud computing has enabled large-scale computing tasks to be efficiently processed on cloud platforms. In a cloud computing environment, task scheduling is a key link to ensure the effective use of system resources and the efficient completion of tasks. The task scheduling system is responsible for allocating tasks submitted by users to computing resources on the cloud platform to maximize resource utilization, shorten task completion time, and ensure the stability and reliability of system performance. Cloud task scheduling often involves multiple optimization objectives, such as reducing energy consumption, shortening task completion time, and improving system throughput. These objectives often have mutual constraints and contradictions, and traditional single-objective optimization algorithms are difficult to handle effectively. Therefore, designing a multi-objective optimization algorithm that can consider multiple objectives at the same time is crucial for cloud task scheduling. This algorithm needs to be able to optimize multiple indicators while ensuring system performance.

[0003] Traditional task scheduling methods usually focus on a single optimization objective, such as minimizing task completion time or maximizing resource utilization. However, in actual cloud task scheduling applications, there are often multiple interrelated optimization objectives, such as reducing energy consumption, improving system throughput, shortening task response time, etc. Traditional methods may have some limitations when dealing with multi-objective optimization. For example, it is often difficult to find the best balance when weighing the trade-offs between different optimization objectives. In addition, traditional methods often require manually designed heuristic rules or rely on experience to make decisions, which may show limitations when facing complex dynamic environments and are difficult to cope with changing demand and resource conditions. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-objective optimization cloud task scheduling method based on the MADDPG algorithm to solve at least one of the above technical problems.

[0005] In one aspect of the present invention, a multi-objective optimization cloud task scheduling method based on the MADDPG algorithm is provided. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm comprises:

[0006] Step 1: Get load information;

[0007] Step 2: Obtain a trained multi-agent reinforcement learning model;

[0008] Step 3: Input the load information into the multi-agent reinforcement learning model to obtain the action set obtained after all agents make decisions;

[0009] Step 4: Send the action sets obtained after all agents make decisions to the cloud task scheduling center module, so that the cloud task scheduling center module can complete the cloud task scheduling.

[0010] Optionally, after step 4, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes:

[0011] Step 5: Obtain cluster load information after task scheduling;

[0012] Step 6: Calculate reward feedback according to the cluster load information after the task scheduling;

[0013] Step 7: Update the multi-agent reinforcement learning model according to the information obtained in steps 3 to 6;

[0014] Step 8: Repeat steps 2 to 7 until all tasks are completed.

[0015] Optionally, before obtaining the trained multi-agent reinforcement learning model, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes:

[0016] The multi-agent reinforcement learning model is trained.

[0017] Optionally, the training of the multi-agent reinforcement learning model includes:

[0018] Construct state space, action space and reward function;

[0019] The multi-agent reinforcement learning model is trained according to the state space, action space and reward function.

[0020] Optionally, constructing the state space includes:

[0021] The cluster load monitoring center transmits the current cluster load information to the cluster load monitoring center for data processing, and designs the reinforcement learning state space State according to the cloud environment. space The formula is as follows:

[0022] State space ={{Vm},Task};

[0023] {Vm}={Vm0,Vm1,…,Vm i ,…,Vm k-1};

[0024] Vm refers to the real-time status information of the virtual machine at any time, Task refers to the information of the current task, and K represents the number of virtual machines in the cluster.

[0025] Optionally, the action space formula is as follows:

[0026] Action space ={Task2Vm0,Task2Vm1,…,Task2Vm i ,…,Task2Vm k-1};

[0027] The length of the action space is equal to the number of virtual machines k, where the action Task2vm i It means assigning the task to virtual machine number i, where i∈[0,k-1].

[0028] Optionally, the reward function formula is as follows:

[0029]

[0030] Among them, reward over is the reward value after each round, makespan is the task processing time, EofCluster is the cluster power consumption, and R1 is the reward value over The discount factor, α, β, γ are weights;

[0031]

[0032] reward t represents the real-time reward after each allocation, where t is the instant after the allocation, μ represents The weight of The weight of is the standard deviation of the cluster CPU utilization at time t after task allocation, It represents the standard deviation of the number of tasks in the task waiting queue of all virtual machines in the cluster at time t after task allocation.

[0033] The present application also provides a multi-objective optimization cloud task scheduling system based on the MADDPG algorithm, and the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm includes:

[0034] A MADDPG decision center design module, wherein the MADDPG decision center design module is used to implement the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as described in any one of claims 1 to 7;

[0035] A simulation environment initialization module, which is used for task generation and cluster deployment;

[0036] A cluster load monitoring center module, which is used to monitor the cloud environment and obtain load information, including CPU load, memory load, the size of each virtual machine task waiting queue, and the number of tasks that have arrived;

[0037] A cloud task scheduling center module, which is used to schedule cloud tasks according to the action sets obtained after all agent decisions generated by the MADDPG decision center design module;

[0038] The power consumption calculation center module is used to calculate the power consumption in a preset period.

[0039] Beneficial Effects

[0040] The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm of the present application realizes the generation of multiple task scheduling actions at one time through learning and collaboration between multiple intelligent agents, solving the problem that a single intelligent agent can only generate one task scheduling action at a time and the scheduling efficiency is low; secondly, the present invention conveniently obtains the power consumption value by introducing the power consumption model, and can obtain the power consumption value in real time; finally, the present invention incorporates multiple optimization objectives into the reward function design of reinforcement learning, and transforms the multi-objective optimization problem into a single objective optimization problem. By optimizing resource utilization, improving load balancing and shortening task execution time, the system can more effectively utilize hardware resources, thereby reducing energy consumption. For example, unnecessary energy consumption when resources are idle is avoided, and the overall power consumption of the system is reduced. The optimized task execution strategy can reduce the execution time of tasks and improve the efficiency of task processing. This means that the system can complete tasks faster, improve production efficiency or service response speed, thereby saving time costs. Through reasonable scheduling and management of resources, the system can make more full use of hardware resources, reduce the idle time of resources, and improve resource utilization. In this way, the potential of resources can be maximized, resource waste can be avoided, and cost-effectiveness can be improved. By optimizing load balancing and task scheduling strategies, the difference in task execution time can be reduced and the standard deviation of the system can be reduced. This means that task execution time is more stable, which improves the reliability and stability of the system, thereby reducing the costs caused by fluctuations in task execution time. These benefits can not only reduce system operating costs, but also improve system performance and reliability, thereby bringing better cost-effectiveness. Through the implementation of comprehensive strategies, system operation can be effectively optimized, resource utilization and performance can be improved, thus achieving a better balance between cost and benefit. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a multi-objective optimization cloud task scheduling method based on the MADDPG algorithm according to an embodiment of the present application.

[0042] Figure 2 It is a schematic diagram of an electronic device for implementing a multi-objective optimization cloud task scheduling method based on a MADDPG algorithm according to an embodiment of the present application.

[0043] Figure 3 It is a system architecture diagram of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application.

[0044] Figure 4 This is a task generation diagram of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application.

[0045] Figure 5 This is a cluster deployment diagram of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application.

[0046] Figure 6 It is a schematic diagram of the design architecture of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application.

[0047] Figure 7 It is a schematic diagram of the MADDPG algorithm of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of the present application.

[0048] Figure 8 It is a program flow diagram of the MADDPG algorithm decision center of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application.

[0049] Fig. 9 It is a schematic diagram of the cloud task scheduling process of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of the present application.

[0050] Fig.10 It is a schematic diagram of the cluster update process of the multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of this application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the implementation of this application clearer, the technical scheme in the embodiment of this application will be described in more detail below in conjunction with the drawings in the embodiment of this application. In the drawings, the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The described embodiments are part of the embodiments of this application, not all of them. The embodiments described below with reference to the drawings are exemplary and are intended to be used to explain this application, and should not be construed as limitations on this application. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application. The embodiments of this application are described in detail below in conjunction with the drawings.

[0052] like Figure 1 The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm shown includes:

[0053] Step 1: Get load information;

[0054] Step 2: Obtain a trained multi-agent reinforcement learning model;

[0055] Step 3: Input the load information into the multi-agent reinforcement learning model to obtain the action set obtained after all agents make decisions;

[0056] Step 4: Send the action sets obtained after all agents make decisions to the cloud task scheduling center module, so that the cloud task scheduling center module can complete the cloud task scheduling.

[0057] In this embodiment, after step 4, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes:

[0058] Step 5: Obtain cluster load information after task scheduling;

[0059] Step 6: Calculate reward feedback according to the cluster load information after the task scheduling;

[0060] Step 7: Update the multi-agent reinforcement learning model according to the information obtained in steps 3 to 6;

[0061] Step 8: Repeat steps 2 to 7 until all tasks are completed.

[0062] In this embodiment, before obtaining the trained multi-agent reinforcement learning model, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes:

[0063] The multi-agent reinforcement learning model is trained.

[0064] In this embodiment, the training of the multi-agent reinforcement learning model includes:

[0065] Construct state space, action space and reward function;

[0066] The multi-agent reinforcement learning model is trained according to the state space, action space and reward function.

[0067] In this embodiment, constructing the state space includes:

[0068] The cluster load monitoring center transmits the current cluster load information to the cluster load monitoring center for data processing, and designs the reinforcement learning state space State according to the cloud environment.space The formula is as follows:

[0069] State space ={{Vm},Task};

[0070] {Vm}={Vm0,Vm1,…,Vm i ,…,Vm k-1};

[0071] Vm refers to the real-time status information of the virtual machine at any time, Task refers to the information of the current task, and K represents the number of virtual machines in the cluster.

[0072] In this embodiment, the action space formula is as follows:

[0073] Action space ={Task2Vm0,Task2Vm1,…,Task2Vm i ,…,Task2Vm k-1};

[0074] The length of the action space is equal to the number of virtual machines k, where the action Task2vm i It means assigning the task to virtual machine number i, where i∈[0,k-1].

[0075] In this embodiment, the reward function formula is as follows:

[0076]

[0077] Among them, reward over is the reward value after each round, makespan is the task processing time, EofCluster is the cluster power consumption, and R1 is the reward value over The discount factor, α, β, γ are weights;

[0078]

[0079] reward t represents the real-time reward after each allocation, where t is the instant after the allocation, μ represents The weight of The weight of is the standard deviation of the cluster CPU utilization at time t after task allocation, It represents the standard deviation of the number of tasks in the task waiting queue of all virtual machines in the cluster at time t after task allocation.

[0080] In this embodiment, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes:

[0081] Perform task generation and cluster deployment;

[0082] Monitor the cloud environment and obtain load information, including CPU load, memory load, the size of each virtual machine task waiting queue, and the number of tasks that have arrived;

[0083] Calculates power consumption at a preset period.

[0084] The present application also provides a multi-objective optimization cloud task scheduling system based on the MADDPG algorithm, which includes a MADDPG decision center design module, a simulation environment initialization module, a cluster load monitoring center module, a cloud task scheduling center module, and a power consumption calculation center module, wherein:

[0085] The MADDPG decision center design module is used to implement the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as described above;

[0086] The simulation environment initialization module is used for task generation and cluster deployment;

[0087] The cluster load monitoring center module is used to monitor the cloud environment and obtain load information, including CPU load, memory load, the size of each virtual machine task waiting queue and the tasks that have arrived;

[0088] The cloud task scheduling center module is used to schedule cloud tasks according to the action set obtained after all intelligent agent decisions generated by the MADDPG decision center design module;

[0089] The power consumption calculation center module is used to calculate the power consumption in a preset period.

[0090] The present application is further described in detail below in combination with the system and the method. It should be understood that this example does not constitute any limitation to the present application.

[0091] The multi-objective optimization cloud task scheduling system based on the MADDPG algorithm of the present application includes a simulation environment initialization module, a cluster load monitoring center module, a MADDPG decision center module, a cloud task scheduling center module, and a power consumption calculation center module. The simulation environment initialization module includes a task generation module and a cluster deployment module, and the MADDPG decision center module has a multi-agent reinforcement learning model based on the MADDPG algorithm; the simulation environment initialization has a signal connection with the cluster load monitoring center, the cluster load monitoring center has a signal connection with the MADDPG decision center, and the MADDPG decision center has a signal connection with the task scheduling center.

[0092] In this embodiment, the simulation environment initialization module includes two submodules: a task generation module and a cluster deployment module.

[0093] In this embodiment, the cluster load monitoring center module can realize CPU load monitoring, memory load monitoring, each virtual machine task waiting queue size monitoring and arrived task monitoring.

[0094] In this embodiment, the power consumption calculation center module is used to calculate power consumption. Specifically, the user can pre-select a cluster power consumption model, and then obtain CPU load monitoring of all virtual machines through the cluster load monitoring center, and complete the power consumption calculation after obtaining the CPU utilization.

[0095] In this embodiment, a multi-agent grouping structure is adopted in the MADDPG decision center design module, and each agent is only responsible for the task scheduling of a group of virtual machines in the cluster.

[0096] In this embodiment, the MADDPG decision center design module includes environment modeling, and the multi-agent environment modeling includes state space design, action space design and reward function design.

[0097] In this embodiment, the functions of the cloud task scheduling center module include a task offloading function and a virtual machine status update function within the cluster.

[0098] The present invention provides a method for operating the system, such as Figure 3 As shown, each module is directly connected to each other, and the transmission of information is triggered by calling the interface. The MADDPG decision center of the present invention is a reinforcement learning model, which requires pre-training to make the model performance meet the requirements. If the present application is trained directly in a real cloud environment, there are problems such as high training difficulty and poor controllability. Therefore, this embodiment performs pre-training based on a simulation platform.

[0099] In this embodiment, task generation and cluster deployment are required in the simulation environment initialization module. Figure 4 As shown in the figure, the task input generation module can realize two ways of task generation. In the first way, users can set the attribute distribution range of the task to generate a probability distribution that meets the user's needs. After the task generation is completed, it is necessary to verify whether the generated probability distribution is accurate. In the second way, users can batch generate by batch importing task information from files or databases. Figure 5 As shown, the cluster deployment module can implement two types of cluster deployment. The first type is suitable for small clusters with a small number of virtual machines, and users can actively set the configuration of virtual machines. The second type is suitable for large clusters with a large number of virtual machines, and users can batch generate virtual machines in the cluster through files or databases to complete cluster deployment.

[0100] In this embodiment, after completing the simulation initialization, the load monitoring center starts to monitor the cloud environment, realizing CPU load monitoring, memory load monitoring, each virtual machine task waiting queue size monitoring and arrived task monitoring.

[0101] In this embodiment, the power consumption calculation center calculates the power consumption for use in the reward function design in the MADDPG decision center design module.

[0102] Specifically, the formula for defining the power value is shown in Formula 1:

[0103] P=P idle +(P busy -P idle )×(2×Cpu utilization -Cpu utilization Pr ) (1)

[0104] Where P idle is the resting power of the server, P busy is the full load power of the server, CPU utilization is the real-time CPU utilization, and Pr is the fitting factor. This paper uses the power calculation method proposed by Fan et al. as the power calculation method of this model. The full name of the paper is as follows: Zhou Y, Liu S, Siow J, et al. Devign: Effective vulnerability identification by learning comprehensive program semantics via graph neural networks [J]. Advances in neural information processing systems, 2019, 32.

[0105] In this embodiment, the MADDPG decision center design module uses a multi-agent grouping strategy to evenly divide the virtual machines in the cluster according to the number of agents, and each agent controls the task scheduling of a group of virtual machines.

[0106] The cluster load monitoring center obtains task information and the number of tasks in the waiting queues of all virtual machines in the cluster, memory utilization, and CPU utilization to implement the design of the reinforcement learning state space.

[0107] The power consumption value is calculated by the power consumption calculation center, and the number of tasks in the waiting queue of all virtual machines in the cluster, memory utilization and CPU utilization are obtained by the cluster load monitoring center to calculate the average CPU utilization, average memory utilization, CPU utilization standard deviation, memory utilization standard deviation, waiting queue size standard deviation and other values, so as to design the reward function to complete the multi-objective optimization of cloud task scheduling.

[0108] The action space of reinforcement learning is designed by selecting the virtual machine functions to be offloaded in the task scheduling center module.

[0109] In this embodiment, the MADDPG decision center design module needs to perform state space design, action space design, reward function design and multi-agent design. The specific design steps of state space design and action space design are as follows:

[0110] The first step is to quantify the state of the cloud computing environment. First, the current cluster load information transmitted by the cluster load monitoring center must be processed. space As shown in Formula 2, it consists of a Vm set and Task information.

[0111] State space ={{Vm},Task} (2)

[0112] {Vm}={Vm0,Vm1,…,Vm i ,…,Vm k-1} (3)

[0113] Where Vm refers to the real-time status information of the virtual machine at a certain moment, and Task refers to the information of the current task. K represents the number of virtual machines in the cluster, and the detailed information of the virtual machine is shown in Formula 4:

[0114]

[0115] Queue waiting ={Task α ,Task β ,…,Task γ} (5)

[0116] CPU corenumber Indicates the number of CPU cores of the virtual machine, indicating the maximum number of tasks that can be run simultaneously on the machine. mips Indicates the length of instructions executed per core processor per second. To be consistent with the Alibaba Cloud dataset, it is quantized to 100. size The memory size of the virtual machine, which is used as a constraint to determine whether a task can be run. transFor transmission speed, CPU utilization is the real-time CPU utilization, as shown in Formula 5, Queue waiting The task waiting queue of the virtual machine stores tasks waiting for CPU processing. idle Indicates the machine standby power, P busy Indicates the full load power of the machine, P r Represents the power consumption model fitting factor.

[0117] The detailed information of the task is shown in Equation 6:

[0118] Task = {Mi, Memory size ,CPU avg_plan} (6)

[0119] Where Mi represents the instruction length Memory of the task size Indicates the memory size and CPU required by the task avg_plan Indicates the average CPU utilization of the task. Because the task needs to perform IO operations during the running time, the CPU usage cannot reach 100%.

[0120] The action space is a set of actions that can be performed by the reinforcement learning agent. The present invention realizes the simultaneous scheduling of multiple tasks through multi-agents. After the reinforcement learning model strategy is calculated, the action is output, that is, a set of tasks is offloaded to multiple virtual machines. The action space is designed as shown in Formula 7:

[0121] Action space ={Task2Vm0,Task2Vm1,…,Task2Vm i ,…,Task2Vm k-1} (7)

[0122] The length of the action space is equal to the number of virtual machines k, where the action Task2vm i It means assigning the task to virtual machine number i, where i∈[0,k-1].

[0123] Reward function design is extremely important for reinforcement learning, which is related to whether the reinforcement learning agent can find the right action to perform elastic scaling of cloud resources. Reward function design is guided by optimization goals. The optimization goal of elastic scaling of cloud resources of the present invention is to optimize cluster load balancing, avoid overload of virtual machine instances, and improve cluster computing resource utilization. Assuming that the total scheduling time makespan for m tasks is T, the current average CPU utilization of the cluster is obtained by using makespan at intervals of 1s, and the total power consumption is obtained by summing up the power model mentioned above. The calculation formula for the average CPU utilization of the cluster at time t is shown in Formula 8:

[0124]

[0125] in represents the CPU utilization of the i-th virtual machine at time t, and k represents the number of virtual machines in the cluster.

[0126] The total power consumption of the cluster is calculated as shown in Formula 9:

[0127]

[0128] The smaller the standard deviation of CPU utilization of each virtual machine in the cluster at any time, the better the cluster load balancing effect. Within the total running time T of a batch of tasks, the CPU utilization of each virtual machine in the cluster is sampled at intervals of 1 second, and a total of T samples are taken to calculate the standard deviation of the CPU utilization of the cluster at a certain time. The calculation formula for the standard deviation of the CPU utilization of the cluster at any time t is shown in Formula 10:

[0129]

[0130] The average value of the cluster CPU utilization standard deviation of T samplings represents the load balancing effect within the total running time T. The smaller the average value of the cluster CPU utilization standard deviation, the better the load balancing effect within the total running time T. The calculation formula for the average value of the cluster CPU utilization standard deviation is shown in Formula 11:

[0131]

[0132] This paper simultaneously optimizes the three objectives of task processing time, cluster power consumption, and cluster load in cloud task scheduling, and weights the three optimization objectives with weights of α, β, and γ respectively, to convert the multi-objective optimization problem of cloud task scheduling into a single-objective optimization problem. Therefore, the overall optimization objective is shown in Formula 12:

[0133] Goal=min(α×makesapn+β×EofCluster+γ×AvgStdCpu utilization ofCluster)(12)

[0134] After all tasks are executed, calculate the task processing time makespan, cluster power consumption EofCluster, and cluster load AvgStdCpu utilization ofCluster then calculates the reward value for each round over , where R1 is the reward value reward over The discount factor is , since the overall optimization goal is to minimize the above sub-goals, so the reward value reward over is a negative value, as shown in Equation 13. utilizationofCluster, makespan, and EofCluster have all been normalized.

[0135] reward over =-R1×(α×makespan+β×EofCluster+γ×AvgStdCpu utilization ofCluster) (13)

[0136] The design of the reinforcement learning reward function has a great impact on the performance of the algorithm. The reward value is calculated once after each round. over , which makes the training cost extremely high. This paper also incorporates the standard deviation of cluster CPU utilization after each task allocation and the number of tasks in the waiting queue of each virtual machine in the cluster into the design of the reward function as an immediate reward.

[0137] In formula 14, reward t represents the real-time reward after each allocation, where t is the instant after the allocation, and μ and ε represent and Note All normalization processing has been completed.

[0138]

[0139] in It is the standard deviation of the cluster CPU utilization at time t after task allocation. The calculation method is described above. represents the standard deviation of the number of tasks in the task waiting queue of all virtual machines in the cluster at time t after task assignment. The calculation method of is shown in formula 15:

[0140]

[0141] Where k is the number of virtual machines in the cluster, represents the size of the task waiting queue of virtual machine i at time t, where i∈[0,k-1]. represents the average number of tasks in the task waiting queue of all virtual machines in the cluster at time t, and the calculation method is shown in Formula 16:

[0142]

[0143] like Figure 5As shown in the figure, the MADDPG algorithm extends the single-agent DDPG algorithm and applies it to multi-agent systems. The core of the algorithm is the Actor-Critic structure, where each agent has an Actor network and a Critic network. The Actor network is responsible for generating actions, while the Critic network evaluates the value of these actions. In the MADDPG algorithm, the agent collects experience through interaction with the environment and stores these experiences in a shared experience replay buffer. Then, a batch of experience is randomly sampled from the experience replay buffer to update the parameters of the Actor and Critic networks of each agent. During the update process, the Critic network learns the state value function by minimizing the error between the actual reward and the predicted reward, while the Actor network updates the strategy by maximizing the state value function estimated by the Critic network. The agents share experience and network parameters, and improve the overall performance through collaborative learning. The experience replay mechanism is used to store and reuse the experience of the agents to improve sample efficiency and stability.

[0144] The MADDPG (Multi-Agent Deep Deterministic Policy Gradient) model is a deep reinforcement learning algorithm used in a multi-agent environment, aiming to solve the problem of cooperation and competition in a multi-agent system. The model is based on the DDPG (Deep Deterministic Policy Gradient) algorithm. By introducing a multi-agent learning framework and a collaborative training mechanism, the agents can learn and adapt to each other in a collaborative and competitive environment. The core idea of ​​the MADDPG model is to regard each agent as an independent learning agent, but its action selection is affected by the behavior of other agents. In a multi-agent environment, the behaviors of agents will affect each other. Therefore, considering the importance of this interaction for overall learning, the MADDPG model realizes collaborative learning between agents through the Actor-Critic structure. In the MADDPG model, each agent has its own Actor and Critic network. The Actor network is responsible for generating actions and selecting appropriate actions based on the current state; while the Critic network evaluates the value of actions and estimates the state value function. Agents collect experience through interaction with the environment, which will be stored in a shared experience replay buffer. During training, the agent randomly samples a batch of experiences from the experience replay buffer and uses these experiences to update the parameters of the Actor and Critic networks. During the update process, the agent's Critic network learns the state value function by minimizing the error between the actual reward and the state value function estimated by the Critic network, while the Actor network updates the strategy by maximizing the state value function estimated by the Critic network. This combination of collaborative learning and experience replay enables agents to collaborate and optimize action selection strategies in a multi-agent environment, thereby achieving better decision-making performance.

[0145] In this embodiment, the cloud task scheduling center implements the task offloading function and the update synchronization of the simulation environment after the task offloading.

[0146] like Figure 6 As shown, in this embodiment, the specific workflow of the MADDPG decision center design module is as follows:

[0147] Step 1: Get load information;

[0148] Step 2: Generate load information according to the load information;

[0149] Step 3: Obtain the trained multi-agent reinforcement learning model;

[0150] Step 4: Input the load information into the multi-agent reinforcement learning model to obtain a set of actions obtained after all agents make decisions;

[0151] Step 5: Send the action sets obtained after all agents make decisions to the cloud task scheduling center module, so that the cloud task scheduling center module can complete the cloud task scheduling.

[0152] Step 6: Obtain cluster load information after task scheduling;

[0153] Step 7: Calculate reward feedback according to the cluster load information after the task scheduling;

[0154] Step 8: Update the multi-agent reinforcement learning model according to the information obtained in steps 4 to 7;

[0155] Step 9: Repeat steps 2 to 8 until all tasks are completed.

[0156] In this embodiment, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm of the present application may further include, before step 1:

[0157] Run the decision center program of the multi-agent deep reinforcement learning MADDPG algorithm and initialize the multi-agent reinforcement learning model of the MADDPG algorithm;

[0158] Send a start monitoring command to the cluster load monitoring center module, and wait for all load information sent by the cluster load monitoring center module;

[0159] Determine whether all tasks have been completed. If not, go to step 1; if so, go to step 9.

[0160] In this embodiment, the cloud task scheduling center implements the task offloading function and the synchronization of the update of the simulation environment after the task offloading. Figure 7 As shown, the working steps of the cloud task scheduling center module are as follows:

[0161] The first step is to start or restart the simulation;

[0162] The second step is to obtain the load information sent by the cluster load monitoring center module;

[0163] The third step is to enter a loop to continuously determine whether all tasks have been assigned;

[0164] In the fourth step, if it is not finished, the action set obtained by the MADDPG decision center will be used to select a virtual machine for task offloading;

[0165] Step 5: After the uninstallation is completed, the status of all virtual machines and tasks will be updated and the process will return to step 2.

[0166] When all tasks are assigned, the sixth step is entered to determine whether all tasks have been completed. If there are still running tasks, the fifth step will be entered and the cycle will continue until all tasks are completed, ending the task scheduling program.

[0167] like Figure 8 As shown, in this embodiment, when it is necessary to update the state function of the virtual machine in the cluster, it is performed by the following method:

[0168] In the first step, the update procedure will be started by calling the interface of the update virtual machine;

[0169] In the second step, all virtual machines will be traversed to determine whether the traversal is completed. If the traversal is not completed, the virtual machine being traversed will be updated;

[0170] In the third step, when updating a specific virtual machine, the running tasks in the virtual machine will be traversed first, and the time step of the traversed tasks will be reduced by 1, which means that the life cycle of the task is reduced by 1;

[0171] The fourth step is to determine whether the current task is alive. When the survival time is equal to 0, it will be removed from the virtual machine and the occupied resources will be released. This cycle will continue until all the running tasks in the virtual machine are traversed;

[0172] Step 5: After all the running tasks in the virtual machine are traversed, determine whether the current virtual machine has free resources to run the tasks in the waiting queue. If there are free resources, the task will be taken out of the waiting queue and started to run. This cycle will continue until there are no free resources to run the task. At this point, the update of this virtual machine is completed. Return to step 2 and continue to cycle until all virtual machines are updated and enter the end state.

[0173] The beneficial effects of the present invention compared with the prior art are as follows: first, the present invention realizes the generation of multiple task scheduling actions at one time through learning and collaboration between multiple intelligent agents, solving the problem that a single intelligent agent can only generate one task scheduling action at a time and the scheduling efficiency is low; second, the present invention conveniently obtains the power consumption value by introducing the power consumption model, and can obtain the power consumption value in real time; finally, the present invention incorporates multiple optimization objectives into the reward function design of reinforcement learning, and transforms the multi-objective optimization problem into a single-objective optimization problem. By optimizing resource utilization, improving load balancing and shortening task execution time, the system can more effectively utilize hardware resources, thereby reducing energy consumption. For example, unnecessary energy consumption when resources are idle is avoided, and the overall power consumption of the system is reduced. The optimized task execution strategy can reduce the execution time of tasks and improve the efficiency of task processing. This means that the system can complete tasks faster, improve production efficiency or service response speed, thereby saving time costs. By reasonably scheduling and managing resources, the system can make more full use of hardware resources, reduce the idle time of resources, and improve resource utilization. In this way, the potential of resources can be maximized, resource waste can be avoided, and cost-effectiveness can be improved. By optimizing load balancing and task scheduling strategies, the difference in task execution time can be reduced and the standard deviation of the system can be reduced. This means that task execution time is more stable, which improves the reliability and stability of the system, thereby reducing the costs caused by fluctuations in task execution time. These benefits can not only reduce system operating costs, but also improve system performance and reliability, thereby bringing better cost-effectiveness. Through the implementation of comprehensive strategies, system operation can be effectively optimized, resource utilization and performance can be improved, thus achieving a better balance between cost and benefit.

[0174] This embodiment uses a deep multi-agent algorithm to implement cloud task scheduling. Compared with the traditional single-agent method, the deep multi-agent algorithm can better adapt to complex cloud task scheduling scenarios through collaboration and learning between agents. This embodiment uses the cloud task scheduling simulation experiment according to the method of Example 1 to simulate and generate the probability distribution of task attributes. The cloud resource hybrid elastic scaling strategy of this embodiment shows excellent results. In the simulation cycle, the high-frequency adjustment of cloud resources greatly reduces the cloud resource investment compared to the threshold-based cloud resource scheduling strategy, thereby reducing the cloud service cost. The experimental parameters are shown in Tables 1 and 2.

[0175] Table 1. Experimental task parameters

[0176]

[0177] Table 2. Experimental virtual machine parameters

[0178]

[0179] The present invention deploys 25 virtual machines in the cluster for this experiment, and the configuration of each virtual machine is shown in Table 2. In a given time period, the task generator generates 1000 tasks, and the probability distribution of task attribute intervals is consistent with Table 1. The random algorithm and the robin algorithm are selected as the benchmark algorithms, and the DQN algorithm and the DDPG algorithm are selected to compare with the MADDPG algorithm proposed in the present invention.

[0180] Table 3. Experimental results

[0181]

[0182] The experimental results are shown in Table 3. It can be seen that the two single-agent models of the present invention, DQN algorithm and DDPG algorithm, have better performance compared with the benchmark algorithms. Among them, MADDPG performs better than DQN algorithm and DDPG algorithm, and has good effects in reducing power consumption, shortening time, improving resource utilization, reducing standard deviation, and improving load balancing.

[0183] The comprehensive strategy of this embodiment can bring better cost-effectiveness. By optimizing resource utilization, improving load balancing and shortening task execution time, the system can use hardware resources more effectively, thereby reducing energy consumption. For example, unnecessary energy consumption when resources are idle is avoided, and the overall power consumption of the system is reduced. The optimized task execution strategy can reduce the execution time of tasks and improve the efficiency of task processing. This means that the system can complete tasks faster, improve production efficiency or service response speed, thereby saving time costs. By reasonably scheduling and managing resources, the system can make more full use of hardware resources, reduce the idle time of resources, and improve resource utilization. In this way, the potential of resources can be maximized, resource waste can be avoided, and cost-effectiveness can be improved. By optimizing load balancing and task scheduling strategies, the difference in task execution time can be reduced and the standard deviation of the system can be reduced. This means that the task execution time is more stable, the reliability and stability of the system are improved, and the cost caused by fluctuations in task execution time is reduced. These benefits can not only reduce the cost of system operation, but also improve the performance and reliability of the system, thereby bringing better cost-effectiveness. Through the implementation of the comprehensive strategy, the system operation can be effectively optimized, resource utilization and performance can be improved, so as to achieve a better balance between cost and benefit.

[0184] It should be noted that the above explanations of the method embodiment are also applicable to the device of this embodiment and will not be repeated here.

[0185] The present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the above laboratory management method is implemented when the processor executes the computer program.

[0186] The present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above laboratory management method can be implemented.

[0187] Figure 2 It is an exemplary structural diagram of an electronic device capable of implementing the laboratory management method provided according to an embodiment of the present application.

[0188] like Figure 2 As shown, the electronic device includes an input device 501, an input interface 502, a central processor 503, a memory 504, an output interface 505, and an output device 506. The input interface 502, the central processor 503, the memory 504, and the output interface 505 are interconnected through a bus 507, and the input device 501 and the output device 506 are connected to the bus 507 through the input interface 502 and the output interface 505, respectively, and then connected to other components of the electronic device. Specifically, the input device 501 receives input information from the outside, and transmits the input information to the central processor 503 through the input interface 502; the central processor 503 processes the input information based on the computer executable instructions stored in the memory 504 to generate output information, temporarily or permanently stores the output information in the memory 504, and then transmits the output information to the output device 506 through the output interface 505; the output device 506 outputs the output information to the outside of the electronic device for use by the user.

[0189] That is to say, Figure 2 The electronic device shown can also be implemented as comprising: a memory storing computer executable instructions; and one or more processors, which can implement the combination when executing the computer executable instructions. Figure 1 Describe the laboratory management approach.

[0190] In one embodiment, Figure 2 The electronic device shown can be implemented to include: a memory 504, configured to store executable program code; one or more processors, configured to run the executable program code stored in the memory 504 to execute the multi-UAV cooperative multi-hop real-time relay link design method in the above-mentioned embodiment.

[0191] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0192] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0193] Computer readable media include permanent and non-permanent, removable and non-removable, and the media can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), data versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0194] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code includes one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes identified in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or the total flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0195] The processor referred to in this embodiment may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0196] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the device / terminal equipment by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0197] In this embodiment, if the module / unit integrated in the device / terminal equipment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. Although the present application is disclosed as above in terms of a preferred embodiment, it is not intended to limit the present application. Any technical personnel in this field may make possible changes and modifications without departing from the spirit and scope of the present application. Therefore, the scope of protection of the present application shall be based on the scope defined by the claims of the present application.

[0198] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0199] In addition, it is obvious that the word "comprising" does not exclude other units or steps. Multiple units, modules or devices stated in the device claim can also be implemented by one unit or the overall device through software or hardware.

[0200] Although the present invention has been described in detail above with general descriptions and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements may be made thereto based on the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all fall within the scope of protection claimed by the present invention.

Claims

1. A multi-objective optimization cloud task scheduling method based on MADDPG algorithm, characterized in that: The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm includes: Step 1: Get load information; Step 2: Obtain a trained multi-agent reinforcement learning model; Step 3: Input the load information into the multi-agent reinforcement learning model to obtain the action set obtained after all agents make decisions; Step 4: Send the action sets obtained after all agents make decisions to the cloud task scheduling center module, so that the cloud task scheduling center module can complete the cloud task scheduling.

2. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 1, characterized in that: After step 4, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes: Step 5: Obtain cluster load information after task scheduling; Step 6: Calculate reward feedback according to the cluster load information after the task scheduling; Step 7: Update the multi-agent reinforcement learning model according to the information obtained in steps 3 to 6; Step 8: Repeat steps 2 to 7 until all tasks are completed.

3. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 2, characterized in that: Before obtaining the trained multi-agent reinforcement learning model, the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm further includes: The multi-agent reinforcement learning model is trained.

4. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 3, characterized in that: The training of the multi-agent reinforcement learning model comprises: Construct state space, action space and reward function; The multi-agent reinforcement learning model is trained according to the state space, action space and reward function.

5. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 4, characterized in that: The constructing state space comprises: The cluster load monitoring center transmits the current cluster load information to the cluster load monitoring center for data processing, and designs the reinforcement learning state space State according to the cloud environment. space The formula is as follows: State space ={{Vm},Task}; {Vm}={Vm0,Vm1,…,Vm i ,…,etc. k-1 }; Vm refers to the real-time status information of the virtual machine at any time, Task refers to the information of the current task, and K represents the number of virtual machines in the cluster.

6. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 5, characterized in that: The action space formula is as follows: Action space ={Task2Vm0,Task2Vm1,…,Task2Vm i ,…,Task2Vm k-1 }; The length of the action space is equal to the number of virtual machines k, where the action Task2vm i It means assigning the task to virtual machine number i, where i∈[0,k-1].

7. The multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as claimed in claim 6, characterized in that: The reward function formula is as follows: reward over =-R1×(α×makespan+β×EofCluster+γ×AvgStdCpu utilization ofCluster); among them, reward over is the reward value after each round, makespa is the task processing time, EofClust is the cluster power consumption, and R1 is the reward value over The discount factor, α, β, γ are weights; reward t represents the real-time reward after each allocation, where t is the instant after the allocation, μ represents ofCluster weight, ε represents StdSizeQuuee waitintg ofCluster weight, StdCpu utilizationt ofCluster is the standard deviation of the cluster CPU utilization at time t after task allocation, StdSizeQueue waitingt ofCluster represents the standard deviation of the number of tasks in the task waiting queues of all virtual machines in the cluster at time t after task allocation.

8. A multi-objective optimization cloud task scheduling system based on MADDPG algorithm, characterized in that: The multi-objective optimization cloud task scheduling system based on the MADDPG algorithm includes: A MADDPG decision center design module, wherein the MADDPG decision center design module is used to implement the multi-objective optimization cloud task scheduling method based on the MADDPG algorithm as described in any one of claims 1 to 7; A simulation environment initialization module, which is used for task generation and cluster deployment; A cluster load monitoring center module, which is used to monitor the cloud environment and obtain load information, including CPU load, memory load, the size of each virtual machine task waiting queue, and the number of tasks that have arrived; A cloud task scheduling center module, which is used to schedule cloud tasks according to the action sets obtained after all agent decisions generated by the MADDPG decision center design module; The power consumption calculation center module is used to calculate the power consumption in a preset period.

Images