A cluster task scheduling method and system based on deep reinforcement learning

Abstract

The application provides a cluster task scheduling method and system based on deep reinforcement learning, which comprises the following steps: using reinforcement learning to perform real-time scheduling on tasks in a heterogeneous computing resource platform; taking physical machine group information in the computing platform, task demand information from a user and task execution cost as a state space of the reinforcement learning; taking a set of available physical machines as an action space of the reinforcement learning; learning, through a deep Q network method, different execution efficiencies of different task types in the heterogeneous resource physical machines that adapt to dynamic changes, and resource utilization efficiency, i.e. resource usage proportion when the physical machines execute tasks, by changing the environment information of the heterogeneous computing platform; and using a genetic algorithm to maximize the resource utilization efficiency according to resource usage of different physical machines in the physical machine groups generated in the previous step, so as to further schedule the tasks.

Description

Technical Field

[0001] This invention relates to the fields of edge computing, resource allocation and task scheduling technology, and in particular to a cluster task scheduling method and system based on deep reinforcement learning. Background Technology

[0002] With the rapid development of IoT and 5G network technologies, the number of "edge" devices such as mobile smart devices and smart homes has increased significantly. This has led to a surge in data traffic and increasingly complex network conditions, making edge computing a key area of ​​research in cutting-edge technologies. The basic principle of edge computing is that computation should occur near the data source. This represents both cloud-based computation on edge data and IoT edge services computation on cloud data. While edge computing shares similarities with fog computing, it focuses more on the collaboration between edge device resources, emphasizing the "things" themselves, whereas fog computing focuses on the construction of gateways or computing nodes, prioritizing infrastructure.

[0003] As a new distributed computing paradigm, edge computing focuses on solving the problem of high latency sensitivity of real-time applications. However, limited resources place higher demands on its collaboration at the edge. Therefore, edge computing must have an efficient and flexible scheduling mechanism that can address challenging issues such as user mobility, changing user needs, and heterogeneous resources.

[0004] Modern computing platforms are mostly composed of large-scale machine clusters to provide ample computing power. These machines may be equipped with different hardware to handle different problems. For example, to handle machine learning tasks, some machines may be equipped with powerful GPUs in addition to CPUs, while others may only have commercial CPUs. In existing technical solutions, the platform management center mostly uses the current usage of computing resources such as CPU and memory of the physical machines in the cluster as indicators to make scheduling decisions and allocate instances to designated physical machines.

[0005] For example, Xi'an University of Electronic Science and Technology disclosed a reinforcement learning-based cross-data center network task scheduling method in its patent application "A Reinforcement Learning-Based Cross-Data Center Network Task Scheduling Method" (Application No.: 201811561700.2, Application Date: 2018.12.20, Publication No.: CN109656702A). This method primarily addresses the problem of achieving adaptive real-time task scheduling and balanced and effective use of three types of resources in cross-data center networks through reinforcement learning. The specific steps of this method are: 1. Generate a training dataset; 2. Generate the state space and action space for reinforcement learning; 3. Calculate the value of possible actions in the action space; 4. Train the reinforcement learning task scheduling model; 5. Schedule tasks arriving in real-time in the cross-data center network. The drawback of this method is that it does not consider the differences in the operating efficiency of heterogeneous resources for different types of tasks, lacks targeted consideration, and fails to fully utilize the capabilities of heterogeneous computing resources, thus failing to effectively utilize the computing resources in the platform. Summary of the Invention

[0006] This invention provides a cluster task scheduling method and system based on deep reinforcement learning. When scheduling tasks from users in a heterogeneous computing resource platform, this invention collects real-time data changes from the physical machine cluster of the computing platform. Based on feedback from the cluster performance effects of different decisions, it adjusts the task scheduling strategy in real time to maximize the utilization efficiency of heterogeneous computing resources, intelligently selects the physical machine for task execution, and effectively improves the computing efficiency of the heterogeneous computing resource platform. Here, heterogeneous computing platform refers to a computing platform based on different hardware acceleration chips such as Nvidia-RTX series, NVIDIA-Jetson series, Raspberry Pi, and Cambricon MLU series. When calculating the value of the action space, this invention fully considers the acceleration effects of different hardware resources on different task types, effectively balancing the utilization efficiency of different resources in the computing cluster and improving the resource utilization rate of the computing cluster.

[0007] To address the shortcomings of existing technologies, this invention proposes a cluster task scheduling method based on deep reinforcement learning, which includes:

[0008] Step 1: Take the task requests, physical machine groups, and task execution costs in the heterogeneous computing platform as the state space, and the set of physical machine groups as the action space. Based on the state space and action space, train a deep Q network to learn the execution efficiency of each physical machine in the heterogeneous computing platform for the task type, and obtain a reinforcement learning model.

[0009] Step 2: Based on the task type of the current task, check the scheduling table to see if there is scheduling experience. If so, schedule the current task to the specified physical unit according to the scheduling experience. Otherwise, input the task request of the current task into the reinforcement learning model. The reinforcement learning model schedules the current task to the specified physical unit with the goal of maximizing the running efficiency.

[0010] Step 3: In the designated physical machine group, based on the resource usage of each physical machine and with the goal of maximizing resource utilization efficiency, schedule the current task to a designated physical machine in the designated physical machine group to run, and obtain the running result of the current task.

[0011] The aforementioned cluster task scheduling method based on deep reinforcement learning, wherein the heterogeneous computing platform includes hardware computing acceleration chips with various architectures.

[0012] The cluster task scheduling method based on deep reinforcement learning includes a scheduling table that comprises a state space consisting of physical unit information, task requirement information from users, and task execution costs, an action space consisting of the set of available physical units, and feedback value. The execution experience is accumulated to calculate the corresponding priority, and the scheduling table sorts the experience sequence according to the priority.

[0013] The cluster task scheduling method based on deep reinforcement learning, wherein step 1 includes: forming the state space based on the user's historical task resource requests and the computing and memory resource information of each physical machine in the computing platform; and forming the action space by aggregating each physical machine group in the computing platform.

[0014] Step 2 includes: If there is no scheduling experience in the scheduling table, the reinforcement learning model calculates the operational efficiency of each action using the following formula and uses it as the value R:

[0015]

[0016] x = (x1, x2, x3…)

[0017] Where A, v, and d represent the amount of resources, latency penalty factor, and task execution time cost consumed when executing the i-th task on a physical machine of type x, respectively; and when calculating the amount of resources A, it is normalized according to the CPU, memory, and GPU performance of the physical machine using the following formula:

[0018]

[0019] Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. l C represents the remaining resources of the l-th physical machine. l M l Gl These represent the total CPU, memory, and GPU resources of the l-th physical machine, respectively. These represent the normalized CPU, memory, and GPU resource quantities of the l-th physical machine, respectively.

[0020] Randomly select a decimal number from 0 to 1. If the selected decimal number is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal number is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action.

[0021] The state space, available actions, the value of available actions, and the state space of the next round after the update are combined into an experience data tuple and stored in the experience data pool. The experience data pool is traversed, and priorities are designed according to value and repetition frequency to sort the experience data tuples in the experience data pool. High-priority data groups are selected as the scheduling table.

[0022] The aforementioned cluster task scheduling method based on deep reinforcement learning includes step 3, which comprises:

[0023] Based on the specified physical unit and the current resource usage status of the specified physical unit, the specified physical machine is obtained using genetic encoding with the objective function of maximizing resource utilization; the objective function includes an objective function of minimizing task execution time and an objective function of minimizing physical machine resource cost.

[0024] Specifically, this genetic coding-based approach includes:

[0025] Step 31: n represents the task sequence of the current task, m represents the number of machines. Encode the task sequence and the specified physical machine group into a two-dimensional matrix (n×m).

[0026] Randomly generate the initial solution group P(t), t=0:

[0027] Step 32: The fitness value is the physical unit resource utilization rate f(x) after task execution. l );

[0028]

[0029]

[0030] Determine f(x) l If the value is greater than the preset threshold, output the solution; otherwise, proceed to step 33.

[0031] Step 33: Use roulette wheel to select the inheritance operator:

[0032] Calculate the fitness value of each solution in the solution group P(t), and sum them to obtain the total fitness value;

[0033]

[0034] The ratio of the fitness value of an individual solution to the sum of fitness values, P(x l ), with P(x l The probability value of selecting an individual;

[0035]

[0036] Based on the probability value, select the two individuals with the highest fitness values ​​from the solution group as parent individuals; according to the set probability Px, perform the crossover operation with probability Px, specifically a single-point crossover.

[0037] Based on experience, a probability Pm is set, and a mutation operation is performed with probability Pm. Specifically, for the part of the two-dimensional matrix that represents the physical machine for task allocation, two mutation positions pos1 and pos2 are randomly generated, and pos1 ≠ pos2. The row code between pos1 and pos2 is reversed.

[0038] Using an elite strategy, retain the best solution in the original solution group: retain the solution with the highest fitness value in the original solution group, denoted as B(t) = Best(P(t)); calculate the fitness value of individual solutions in the new solution group, and denote the best solution in the new solution group as B(t+1) = Best(P(t+1)); replace the solution with the lowest fitness value in P(t+1) with the best solution in P(t); calculate the fitness value of each solution in the new solution group, and return to execute step 32.

[0039] This invention also proposes a cluster task scheduling system based on deep reinforcement learning, which includes:

[0040] The initial module is used to take the task requests, physical machine groups and task execution costs in the heterogeneous computing platform as the state space and the set of physical machine groups as the action space. Based on the state space and action space, a deep Q network is trained to learn the execution efficiency of each physical machine in the heterogeneous computing platform for the task type, and a reinforcement learning model is obtained.

[0041] The lookup module is used to search the scheduling table for scheduling experience based on the task type of the current task. If the experience exists, the current task is scheduled to the specified physical unit. Otherwise, the task request of the current task is input into the reinforcement learning model. The reinforcement learning model, with the goal of maximizing operating efficiency, schedules the current task to the specified physical unit.

[0042] The scheduling module is used to schedule the current task to a specified physical machine in the specified physical group based on the resource usage of each physical machine, with the goal of maximizing resource utilization efficiency, and obtain the running result of the current task.

[0043] The aforementioned cluster task scheduling system based on deep reinforcement learning, wherein the heterogeneous computing platform includes hardware computing acceleration chips with multiple architectures;

[0044] The scheduling table contains a state space consisting of physical unit information, task requirements from users, and task execution costs; an action space consisting of the set of available physical units; and feedback value. The execution experience is accumulated to calculate the corresponding priority, and the scheduling table sorts the experience sequence according to the priority.

[0045] The cluster task scheduling system based on deep reinforcement learning, wherein the initial module is used to: form the state space based on the user's historical task resource requests and the computing and memory resource information of each physical machine in the computing platform; and to form the action space by aggregating each physical machine group in the computing platform.

[0046] The lookup module is used to: if no scheduling experience exists in the scheduling table, the reinforcement learning model calculates the operational efficiency of each action using the following formula and uses it as the value R:

[0047]

[0048] x = (x1, x2, x3…)

[0049] Where A, v, and d represent the amount of resources, latency penalty factor, and task execution time cost consumed when executing the i-th task on a physical machine of type x, respectively; and when calculating the amount of resources A, it is normalized according to the CPU, memory, and GPU performance of the physical machine using the following formula:

[0050]

[0051] Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. l C represents the remaining resources of the l-th physical machine. l M l G l These represent the total CPU, memory, and GPU resources of the l-th physical machine, respectively. These represent the normalized CPU, memory, and GPU resource quantities of the l-th physical machine, respectively.

[0052] Randomly select a decimal number from 0 to 1. If the selected decimal number is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal number is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action.

[0053] The state space, available actions, the value of available actions, and the state space of the next round after the update are combined into an experience data tuple and stored in the experience data pool. The experience data pool is traversed, and priorities are designed according to value and repetition frequency to sort the experience data tuples in the experience data pool. High-priority data groups are selected as the scheduling table.

[0054] This scheduling module is used for:

[0055] Based on the specified physical unit and the current resource usage status of the specified physical unit, the specified physical machine is obtained using genetic encoding with the objective function of maximizing resource utilization; the objective function includes an objective function of minimizing task execution time and an objective function of minimizing physical machine resource cost.

[0056] This genetic coding-based method specifically includes:

[0057] Step 31: n represents the task sequence of the current task, m represents the number of machines. Encode the task sequence and the specified physical machine group into a two-dimensional matrix (n×m).

[0058] Randomly generate the initial solution group P(t), t=0:

[0059] Step 32: The fitness value is the physical unit resource utilization rate f(x) after task execution. l );

[0060]

[0061]

[0062] Determine f(x) l If the value is greater than the preset threshold, output the solution; otherwise, proceed to step 33.

[0063] Step 33: Use roulette wheel to select the inheritance operator:

[0064] Calculate the fitness value of each solution in the solution group P(t), and sum them to obtain the total fitness value;

[0065]

[0066] The ratio of the fitness value of an individual solution to the sum of fitness values, P(x l ), with P(x l The probability value of selecting an individual;

[0067]

[0068] Based on the probability value, select the two individuals with the highest fitness values ​​from the solution group as parent individuals; according to the set probability Px, perform the crossover operation with probability Px, specifically a single-point crossover.

[0069] Based on experience, a probability Pm is set, and a mutation operation is performed with probability Pm. Specifically, for the part of the two-dimensional matrix that represents the physical machine for task allocation, two mutation positions pos1 and pos2 are randomly generated, and pos1 ≠ pos2. The row code between pos1 and pos2 is reversed.

[0070] Using an elite strategy, retain the best solution in the original solution group: retain the solution with the highest fitness value in the original solution group, denoted as B(t) = Best(P(t)); calculate the fitness value of individual solutions in the new solution group, and denote the best solution in the new solution group as B(t+1) = Best(P(t+1)); replace the solution with the lowest fitness value in P(t+1) with the best solution in P(t); calculate the fitness value of each solution in the new solution group, and return to execute step 32.

[0071] The present invention also proposes a storage medium for storing programs that execute any of the deep reinforcement learning-based cluster task scheduling methods.

[0072] The present invention also proposes a client for any of the aforementioned deep reinforcement learning-based cluster task scheduling systems.

[0073] As can be seen from the above solutions, the advantages of the present invention are:

[0074] First, because this invention employs reinforcement learning for task scheduling on heterogeneous computing platforms, it overcomes the shortcomings of existing technologies that do not fully consider the varying acceleration effects of different hardware resources on different task types. This allows the invention to specifically optimize task scheduling for heterogeneous resource computing platforms, effectively balancing the utilization efficiency of different resources within the computing cluster and improving the cluster's resource utilization rate. Simultaneously, leveraging the characteristics of reinforcement learning, this invention also features adaptive real-time optimization of scheduling strategies, contributing to improved computational efficiency of the computing platform.

[0075] Secondly, in designing the reinforcement learning model, this invention uses a data experience pool as a foundation. By comparing experience priorities and action values, a scheduling experience table is formed. This table records the scheduling strategies and values ​​of scheduling actions for high-frequency tasks, serving as a reference for future task scheduling. In large-scale, high-concurrency, and multi-task dispatch scenarios, it effectively avoids meaningless computations, reduces the computational load of reinforcement learning, and improves task scheduling efficiency.

[0076] Third, this invention uses a genetic algorithm in the second intelligent node to further schedule task instances within the physical machine group. By normalizing the CPU, memory, and computing resources as fitness indicators, it can avoid problems such as resource fragmentation and reduced resource utilization caused by uneven resource use in task scheduling. This can reduce the idle time of physical machines, effectively improve the parallel execution efficiency of tasks, and improve the resource utilization of heterogeneous computing platforms. Attached Figure Description

[0077] Figure 1 This is an overall flowchart of the present invention;

[0078] Figure 2 This is a flowchart of the decision-making process of the first intelligent node in this invention;

[0079] Figure 3 This is a flowchart of the decision-making process for the second intelligent node in this invention. Detailed Implementation

[0080] The specific implementation idea of ​​this invention is as follows: First, an improved reinforcement learning method is used to schedule tasks in a heterogeneous computing resource platform in real time. The physical machine group information (based on the hardware acceleration resources (CPU, memory, etc.) of the physical machines, user task requirements, and task execution cost (the amount of resources requested by the task) are used as the state space for reinforcement learning. The set of available physical machines is used as the action space for reinforcement learning. Through a deep Q-network method, by adapting to changes in the heterogeneous computing platform environment information (environment information refers to the waiting time of task requests, the waiting time of physical machines, and resource usage), the system learns to adapt to the different execution efficiencies of dynamically changing task types on heterogeneous resource physical machines. Resource utilization efficiency refers to the proportion of resources used by a physical machine when executing a task. Different execution efficiencies mean that different hardware computing resources have different acceleration effects on the same task. Second, a genetic algorithm is used to further schedule tasks based on the resource usage of different physical machines within the physical machine groups generated by the previous step's decision, maximizing resource utilization efficiency. Available physical machines can be those that meet the computing resources required for task execution or whose waiting time exceeds a threshold.

[0081] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0082] See attached document Figure 1 The following is a further detailed description of the specific steps of the present invention.

[0083] 1. Collect real-time task request sequences, including CPU, memory (mem), and GPU usage.

[0084] The user's task request data is transmitted to the first intelligent node, which schedules the task execution resource group type to select the physical machine group for the current task. This intelligent node can be any hardware machine in the physical machine group executing the deep learning inference task, such as NVIDIA-TX2, ML220, or ML270.

[0085] 2. Compare the Task scheduling table to see if there is any relevant task scheduling experience to refer to.

[0086] After receiving a task, the first intelligent node refers to the task scheduling table, which is composed of accumulated experience. If there is relevant scheduling experience, the task is scheduled directly based on the experience, saving the computation time of the policy network in that instance. The task scheduling table includes a state space (S) consisting of physical unit information, task requirement information from the user, and task execution cost; an action space (α) consisting of the set of available physical units; and feedback value (R). The priority corresponding to the accumulated execution experience is calculated, and the task scheduling table sorts the experience sequence according to the priority.

[0087] For instances where there is no reference scheduling process, the task request sequence and resource information constitute the state space of reinforcement learning.

[0088] 3. Based on the currently collected operational data of this type of task instance on different physical units, schedule it with the goal of maximizing operational efficiency.

[0089] The reinforcement learning model formulates a task scheduling strategy with the goal of maximizing operational efficiency (scheduling tasks to specific physical units) and calculates the value of possible actions in the action space.

[0090] 4. The first intelligent node completes task scheduling based on the calculation results.

[0091] Based on the calculation results, the task instance is assigned to the specified physical unit, and the corresponding task data is transmitted to the second intelligent node.

[0092] 5. The second intelligent node encodes the received task request sequence.

[0093] The algorithm encodes the task sequence with physical units and transforms it into a two-dimensional matrix.

[0094] 6. Find the optimal scheduling scheme to maximize resource utilization.

[0095] The second intelligent node performs scheduling calculations based on the current resource occupancy status of machines in the physical unit, with the goal of maximizing resource utilization.

[0096] 7. Based on the optimal solution, schedule the task to the designated physical machine.

[0097] Using a genetic algorithm, the optimal scheduling scheme is calculated, and task instances are scheduled to the specified physical machines based on the results.

[0098] See attached document Figure 2 The improved deep Q-network method described in this invention will be further described below.

[0099] 1. Generate a training dataset by combining users' historical task resource requests over a period of time into a training dataset;

[0100] 2. Generate the state space and action space for reinforcement learning:

[0101] a. The user's historical task resource requests and the computing and memory resource information of each physical machine on the computing platform constitute the state space of reinforcement learning;

[0102] b. Collect all physical units in the computing platform to form the action space for reinforcement learning; that is, the data in the action space are the numbers of different physical units. If there are two types of machines in the platform, namely Jetson-TX2 and Raspberry, then the action space is (0, 1).

[0103] 3. Referring to the task scheduling table Ttask, which is based on accumulated experience, identical instances are directly scheduled to the designated physical machine resource group. Proceed to step 6;

[0104] 4. When there is no scheduling experience to refer to in the table, calculate the value of each action according to the following formula:

[0105]

[0106] x=(x1,x2,x3…) (2)

[0107] Where A, v, and d represent the resource consumption, latency penalty factor, and task execution time cost respectively when executing the i-th task on a physical unit of type x (representing different types of resource units). Specifically, when calculating the resource consumption (A), the performance indicators of the physical machine, such as CPU, memory, and GPU, are considered, and normalization is performed using the following formula:

[0108]

[0109] Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. l C represents the remaining resources of the l-th physical machine. l M l G l These represent the total CPU, memory, and GPU resources of the l-th physical machine, respectively. These represent the normalized CPU, memory, and GPU resource quantities of the l-th physical machine, respectively.

[0110] 5. Randomly select a decimal number from 0 to 1. If the selected decimal is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action. Explore the value feedback of different actions through random actions; if the maximum value is always selected, the algorithm is prone to getting trapped in local optima.

[0111] 6. Combine the state space, available actions, the value of available actions, and the state space of the next round after the update into an experience data tuple and store it in the experience data pool.

[0112] 6-1. Traverse and compare the experience data pool, design priorities based on value and repetition frequency, and sort the data tuples.

[0113] 6-2. Select high-priority data groups as scheduling experience tables Ttask for comparison and reference in the third step.

[0114] 7. Output action selection and update network parameters. Network parameters refer to the deep reinforcement learning model, which consists of three fully connected layers. Specifically, they are used in the neural network's action value calculation stage.

[0115] See attached document Figure 3 The genetic algorithm used in the second intelligent node described in this invention will be further described.

[0116] Based on the physical unit selected by the first intelligent node, and considering the current resource usage status of the physical unit, the node uses the highest resource utilization rate as the objective function, and encodes it using a genetic algorithm to solve the multi-objective optimization model, obtaining a collaborative optimization result. The objective functions include minimizing task execution time and minimizing physical machine resource costs.

[0117] The specific execution steps of the genetic-based coding algorithm are as follows:

[0118] 1. Each chromosome is encoded using natural numbers, converting the task request sequence and physical units into a two-dimensional matrix.

[0119] n represents the task sequence and m represents the number of machines. The task sequence and physical machine group are binary encoded and converted into a two-dimensional matrix (n×m). The two-dimensional matrix represents the physical machine to which each task is assigned. For example, (0,0,1,0) means that the task is assigned to physical machine 3.

[0120] 2. Randomly generate an initial solution group P(t), t=0; the solution group P is composed of the two-dimensional matrix mentioned above, and the physical machine to be allocated is determined by matrix encoding.

[0121] 3. Fitness calculation.

[0122] The fitness value of this method is the physical unit resource utilization rate after all tasks have been executed.

[0123]

[0124]

[0125] 4. Detect f(x) l Check if the iteration convergence termination condition is met. If it is met, jump to step 10 and output the solution; otherwise, proceed to the next step.

[0126] 5. Use roulette wheel to select the inheritance operator:

[0127] Genetic selection is a process of selecting and retaining superior individuals and eliminating inferior individuals based on their fitness values. This invention uses a roulette wheel method to determine the parent individuals for crossbreeding in each iteration.

[0128] (1) Calculate the fitness value of each solution in the solution group and add them together to obtain the total fitness value;

[0129]

[0130] (2) Calculate the ratio P(x) of the fitness value of an individual solution to the sum of fitness values. l The resulting ratio is the probability value of selecting an individual;

[0131]

[0132] (3) Select the two most adapted (highest fitness value) individual solutions from the solution group as parent individuals based on the probability value (the fitness value algorithm of this invention is better the larger it is).

[0133] 6. Based on experience, set a probability Px and perform crossover operation with probability Px. The crossover method is single-point crossover. Specifically, in the binary encoding, generate a random number pos, and the parent individuals exchange rows pos with each other.

[0134] 7. Based on experience, set a probability Pm and perform mutation operation with probability Pm. Specifically, for the chromosome part representing the physical machine for task allocation in the two-dimensional matrix, randomly generate two mutation positions pos1 and pos2. These two positions must satisfy the following condition: pos1≠pos2. Reverse the row encoding between these two mutation positions pos1 and pos2.

[0135] 8. Use an elite strategy to retain the best solution in the original solution group: retain the solution with the highest fitness value in the original solution group, denoted as B(t) = Best(P(t)); calculate the fitness value of individual solutions in the new solution group, and denote the best solution in the new solution group as B(t+1) = Best(P(t+1)); replace the worst solution in P(t+1) with the best solution in P(t).

[0136] 9. A new solution group is generated as P(t+1), t=t+1. Return to step 4.

[0137] 10. Output the approximate optimal solution.

[0138] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0139] This invention also proposes a cluster task scheduling system based on deep reinforcement learning, which includes:

[0140] The initial module is used to take the task requests, physical machine groups and task execution costs in the heterogeneous computing platform as the state space and the set of physical machine groups as the action space. Based on the state space and action space, a deep Q network is trained to learn the execution efficiency of each physical machine in the heterogeneous computing platform for the task type, and a reinforcement learning model is obtained.

[0141] The lookup module is used to search the scheduling table for scheduling experience based on the task type of the current task. If the experience exists, the current task is scheduled to the specified physical unit. Otherwise, the task request of the current task is input into the reinforcement learning model. The reinforcement learning model, with the goal of maximizing operating efficiency, schedules the current task to the specified physical unit.

[0142] The scheduling module is used to schedule the current task to a specified physical machine in the specified physical group based on the resource usage of each physical machine, with the goal of maximizing resource utilization efficiency, and obtain the running result of the current task.

[0143] The aforementioned cluster task scheduling system based on deep reinforcement learning, wherein the heterogeneous computing platform includes hardware computing acceleration chips with multiple architectures;

[0144] The scheduling table contains a state space consisting of physical unit information, task requirements from users, and task execution costs; an action space consisting of the set of available physical units; and feedback value. The execution experience is accumulated to calculate the corresponding priority, and the scheduling table sorts the experience sequence according to the priority.

[0145] The cluster task scheduling system based on deep reinforcement learning, wherein the initial module is used to: form the state space based on the user's historical task resource requests and the computing and memory resource information of each physical machine in the computing platform; and to form the action space by aggregating each physical machine group in the computing platform.

[0146] The lookup module is used to: if no scheduling experience exists in the scheduling table, the reinforcement learning model calculates the operational efficiency of each action using the following formula and uses it as the value R:

[0147]

[0148] x = (x1, x2, x3…)

[0149] Where A, v, and d represent the amount of resources, latency penalty factor, and task execution time cost consumed when executing the i-th task on a physical machine of type x, respectively; and when calculating the amount of resources A, it is normalized according to the CPU, memory, and GPU performance of the physical machine using the following formula:

[0150]

[0151] Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. l C represents the remaining resources of the l-th physical machine. l M l G l These represent the total CPU, memory, and GPU resources of the l-th physical machine, respectively. These represent the normalized CPU, memory, and GPU resource quantities of the l-th physical machine, respectively.

[0152] Randomly select a decimal number from 0 to 1. If the selected decimal number is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal number is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action.

[0153] The state space, available actions, the value of available actions, and the state space of the next round after the update are combined into an experience data tuple and stored in the experience data pool. The experience data pool is traversed, and priorities are designed according to value and repetition frequency to sort the experience data tuples in the experience data pool. High-priority data groups are selected as the scheduling table.

[0154] This scheduling module is used for:

[0155] Based on the specified physical unit and the current resource usage status of the specified physical unit, the specified physical machine is obtained using genetic encoding with the objective function of maximizing resource utilization; the objective function includes an objective function of minimizing task execution time and an objective function of minimizing physical machine resource cost.

[0156] This genetic coding-based method specifically includes:

[0157] Step 31: n represents the task sequence of the current task, m represents the number of machines. Encode the task sequence and the specified physical machine group into a two-dimensional matrix (n×m).

[0158] Randomly generate the initial solution group P(t), t=0:

[0159] Step 32: The fitness value is the physical unit resource utilization rate f(x) after task execution. l );

[0160]

[0161]

[0162] Determine f(x) l If the value is greater than the preset threshold, output the solution; otherwise, proceed to step 33.

[0163] Step 33: Use roulette wheel to select the inheritance operator:

[0164] Calculate the fitness value of each solution in the solution group P(t), and sum them to obtain the total fitness value;

[0165]

[0166] The ratio of the fitness value of an individual solution to the sum of fitness values, P(x l ), with P(x l The probability value of selecting an individual;

[0167]

[0168] Based on the probability value, select the two individuals with the highest fitness values ​​from the solution group as parent individuals; according to the set probability Px, perform the crossover operation with probability Px, specifically a single-point crossover.

[0169] Based on experience, a probability Pm is set, and a mutation operation is performed with probability Pm. Specifically, for the part of the two-dimensional matrix that represents the physical machine for task allocation, two mutation positions pos1 and pos2 are randomly generated, and pos1 ≠ pos2. The row code between pos1 and pos2 is reversed.

[0170] Using an elite strategy, retain the best solution in the original solution group: retain the solution with the highest fitness value in the original solution group, denoted as B(t) = Best(P(t)); calculate the fitness value of individual solutions in the new solution group, and denote the best solution in the new solution group as B(t+1) = Best(P(t+1)); replace the solution with the lowest fitness value in P(t+1) with the best solution in P(t); calculate the fitness value of each solution in the new solution group, and return to execute step 32.

[0171] The present invention also proposes a storage medium for storing programs that execute any of the deep reinforcement learning-based cluster task scheduling methods.

[0172] The present invention also proposes a client for any of the aforementioned deep reinforcement learning-based cluster task scheduling systems.

Claims

1. A cluster task scheduling method based on deep reinforcement learning, characterized in that, include: Step 1: Take the task requests, physical machine groups, and task execution costs in the heterogeneous computing platform as the state space, and the set of physical machine groups as the action space. Based on the state space and action space, train a deep Q network to learn the execution efficiency of each physical machine in the heterogeneous computing platform for the task type, and obtain a reinforcement learning model. Step 2: Based on the task type of the current task, check the scheduling table to see if there is scheduling experience. If so, schedule the current task to the specified physical unit according to the scheduling experience. Otherwise, input the task request of the current task into the reinforcement learning model. The reinforcement learning model schedules the current task to the specified physical unit with the goal of maximizing the running efficiency. Step 3: In the designated physical machine group, based on the resource usage of each physical machine and with the goal of maximizing resource utilization efficiency, schedule the current task to a designated physical machine in the designated physical machine group to run, and obtain the running result of the current task. Step 1 includes: forming the state space based on the user's historical task resource requests and the computing and memory resource information of each physical machine on the computing platform; and assembling the physical machine groups in the computing platform to form the action space. Step 2 includes: If there is no scheduling experience in the scheduling table, the reinforcement learning model calculates the operational efficiency of each action using the following formula and uses it as the value. : Where A, v, and d represent the amount of resources, latency penalty factor, and task execution time cost consumed when executing the i-th task on a physical machine of type x, respectively; and when calculating the amount of resources A, it is normalized according to the CPU, memory, and GPU performance of the physical machine using the following formula: Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. Indicates the first The remaining resources of each physical machine They represent the first The total CPU, memory, and GPU resources of a physical machine. They represent the first Normalized CPU, memory, and GPU resource quantities for each physical machine; Randomly select a decimal number from 0 to 1. If the selected decimal number is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal number is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action. The state space, available actions, the value of available actions, and the state space of the next round after the update are combined into an experience data tuple and stored in the experience data pool. The experience data pool is traversed, and priorities are designed according to value and repetition frequency to sort the experience data tuples in the experience data pool. High-priority data groups are selected as the scheduling table.

2. The cluster task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, This heterogeneous computing platform includes hardware computing acceleration chips with various architectures.

3. The cluster task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, The scheduling table contains a state space consisting of physical unit information, task requirements from users, and task execution costs; an action space consisting of the set of available physical units; and feedback value. The execution experience is accumulated to calculate the corresponding priority, and the scheduling table sorts the experience sequence according to the priority.

4. The cluster task scheduling method based on deep reinforcement learning as described in claim 1, characterized in that, Step 3 includes: Based on the specified physical unit and the current resource usage status of the specified physical unit, the specified physical machine is obtained using genetic encoding with the objective function of maximizing resource utilization; the objective function includes an objective function of minimizing task execution time and an objective function of minimizing physical machine resource cost. Specifically, this genetic coding-based approach includes: Step 31 This represents the task sequence of the current task. The number of machines is represented by the binary encoding of the task sequence and the specified physical unit, which is then converted into a two-dimensional matrix. ); Randomly generate the initial solution group P(t), t=0: Step 32: The fitness value is the physical unit resource utilization rate after task execution. ; judge If the value is greater than the preset threshold, output the solution; otherwise, proceed to step 33. Step 33: Use roulette wheel to select the inheritance operator: Calculate the fitness value of each solution in the solution group P(t), and sum them to obtain the total fitness value; Based on the ratio of the fitness value of an individual solution to the sum of fitness values ,by The probability value of selecting an individual; Based on this probability value, select the two individuals with the highest fitness values ​​from the solution group as parent individuals; based on the set probability... With probability Perform a crossover operation, specifically a single-point crossover; Probability is set based on experience. With probability The mutation operation is performed by randomly generating two mutation locations for the portion of the two-dimensional matrix representing the physical machines used for task allocation. and ,and ,Will and Reverse the row codes between them; Using an elite strategy, the optimal solution in the original solution group is retained: the solution with the highest fitness value in the original solution group is retained, denoted as... Calculate the fitness values ​​of individual solutions in the new solution group, and denote the optimal solution in the new solution group as . ;Will The solution with the lowest fitness value is replaced with Calculate the fitness values ​​of each solution in the new solution group and return to execute step 32.

5. A cluster task scheduling system based on deep reinforcement learning, characterized in that, include: The initial module is used to take the task requests, physical machine groups and task execution costs in the heterogeneous computing platform as the state space and the set of physical machine groups as the action space. Based on the state space and action space, a deep Q network is trained to learn the execution efficiency of each physical machine in the heterogeneous computing platform for the task type, and a reinforcement learning model is obtained. The lookup module is used to search the scheduling table for scheduling experience based on the task type of the current task. If the experience exists, the current task is scheduled to the specified physical unit. Otherwise, the task request of the current task is input into the reinforcement learning model. The reinforcement learning model, with the goal of maximizing operating efficiency, schedules the current task to the specified physical unit. The scheduling module is used to schedule the current task to a specified physical machine in the specified physical machine group based on the resource usage of each physical machine, with the goal of maximizing resource utilization efficiency, and obtain the running result of the current task. This initial module is used to: form the state space based on the user's historical task resource requests and the computing and memory resource information of each physical machine on the computing platform; This action space is formed by combining the physical units in the computing platform. The lookup module is used to: if no scheduling experience exists in the scheduling table, the reinforcement learning model calculates the operational efficiency of each action using the following formula and uses it as the value. : Where A, v, and d represent the amount of resources, latency penalty factor, and task execution time cost consumed when executing the i-th task on a physical machine of type x, respectively; and when calculating the amount of resources A, it is normalized according to the CPU, memory, and GPU performance of the physical machine using the following formula: Where C, M, and G represent the physical machine's CPU, memory, and GPU resources, respectively. Indicates the first The remaining resources of each physical machine They represent the first The total CPU, memory, and GPU resources of a physical machine. They represent the first Normalized CPU, memory, and GPU resource quantities for each physical machine; Randomly select a decimal number from 0 to 1. If the selected decimal number is between 0 and 0.9, select the action with the highest action value as the available action. If the selected decimal number is between 0.9 and 1, randomly select an action other than the action with the highest action value as the available action. The state space, available actions, the value of available actions, and the state space of the next round after the update are combined into an experience data tuple and stored in the experience data pool. The experience data pool is traversed, and priorities are designed according to value and repetition frequency to sort the experience data tuples in the experience data pool. High-priority data groups are selected as the scheduling table.

6. The cluster task scheduling system based on deep reinforcement learning as described in claim 5, characterized in that, This heterogeneous computing platform includes hardware computing acceleration chips with various architectures; The scheduling table contains a state space consisting of physical unit information, task requirements from users, and task execution costs; an action space consisting of the set of available physical units; and feedback value. The execution experience is accumulated to calculate the corresponding priority, and the scheduling table sorts the experience sequence according to the priority.

7. The cluster task scheduling system based on deep reinforcement learning as described in claim 5, characterized in that, This scheduling module is used for: Based on the specified physical unit, and taking into account the current resource usage status of the specified physical unit, the specified physical unit is obtained using the highest resource utilization rate as the objective function and based on genetic encoding. The objective function includes the objective function of minimizing task execution time and the objective function of minimizing physical machine resource cost; This genetic coding-based method specifically includes: Step 31 This represents the task sequence of the current task. The number of machines is represented by the binary encoding of the task sequence and the specified physical unit, which is then converted into a two-dimensional matrix. ); Randomly generate the initial solution group P(t), t=0: Step 32: The fitness value is the physical unit resource utilization rate after task execution. ; judge If the value is greater than the preset threshold, output the solution; otherwise, proceed to step 33. Step 33: Use roulette wheel to select the inheritance operator: Calculate the fitness value of each solution in the solution group P(t), and sum them to obtain the total fitness value; Based on the ratio of the fitness value of an individual solution to the sum of fitness values ,by The probability value of selecting an individual; Based on this probability value, select the two individuals with the highest fitness values ​​from the solution group as parent individuals; based on the set probability... With probability Perform a crossover operation, specifically a single-point crossover; Probability is set based on experience. With probability The mutation operation is performed by randomly generating two mutation locations for the portion of the two-dimensional matrix representing the physical machines used for task allocation. and ,and ,Will and Reverse the row codes between them; Using an elite strategy, the optimal solution in the original solution group is retained: the solution with the highest fitness value in the original solution group is retained, denoted as... Calculate the fitness values ​​of individual solutions in the new solution group, and denote the optimal solution in the new solution group as . ;Will The solution with the lowest fitness value is replaced with Calculate the fitness values ​​of each solution in the new solution group and return to execute step 32.

8. A storage medium for storing a program that executes any of the deep reinforcement learning-based cluster task scheduling methods described above.

9. A client for any of the deep reinforcement learning-based cluster task scheduling systems described above.

Images