Batch job load balancing method and device based on distributed reinforcement learning

Abstract

The invention relates to the technical field of intelligent operation and maintenance resource management, in particular to a batch job load balancing method and device based on distributed reinforcement learning. The method comprises the following steps: submitting user requests in a batch job form according to an arrival time sequence; according to the cluster resource information and the resource request of the current batch job, the intelligent agent executes the action of creating and running the optimal virtual machine of the current job, and obtains a return signal; inputting the load information of the current cluster into the quantile regression network based on the action of selecting the virtual machine to obtain the value distribution of each action, respectively obtaining the expectation of the value distribution of each action to obtain a corresponding target Q value, and determining the optimal action selected by the intelligent agent in the next step according to the maximum target Q value; and optimizing the quantile regression network according to the value distribution corresponding to the optimal action in combination with the return signal. The method can solve the problem that it is difficult to converge to be close to the optimal strategy well in the face of complex batch job loads.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent operation and maintenance resource management, and in particular to a batch job load balancing method and device based on distributed reinforcement learning. Background Art

[0002] Cloud computing has the characteristics of resource sharing, on-demand access, and flexible deployment. Cloud computing platforms allocate configurable computing resource sharing pools to different users through the Internet. Users submit various service requests according to their own task requirements and quickly deploy applications. Load balancing scheduling reasonably distributes service load requests, which plays a vital role in maintaining the stability of cloud computing platforms and improving user experience. With the widespread use of cloud computing services, the load of batch job types has increased significantly. How to effectively schedule batch jobs is crucial to maintaining cloud computing load balance. Load balancing faces challenges from multiple levels and aspects. In terms of user service requests, the running time of batch jobs submitted by users ranges from a few seconds to a few days, the arrival time is random, and the resource utilization rate fluctuates significantly over time. In addition, batch jobs are combined according to different applications submitted and run by users, and each job contains multiple computing tasks with different resource requests. In terms of cluster service resources, due to the complexity and unpredictability of user requests, clusters need to dynamically adjust resources to meet user needs. Cloud computing clusters are composed of machines with various resource configurations, providing multi-dimensional resources including CPU, memory, and storage. It is difficult to achieve load balancing in multiple resource dimensions. In terms of user service quality, violation of service level agreements means that the service cannot be guaranteed, which leads to unnecessary consequences. The goal of load balancing scheduling in cloud computing platforms is to prevent various violations of service level agreements, improve service quality as much as possible, and increase the success rate of completing user requests. Therefore, designing an adaptive and efficient solution for load balancing scheduling in cloud computing is a very challenging task.

[0003] Rule-based and heuristic algorithms can solve the problem to a certain extent, but they usually use the system's prior knowledge and expert experience to build rules and formulate strategies for different scenarios and conditions. With the expansion of cluster size and the growth of users, this method is only applicable to specific scenarios and cannot adapt to cloud computing environments with dynamic system states and user needs. In recent years, in order to cope with the problems and shortcomings of rule-based and heuristic methods, scheduling algorithms based on deep reinforcement learning have become an effective way to solve load balancing problems. However, in the face of complex batch job loads, it is difficult to converge to an optimal strategy. Summary of the invention

[0004] In view of the defects existing in the prior art, the purpose of the present invention is to provide a batch job load balancing method and device based on distributed reinforcement learning, which can solve the problem in the prior art that it is difficult to converge to an optimal strategy when facing complex batch job loads.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] In one aspect, the present invention provides a batch job load balancing method based on distributed reinforcement learning, comprising the following steps:

[0007] Submit user requests in batch jobs in the order of their arrival time;

[0008] The agent creates and runs the best virtual machine for the current job based on the cluster resource information and the resource request of the current batch job, and obtains a reward signal;

[0009] Based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action. The expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value. The optimal action selected by the agent in the next step is determined according to the maximum target Q value.

[0010] According to the value distribution corresponding to the optimal action and combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network.

[0011] In some optional solutions, the obtaining reward signal includes:

[0012] Set service level agreement constraints based on the average load of the cluster in terms of CPU and memory resources;

[0013] Based on the service level agreement constraints, the effectiveness of the action is judged, and the reward signal is determined based on the current cluster resource load and the average cluster load in the CPU and memory resource dimensions.

[0014] In some optional solutions, the service level agreement constraints are: Among them, insCpu p The CPU resource request for the pth instance in the current batch job task, insMem p is the memory resource request of the pth instance in the current batch job task, is the currently available CPU resource capacity, is the currently available memory resource capacity, p represents the pth instance of the current task, and P represents the total number of instances that need to be created for the current task.

[0015] In some optional solutions, a reward signal function is designed based on minimizing the cluster load balance and maximizing the instance creation success rate.

[0016] In some alternative schemes, according to the reward signal function Determine the return signal, where lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg The average load of the cluster in terms of memory resources.

[0017] In some optional solutions, the minimum cluster load balancing degree is:

[0018] f1=min(|lCpu i -lCpu avg |+|lMem i -lMem avg |);

[0019] The maximum instance creation success rate is:

[0020]

[0021] Among them, lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg is the average load of the cluster in the memory resource dimension, insNum j,o To create a task task j,o The number of instances required, insSucc j,o To create a task task j,o The number of successful instances, N is the number of batch jobs, O is the number of jobs j The number of tasks in, j is the batch job number, and o is the task number.

[0022] In some alternative schemes, according to the formula Determine the current CPU load lCpu of the i-th virtual machine i And the memory load lMem of the current i-th virtual machine i ;

[0023] According to the formula Determine the average load of the cluster in the CPU resource dimension lCpu avg, the average load of the cluster in the memory resource dimension lMem avg ;

[0024] in, is the total CPU resource capacity, is the total memory resource capacity, and M is the total number of virtual machines.

[0025] In some alternative schemes, according to the formula Determine the target distribution for training r is the return signal, Z target is the value distribution of selecting virtual machine actions, and γ is the discount factor.

[0026] In some optional schemes, the optimized quantile regression network includes:

[0027] Based on the quantile regression loss function, the quantile regression loss of the quantile regression network is obtained according to the target distribution used for training;

[0028] Optimize a quantile regression network based on its quantile regression loss.

[0029] On the other hand, the present invention also provides a batch job load balancing method based on distributed reinforcement learning, comprising:

[0030] Batch job arrival module, which is used to submit user requests in the form of batch jobs in the order of arrival time;

[0031] The deep reinforcement learning load balancing module is used to enable the intelligent agent to execute the action of creating and running the best virtual machine for the current job according to the cluster resource information and the resource request of the current batch job, and obtain the reward signal; based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action, and the expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value, and the optimal action selected by the intelligent agent in the next step is determined according to the maximum target Q value; according to the value distribution corresponding to the optimal action, combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network.

[0032] Compared with the prior art, the advantages of the present invention are: this scheme uses an intelligent agent to execute the action of creating and running the best virtual machine for the current job according to the cluster resource information and the resource request of the current batch job, and obtains a reward signal; based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action, and the expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value, and the optimal action selected by the intelligent agent in the next step is determined according to the maximum target Q value, which can effectively improve the performance of batch job load balancing scheduling, and according to the value distribution corresponding to the optimal action, combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network, which can guide the intelligent agent to make the optimal action in the next selection, so as to improve the performance of batch job load balancing scheduling and better converge to the optimal strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0034] Figure 1 Flow chart of a batch job load balancing method based on distributed reinforcement learning in an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a load balancing system model diagram in an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of a deep reinforcement learning load balancing module in an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of a quantile regression network in an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0039] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0040] like Figure 1As shown, on the one hand, the present invention provides a batch job load balancing method based on distributed reinforcement learning, comprising the following steps:

[0041] S1: Submit user requests in the form of batch jobs in the order of their arrival time.

[0042] Aiming at the scenario of batch job load balancing scheduling in cloud computing, a batch job load balancing system model is constructed, such as Figure 2 As shown in Figure 1, a cloud computing cluster consists of virtual machines with different resource configurations. Requests submitted by users arrive in the form of batch jobs, which can be described by the "job-task-instance" model.

[0043] Batch jobs are represented as: {job1, job2, job j ,…,job N},j∈[1,N]

[0044] Among them, job j is the job with serial number j, and N is the number of batch jobs.

[0045] The relationship between jobs and tasks is:

[0046] job j ={task j,1 ,task j,2 ,task j,o ,…,task j,O},i∈[1,O]

[0047] Among them, task j,o For job j The task with sequence number o is O, and job is job j The number of tasks in .

[0048] Instances of the same task require the same CPU and memory resources when they are created, and have the same execution duration after successful creation, and can be created on different virtual machines. The most important attribute of a task is the number of instances that need to be created. In our system, an instance is the smallest scheduling unit, so under the same conditions, a task with a larger number of instances requires more scheduling times than a task with a smaller number of instances. j,o ,insSucc j,o ,insCpu j,o ,insMem j,o Respectively represent the creation of tasks j,o The number of instances required, the number of instances created successfully, and the CPU and memory resources required to create an instance.

[0049] A cloud computing cluster can be represented as: VMs = {VM1, VM2, VM i ,…,VM M},i∈[1,M]

[0050] Among them, VM i Indicates the virtual machine with serial number i. The most important attributes of a cluster are the resource capacity of the virtual machine, such as CPU and memory. and Represents a VM i Total CPU and memory resource capacity. and Represents a VM i The current available CPU and memory resource capacity.

[0051] S2: The agent creates and runs the best virtual machine for the current job based on the cluster resource information and the resource request of the current batch job, and obtains a reward signal.

[0052] In some optional embodiments, step S2 includes the following steps:

[0053] S21: Set service level agreement constraints based on the average load of the cluster in the CPU and memory resource dimensions.

[0054] In this example, the service level agreement constraints are: Among them, insCpu p The CPU resource request for the pth instance in the current batch job task, insMem p is the memory resource request of the pth instance in the current batch job task, is the currently available CPU resource capacity, is the currently available memory resource capacity, p represents the pth instance of the current task, and P represents the total number of instances that need to be created for the current task.

[0055] S22: Based on the service level agreement constraints, determine the effectiveness of the action, and determine the return signal according to the current cluster resource load and the average cluster load in the CPU and memory resource dimensions.

[0056] In this case, according to the return signal function Determine the return signal, where lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg The average load of the cluster in terms of memory resources.

[0057] According to the formula Determine the current CPU load lCpu of the i-th virtual machine i And the memory load lMem of the current i-th virtual machine i ;

[0058] According to the formula Determine the average load of the cluster in the CPU resource dimension lCpu avg , the average load of the cluster in the memory resource dimension lMem avg ;

[0059] in, is the total CPU resource capacity, is the total memory resource capacity, and M is the total number of virtual machines.

[0060] In some optional embodiments, a reward signal function is designed based on minimizing the cluster load balance and maximizing the instance creation success rate.

[0061] The minimum cluster load balancing degree is:

[0062] f1=min(|lCpu i -lCpu avg |+|lMem i -lMem avg |).

[0063] The maximum instance creation success rate is:

[0064]

[0065] Among them, lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg is the average load of the cluster in the memory resource dimension, insNum j,o To create a task task j,o The number of instances required, insSucc j,o To create a task task j,o The number of successful instances, N is the number of batch jobs, O is the number of jobs j The number of tasks in, j is the batch job number, and o is the task number.

[0066] In addition, the load balance of the cluster in terms of CPU and memory resources is expressed as follows:

[0067]

[0068] The definition of load balance is obtained by calculating the standard deviation of the load of all machines in the cluster. The larger the load balance, the more unbalanced the cluster load is. On the contrary, the smaller the load balance, the more balanced the cluster load is.

[0069] S3: Based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action, and the expectation of the value distribution of each action is obtained to obtain the corresponding target Q value. The optimal action for the next step of the intelligent agent is determined according to the maximum target Q value.

[0070] The Q value is used to represent the expected return that can be obtained by taking a certain action in a certain state. Specifically, the Q value function is the expected value of the cumulative return obtained in the future after taking action a in state s. The target Q value is a reference value used to update the Q value function and is used to guide the update of the Q value function.

[0071] The agent is a model that is trained using a dynamic load balancing algorithm based on a quantile regression network and reaches convergence. Figure 4 First, the agent collects job and cluster resource information, makes an action to select a virtual machine, then determines the effectiveness of the action based on the service level agreement, and finally calculates the reward value to be returned to the agent, completing an interaction between the agent and the environment, as shown in Figure 3 shown.

[0072] The action of the agent to select a virtual machine is represented by the action space:

[0073] A={a t |a t ∈{0,1,2,…,M}}

[0074] Among them, a t =0 means not to select a virtual machine in the cluster or a certain virtual machine, M is the total number of virtual machines, a t =M indicates that the Mth virtual machine is selected.

[0075] The current cluster load information is represented by the state space S:

[0076] s1=[lCpu1,lMem1,…,lCpu N ,lMem N ,lCpu avg ,lMem avg ]

[0077] s2=[taskType,ins_cpu p ,ins_mem p ,ins_nump ]

[0078] Among them, lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg is the average load of the cluster in the memory resource dimension, taskType is the task type in the current job, ins_cpu p CPU request for the current task instance, ins_mem p ins_cpu p The memory request of the current task instance, ins_num p The number of current task instances.

[0079] The quantile regression algorithm is introduced to learn the value distribution of cumulative returns in the load balancing process, and a dynamic load balancing algorithm based on the quantile regression deep Q network is proposed. The representation of cumulative returns is improved from value calculation to value distribution modeling, and the dimension of the load balancing change information represented by the cumulative returns is increased. The intrinsic randomness of the load balancing process caused by random changes in the environment can be more comprehensively obtained. The quantile regression network is as follows Figure 4 shown.

[0080] Using quantile regression, the cumulative return is regarded as a random variable Q, and the distribution Z of the random variable Q is approximately modeled. The range of the random variable Q is divided into L parts. The purpose is to make the random variable Q appear with equal probability in L intervals, that is, all are That is, the quantile weight. The purpose of this approximation is to estimate the quantile of the distribution Z of the random variable Q, that is, the quantile distribution. The quantile distribution space of fixed L is represented by Z Q The cumulative probability of the quantile distribution is expressed as τ1,…,τ i ,…,τ L , where i∈[0,1,…,L],

[0081] The quantile regression deep Q-network uses L estimated parameters {θ1,...,θ i ,...,θ L} describes a distribution, so the quantile distribution Z θ ∈Z Q It is expressed as:

[0082]

[0083] Among them, L is the number of distribution support points, δ zrepresents the Dirac function of z∈R, z is a variable, R is the real number space, a is the action space, s is the state space, Z θ (s,a) represents the value distribution of each action.

[0084] S4: Based on the value distribution corresponding to the optimal action and the reward signal, we obtain the target distribution for training to optimize the quantile regression network.

[0085] In some optional embodiments, optimizing the quantile regression network includes the following steps:

[0086] S41: Based on the quantile regression loss function, the quantile regression loss of the quantile regression network is obtained according to the target distribution used for training.

[0087] In this example, according to the formula Determine the target distribution for training r is the return signal, Z target is the value distribution of selecting virtual machine actions, and γ is the discount factor.

[0088] The quantile regression loss used to train quantile neural networks in quantile reinforcement learning is expressed as:

[0089]

[0090] Among them, ρ τ represents the function for calculating the quantile regression loss for the parameter θ of the quantile neural network, and E represents the expectation of the content of []. is the empirical distribution sampled from the real samples of distribution Z, is the quantile regression loss.

[0091] S42: Optimize the quantile regression network according to its quantile regression loss.

[0092] In this example, there is no sequence between the above steps S2, S3 and S4, and the above embodiment is only for the purpose of explaining the present solution logically and clearly.

[0093] On the other hand, the present invention also provides a batch job load balancing device based on distributed reinforcement learning in cloud computing, including: a batch job load balancing system model including a batch job arrival module and a deep reinforcement learning load balancing module.

[0094] Among them, the batch job arrival module is used to submit user requests in the form of batch jobs in the order of arrival time; the deep reinforcement learning load balancing module includes an agent execution unit, an execution action determination unit and an optimization feedback unit. The agent execution unit is used to enable the agent to execute the action of creating and running the best virtual machine for the current job according to the cluster resource information and the resource request of the current batch job, and obtain a reward signal; the execution action determination unit is used to input the load information of the current cluster into the quantile regression network based on the action of selecting the virtual machine, obtain the value distribution of each action, obtain the expectation of the value distribution of each action, obtain the corresponding target Q value, and determine the optimal action selected by the agent in the next step according to the maximum target Q value; the optimization feedback unit is used to obtain the target distribution for training according to the value distribution corresponding to the optimal action and the reward signal, so as to optimize the quantile regression network.

[0095] Among them, the functional implementation of each module in the above-mentioned batch job load balancing device based on distributed reinforcement learning in cloud computing corresponds to the steps in the above-mentioned batch job load balancing method based on distributed reinforcement learning. Its functions and implementation processes will not be repeated here one by one.

[0096] In summary, this scheme uses an intelligent agent to execute the action of creating and running the best virtual machine for the current job according to the cluster resource information and the resource request of the current batch job, and obtains a reward signal; based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action, and the expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value, and the optimal action selected by the intelligent agent in the next step is determined according to the maximum target Q value, which can effectively improve the performance of batch job load balancing scheduling, and according to the value distribution corresponding to the optimal action, combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network, which can guide the intelligent agent to make the optimal action in the next selection to improve the performance of batch job load balancing scheduling.

[0097] In the description of the present application, it should be noted that the terms "upper", "lower", etc. indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. Unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present application can be understood according to the specific circumstances.

[0098] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0099] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A batch job load balancing method based on distributed reinforcement learning, characterized in that: The following steps are involved: Submit user requests in batch jobs in the order of their arrival time; The agent creates and runs the best virtual machine for the current job based on the cluster resource information and the resource request of the current batch job, and obtains a reward signal; Based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action. The expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value. The optimal action selected by the agent in the next step is determined according to the maximum target Q value. According to the value distribution corresponding to the optimal action and combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network.

2. The batch job load balancing method based on distributed reinforcement learning according to claim 1, characterized in that: The reward signal includes: Set service level agreement constraints based on the average load of the cluster in terms of CPU and memory resources; Based on the service level agreement constraints, the effectiveness of the action is judged, and the reward signal is determined based on the current cluster resource load and the average cluster load in the CPU and memory resource dimensions.

3. The batch job load balancing method based on distributed reinforcement learning according to claim 2, characterized in that: The service level agreement constraints are: Among them, insCpu p The CPU resource request for the pth instance in the current batch job task, insMem p is the memory resource request of the pth instance in the current batch job task, is the currently available CPU resource capacity, is the currently available memory resource capacity, p represents the pth instance of the current task, and P represents the total number of instances that need to be created for the current task.

4. The batch job load balancing method based on distributed reinforcement learning according to claim 3, characterized in that: Design a reward signal function based on minimizing cluster load balancing and maximizing instance creation success rate.

5. The batch job load balancing method based on distributed reinforcement learning according to claim 4, characterized in that: According to the return signal function Determine the return signal, where lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg The average load of the cluster in terms of memory resources.

6. The batch job load balancing method based on distributed reinforcement learning according to claim 5, characterized in that: The minimum cluster load balancing degree f1 is: f1=min(|lCpu i -lCpu avg |+|lMem i -lMem avg |); The maximum instance creation success rate f2 is: Among them, lCpu i Indicates the CPU load of the current i-th virtual machine, lMem i is the memory load of the current i-th virtual machine, lCpu avg is the average load of the cluster in the CPU resource dimension, lMem avg is the average load of the cluster in the memory resource dimension, insNum j,o To create a task task j,o The number of instances required, insSucc j,o To create a task task j,o The number of successful instances, N is the number of batch jobs, O is the number of jobs j The number of tasks in, j is the batch job number, and o is the task number.

7. The batch job load balancing method based on distributed reinforcement learning according to claim 6, characterized in that: According to the formula Determine the current CPU load lCpu of the i-th virtual machine i And the memory load lMem of the current i-th virtual machine i ; According to the formula Determine the average load of the cluster in the CPU resource dimension lCpu avg , the average load of the cluster in the memory resource dimension lMem avg ; in, is the total CPU resource capacity, is the total memory resource capacity, and M is the total number of virtual machines.

8. The batch job load balancing method based on distributed reinforcement learning according to claim 1, characterized in that: According to the formula Determine the target distribution for training r is the return signal, Z target is the value distribution of selecting virtual machine actions, and γ is the discount factor.

9. The batch job load balancing method based on distributed reinforcement learning according to claim 8, characterized in that: The optimized quantile regression network includes: Based on the quantile regression loss function, the quantile regression loss of the quantile regression network is obtained according to the target distribution used for training; Optimize a quantile regression network based on its quantile regression loss.

10. A batch job load balancing method based on distributed reinforcement learning, characterized in that: include: Batch job arrival module, which is used to submit user requests in the form of batch jobs in the order of arrival time; The deep reinforcement learning load balancing module is used to enable the intelligent agent to execute the action of creating and running the best virtual machine for the current job according to the cluster resource information and the resource request of the current batch job, and obtain the reward signal; based on the action of selecting the virtual machine, the load information of the current cluster is input into the quantile regression network to obtain the value distribution of each action, and the expectation of the value distribution of each action is obtained respectively to obtain the corresponding target Q value, and the optimal action selected by the intelligent agent in the next step is determined according to the maximum target Q value; according to the value distribution corresponding to the optimal action, combined with the reward signal, the target distribution for training is obtained to optimize the quantile regression network.

Images