Cluster load balancing method, device, equipment and readable storage medium

Abstract

The present invention provides a cluster load balancing method, apparatus, device, and readable storage medium. The cluster load balancing method includes: a meta-controller generates sub-goals based on the current utilization of cluster resources; a job scheduling controller allocates cluster resources to jobs to be scheduled; an elastic scaling controller expands or recycles cluster resources based on the current utilization of cluster resources; and the meta-controller rewards the meta-controller based on changes in the cluster load balancing degree to update the strategy for generating sub-goals. Through the present invention, the meta-controller generates sub-goals based on the current utilization of cluster resources. While the job scheduling controller allocates cluster resources to jobs to be scheduled, the elastic scaling controller expands or recycles cluster resources. The meta-controller is then rewarded to update and generate a better strategy for generating sub-goals, thereby better maintaining the cluster load balancing state and resolving resource bottleneck problems.

Description

Technical Field

[0001] The present invention relates to the technical field of cluster scheduling, and in particular to a cluster load balancing method, apparatus, device and readable storage medium. Background Art

[0002] Cloud computing, as a new computing paradigm, uses virtualization technology to aggregate nodes into a resource pool, providing customers with configurable, on-demand computing resources on a pay-as-you-go basis. Cloud service providers offer different instance types for virtual machines, each with varying resource capacities. Compared to physical resources, the flexibility and virtual nature of cloud resources make them easier to manage and deploy. However, effective resource management also poses significant challenges. Currently, job scheduling and resource management are hot topics in cloud computing research. Job scheduling in cloud computing allocates computing resources from cluster nodes to jobs. In this process, inappropriate job scheduling strategies and unreasonable cluster sizing can affect cluster load balancing. Proper resource sizing depends on a comprehensive understanding of environmental changes and the dynamic factors that may affect system performance. Many resource allocation schemes in the cloud are based on thresholds, time series analysis, or heuristics. However, most of these works assume homogeneous cluster configurations and leverage prior knowledge of the cloud system. While they can address some of the challenges of cloud resource management, they are unable to adapt to the complexity and uncertainty of cloud environments.

[0003] Reinforcement learning (RL) is used to solve complex real-world problems. In RL, an intelligent agent continuously interacts with the environment to collect system states and develop an updated knowledge base. This knowledge base is then used to update the agent. The agent is modeled as a control loop, which incrementally learns and updates the optimal strategy through trial and error. However, traditional reinforcement learning methods suffer from dimensionality explosion when dealing with problems with large state and action spaces. To address this issue, a deep reinforcement learning (DRL) method was proposed by combining deep neural networks with reinforcement learning. However, due to the uncertainty of the number and arrival time of jobs, a sharp increase in the workload can still easily lead to load imbalance and resource bottlenecks in cluster resources. Summary of the Invention

[0004] The main purpose of the present invention is to provide a cluster load balancing method, device, equipment and readable storage medium, aiming to solve the technical problem that when the arrival amount and arrival time of jobs are uncertain, the existing methods are still likely to cause load imbalance and resource bottlenecks in cluster resources when the job volume increases sharply.

[0005] In a first aspect, the present invention provides a cluster load balancing method, the cluster load balancing method comprising:

[0006] The meta-controller generates sub-goals based on the current cluster resource utilization;

[0007] When the generated sub-goal is job scheduling, the job scheduling controller allocates cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource requirements of the job to be scheduled;

[0008] When the generated sub-goal is elastic scaling, the elastic scaling controller expands or recycles cluster resources based on the current cluster resource utilization;

[0009] After the sub-goal is executed, the meta-controller is rewarded according to the change of cluster load balance so that the meta-controller can update the strategy of generating sub-goals.

[0010] Optionally, the cluster load balance includes CPU load balance and memory load balance, and the rewarding of the meta-controller according to the change of the cluster load balance includes:

[0011] If both the CPU load balance and the memory load balance increase, the meta-controller is given the first preset reward;

[0012] If one of the CPU load balance degree and the memory load balance degree increases and the other decreases, the meta-controller is given a second preset reward;

[0013] If both the CPU load balance and the memory load balance are reduced, a third preset reward is given to the meta-controller, where the third preset reward is greater than the second preset reward, and the second preset reward is greater than the first preset reward.

[0014] Optionally, the CPU load balancing degree is calculated as follows:

[0015]

[0016] The calculation formula of the memory load balance is:

[0017]

[0018] Among them, lb cpu is the CPU load balance, std cpu is the standard deviation of CPU utilization, is the CPU utilization of the i-th virtual machine in the cluster, is the average CPU utilization of the cluster virtual machines, N is the total number of virtual machines in the cluster, lb mem is the memory load balance, std memis the standard deviation of memory utilization, is the memory utilization of the i-th virtual machine in the cluster, The average memory utilization of the cluster's virtual machines.

[0019] Optionally, the cluster resources include multiple virtual machines, each of which includes a CPU and memory, the job includes one or more tasks, each of which includes one or more instances, and allocating cluster resources to the job to be scheduled includes:

[0020] Allocate the CPU and memory of the virtual machine to each instance of each task of the job to be scheduled.

[0021] Optionally, after the job scheduling controller allocates cluster resources to the job to be scheduled according to the current utilization of cluster resources and the resource demand of the job to be scheduled, the process includes:

[0022] If the CPU required by the instance is greater than the available CPU of the allocated virtual machine, or the memory required by the instance is greater than the available memory of the allocated virtual machine, the allocation request is rejected and a negative reward is given to the job scheduling controller so that the job scheduling controller can update the job scheduling strategy;

[0023] If the CPU required by the instance is not greater than the available CPU of the assigned virtual machine, and the memory required by the instance is not greater than the available memory of the assigned virtual machine, then after the scheduled job is executed, the CPU and memory are released. The job scheduling controller rewards the job scheduling controller based on the change in the cluster load balance so that the job scheduling controller can update the job scheduling strategy.

[0024] Optionally, after the elastic scaling controller expands or reclaims cluster resources according to current cluster resource utilization, the following steps may be performed:

[0025] The elastic scaling controller rewards the elastic scaling controller based on changes in cluster load balancing so that the elastic scaling controller can update its elastic scaling strategy.

[0026] Optionally, the meta-controller, the job scheduling controller, and the elastic scaling controller are constructed using a deep Q network, and each of the meta-controller, the job scheduling controller, and the elastic scaling controller includes a target network and an evaluation network;

[0027] Among them, the update formula of the Q function is:

[0028] Q(s t ,a t )=Q(s t ,a t )+α[r t +δmax a′ Q(st+1 ,a′)-Q(s t ,a t )];

[0029] Among them, s t is the state of the current time step t, a t For action, r t is the immediate reward, α is the discount factor, δ is the learning rate, α,δ∈[0,1], r t +δmax a′ Q(s t+1 ,a′) approximates the true value of the Q function and is used to estimate the Q value by function approximation, i.e., Q(s t ,a t ).

[0030] In a second aspect, the present invention further provides a cluster load balancing device, the cluster load balancing device comprising:

[0031] The generation module is used by the meta-controller to generate sub-goals based on the current cluster resource utilization;

[0032] The job scheduling module is used to allocate cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource demand of the job to be scheduled when the generated sub-goal is job scheduling.

[0033] The elastic scaling module is used to expand or recycle cluster resources based on the current cluster resource utilization when the generated sub-goal is elastic scaling.

[0034] The reward module is used to reward the meta-controller after the sub-goal is executed according to the change of cluster load balance, so that the meta-controller can update the strategy of generating sub-goals.

[0035] In a third aspect, the present invention also provides a cluster load balancing device, which includes a processor, a memory, and a cluster load balancing program stored on the memory and executable by the processor, wherein when the cluster load balancing program is executed by the processor, the steps of the cluster load balancing method described above are implemented.

[0036] In a fourth aspect, the present invention further provides a readable storage medium, on which a cluster load balancing program is stored, wherein when the cluster load balancing program is executed by a processor, the steps of the cluster load balancing method as described above are implemented.

[0037] In the present invention, the meta-controller generates sub-goals according to the current utilization rate of cluster resources; when the generated sub-goal is job scheduling, the job scheduling controller allocates cluster resources to the job to be scheduled according to the current utilization rate of cluster resources and the resource demand of the job to be scheduled; when the generated sub-goal is elastic scaling, the elastic scaling controller performs resource expansion or resource recovery on the cluster resources according to the current utilization rate of cluster resources; after the sub-goal is executed, the meta-controller rewards the meta-controller according to the change in the cluster load balance, so that the meta-controller can update the strategy for generating sub-goals. The present invention is achieved by the meta-controller generating sub-goals according to the current utilization rate of cluster resources, the sub-goals being job scheduling or elastic scaling, the job scheduling controller allocating cluster resources to the job to be scheduled, the elastic scaling controller performing resource expansion or resource recovery on the cluster resources, and then rewarding the meta-controller, updating and generating a better strategy for generating sub-goals, thereby better maintaining the cluster load balance state and solving the resource bottleneck problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of an embodiment of a cluster load balancing method of the present invention;

[0039] Figure 2 This is a schematic diagram of agent interaction in an embodiment of a cluster load balancing method of the present invention;

[0040] Figure 3 A schematic diagram of a hierarchical intelligent agent architecture according to an embodiment of a cluster load balancing method of the present invention;

[0041] Figure 4 for Figure 1 Detailed flow chart of step S40;

[0042] Figure 5 A schematic diagram of CPU load balancing degree changes in an embodiment of a cluster load balancing method of the present invention;

[0043] Figure 6 A schematic diagram of the functional modules of an embodiment of a cluster load balancing device of the present invention;

[0044] Figure 7 This is a schematic diagram of the hardware structure of an embodiment of a cluster load balancing device of the present invention.

[0045] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0046] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0047] In a first aspect, an embodiment of the present invention provides a cluster load balancing method.

[0048] In order to more clearly demonstrate the cluster load balancing method provided in the embodiment of the present application, the application scenario of the cluster load balancing method provided in the embodiment of the present application is first introduced.

[0049] The cluster load balancing method provided in the embodiment of the present application is applied to job scheduling and resource management issues, which have become hot issues in cloud computing research. Job scheduling in cloud computing allocates computing resources of cluster nodes to jobs. In this process, since the arrival volume and arrival time of jobs are uncertain, inappropriate job scheduling strategies and unreasonable cluster size will affect the load balancing state of the cluster. When the amount of jobs increases sharply, it is easy to cause load imbalance of cluster resources and resource bottleneck problems.

[0050] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the cluster load balancing method of the present invention. Figure 1 As shown, the cluster load balancing method includes:

[0051] Step S10: The meta-controller generates a sub-goal according to the current utilization of cluster resources.

[0052] In this embodiment, the meta-controller outputs the sub-goal goal based on the evaluation of the current cluster resource availability status. Specifically, the input of the meta-controller is the vector of the current cluster virtual machine resource availability, that is, Where N represents the total number of virtual machines in the cluster. represents the CPU availability of the i-th virtual machine, represents the memory availability of the i-th virtual machine. The output of the meta-controller is the sub-goal goal, goal = {0, 1}. That is, the output sub-goal goal = 0 or 1 is used to instruct the job scheduling controller to execute the job scheduling task or the elastic scaling controller to execute the elastic scaling task.

[0053] Step S20: When the generated sub-goal is job scheduling, the job scheduling controller allocates cluster resources to the job to be scheduled according to the current utilization of cluster resources and the resource demand of the job to be scheduled.

[0054] In this embodiment, the meta-controller outputs the sub-goal goal=0 to instruct the job scheduling controller to perform the job scheduling task. Specifically, the job scheduling controller allocates cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource demand of the job to be scheduled.

[0055] Step S30: When the generated sub-goal is elastic scaling, the elastic scaling controller performs resource expansion or resource recovery on the cluster resources according to the current utilization rate of the cluster resources.

[0056] In this embodiment, the meta-controller outputs a sub-goal goal = 1, indicating that the current cluster load state is unbalanced, to instruct the elastic scaling controller to perform elastic scaling tasks. Specifically, the elastic scaling controller expands or recycles cluster resources based on the current cluster resource utilization.

[0057] In step S40 , after the sub-goal is executed, the meta-controller is rewarded according to the change of the cluster load balance so that the meta-controller can update the strategy for generating the sub-goal.

[0058] In this embodiment, after the sub-goal is executed, the cluster load balancing may improve or deteriorate due to job scheduling or elastic scaling. The meta-controller is rewarded accordingly, and the meta-controller updates the strategy for generating sub-goals to optimize and obtain a better strategy for generating sub-goals, thereby better maintaining the cluster load balancing state and solving resource bottleneck problems.

[0059] In this embodiment, refer to Figure 2 , Figure 2 This is a schematic diagram of agent interaction in an embodiment of the cluster load balancing method of the present invention. Figure 2 As shown in Figure 2, in a reinforcement learning environment, the agent outputs corresponding actions based on the state of the external environment, obtains rewards by interacting with the external environment, and then judges the actions to learn better strategies. Figure 3 , Figure 3 This is a schematic diagram of a hierarchical intelligent agent architecture according to an embodiment of the cluster load balancing method of the present invention. Figure 3 As shown in the figure, it includes the upper-layer meta-controller, and the lower-layer job scheduling controller 1 and elastic scaling controller 2. The meta-controller generates a sub-goal goal according to the current cluster resource utilization, goal = {0, 1}, that is, the output sub-goal goal = 0 or 1 is used to instruct the job scheduling controller to execute the job scheduling task or the elastic scaling controller to execute the elastic scaling task. After that, the cluster load balancing may improve or deteriorate, and the meta-controller is rewarded accordingly. The meta-controller updates the strategy for generating sub-goals to optimize and obtain a better strategy for generating sub-goals, thereby better maintaining the cluster load balancing state and solving the resource bottleneck problem.

[0060] Furthermore, in one embodiment, the cluster load balance includes the CPU load balance and the memory load balance, referring to Figure 4 , Figure 2 for Figure 1 The detailed flow chart of step S40 is as follows: Figure 4 As shown, step S40 includes:

[0061] Step S401: If both the CPU load balance and the memory load balance are increased, a first preset reward is given to the meta-controller;

[0062] Step S402: If one of the CPU load balance degree and the memory load balance degree increases and the other decreases, a second preset reward is given to the meta-controller;

[0063] Step S403: If both the CPU load balance and the memory load balance are reduced, a third preset reward is given to the meta-controller, where the third preset reward is greater than the second preset reward, and the second preset reward is greater than the first preset reward.

[0064] In this embodiment, cluster load balance can be considered from two aspects: CPU load balance and memory load balance. Based on the changes in CPU load balance and memory load balance, that is, based on changes in the external environment, the meta-controller is given corresponding rewards. The setting of the reward function is crucial. The reward function of the meta-controller can be expressed by the following formula:

[0065]

[0066] Among them, r t For immediate rewards, Indicates the CPU load balance status at time t, Indicates the CPU load balancing status at time t+1. Indicates the load balancing status of the memory at time t. Represents the load balance status of the memory at time t+1. We reward the agent by comparing the changes in the system load balance status at time t and t+1, that is, after the meta-controller makes the sub-goal decision. After the sub-goal is executed, if both the CPU load balance and the memory load balance increase, it means that the decision is poor, and a zero reward is given. If one of the CPU load balance and the memory load balance increases and the other decreases, it means that the decision is average, and a 1.0 reward is given. If both the CPU load balance and the memory load balance decrease, it means that the decision is good, and a 2.0 reward is given.

[0067] Furthermore, in one embodiment, the calculation formula for the CPU load balancing degree is:

[0068]

[0069] The calculation formula of the memory load balance is:

[0070]

[0071] Among them, lb cpu is the CPU load balance, stdcpu is the standard deviation of CPU utilization, is the CPU utilization of the i-th virtual machine in the cluster, is the average CPU utilization of the cluster virtual machines, N is the total number of virtual machines in the cluster, lb mem is the memory load balance, std mem is the standard deviation of memory utilization, is the memory utilization of the i-th virtual machine in the cluster, The average memory utilization of the cluster's virtual machines.

[0072] In this embodiment, the state of cluster load balancing can be represented by the standard deviation of the utilization of the CPU and memory resources of the cluster virtual machines, that is, lb cpu =std cpu and lb mem =std mem , lb cpu and lb mem The smaller the value, the more balanced the cluster load.

[0073] Furthermore, in one embodiment, the cluster resources include multiple virtual machines, each of which includes a CPU and memory, and the job includes one or more tasks, each of which includes one or more instances. Step S20 includes:

[0074] Allocate the CPU and memory of the virtual machine to each instance of each task of the job to be scheduled.

[0075] In this embodiment, specifically, the input of the job scheduling controller is the vector of the current cluster virtual machine resource availability, that is, And the resource requirements of the job to be scheduled [job cpu ,job mem ,job type ,job instance ,job duration ], respectively representing the CPU demand of the job, the memory demand of the job, the type of job, the number of job instances and the duration of the job. The output of the job scheduling controller is the action set Indicates the situation where the instance of the task of the current job to be scheduled is assigned to the virtual machines in the cluster. When controller 1 selects a virtual machine for the job, the action value corresponding to the virtual machine is 1, and the action values corresponding to other virtual machines are all 0. For example, if the Nth virtual machine is selected to execute the task, then a = (0, 0, ..., 1).

[0076] Furthermore, in one embodiment, after step S20, the following steps are included:

[0077] If the CPU required by the instance is greater than the available CPU of the allocated virtual machine, or the memory required by the instance is greater than the available memory of the allocated virtual machine, the allocation request is rejected and a negative reward is given to the job scheduling controller so that the job scheduling controller can update the job scheduling strategy;

[0078] If the CPU required by the instance is not greater than the available CPU of the assigned virtual machine, and the memory required by the instance is not greater than the available memory of the assigned virtual machine, then after the scheduled job is executed, the CPU and memory are released. The job scheduling controller rewards the job scheduling controller based on the change in the cluster load balance so that the job scheduling controller can update the job scheduling strategy.

[0079] In this embodiment, if the CPU required by the instance is greater than the available CPU of the assigned virtual machine, or the memory required by the instance is greater than the available memory of the assigned virtual machine, the allocation request is rejected and a negative reward is given to the job scheduling controller. After the execution of the scheduled job is completed, the job scheduling controller rewards the action of allocating the job to a virtual machine with a relatively light load and punishes the action of allocating the job to a virtual machine with a heavier load based on feedback from the external environment.

[0080] When the action selected by the agent is a t =i, that is, when the current job is assigned to the i-th virtual machine, the reward function of the job scheduling controller can be expressed by the following formula:

[0081]

[0082] in, represents the CPU utilization of the i-th virtual machine, represents the memory utilization of the i-th virtual machine, Indicates the average CPU utilization of the cluster virtual machines. Indicates the average memory utilization of the cluster virtual machines. When we assign a job to a virtual machine whose CPU utilization and memory utilization are both higher than the average, the decision action a t Poor, giving a zero reward to indicate a penalty, when we assign the job to a virtual machine with a CPU utilization lower than the average and a memory utilization higher than the average, this decision returns a positive value That is, the difference between the average CPU utilization and the CPU utilization of the current virtual machine. When we assign a job to a virtual machine with a CPU utilization higher than the average and a memory utilization lower than the average, this decision returns a positive value. That is, the difference between the average memory utilization and the memory utilization of the current virtual machine. When we assign a job to a virtual machine with both CPU utilization and memory utilization lower than the average, this decision returns a positive value. That is, the sum of the difference between the average CPU utilization and the CPU utilization of the current virtual machine, and the difference between the average memory utilization and the memory utilization of the current virtual machine.

[0083] Furthermore, in one embodiment, after step S30, the following steps are included:

[0084] The elastic scaling controller rewards the elastic scaling controller based on changes in cluster load balancing so that the elastic scaling controller can update its elastic scaling strategy.

[0085] In this embodiment, the input of the elastic scaling controller is the vector of the current cluster virtual machine resource availability, that is, The output is a two-dimensional vector [vmID, a], where vmID indicates the serial number of the virtual machine that needs resource adjustment. a has four candidate values: a = {0, 1, 2, 3}, which respectively represent adding one unit of CPU resources, adding one unit of memory resources, reclaiming one unit of CPU resources, and reclaiming one unit of memory resources. That is, the dimension of the action space of the elastic scaling controller agent is 4N.

[0086] The reward function of the elastic scaling controller can be expressed as follows:

[0087] When a={0,2}, that is, when the elastic scaling controller operates on CPU resources,

[0088]

[0089] If the CPU load balancing status value increases after executing the elastic scaling action, it means that the decision is poor and a negative reward is given. If the CPU load balancing status value remains unchanged after executing the elastic scaling action, it means that the decision is average and a zero reward is given. If the CPU load balancing status value decreases after executing the elastic scaling action, it means that the decision is good and a positive reward is given.

[0090] Similarly, when a={1,3}, that is, when the elastic scaling controller operates on memory resources,

[0091]

[0092] If the memory load balancing status value increases after the elastic scaling action is executed, it means that the decision is poor and a negative reward is given. If the memory load balancing status value remains unchanged after the elastic scaling action is executed, it means that the decision is average and a zero reward is given. If the memory load balancing status value decreases after the elastic scaling action is executed, it means that the decision is good and a positive reward is given.

[0093] Furthermore, in one embodiment, the meta-controller, the job scheduling controller, and the elastic scaling controller are constructed using a deep Q network, and each of the meta-controller, the job scheduling controller, and the elastic scaling controller includes a target network and an evaluation network;

[0094] Among them, the update formula of the Q function is:

[0095] Q(s t ,a t )=Q(s t ,a t )+α[r t +δmax a′ Q(s t+1 ,a′)-Q(s t ,a t )];

[0096] Among them, s t is the state of the current time step t, a t For action, r t is the immediate reward, α is the discount factor, δ is the learning rate, α,δ∈[0,1], r t +δmax a′ Q(s t+1 ,a′) approximates the true value of the Q function and is used to estimate the Q value by function approximation, i.e., Q(s t ,a t ).

[0097] In this embodiment, DQN (Deep Q-network) is an improvement on the Q-learning method, which introduces deep neural network (DNN) into reinforcement learning (RL) method. It is a specific implementation of DRL (deep reinforcement learning). In reinforcement learning, the intelligent agent we train at each time step t, according to the current environment state s t , according to the current strategy π, select the action a in the current state t , take action a t After interacting with the environment, the environment will feedback an immediate reward r t , the agent receives an immediate reward r t The value of updates the policy of the current policy network.

[0098] In the DQN algorithm, we use a neural network to generate an approximation of the state-action function Q function to select the optimal action. Among them, the Q function, also known as the state-action-value function, is used to approximate the long-term potential value of the current state-action pair. In the DQN algorithm, we select the action a with the largest state-action value at each step. t Specifically, the Q function refers to Q π(s,a), which estimates that under policy π, the state s t And perform action a t The cumulative discount value of

[0099] The meta-controller, job scheduling controller, and elastic scaling controller are all constructed using a deep Q network. The meta-controller, job scheduling controller, and elastic scaling controller all include a target network and an evaluation network, each consisting of a three-layer fully connected network. Each layer is activated using the Relu activation function. Initially, the two networks are initialized with the same parameters. Then, after every M steps, the parameters of the evaluation network are copied to the target network. The target network uses the parameters of the evaluation network for parameter update. The target network provides an approximate true value of the Q function, i.e., r t +δmax a′ Q(s t+1 ,a′), so that the output Q(s t ,a t ) infinitely approaches the true Q value of the target network. In addition to function approximation, the DQN algorithm uses experience replay to convert (s t ,a t ,r t ,s t+1 ) is stored in the experience pool. When the pool has sufficient experience, a small batch of experience samples is randomly selected to train the DQN. If the pool reaches its upper capacity N, i.e., it is full, the previous experience is deleted and only the most recent N experiences are retained. This random selection of experience from the pool each time avoids the impact of correlation between experiences from the same episode on learning, and reduces the oscillation and divergence of reinforcement learning caused by the correlation between previous and subsequent samples.

[0100] The exploration strategy is the ε-greedy strategy, which means that at each time step, a random action is selected with probability ε, and the action that maximizes the Q value is selected using the current agent's policy with probability 1-ε. The initial value of ε is 1, and it gradually decreases to 0.01 as the agent interacts with the environment and gradually trains to develop a better policy. The main purpose of this strategy is to increase the probability of selecting random actions when the agent is not fully trained, thereby achieving a wider exploration of the environment and learning a better policy.

[0101] In this embodiment, the specific implementation of the DQN algorithm can be divided into the following steps:

[0102] (1) Network parameter and hyperparameter initialization

[0103] First, create an experience pool D and set its capacity N. Then, create two neural network models with identical structures: the evaluation network Q and the target network Q′. Initialize the parameters θ of the evaluation network Q using random weights. Then, copy the parameters θ of the evaluation network Q to the parameters θ′ of the evaluation network Q′, ensuring that both network models have the same initialization parameters. Set the initial value of ε to 1, the discount factor α to 0.99, and the learning rate δ to 0.001. Set the target network parameter update interval M.

[0104] (2) The agent interacts with the environment

[0105] At each time step t, we make a probability selection of ε, that is, generate a random number k in (0,1). If the number is less than ε, we select a random action as a t ; If the number is greater than ε, we evaluate the Q(s t ,a t ; θ) value is selected to maximize the Q value t As the action value. The specific action selection formula is as follows:

[0106]

[0107] Select action a t Then, execute the action a in the environment t , the environmental state changes from s t becomes s t+1 , we generate an immediate reward r according to the state change of the environment t , used to update the agent policy network.

[0108] After completing the above steps, t ,a t ,r t ,s t+1 ) is stored in the experience buffer pool D. When the experience pool is full and reaches the maximum capacity N, the earliest stored experience is deleted and the latest N experiences are retained.

[0109] (3) Neural network parameter update

[0110] Randomly select a small batch of experiences (s t ,a t ,r t ,s t+1 ), using the target network, calculate r t +δmax a′ Q(s t+1 ,a') to update the evaluation network Q(s t ,a t ;θ) parameter θ, so that the evaluation network Q(st ,a t ;θ) approaches r t +δmax a′ Q(s t+1 ,a′). Each time the evaluation network parameters are updated, the target network parameters remain unchanged, and after every M steps, the evaluation network parameters are copied to the target network.

[0111] In this example, we used Alibaba's real user load tracking data to conduct experiments. We used 9 virtual machines and considered resource management scheduling in two dimensions: CPU and memory. Figure 5 , Figure 5 FIG. 1 is a schematic diagram showing changes in CPU load balancing degree according to an embodiment of a cluster load balancing method of the present invention. Figure 5 As shown, we can observe that after 1000 rounds of training, the CPU load balance of the cluster is maintained at a relatively low level, that is, in the range of 0.10-0.13, indicating that the present invention effectively balances the job scheduling and resource capacity balance in the job scheduling process, and can effectively maintain the load balance state of the cluster when the job arrival time and arrival rate are unknown.

[0112] In a second aspect, an embodiment of the present invention further provides a cluster load balancing device.

[0113] Reference Figure 6 , Figure 6 This is a schematic diagram of the functional modules of an embodiment of a cluster load balancing device of the present invention.

[0114] In this embodiment, the cluster load balancing device includes:

[0115] A generation module 10 is used by the meta-controller to generate sub-goals according to the current utilization of cluster resources;

[0116] The job scheduling module 20 is configured to allocate cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource demand of the job to be scheduled when the generated sub-goal is job scheduling.

[0117] The elastic scaling module 30 is used to, when the generated sub-goal is elastic scaling, cause the elastic scaling controller to expand or recycle cluster resources based on the current utilization of cluster resources;

[0118] The reward module 40 is used to reward the meta-controller according to the change of the cluster load balance after the sub-goal is executed, so that the meta-controller can update the strategy of generating sub-goals.

[0119] Furthermore, in one embodiment, the cluster load balance includes the CPU load balance and the memory load balance, and the reward module 40 is configured to:

[0120] If both the CPU load balance and the memory load balance increase, the meta-controller is given the first preset reward;

[0121] If one of the CPU load balance degree and the memory load balance degree increases and the other decreases, the meta-controller is given a second preset reward;

[0122] If both the CPU load balance and the memory load balance are reduced, a third preset reward is given to the meta-controller, where the third preset reward is greater than the second preset reward, and the second preset reward is greater than the first preset reward.

[0123] Furthermore, in one embodiment, the calculation formula for the CPU load balancing degree is:

[0124]

[0125] The calculation formula of the memory load balance is:

[0126]

[0127] Among them, lb cpu is the CPU load balance, std cpu is the standard deviation of CPU utilization, is the CPU utilization of the i-th virtual machine in the cluster, is the average CPU utilization of the cluster virtual machines, N is the total number of virtual machines in the cluster, lb mem is the memory load balance, std mem is the standard deviation of memory utilization, is the memory utilization of the i-th virtual machine in the cluster, The average memory utilization of the cluster's virtual machines.

[0128] Furthermore, in one embodiment, the cluster resources include multiple virtual machines, each of which includes a CPU and memory, and the job includes one or more tasks, each of which includes one or more instances. The job scheduling module 20 is configured to:

[0129] Allocate the CPU and memory of the virtual machine to each instance of each task of the job to be scheduled.

[0130] Furthermore, in one embodiment, the cluster load balancing device further includes a job scheduling reward module, which is used to:

[0131] If the CPU required by the instance is greater than the available CPU of the allocated virtual machine, or the memory required by the instance is greater than the available memory of the allocated virtual machine, the allocation request is rejected and a negative reward is given to the job scheduling controller so that the job scheduling controller can update the job scheduling strategy;

[0132] If the CPU required by the instance is not greater than the available CPU of the assigned virtual machine, and the memory required by the instance is not greater than the available memory of the assigned virtual machine, then after the scheduled job is executed, the CPU and memory are released. The job scheduling controller rewards the job scheduling controller based on the change in the cluster load balance so that the job scheduling controller can update the job scheduling strategy.

[0133] Furthermore, in one embodiment, the cluster load balancing device further includes an elastic scaling reward module, which is configured to:

[0134] The elastic scaling controller rewards the elastic scaling controller based on changes in cluster load balancing so that the elastic scaling controller can update its elastic scaling strategy.

[0135] Furthermore, in one embodiment, the meta-controller, the job scheduling controller, and the elastic scaling controller are constructed using a deep Q network, and each of the meta-controller, the job scheduling controller, and the elastic scaling controller includes a target network and an evaluation network;

[0136] Among them, the update formula of the Q function is:

[0137] Q(s t ,a t )=Q(s t ,a t )+α[r t +δmax a' Q(s t+1 ,a′)-Q(s t ,a t )];

[0138] Among them, s t is the state of the current time step t, a t For action, r t is the immediate reward, α is the discount factor, δ is the learning rate, α,δ∈[0,1], r t +δmax a′ Q(s t+1 ,a′) approximates the true value of the Q function and is used to estimate the Q value by function approximation, i.e., Q(s t ,a t ).

[0139] The functional implementation of each module in the above cluster load balancing device corresponds to each step in the above cluster load balancing method embodiment, and their functions and implementation processes are not described here one by one.

[0140] In a third aspect, an embodiment of the present invention provides a cluster load balancing device.

[0141] Reference Figure 7 , Figure 7 This is a hardware structure diagram of an embodiment of a cluster load balancing device of the present invention. In an embodiment of the present invention, the cluster load balancing device may include a processor 1001 (e.g., a central processing unit (CPU)), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components; the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard); the network interface 1004 may optionally include a standard wired interface, a wireless interface (such as a wireless fidelity WIreless-FIdelity, WI-FI interface); the memory 1005 may be a high-speed random access memory (random access memory, RAM), or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001. Those skilled in the art will understand that, Figure 7 The hardware structure shown in the figure does not constitute a limitation of the present invention, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0142] Continue to refer to Figure 7 , Figure 7 The memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a cluster load balancing program. The processor 1001 may call the cluster load balancing program stored in the memory 1005 and execute the cluster load balancing method provided in the embodiment of the present invention.

[0143] In a fourth aspect, an embodiment of the present invention further provides a readable storage medium.

[0144] The readable storage medium of the present invention stores a cluster load balancing program, wherein when the cluster load balancing program is executed by a processor, the steps of the cluster load balancing method described above are implemented.

[0145] The method implemented when the cluster load balancing program is executed may refer to the various embodiments of the cluster load balancing method of the present invention, and will not be described in detail here.

[0146] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0147] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, or of course by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device to execute the methods described in various embodiments of the present invention.

[0149] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A cluster load balancing method, characterized in that: The cluster load balancing method includes: The meta-controller generates sub-goals based on the current cluster resource utilization; When the generated sub-goal is job scheduling, the job scheduling controller allocates cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource requirements of the job to be scheduled; When the generated sub-goal is elastic scaling, the elastic scaling controller expands or recycles cluster resources based on the current cluster resource utilization; After the sub-goal is executed, the meta-controller is rewarded based on the change in cluster load balance so that the meta-controller can update the strategy for generating sub-goals; The cluster load balance includes the CPU load balance and the memory load balance. The rewarding of the meta-controller according to the change of the cluster load balance includes: If both the CPU load balance and the memory load balance increase, the meta-controller is given the first preset reward; If one of the CPU load balance degree and the memory load balance degree increases and the other decreases, the meta-controller is given a second preset reward; If both the CPU load balance and the memory load balance are reduced, a third preset reward is given to the meta-controller, wherein the third preset reward is greater than the second preset reward, and the second preset reward is greater than the first preset reward; The calculation formula of the CPU load balancing degree is: ； The calculation formula of the memory load balance is: ； Among them, Ib cpu is the CPU load balance, std cpu is the standard deviation of CPU utilization, is the CPU utilization of the i-th virtual machine in the cluster, is the average CPU utilization of the cluster virtual machines, N is the total number of virtual machines in the cluster, and Ib mem is the memory load balance, std mem is the standard deviation of memory utilization, is the memory utilization of the i-th virtual machine in the cluster, The average memory utilization of the cluster's virtual machines.

2. The cluster load balancing method according to claim 1, wherein: The cluster resources include multiple virtual machines, each of which includes a CPU and memory. The job includes one or more tasks, each of which includes one or more instances. Allocating cluster resources to the job to be scheduled includes: Allocate the CPU and memory of the virtual machine to each instance of each task of the job to be scheduled.

3. The cluster load balancing method according to claim 2, wherein: After the job scheduling controller allocates cluster resources to the job to be scheduled according to the current utilization of cluster resources and the resource demand of the job to be scheduled, the method includes: If the CPU required by the instance is greater than the available CPU of the allocated virtual machine, or the memory required by the instance is greater than the available memory of the allocated virtual machine, the allocation request is rejected and a negative reward is given to the job scheduling controller so that the job scheduling controller can update the job scheduling strategy; If the CPU required by the instance is not greater than the available CPU of the assigned virtual machine, and the memory required by the instance is not greater than the available memory of the assigned virtual machine, then after the scheduled job is executed, the CPU and memory are released. The job scheduling controller rewards the job scheduling controller based on the change in the cluster load balance so that the job scheduling controller can update the job scheduling strategy.

4. The cluster load balancing method according to claim 1, wherein: After the elastic scaling controller expands or reclaims cluster resources according to the current utilization of cluster resources, the following steps are included: The elastic scaling controller rewards the elastic scaling controller based on changes in cluster load balancing so that the elastic scaling controller can update its elastic scaling strategy.

5. The cluster load balancing method according to claim 1, wherein: The meta-controller, job scheduling controller, and elastic scaling controller are constructed using a deep Q network, each of which includes a target network and an evaluation network. Among them, the update formula of the Q function is: ； Among them, s t is the state of the current time step t, a t For action, r t is the immediate reward, α is the discount factor, δ is the learning rate, , Approximately represents the true value of the Q function, which is used to estimate the Q value by function approximation, that is, .

6. A cluster load balancing device, characterized in that: The cluster load balancing device includes: The generation module is used by the meta-controller to generate sub-goals based on the current cluster resource utilization; The job scheduling module is used to allocate cluster resources to the job to be scheduled based on the current cluster resource utilization and the resource demand of the job to be scheduled when the generated sub-goal is job scheduling. The elastic scaling module is used to expand or recycle cluster resources based on the current cluster resource utilization when the generated sub-goal is elastic scaling. The reward module is used to reward the meta-controller after the sub-goal is executed, based on the change in cluster load balance, so that the meta-controller can update the strategy for generating sub-goals; The cluster load balance includes the CPU load balance and the memory load balance. The reward module is used to: If both the CPU load balance and the memory load balance increase, the meta-controller is given the first preset reward; If one of the CPU load balance degree and the memory load balance degree increases and the other decreases, the meta-controller is given a second preset reward; If both the CPU load balance and the memory load balance are reduced, a third preset reward is given to the meta-controller, wherein the third preset reward is greater than the second preset reward, and the second preset reward is greater than the first preset reward; The calculation formula of the CPU load balancing degree is: The calculation formula of the memory load balance is: Among them, Ib cpu is the CPU load balance, std cpu is the standard deviation of CPU utilization, is the CPU utilization of the i-th virtual machine in the cluster, is the average CPU utilization of the cluster virtual machines, N is the total number of virtual machines in the cluster, and Ib mem is the memory load balance, std mem is the standard deviation of memory utilization, is the memory utilization of the i-th virtual machine in the cluster, The average memory utilization of the cluster's virtual machines.

7. A cluster load balancing device, characterized in that: The cluster load balancing device includes a processor, a memory, and a cluster load balancing program stored in the memory and executable by the processor, wherein when the cluster load balancing program is executed by the processor, the steps of the cluster load balancing method according to any one of claims 1 to 5 are implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores a cluster load balancing program, wherein when the cluster load balancing program is executed by a processor, the steps of the cluster load balancing method according to any one of claims 1 to 5 are implemented.

Images