Multi-agent system and deep reinforcement learning load balancing method thereof

Abstract

The invention discloses a multi-agent system and a deep reinforcement learning load balancing method thereof, which reduce the complexity of agent selection action and can more easily achieve the optimal performance on the whole. According to the method, the request distribution is divided into two stages, the central agent selects the data center in the first stage, the local agent selects the node in the second stage, and therefore the problem that the multi-agent environment is unstable is avoided. In addition, request distribution of each agent is modeled into a Markov decision process, and definition of a state space, an action space, a reward function and state transition is described in detail. In the aspect of a network structure, a decision network is adopted, and compared with a single-flow Q network, the decision network has the advantage of being faster and better in learning. In the aspect of exploration and utilization, an improved load balancing-oriented action exploration strategy is used. Finally, a pseudo code and a learning process chart of the algorithm are given, and the training process of the algorithm is explained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of software load balancing, and more particularly, to a multi-agent system and a deep reinforcement learning load balancing method thereof. BACKGROUND

[0002] Load balancing is one of the key technologies to improve system resource utilization and parallel computing performance. The core of load balancing technology is the scheduling algorithm, that is, to distribute each request to different server nodes for parallel computing as evenly as possible, so as to maximize the utilization of each node. The load balancing strategy of external requests can be divided into two categories: static load balancing algorithm and dynamic load balancing algorithm. Among them, the current static load balancing algorithm research status and existing problems are as follows: (1) the current static load balancing algorithm usually distributes requests to nodes based on pre-determined rules, without considering the real-time load state of nodes and environmental changes. Therefore, the fault tolerance of these algorithms is low, especially in the case of sudden load changes. (2) The improved algorithm of existing static load balancing is still limited by the defects of static algorithm, and cannot make decisions according to the real-time data center state, lacking comprehensive consideration of load balancing. The research status and existing problems of dynamic load balancing algorithm are as follows: (1) most of the current load balancing algorithms focus on the single data center scenario, ignoring the distributed data center scenario. (2) The time and number of user-initiated requests are random, making it difficult to efficiently and flexibly balance external requests in the distributed data center scenario. (3) The algorithm goal is relatively single, lacking consideration of the performance difference between different nodes and the energy cost difference between data centers. SUMMARY

[0003] In view of the existing problems in the prior art, the present application provides a multi-agent system and a deep reinforcement learning load balancing method based on the system, including multi-agent system design, Markov decision process modeling of agent request distribution, and load balancing algorithm MDDLB based on multi-agent deep reinforcement learning. Specifically, it is: 1) Multi-agent system design. 2) Markov decision process modeling of agent request distribution: the request distribution of local agents and central agents is modeled as a Markov decision process, and their state space, action space, reward function and state transition are defined. 3) Load balancing algorithm MDDLB based on multi-agent deep reinforcement learning.

[0004] Specifically, the first aspect of the present application provides a multi-agent system.

[0005] Based on the same inventive concept, the second aspect of the present application provides a deep reinforcement learning load balancing method based on a multi-agent system, which is realized based on the multi-agent system of the first aspect.

[0006] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows: The present invention provides a multi-agent system and a deep reinforcement learning load balancing method based on the system. The multi-agent system includes a user layer, an agent layer and a scheduling layer. There is a central agent in the agent layer, and each scheduler in the scheduling layer has a local agent. There is a leader-follower relationship between the two. This method reduces the complexity of the agent's action selection and can more easily achieve the overall optimal performance. The local agents in the scheduling layer are independent of each other. The deep reinforcement learning load balancing method based on the multi-agent system mainly includes two stages. In the first stage, the central agent of the agent layer uses the Markov decision process and the deep reinforcement learning algorithm to select the data center based on the observed resource conditions; in the second stage, the local agent of the scheduling layer uses the Markov decision process and the deep reinforcement learning algorithm to select the node based on the observed resource conditions, thereby avoiding the problem of instability in the multi-agent environment. Furthermore, in terms of network structure, the present invention adopts a duel network. Compared with the single-stream 𝑄 network, the duel network has the characteristics of faster and better learning. In terms of exploration and utilization, the present invention uses an improved load balancing-oriented Action exploration strategy. Finally, the present invention provides the pseudo code of the algorithm and the learning process diagram, which illustrates the training process of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 This is a diagram of a load balancing framework based on a distributed data center in an embodiment of the present invention; Figure 2 This is a diagram of the proxy structure in an embodiment of the present invention; Figure 3 This is a structural diagram of a scheduler in an embodiment of the present invention; Figure 4 This is a diagram of a multi-agent architecture in an embodiment of the present invention; Figure 5 Schematic diagram of state changes in an embodiment of the present invention; Figure 6 This is the network structure change from DQN to Dueling DQN in the embodiment of the present invention; Figure 7 Schematic diagram of the duel neural network structure in an embodiment of the present invention; Figure 8 This is the multi-agent learning process in an embodiment of the present invention; Figure 9 This is the reward change curve of the central agent in the embodiment of the present invention; Figure 10 The reward change curve of the local agent in the embodiment of the present invention; Figure 11 : is a curve showing a change in the standard deviation of CPU utilization in a distributed data center according to an embodiment of the present invention; Figure 12 is a curve showing a change in the standard deviation of memory utilization in a distributed data center according to an embodiment of the present invention; Figure 13 is the standard deviation of the average CPU utilization of a single data center in an embodiment of the present invention; Figure 14The average memory utilization standard deviation of a single data center in the embodiment of the present application; Figure 15 The Gantt chart of the completion time in the embodiment of the present application; Figure 16 The energy cost Gantt chart in the embodiment of the present application. DETAILED DESCRIPTION

[0008] Embodiment one The embodiment provides a multi-agent system, which comprises: a user layer for initiating a request; an agent layer comprising a central agent, which can observe the resource conditions of all local agents and data centers, and is used for selecting a data center to respond to the request initiated by the user layer based on the observed resource conditions by using a Markov decision process and a deep reinforcement learning algorithm; and a scheduling layer comprising a plurality of schedulers and a plurality of nodes, wherein each scheduler comprises a local agent, the local agent is configured in a data center and can observe the resource conditions of the data center where the local agent is located, the local agents are independent of each other, and after the central agent selects a data center corresponding to the local agent, the local agent selects a corresponding node to respond to the request initiated by the user layer based on the observed resource conditions by using a Markov decision process and a deep reinforcement learning algorithm to perform load balancing.

[0009] Specifically, the multi-agent is a main way for the present application to solve the load balancing problem in the distributed data center scenario. The request of the multi-agent system of the present application comes from the user layer. The load balancing of the present application is divided into two-stage tasks, the first-stage task is executed by the central agent of the agent layer, and the second-stage task is executed by the local agent of the scheduling layer, the local agent must obey the command of the central agent, and only when the central agent selects the data center under the jurisdiction of the local agent, the local agent can perform the next action. The local agents of the scheduling layer are independent of each other and only have limited observation ability, and the environment where the local agent is located is part of the environment where the central agent is located. The local agent is configured in the data center, and it can only observe the environment of the data center where it is located, and the environments of the local agents are isolated from each other, while the central agent can observe the resource conditions of all local agents and data centers. The Markov decision process modeling of the agent request allocation: the request allocation of the local agent and the central agent is modeled as a Markov decision process respectively, and the state space, action space, reward function and state transition of them are defined.

[0010] The system proposed by the present application is described in detail in the following three parts.

[0011] I. Multi-agent system design The multi-agent is a main way for the present application to solve the load balancing problem in the distributed data center scenario. The multi-agent system design refers to the attached Figure 1 , the attached Figure 2 and the attachedFigure 3 In the framework, there is a central agent in the agent layer, and each scheduler in the scheduling layer has a local agent, and the two have a leader-follower relationship. The environments that the agents in the agent and the scheduler can observe are different, so the state space and action space of the two agents are also different, but they have similarities. In order to distinguish the two agents, the present invention uses the central agent as Indicates that the agent in the scheduler is called a local agent, and is represented by express.

[0012] Distributed multi-agent learning algorithm uses independent Learning Algorithms. Independent Each agent in the learning algorithm maintains a value table, and based on the single agent -learning algorithm is updated as shown in formula (1) Value table.

[0013] (1) The multi-agent architecture is as follows Figure 4 As shown, Input Layer, Hidden Layer and Output Layer represent the input layer, hidden layer and output layer respectively, and V and ADV represent the state value and advantage value generated based on the duel network respectively. The load balancing method designed by the present invention has the characteristics of a distributed method. The local agents in the scheduling layer are independent of each other and have only limited observation capabilities. The environment they are in is part of the environment where the central agent is located. They can only output actions based on their own observations. They cannot communicate with each other and cannot exchange observation information and learned knowledge with other local agents. The central agent is connected to the local agents. For the local agent, it cannot see the existence of the central agent, but the central agent can observe the resource usage of each data center and the local agent. The environments between local agents are isolated from each other. Local agents will not learn the status and actions of other local agents, so local agents will not interfere with each other. The load balancing method of the present invention is divided into two phases. The first phase is performed by the central agent in the proxy layer to select the data center, and the second phase is performed by the local agent in the scheduling layer to select the node. The central agent and the local agent have a sequential relationship. The local agent can only take action after the central agent selects the data center. Therefore, only one agent is taking action during each scheduling. This can avoid the problem of instability in multi-agent environments.

[0014] 2. The specific design of the local agent is as follows: 1. State Space For the local agent, It's the environment The space of all possible states, each individual state They are all descriptions of the current environment.

[0015] The state of the environment in which the local agent is located is defined as the resource information of all nodes in the data center and the resource information of the requests that need to be scheduled in the current time period t, which can be represented by a one-dimensional vector: (2) in, Indicates the number of CPU cores remaining available on this node. Indicates the remaining available memory capacity of this node. Indicates the computing performance of the node (that is, how many millions of CPU instructions the node can process per second), Indicates the percentage of a CPU core that the request will occupy (percentage system, for example, 80 means 80% of a CPU core is occupied). Indicates the size of the memory occupied. Indicates the number of CPU instructions in millions.

[0016] 2. Action Space The action space is a space that contains all the actions that the agent can choose. The actions are necessarily discrete. In the scenario of the present invention, each action selected by the agent is the result of the self-learning of the reinforcement learning agent. For the local agent, is the action space that the local agent can choose. The action in the request allocation problem is defined as selecting a suitable node for a request, so the action in the action space corresponds to the sequence number of the selected node. In fact, the size of the action space is the same as the number of nodes. Assume that the agent Select the node selected in time period t , then the action can be expressed as: (3) 3. Reward Function The reward function of the local agent is divided into two stages in the application. The first stage judges whether the scheduling meets the requirement of SLA constraint. If not, a penalty should be given. Otherwise, the second stage is entered. The reward function should guide the local agent to maintain the load balance of the data center. If the load balance degree of the data center decreases after the scheduling is executed, it means that the scheduling is beneficial to the load balance, and a reward should be given. The reward value is the difference of the load balance before and after the scheduling. Similarly, if the load balance degree increases after the request is allocated, a penalty should also be given. Meanwhile, for a single data center, the load balance degree should be considered while the request is completed as soon as possible.

[0017] If the node is selected by the action this time, the application defines the change degree of the CPU utilization standard deviation and the memory utilization standard deviation of the data center respectively, and is defined as follows: (4) (5) Wherein, represents the CPU utilization standard deviation of the data center before the scheduling, represents the CPU utilization standard deviation of the data center after the scheduling to the node , and the memory size is the same. The numerator is a positive value, which means that the CPU utilization standard deviation of the local agent is reduced by the scheduling this time, and the load balance degree is improved, which means that the scheduling is beneficial to the load balance degree of the data center. Otherwise, it means that the scheduling makes the load of the data center more unbalanced. The size of the value divided by indicates the change degree of the load balance degree after the allocation. If the positive value is larger, it means that the effect of the allocation is better, and the reward is naturally higher. If the negative value is larger, it means that the effect is worse, and the penalty is heavier. The change degree of the memory utilization standard deviation is also defined similarly.

[0018] Therefore, the formula of the reward function is as follows: (6) Wherein, is a negative value, indicating a penalty to the local agent. , and respectively represent the weight of the CPU utilization standard deviation, the memory utilization standard deviation and the request execution time. represents the request allocated to the node , and the reciprocal of the running time is required. The larger the number is, the shorter the running time is, which means that the request can be completed faster. Otherwise, it means that the request is completed for a long time.

[0019] In order to avoid the interference of human factors, the present invention introduces an objective weighting method: entropy weight method.

[0020] Assume that N steps have been trained and the indicator value is (1 ≤ m ≤ 3), each indicator value is defined as follows: (7) (8) (9) The present invention saves two indicator values ​​during each episode training. Based on these indicator values, the entropy weight method assigns weights to each of the two indicators. The entropy weight method has four steps: (1) data normalization; (2) calculation of entropy value; (3) calculation of variance coefficient; (4) calculation of weight. The following describes the process of each step in turn. The first step is value normalization, which is calculated as follows: (10) In the entropy weight method, the indicator entropy value is defined as: (11) In actual evaluation using the entropy weight method, in order to facilitate calculation, it is generally Time setting The entropy value ranges from [0,1]. The smaller the entropy value, the greater the degree of difference of this indicator, the more information can be obtained, and the higher the weight should be given to this index. Therefore, in the entropy weight method, the weight is calculated as follows: (12) 4. State Transfer State transition is the agent's transition from the current time period t to the execution of action After that, the current state Change to the next state In general scenarios, reinforcement learning usually assumes that state transitions are random, and the transitions between states are affected by the current state. and actions The state transition probability function is also affected by the environment. The state transition probability function is determined by the environment and is usually stable, that is, the function does not change with the time period t. If you build a Super Mario game environment, the monster in the game should go left or right at the current moment. The choice here has a certain probability. The state transition can also be deterministic, and the new state can be calculated using a certain function 𝑓 For example, the state transition in Go is deterministic, and the next state Completely by and actions Decide that there is no randomness in the environment.

[0021] In the scenario of request allocation, the state transition is obviously deterministic. Because when a request is scheduled to a node, the node will occupy a certain amount of resources, which is explicitly visible, and the resource information about this node in the state will change accordingly, which is deterministic. Since the state transition here is difficult to express in the form of a function, the present application uses the form of pseudo code for illustration, as shown in Algorithm 1. In Algorithm 1, at time period t, the scheduler has taken action After that, the CPU core and memory amount of the node should be reduced by the CPU core and memory amount required by the request (1-2 lines), while the state updates its available resource situation (3 line), and finally returns the new state (5 line).

[0022]

[0023] However, it needs to be further explained that in the entire scheduling process, the change between states is not only the state transition. The state transition occurs after the local agent makes an action, the resources on the node are occupied, and the state naturally changes. However, as time flows, when a new request is proposed by a new cloud user, the local agent needs to make a decision again, and the state of the environment will also change, because between the two time periods, the life cycle of the request will be completed, and the resources occupied by them need to be released, so the state of the environment will also change accordingly. Therefore, the state transition is determined by the decision result of the local agent itself, and the second state change is related to the completion time of the request itself. So in the process of load balancing, the state needs to go through two stages of changes, as shown in the attached Figure 5 .

[0024] The second state change occurs before the local agent schedules the next request. Suppose at time period t+1, the local agent needs to allocate the next request. First, traverse all the requests running on the nodes in the data center (1-2 lines), check whether the end time period of these requests is less than or equal to time period t+1 (3 line), if yes, it means that the request has been executed before time period t+1, then release the resources occupied by these requests (4-5 lines), delete these requests from the node list (6 line), and update the corresponding part of the state (7 line), and then assign the state to (11 line), and finally return the new state of this time period.

[0025]

[0026] Through the two algorithms, it can be found that the state transition is only related to the current state and the action , and the second change is related to the request execution time on the node. In fact, as long as the state, action and request completion time are determined, the change between the states is also determined and is not random, but this process is transparent to the local agent, because the local agent knows how the state will change after it makes an action, but it cannot remember when the request it allocates will end, and it does not know what the state will become when it allocates the request in the next time period, because it cannot predict when the next request will be initiated, and it will not know how many requests need to be allocated at that time point, which is the randomness of the request initiation time and the number of requests.

[0027] III. The specific design of the central agent is as follows: 1. State space For the central agent, its state space is is the space containing all possible states in the environment . Similarly, the state of the central agent is defined as the resource information of each data center and the resource information of the request to be scheduled at the current time period t, which can be represented by a one-dimensional vector: (13) wherein represents the number of CPU cores remaining available in this data center, represents the remaining available memory capacity of this data center. 2. Action space For the central agent, is the action space that the central agent can choose. The action that the central agent can choose is defined as selecting a suitable data center for the request, so the action in the action space corresponds to the serial number of the selected data center. In fact, the size of the action space is the same as the number of data centers. Therefore, the action made by the central agent at time period t can be defined as: (14) 3. Reward function Similarly, the reward function of the central agent is also divided into two parts. The first part is related to the scheduling of the local agent. If the scheduling of the local agent does not meet the SLA constraint, in addition to the punishment that the local agent needs to bear, the central agent also needs to bear the punishment. If the SLA constraint is met, the degree of change of the load balancing of the entire system is calculated. Suppose the central agent selects data center , respectively define the degree of change of the CPU utilization standard deviation and the memory utilization standard deviation of the distributed data center, and are defined as follows: (15) (16) wherein, and are the CPU utilization standard deviation and the memory utilization standard deviation of the distributed data center, and are the CPU utilization standard deviation and the memory utilization standard deviation of the distributed data center after the selection of .

[0028] Meanwhile, the central agent also needs to consider the impact of energy consumption cost, and the reward function of the central agent is defined as follows (17) wherein, is the reciprocal of the energy consumption cost required after the two-stage allocation by the central agent and the local agent, i.e. the energy consumption cost spent by the central agent selecting the data center and the local agent selecting the node , , and are three weights of , and . Similar to the weights of the local agent, the weights are determined using the entropy weight method.

[0029] 4. State transition For the central agent, all local agents belong to its environment, and the local agent can only act after the central agent selects it. Therefore, the state of the central agent changes only after the local agent allocates the request to the node, i.e. the state does not change immediately after the central agent acts. Since there is a relationship between the data center and the node as shown in equation 18, when the state of the local agent changes, the state of the central agent will also change, so the state transition of the central agent is not described again.

[0030] (18) wherein, represents the number of CPU cores remaining available for this node, represents the remaining available memory capacity of this node, represents the maximum number of CPU cores possessed by this machine, represents the maximum memory capacity possessed by this node. Indicates the number of CPU cores remaining available in this data center. Indicates the remaining available memory capacity of this data center. Indicates the maximum number of CPU cores in this data center. Indicates the maximum memory capacity of this data center.

[0031] Example 2 Based on the same inventive concept, this embodiment discloses a deep reinforcement learning load balancing method based on a multi-agent system, which is implemented based on the multi-agent system of embodiment 1. The method includes: Receive requests initiated by users; The central agent in the proxy layer selects a data center based on observed resource conditions using a Markov decision process and deep reinforcement learning algorithm; The local agent in the scheduling layer selects nodes based on the observed resource conditions using Markov decision process and deep reinforcement learning algorithm.

[0032] The following is a detailed introduction to the deep reinforcement learning load balancing method based on multi-agent system.

[0033] 1. Network structure In the case of a finite state space, -learning performs well, but as the state space and action space continue to expand, -learning performance becomes difficult because traditional reinforcement learning combines states, actions, and the The value is stored in a table and obtained by looking up the table However, as the number of states and actions increases, the table will become very large, and the solution and iteration process will be very long. Therefore, a neural network is used to approximate a value function, that is, to use a deep neural network. Network DQN. Considering that the scheduling scenario of the present invention is not as complex as image or video processing, the input dimensions are not many and do not require a very complex neural network, so the present invention uses a fully connected neural network to parameterize the approximate value function ,in yes The parameters of the network. A major improvement of DQN is that it has two neural networks with the same structure but different parameters. One is the evaluation network, which is used to evaluate the current state-action pair generated. value ; The other is the target network, which is used to obtain the target value ,in are the parameters of the target network. The evaluation network uses the latest parameters, and the target network uses the older parameters. After the step, the parameters are cloned from the evaluation network to the target network to update the target network. The essence of DQN algorithm training is to update the parameters of the evaluation network to the target network , and finally reach a state of convergence. Since there is a delay between the old parameters of the target network and the new parameters of the evaluation network, the correlation between the generated values and the target values can be reduced, thereby greatly reducing divergence and oscillation and improving the stability of the algorithm.

[0034] The method for optimizing the parameters of the specific update neural network in DQN according to the loss function by using an optimizer, including gradient descent (GD), AdaGrad, and Adam methods. Among them, Adam is a commonly used optimizer for minimizing risk functions and loss functions; therefore, the Adam optimizer is used in the present application, and the mean squared error (MSE) is used as the loss function, which is defined as follows: (19) where and are the parameters of the evaluation network and the target network, respectively. The loss function value is the average of the squared differences between the evaluation values and the predicted values. The target values are ; the evaluation values are

[0035] In addition to adding a target network, the present application also uses the Dueling DQN structure. Dueling DQN is similar to DQN, as shown in the accompanying Figure 6 . This figure shows the difference between the Dueling DQN structure and the network structure of DQN. The left half of the figure is a traditional single-stream network that outputs the values of actions at the end, and the right side is the Dueling network.

[0036] The first stream is only related to the state and has nothing to do with the action to be selected, which is called the value function , and the second part is related to both the state and the action , which is called the advantage function . Among them, represents the network parameters of the common part, and respectively represent the network parameters unique to the value function and the advantage function. Thus, the mathematical representation of the value function can be defined as: (20) This equation will produce an indistinguishable problem, given a Q, it is impossible to get a unique V and A, so it will lack distinguishability, and when using the equation directly, the actual performance is poor, so the more commonly used method is to decentralize the advantage function, as follows: (21) wherein, is the total number of all possible actions in the action space is the advantage function of taking a specific action

[0037] Therefore, the network structure adopted by the intelligent agent of the present application is finally shown in the accompanying Figure 7 It can be found that the entire network structure is composed of seven parts: input layer, hidden layer, branch layer, state layer, advantage layer, aggregation layer and output layer. The input of the input layer is the current state, which is D x 2 + 3 for the central intelligent agent, wherein D represents the number of data centers, 2 represents two indexes, and 3 represents three attributes of the request; for the local intelligent agent, the size is J x 3 + 3, wherein J represents the number of nodes in this data center, the first 3 represents the available CPU core number, available memory amount and calculation speed of each node, and the second 3 represents three attributes of the request. The hidden layer is composed of two fully connected layers, and the output of the hidden layer will branch into two branches, both of which are fully connected layers with the same number of neurons. The first branch outputs a scalar, i.e. the value, and the second branch outputs the advantage value, which is a vector. The two values will be aggregated in the aggregation layer, as shown in equation (20). The output layer outputs the aggregated result, which is D for the central intelligent agent, i.e. the Q value that can be obtained by selecting each data center; for the local intelligent agent, the output size is J, i.e. the Q value that can be obtained by selecting each node.

[0038] 2. Experience replay The second improvement of DQN is called the experience replay buffer, which is a biological mechanism called experience replay. At each time step t, each action of the agent will produce an experience ​​and a memory (called a replay buffer) is used in DQN to store the generated experiences. When there are enough experiences (e.g., N) in the replay buffer, then a mini-batch of samples is uniformly drawn from the buffer to train the DQN. If the capacity of the replay buffer is larger than N, then the oldest experiences are deleted and only the most recent N experiences are kept. The replay buffer has several advantages over fitting the Q iteratively. First, the experiences in each step can be reused to learn Q the function, thus improving data efficiency. Second, without the replay buffer, if in the fitting of the Q iteratively, then the mini-batch of samples are collected consecutively, i.e., they are highly correlated, which increases the variance of the update. Third, due to the strong correlation between the samples, learning directly from the consecutive samples is inefficient, and drawing a mini-batch of samples randomly from the replay buffer breaks these correlations, thus reducing the variance of the update, avoiding the situation that the samples used for training are determined by the previous parameters, the action distribution over many states before it is averaged out, thus smoothing the learning and reducing the oscillation or divergence in the parameters. Finally, the size of the replay buffer is constrained, only the most recent N experience tuples are stored in the buffer to save memory. Q

[0039] 3. Action selection Now a commonly used strategy is the epsilon-greedy strategy. In this strategy, is a hyper-parameter. In each round of action selection, the agent does not greedily select the action that maximizes the Q value. The algorithm randomly draws a number between 0 and 1, if the number is greater than the hyper-parameter, then an exploration operation is performed, i.e., a random action is selected from the action space and executed, otherwise, the action that maximizes the expected reward is selected and executed.

[0040] In order to converge faster in exploration, the present application proposes a load balancing oriented strategy. If the random number is less than , then a resource entity (i.e., a data center or a node) that does not violate the SLA penalty is first drawn as a candidate set, and then the resource entity that is most beneficial to load balancing is selected from the candidate set. The present application defines the load balancing score as follows: (22) where d = 0 means that this is the load balancing score at the distributed data center level, and at this time represents the action made by the central agent, i.e., the serial number of the selected data center, when d ≠ 0 means that this is the load balancing score inside the data center , and at this time represents the action made by the local agent, i.e., the serial number of the selected node. Therefore, this action selection strategy can be expressed as formula 23: ​ (23) where, denotes the action space of the central agent or local agent, denotes the set of indices of the resource entities that satisfy the SLA constraints, since is extracted from the action space, so . is a predefined parameter of the policy. This policy can also incorporate a decay factor, which will become smaller and smaller as the number of training increases, until it degenerates into a full-greedy policy. The update rule is as follows: (24) where, is the decay rate, is a predefined parameter, which denotes the minimum value that can be decayed. If the parameter is not set, then will become 0 sooner or later, and the policy will degenerate into a full-greedy policy. Besides the policy, there are also the Boltzmann exploration, UCB, entropy regularization, etc.

[0041] where one specific training process of the MDDLB algorithm contains the following steps, as shown in Algorithm 3.

[0042]

[0043] Line 1 of the algorithm indicates that the learning of the algorithm enters into the loop iteration, in which the agents will gradually accumulate experience, update parameters, and learn how to accumulate the best expected reward value, so as to achieve the goal of load balancing. Lines 2-35 describe one training process of the MDDLB proposed by the present application. For a given input, in order to find the optimal request allocation scheme, a plurality of episodes need to be trained. Therefore, lines 6-35 will be executed iteratively until the maximum number of episodes M is reached.

[0044] At the beginning of each episode, all agents need to be reset, and the initial state of the central agent is re-observed (line 2). Line 4 embodies the improved policy adopted when selecting actions, and line 5 decays. From line 6, the learning process of the local agent is entered. Similarly, line 7 is the improved policy of the local agent. Lines 9-10 are the rewards and state transitions​ In line 11, the experience of the local agent is Save to Experience Replay In lines 12-14, if the experience replay has exceeded , the oldest half of the elements are deleted so that new experience can be stored. In the initial stage of the training process, because there is little experience in the replay buffer, the network does not have enough experience to learn. Therefore, a parameter is set To indicate the number of steps at the beginning of learning. In the scene, the local agent tends to explore more environments and the experience gained is stored in the replay buffer. When , the evaluation network starts to learn from past experiences. In this stage (lines 15-21), a small batch of size k is randomly sampled from the replay buffer, but here, to simplify the process, only one experience is randomly sampled each time. The estimated Q value is The network predicts. In lines 18-19, a gradient descent step is performed to minimize the loss and update Network parameters Finally, in order to ensure the stability of the training phase, in line 20, each step Step by step, The weights of the network are cloned to the target In the network, lines 22 to 34 are the learning process of the central agent, which is not much different from the learning process of the local agent. The above process is repeated until until.

[0045] Learning process, refer to the attached Figure 8 . Looking at the central agent separately With local agent , their network structure and learning process are similar. The network structure is a duel network. The central agent first observes the state of the environment. , then make the move , that is, a local agent is selected. For the central agent, it does not get rewarded immediately after taking action. , but enters the learning process of the local agent. The local agent observes the state of its environment , then make the move , then get rewarded The newly gained experience is stored in the experience playback buffer area In the process, a batch of experiences are extracted from the experience replay to update the evaluation network. The evaluation network and the target network calculate the evaluation Values ​​and Goals value, both of which compute losses for updating the evaluation network, which updates the target network every steps. The local agent gets the reward only after the central agent has assigned the nodes for the request , which is also the process of generating experience and storing it in the experience replay buffer. The following steps are not described in detail.

[0046] The connection between the central agent and the local agent is reflected in the fact that the central agent After making an action , the selected local agent only begins to observe the state of the environment it is in . The difference between the two is that the local agent updates its state immediately after making an action , while the central agent updates its state only after the local agent's request assignment for this time is complete. This is easy to imagine because only after the request occupies the resource, the state of the local agent and the central agent will change in succession. In addition to this difference, the size of the experience replay buffer of the local agent is that of the central agent , and as can be seen from the subscript, since the local agent is not necessarily selected continuously, the subscript of the experience stored in the buffer will jump, such as the first experience in the experience replay buffer in the figure jumps to the second experience . It should be noted that the figure is only an example showing the learning process of the two agents when the central agent selects the local agent in the process of a request assignment. In the process of continuous request assignment, different local agents may be selected.

[0047] Four, experimental results 1. Experimental environment Operating system: Windows 10 Hardware configuration: CPU is i9-9900K, memory is 32GB, main frequency 3.6GHZ Integrated development environment: JetBrains Pycharm 2020 Development tools: Python 3.7, Pytorch1.10.2 2. Experimental data set The data set contains three separate traces, namely server, online service and batch job. Each trace contains two files, the meta file records the basic information, and the usage file stores the runtime data, and six csv files total 570GB, and the description of each file is shown in Table 1: Table 1 Description of each table in the data set

[0048] Because the original dataset contained numerous unused columns and incomplete records, data preprocessing was required before the experiment to prune unused rows and remove invalid records. The data required for the experiments in this paper primarily came from the machine_meta and batch_instance tables. Required fields were selected from these tables, and unnecessary fields were deleted, resulting in the node data table and the request data table. Five machines with varying computing power were selected from the machine_meta table as experimental nodes. The node information table is shown in the table below. As shown in Table 2, the node information table primarily consists of three fields: the node's CPU, memory size, and computing power. CPU and memory size represent the resources available to the node. For a CPU, 100 represents one core, and 80 represents 80% of a core. The number of machine language instructions processed per second (MLIs) by the CPU represents the computing power of the node. In this study, 50 nodes were selected to execute requests. Generally speaking, machines with average performance are the majority, while machines with high or low performance are typically in the minority. Therefore, the number of nodes of the two types, the worst and the best, in terms of computing power, is set to 5, the second best is set to 10, and the medium performance is set to 20.

[0049] Table 2 Node Information Table Description

[0050] The experiment is conducted with five different data centers, each with 10 nodes. The allocation of the 50 nodes is random. The unit time usage price of the resources in the five data centers is shown in Table 3.

[0051] Table 3 Data center resource price per unit time

[0052] As shown in Table 4, the request information table mainly has three fields: the request number, which is used to uniquely identify the request; the number of millions of instructions in the request, which is used to indicate the size of the data volume and is used to calculate the execution time; the submission time of the request, which indicates the time when the cloud user submits this request; and the number of CPU cores and memory size required by the request, which are used to indicate the resources required by the request.

[0053] Table 4 Request field table

[0054] Table 5 shows the parameter settings of the MDDLB algorithm proposed in the present invention.

[0055] Table 5 MDDLB algorithm parameter settings

[0056] In order to comprehensively compare the method proposed in the present application with the comparative method, the present application adopts multiple indexes, which include load balancing degree, SLA conflict times, completion time and cost. The load balancing degree reflects the ability of the agent to fully utilize the computing resources, and the last three indexes reflect the goal that the load balancing can finally reach through the guidance of the reward function, reflecting the ability of the agent to quickly and effectively process requests.

[0057] 3. Algorithm effect In order to evaluate the performance of the proposed method, the present application compares it with some classic load balancing algorithms and traditional deep reinforcement learning-based methods. They include random scheduling algorithm (Random), genetic-polling scheduling algorithm (GA-RR), DDMTS and IQL.

[0058] (1) Convergence In order to verify the convergence and stability of the agent, in the scenario of 4500 request quantities, the present application collects the reward values accumulated by each episode of MDDLB, single-agent Dueling DQN and multi-agent DQN, and illustrates the convergence and stability of the algorithm through the change curve of the reward value. First, compare the reward changes between the central agents. For MDDLB and multi-agent DQN, the models between them are consistent, which are multi-agent models, and the only difference is the algorithm used. The reward change curve of the central agent is as follows: Figure 9It can be found that the central agent of MDDLB reaches the convergence state at about 250 episodes, the central agent of IQL reaches the convergence state at about 1250 episodes, and the DDMTS reaches the convergence state at about 1750 episodes, which indicates that the convergence speed of the central agent of MDDLB and IQL is faster than that of DDMTS. This is because the action space of a single agent is large, and a total of 50 actions can be selected, and a larger action space is prone to cause the effect of the agent to be suboptimal, while in the multi-agent model, the central agent only has 5 actions to choose from, and the local agent only has 10 actions to choose from, and the central agent and the local agent need to consider smaller actions and are more likely to make beneficial actions. After the three agents reach the convergence state, the average reward values of them are 1609.78, 1459.83 and 1360.45 respectively, and the reward values that can be reached by MDDLB and IQL are also higher, which indicates that the multi-agent model of MDDLB and IQL is more effective than the single-agent model. Secondly, the reward changes between local agents also need to be compared. For MDDLB and IQL, since there are multiple local agents, and only the local agent of one scheduler will respond in each request allocation process, and the local agents of other schedulers are in a static state, the training degrees of the local agents are completely different, and the number of times each scheduler is selected after each episode is also different, and the final rewards of them are also different, so it is difficult to compare them. Therefore, in order to make a unified comparison, the sum of the rewards of all local agents in each episode is calculated, that is, , where T is a set of time periods of event occurrence, and finally the reward curves are compared, as shown in FIG. 6. Figure 10 It can be seen from the figure that the local agents of MDDLB and IQL reach the convergence at about 250 episodes and 1250 episodes respectively, because MDDLB uses a duel network, and IQL uses a traditional single-flow Q network. The former has certain advantages over the latter, and it can effectively learn the state value function. With each update of the Q value in the duel network, the network connection parameters related to the state value can be gradually learned regardless of how the action A is selected, so the learning under multiple action selection is more stable, the reward value obtained is larger, and the number of learning rounds is less. With the increase of the number of action selection, the advantage is more obvious. Similarly, after convergence, the average reward values of the two methods are 1434.47 and 1289.17, which also indicates that the duel network structure has more advantages than the single-flow network structure.

[0059] (2) Evaluation index-load balancing degree From each episode, the load balancing degree of the distributed data center is constantly changing. In order to compare the pros and cons of the load balancing algorithm in this process, 10 episodes of Dueling DQN, IQL and DDMTS that have reached the convergence condition are randomly selected under the scenario of 4500 requests. The load balancing degree of the distributed data center is calculated at each step of scheduling in the 10 episodes. The average load balancing degree of the 10 episodes is taken when drawing the graph. Finally, the variation curve of the CPU utilization standard deviation and the variation curve of the memory utilization standard deviation of the distributed data center are obtained respectively.

[0060] The variation curve of the CPU utilization standard deviation and the variation curve of the memory utilization standard deviation of the distributed data center are shown in Figs. 2 and 3. Figure 11 and Figs. 4 and 5. Figure 12 The horizontal axis of the two graphs represents the first 5000 training steps in the episode, and the vertical axis is the CPU utilization standard deviation and the memory utilization standard deviation of the distributed data center. The smaller the standard deviation, the better the load balancing of the current distributed data center. From the graph, it can be intuitively felt that the difference between most algorithms in the early stage is not large, but as the training deepens, the number of requests that need to be scheduled is also constantly changing, and the resources required by the requests are various, which challenges the adaptability of the algorithm. It can be found that the curve of the method MDDLB proposed by the present application is in the lowest position in most training steps, and the change is relatively stable. The second best performance is IQL, followed by DDMTS. The worst two methods are the round robin algorithm and the random algorithm, because the scheduling rules of these two algorithms are simple and cannot adapt to various scheduling scenarios. In this experiment, the average CPU utilization standard deviation of each algorithm in this episode is: 0.02696, 0.3264, 0.03664, 0.04519 and 0.05453, and the average memory utilization standard deviation is: 0.0927, 0.1098, 0.1638, 0.1991 and 0.2639. This fully shows that the central agent of MDDLB has good scheduling performance and self-adaptability at the level of the distributed data center. In order to illustrate the load balancing situation inside each data center in this process, the average resource utilization standard deviation of each data center is calculated, that is, wherein, res represents the resource, which can be CPU resource or memory resource, represents the resource utilization of the data center at the nth training step. In addition, the average resource utilization of the five data centers is also calculated, that is,

[0061] Figs. 6 and 7. Figure 13 and Figs. 8 and 9. Figure 14 ​It is noted that the results are multiplied by 100 times for better display. The two above figures show the load balancing effect of each method in five data centers, and the left vertical axis corresponds to the column chart, and the right vertical axis corresponds to the line chart. It can be found that the agent using the MDDLB algorithm can keep the load balancing of the data center low, which shows that each local agent can well complete its own work and can better balance the load of the data center it governs, followed by IQL and DDMTS, which shows the superiority of the multi-agent model and the ability of the Dueling DQN algorithm to improve the performance of the traditional DQN.

[0062] (3) Evaluation index - SLA conflict times Table 6 SLA conflict times table

[0063] The SLA conflict times reflect the user's requirements for service quality. Obviously, the user hopes that the request can be responded in the first time, and if not, at least the SLA conflict times of each request should be reduced as much as possible. Comparative experiments of five methods in different request quantity scenarios are carried out, and the total number of all requests rejected in each scenario is obtained, and the results are shown in Table 6. The bold part in the table represents the lowest SLA conflict times in each scenario. It can be seen that the MDDLB proposed in the application can achieve the lowest SLA conflict times in the scenarios of 1500 requests, 3000 requests, 4500 requests and 6000 requests, because the method proposed in the application can ensure that the data center is in a relatively stable load balancing state, can better utilize the available computing resources, and each node can be fully utilized, thereby reducing the possibility of request rejection. The IQL performs best, because the multi-agent model can better balance the load. The DDMTS has little difference with the polling scheduling algorithm in the scenario of small number of requests, but as the number of requests increases, the number of requests to be processed at the same time increases, and the adaptive ability based on deep reinforcement learning algorithm is better than the static load balancing algorithm.

[0064] (4) Evaluation index - completion time and energy consumption cost Comparative experiments of the completion time and energy consumption cost of five methods in different request quantity scenarios are carried out, and the experimental results are shown in Figures 7 and 8. Figure 15 and Figure 8. Figure 16It can be seen from the two figures that the MDDLB algorithm proposed in the application can realize shorter completion time and lower energy consumption cost in different scenarios. This is because, compared with the traditional load balancing algorithm, the MDDLB can consider the different computing capabilities of each node and the different use prices of each data center, reduce the number of SLA conflicts of the request, and distribute the request to the most suitable node for running. For the completion time, the average completion time of the five load balancing algorithms in the four scenarios is 452.75 seconds, 470.75 seconds, 481.75 seconds, 513.25 seconds and 540.75 seconds, respectively, and the completion time of the MDDLB algorithm is reduced by 3.98%, 6.04%, 13.36% and 19.44% compared with other comparative algorithms, respectively. The energy consumption cost is 19792.78 yuan, 20995.24 yuan, 21293.56 yuan, 22095.20 yuan and 23368.74 yuan, respectively, and the energy consumption cost of the MDDLB algorithm is reduced by 6.08%, 7.58%, 11.63% and 18.07% compared with other comparative algorithms, respectively.

[0065] From the above experiments, it can be found that the performance of the round-robin scheduling algorithm is better than that of the random algorithm. This is because the allocation method of the random algorithm is relatively random and does not consider the load of each virtual machine, which can easily cause load imbalance. Although the round-robin scheduling algorithm does not consider the work load on each node and the required load of the request to be scheduled, it at least ensures that each node has a chance to be selected, so the load balancing effect is better. Of course, the algorithm based on deep reinforcement learning is more suitable for complex task scenarios than the traditional algorithm. Therefore, the application proves through several experiments that the load balancing framework based on distributed data centers and the load balancing algorithm based on multi-agent deep reinforcement learning can more evenly distribute the load, reduce the number of request rejections, shorten the completion time, save the energy consumption cost, and prove the effectiveness, superiority and adaptability of the algorithm.

[0066] (5) Ablation experiment The improvement of the application compared with the traditional deep reinforcement learning algorithm mainly has three parts: multi-agent system, competition network and action exploration strategy. In order to explain the effectiveness of these three parts, it is necessary to carry out ablation experiment, and the application selects the scene of 4500 requests to ablate different components, and the experimental results are shown in Table 7.

[0067] Table 7 Ablation experiment table

[0068] As can be seen from Table 7, the first row represents all components of the MDDLB algorithm, the second row represents the MDDLB algorithm without multi-agent, the third row represents the single stream Q network, and the fourth row represents the most basic From the experimental results, it can be clearly seen that the performance will decrease if one component is ablated, and the greater the decrease, the more important the ablated part is. Therefore, according to the table, the importance of the three components from high to low is: multi-agent, action exploration strategy and competition network. At the same time, in order to illustrate the correctness of the parameter selection in the previous experiment, the present application also carries out experimental comparison on the hyperparameters with different values, and the results are shown in Table 8.

[0069] Table 8 Hyperparameter selection table

[0070] From the results, when , and are respectively 0.001, 0.9 and 0.95, the reward value can reach a good level, which shows that the parameter selection in the previous experiment is correct and can better reflect the performance of the MDDLB algorithm.

[0071] The present application carries out simulation experiments on the cluster tracking data set disclosed by Alibaba, evaluates and analyzes the MDDLB algorithm proposed by the present application and other comparative algorithms, and the experimental results verify that the MDDLB method can converge faster than the DQN algorithm of the single-agent model, can reach a steady state at a higher reward value, and can better complete the load balancing task, so as to achieve the goal of maintaining a lower load balancing degree, reducing the number of SLA conflicts, shortening the completion time and saving energy cost, which shows the effectiveness and superiority of the load balancing algorithm proposed by the present application. Through the ablation experiment and the hyperparameter experiment, the effectiveness of the components of the MDDLB algorithm and the correctness of the hyperparameter selection are illustrated.

Claims

1. A multi-agent system, characterized in that: include: User layer, used to initiate requests; The proxy layer, including the central agent, can observe the resource status of all local agents and data centers, and is used to select a data center to respond to requests initiated by the user layer based on the observed resource status using Markov decision process and deep reinforcement learning algorithm; The scheduling layer includes multiple schedulers and multiple nodes, where each scheduler includes a local agent. The local agent is configured in the data center and can observe the resource status of the data center where it is located. Each local agent is independent. After the central agent selects the corresponding data center, the local agent uses the Markov decision process and deep reinforcement learning algorithm to select the corresponding node based on the observed resource status to respond to the request initiated by the user layer for load balancing.

2. The multi-agent system according to claim 1, wherein: The local agent includes a state space, an action space, a reward function, and a state transition. The state space is a space containing all possible states in the environment of the local agent. The state of the local agent is the resource information of all nodes contained in the corresponding data center and the resource information of the requests that need to be scheduled in the current time period. The action space is a space containing all actions that the local agent can choose. The reward function consists of two parts. The first part is used to determine whether the scheduling meets the requirements of the SLA constraints. If not, a penalty is given. Otherwise, the second part is entered to determine whether the scheduling is conducive to the load balancing of a single data center. The state transition includes the first state change in which the resources on the node are occupied after the local agent takes an action and the second state change before the next scheduling request of the local agent.

3. The multi-agent system according to claim 2, wherein: The reward function of the local agent is: in, A negative value indicates a penalty for the local agent. 、 and Respectively represent the selection nodes Standard deviation of CPU utilization in the post-data center , select Select Node Standard deviation of memory utilization in the post-data center and request execution time The weight of Express a request Assigned to the node What is needed is the inverse of the running time.

4. The multi-agent system according to claim 3, wherein: The weights in the reward function of the local agent are calculated using the entropy weight method.

5. The multi-agent system according to claim 1, wherein: The central agent includes a state space, an action space, a reward function, and a state transition. The state space is the space containing all possible states in the environment of the central agent. The state of the central agent is the resource information of each data center and the resource information of the requests that need to be scheduled in the current time period. The action space is a space containing all actions that the central agent can choose. The reward function consists of two parts. The first part is related to the scheduling of the local agent. If the scheduling of the local agent does not meet the SLA constraints, both the local agent and the central agent will be punished. If the SLA constraints are met, the second part is entered to calculate the degree of change in the load balancing of the entire system. The state transition is the state change after the local agent allocates the request to the node.

6. The multi-agent system according to claim 5, wherein: The reward function of the central agent is: in, is the inverse of the energy cost required after the two-stage allocation between the central agent and the local agent, indicating that the central agent has selected the data center , the local agent selects the node The energy consumption, 、 and is the standard deviation of CPU utilization of distributed data centers after data center selection , Standard deviation of memory utilization of distributed data centers after selecting data centers and The three weights.

7. A deep reinforcement learning load balancing method based on a multi-agent system, characterized in that: The method is implemented based on the multi-agent system according to any one of claims 1 to 6, comprising: Receive requests initiated by users; The central agent in the proxy layer selects a data center based on observed resource conditions using a Markov decision process and deep reinforcement learning algorithm; The local agent in the scheduling layer selects nodes based on the observed resource conditions using Markov decision process and deep reinforcement learning algorithm.

8. The deep reinforcement learning load balancing method based on a multi-agent system according to claim 7, characterized in that: The deep reinforcement learning algorithm uses a duel network, which includes an evaluation network and a target network. The evaluation network is used to evaluate the current state-action pair generated. value ,in, is the current state, For action, To evaluate the parameters of the network; the target network is used to obtain the target value ,in are the parameters of the target network.

9. The deep reinforcement learning load balancing method based on a multi-agent system according to claim 7, characterized in that: The structure of the duel network consists of seven parts: input layer, hidden layer, branch layer, state layer, advantage layer, aggregation layer and output layer. The input of the input layer is the current state. The hidden layer consists of two fully connected layers. The output of the hidden layer will be divided into two branches. Both branches are fully connected layers with the same number of neurons. The first branch will output a scalar, which is the value, and the second branch will output the advantage value, which is a vector. These two values ​​will be aggregated in the aggregation layer; the output layer outputs the aggregation result. For the central agent, the output selects what can be obtained by each data center. For the local agent, the output is the value that each node can obtain. value.

10. The deep reinforcement learning load balancing method based on a multi-agent system according to claim 7, characterized in that: When the central agent and the local agent perform load balancing, they use an improved Strategy: in, express value, represents the current state space, represents the action space of the central agent or local agent, A set of serial numbers representing resource entities that meet SLA constraints. yes The predefined parameters of the strategy, the above formula means that if the random number is less than , we first extract resource entities that will not violate the SLA as the candidate set, and then select the resource entity that is most conducive to load balancing from the candidate set, where the resource entity includes data centers and nodes. is the load balancing score, which is calculated as follows: Among them, when d=0, Indicates the load balancing score at the distributed data center level. Indicates that this is the action taken by the central agent, that is, selecting the sequence number of the data center, , Indicates the weight of the standard deviation of CPU utilization and memory utilization after selecting the node; when d≠0, Indicates data center Internal load balancing score, at this time Indicates the action taken by the local agent, that is, the sequence number of the selected node, , Indicates the weight of the standard deviation of CPU utilization and the standard deviation of memory utilization after selecting a data center.

Images