Computer load balancing method based on multi-agent reinforcement learning

Abstract

The invention provides a computer load balancing method based on multi-agent reinforcement learning, which introduces a De-POMDP model concept, proposes a dual-cycle communication network, and adopts a dual-cycle network training load strategy of a deterministic strategy gradient algorithm. The invention designs a more effective and intelligent load balancing method. The invention provides a distributed dual-circulation framework based on multi-agent reinforcement learning to simulate a load strategy, and experiments show that when a computing system becomes more complex and many autonomous learning causes no convergence of the load strategy and causes great delay, the dual-circulation framework provides a communication channel for a distributed computer, so that the load strategy is more stable. And coupling between different computers is ensured. Compared with other methods based on reinforcement learning or multi-agent reinforcement learning, the method provided by the invention is more suitable for a large-scale computing system.

Description

Technical Field

[0001] This invention relates to the field of networking, and in particular to a load balancing method for computers. Background Technology

[0002] With the development of network technology, more and more applications have emerged, such as smart cities, big data, intelligent vehicles, and adaptive drones. However, the development of these products has placed new demands on computing and processing capabilities. How to achieve real-time and large-scale computing is the focus of this invention. Based on these problems, and the latency and reliability issues of network transmission and task computing loads in large systems, a computer load balancing strategy based on multi-agent reinforcement learning is designed and implemented to alleviate network transmission latency and the uneven distribution of task computing resources, thereby achieving low-latency and efficient transmission while improving reliability.

[0003] Modern large-scale applications have high latency requirements. When they process tasks on a computer, latency stems from both computational and communication capabilities. Computational tasks and results are transmitted via uplink and downlink, respectively, so communication latency depends on the distance between the application and the corresponding computer. For these reasons, load balancing strategies play a crucial role in the computing model. An excellent computing model should fully utilize computing resources and efficiently distribute the load to applications.

[0004] In earlier research, heuristic methods were used to generate load balancing strategy algorithms, where the computational task was treated as a traveling salesman problem. However, heuristic algorithms face the challenge of nondeterministic polynomials (NP) and can take a long time to compute in complex systems. With the success of deep reinforcement learning in unmanned scenarios, some researchers have attempted to simulate load balancing strategies using deep reinforcement learning methods. For example, Chen Xianfu et al. proposed a load balancing strategy based on a deep Q-learning model in "Mobile Network Optimization Selection Technology Based on Deep Reinforcement Learning," establishing channel quality as a state between mobile users and base stations; Ning Zhaolong et al. combined a two-sided matching scheme in "Intelligent Computing System Based on Deep Reinforcement Learning" and used deep reinforcement learning to schedule computation requests and network resource allocation separately; Wang Jiadai et al., in "Intelligent Resource Allocation Load Balancing Algorithm Based on Deep Reinforcement Learning," developed a resource allocation-based load deep reinforcement learning scheme to address abrupt changes in computational conditions.

[0005] Deep reinforcement learning methods can achieve timely decision-making after simulation; however, the state and action spaces of Markov Decision Process (MDP) models grow exponentially on a computer, leading to unbearable simulation costs. Therefore, multi-agent reinforcement learning techniques have gained popularity. This means that each computer or computing module can be simulated as an agent, assigned an independent load policy. Using multi-agent reinforcement learning, load policies can be simulated distributed across multiple computers, thereby optimizing the model, reducing system load, and serving the system more effectively. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention provides a computer load balancing method based on multi-agent reinforcement learning.

[0007] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0008] Step 1. Introduce the concept of the Dec-POMDP model;

[0009] Due to the uncertainty of application and computation tasks, it is modeled as a locally observable Markov decision process. Each agent treats the uncertain state transition as part of the coupling between itself and other agents, and uses a decentralized mechanism to simulate each independent policy. This is called a decentralized Markov decision process.

[0010] Since multiple agents collaborate in a partially observable, decentralized decision-making environment, the Dec-POMDP model is adopted to simulate the strong coupling of computers in decentralized management. At the same time, the recurrent neural network based on the communication channel requires that the state and work of the computer are shared. In addition, considering the state sequence of the computer, the dependency between communication and state sequence is grasped. Each computer perceives a complete environment through local observation. Therefore, a decentralized framework with a double recurrent network is established on the Dec-POMDP model so that the generated load balancing strategy can adapt to uncertain tasks.

[0011] Step 2. Propose a dual-loop communication network;

[0012] Given a system with N computers, each computer is deployed in its own location and has its own processing unit. <w i f i The set represents the i-th computer, where w i f represents the computer's operating status. i This refers to the computer's load capacity, measured in cycles per second or megahertz; the tasks to be performed are represented as...<t1,t2,···> And each task's tuple is represented as j ,d j ,l j ​,u j ,k j >, where b j Let d represent the priority of the j-th task. j Let l represent the importance of the j-th task. j For the time required for uplink and downlink, u j k represents the workload of performing the j-th task. j This represents the waiting delay, which increases over time; the goal is to find an overall load strategy <π1, π2, ..., π N >The minimum total delay K = ∑ i k i Since each computer has a different strategy, the task set operates on each computer to obtain the total latency k for each computer. i ;

[0013] At each time point, the task set includes new tasks and waiting tasks. First, the first task set s is executed. t = <t1,t2,…,t M Because the number of tasks is uncertain, a maximum number of tasks M is set. When the number of tasks is less than M, s t Will be filled with 0, s t The task status is represented and transmitted to all computers, and a separate load balancing strategy is assigned to each computer; the load balancing strategy π for the i-th computer. i It is a series of task states s t and computer status s = <w i f i >To load behavior a i The mapping, load behavior a i It is divided into three states: forced execution, normal execution, and non-execution. Forced execution means that the current task is forced to wait.

[0014] The load balancing strategy is distributed, with each load action a... i =π i (s t s s θ i , ρ i ), ρ i θ i For π i The parameter, ρ i Edge-constrained index, θ i Resource allocation granularity; In order to capture the coupling between computers, a communication channel is constructed to share messages, and considering the dependency of task sequences, a double-loop network is developed for communication;

[0015] Step 3. Perform gradient calculation;

[0016] A dual-loop network using a deterministic policy gradient algorithm is used to train the load strategy;

[0017] Step 4. Model running;

[0018] In the dual-loop model, the state of the encoding module and the state of the task are both S-shaped growth curves. Each time there are n nodes in the dual-loop network, while the action network and the critical network have hidden layers with 2n nodes.

[0019] Next, scale up the model, simulate a computer, set its computing power and the maximum number of tasks, and record the final average waiting time.

[0020] In step 2, firstly, the task state s is encoded using two encoders. t,i and computer status s,i Perform encoding, and connect the encoded task state and computer state to s i Then s i With the previous state sequence h' i A connection, or state sequence, refers to the trajectory of an agent's observed states arranged chronologically over a period of time. It reflects the complete process of dynamic environmental changes in a multi-agent system and serves as a crucial basis for policy learning, value function estimation, and credit allocation. Processing the fully connected layer with a normalized exponential function yields...

[0021]

[0022] Where f represents the fully connected layer processing used to compute h' i and s i The correlation strength; Softmax is used to normalize the values ​​generated by f into a probability distribution; the generated It is a temporary state, and the sequence of states is integrated through a fully connected layer:

[0023]

[0024] The hyperbolic tangent function is used for nonlinear activation.

[0025] The first loop integrates the state sequence for each server, while the second loop combines the communication sequences. These sequences are time-series records of information exchanged by multiple agents during cooperation or competition, either explicitly or implicitly. They describe how agents coordinate strategies through message passing, signal sharing, or environmental mediation, and are a key mechanism for solving partial observability issues and achieving efficient cooperation. In Dec-POMDP, communication approximates a virtual global state; the load policy from another computer serves as the basis for the previous communication c'. i The input loop model is passed through a fully connected layer to generate a new communication result c. i ;

[0026] c i =tanh[f(c' i ,h i (3)

[0027] Using the established recurrent network framework, the state sequence h generated by the first loop is integrated. i The communication sequence c generated by the second loop i Compared to other recurrent network models, the dual-loop network model has a controller in the first loop that is responsible for generating actions, while the second loop is separate from the first and is used for state evaluation and feedback adjustment. This network provides the computer with a communication channel for sharing state, and the dual-loop network captures the dependencies between state and work sequence.

[0028] In step 3, the function value is estimated using the second recurrent network, and the probability distribution of the selection behavior is calculated using the first recurrent network; for the i-th server, its first recurrent network parameters are... Let i be the policy parameter for the i-th server, and let its second cyclic network parameter be denoted as: Let K be the value function parameter for the i-th server, and let the compensation K be a negative of the total waiting time. Therefore, the loss function of the second recurrent network is:

[0029]

[0030] Where γ is the discount factor, and the state-value function Estimate the current state c i Long-term returns For the long-term reward of the next state, the bicyclic network is updated using the policy gradient algorithm, and the corresponding gradient is calculated as follows:

[0031]

[0032] Where V(c) i) is the output of the second recurrent network, equivalent to the state-value function. Representing strategy π i The average return of all actions a, π i (c i |a) represents the probability distribution of the strategy.

[0033] The specific process for calculating the loss function and gradient is as follows:

[0034] a) The first loop is based on the current state c i Select action a;

[0035] b) The environment returns reward K and the next state c. i+1 ;

[0036] c) The second loop calculates the current state value. and target value

[0037] d) Update the parameters of the second recurrent network To minimize the mean squared error loss;

[0038] e) Update the parameters of the first recurrent network. To maximize expected return;

[0039] The loss function measures the difference between the model's predicted value and the true value. It is the objective to be minimized during the optimization process. The gradient is the derivative of the loss function with respect to the model parameters, indicating how to adjust the parameters to minimize the loss function. Only through continuous updates and iterations of the gradient and loss function can the optimal solution of the system be found.

[0040] An electronic device includes: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.

[0041] A computer-readable storage medium storing program code that can be invoked by a processor to perform the method described above.

[0042] The beneficial effects of this invention lie in its improved computational efficiency and the design of a more efficient and intelligent load balancing method. This invention proposes a distributed, multi-agent reinforcement learning-based dual-loop framework to simulate load balancing strategies. Experiments show that as computing systems become more complex, numerous autonomous learning processes cause load balancing strategies to fail to converge, resulting in significant latency. The dual-loop framework of this invention provides a communication channel for distributed computers, ensuring coupling between different computers. Compared to other reinforcement learning-based or multi-agent reinforcement learning-based methods, the method of this invention is more suitable for large-scale computing systems. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the model framework of the present invention.

[0044] Figure 2 This is the flowchart of the dual-loop network of the present invention.

[0045] Figure 3 This is a simulation result diagram of 10 computers in an embodiment of the present invention. Detailed Implementation

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0047] The specific steps of the embodiments of the present invention are as follows:

[0048] Step 1. Introduce the concept of the Dec-POMDP model;

[0049] Due to the uncertainty of applications and computational tasks, it is modeled as a locally observable Markov decision process (POMDP). Each agent treats the uncertain state transition as part of the coupling between itself and other agents, and uses a decentralized mechanism to simulate each independent policy, which is called a decentralized Markov decision process (Dec-POMDP).

[0050] Since multiple agents collaborate in a partially observable, decentralized decision-making environment, the Dec-POMDP model is adopted to simulate the strong coupling of computers in decentralized management. At the same time, the recurrent neural network based on the communication channel requires that the state and work of the computer are shared. In addition, considering the state sequence of the computer, the dependency between communication and state sequence is grasped. Each computer perceives a complete environment through local observation. Therefore, a decentralized framework with a double recurrent network is established on the Dec-POMDP model so that the generated load balancing strategy can adapt to uncertain tasks.

[0051] Step 2. Propose a dual-loop communication network;

[0052] In large systems, multiple computers are deployed in different locations to share the load and handle computationally intensive application tasks, thereby reducing computational latency. Most importantly, the total latency caused by the computational load of a task depends on the allocation of load resources; therefore, a prerequisite for an effective load balancing strategy is low latency. When designing a load balancing strategy, latency, transmission latency, and computation latency must be considered. Transmission latency is caused by the tasks and corresponding results of the uplink and downlink, while computation latency depends on the computation cycle of the required tasks and the load capacity of the allocated computers. A good load balancing strategy should achieve the minimum total latency.

[0053] Given a system with N computers, each computer is deployed in its own location and has its own processing unit. <w i f i The set represents the i-th computer, where w i f represents the computer's operating status. i This refers to the computer's load capacity, measured in cycles per second or megahertz; the tasks to be performed are represented as...<t1,t2,···> And each task's tuple is represented as j ,d j ,l j ,u j ,k j >, where b j Let d represent the priority of the j-th task. j Let l represent the importance of the j-th task. j For the time required for uplink and downlink, u j k represents the workload of performing the j-th task. j This represents the waiting delay, which increases over time; the goal is to find an overall load strategy <π1, π2, ..., π N >The minimum total delay K = ∑ i k i Since each computer has a different strategy, the task set operates on each computer to obtain the total latency k for each computer. i ;

[0054] like Figure 1 As shown, at each time point, the task set includes new tasks and waiting tasks. First, the first task set s is executed. t = <t1,t2,…,t M Because the number of tasks is uncertain, a maximum number of tasks M is set. When the number of tasks is less than M, s t Will be filled with 0, s t The task status is represented and transmitted to all computers, and a separate load balancing strategy is assigned to each computer; the load balancing strategy π for the i-th computer.​i It is a series of task states s t and computer status s = <w i f i >To load behavior a i The mapping, load behavior a i It is divided into three states: forced execution, normal execution, and non-execution. Forced execution means that the current task is forced to wait.

[0055] The load balancing strategy is distributed, with each load action a... i =π i (s t s s θ i , ρ i ), ρ i θ i For π i The parameter, ρ i Edge-constrained index, θ i Resource allocation granularity; to capture coupling between computers, a communication channel is constructed to share messages. Figure 1 The communication channel shown in the connecting line section employs a cyclic and embedded strategy. Considering the dependencies between task sequences, a double-loop network is developed for communication. The developed network flow is as follows: Figure 2 As shown;

[0056] First, the task state s is processed using two encoders respectively. t,i and computer status s,i Perform encoding, and connect the encoded task state and computer state to s i Then s i With the previous state sequence h' i A connection, or state sequence, refers to the trajectory of an agent's observed states arranged chronologically over a period of time. It reflects the complete process of dynamic environmental changes in a multi-agent system and serves as a crucial basis for policy learning, value function estimation, and credit allocation. Processing the fully connected layer with a normalized exponential function yields...

[0057]

[0058] Where f represents the fully connected layer processing used to compute h' i and s i The correlation strength; Softmax is used to normalize the values ​​generated by f into a probability distribution; the generated It is a temporary state, and the sequence of states is integrated through a fully connected layer:

[0059]

[0060] The hyperbolic tangent function is used for nonlinear activation.

[0061] The first loop integrates the state sequence for each server, while the second loop combines the communication sequences. These sequences are time-series records of information exchanged by multiple agents during cooperation or competition, through explicit or implicit communication. They describe how agents coordinate strategies through message passing, signal sharing, or environmental mediation, and are a key mechanism for solving partial observability issues and achieving efficient cooperation. In Dec-POMDP, communication approximates the virtual global state; for example... Figure 2 As shown, the load strategy from another computer is used as the previous communication c' i The input loop model is passed through a fully connected layer to generate a new communication result c. i ;

[0062] c i =tanh[f(c' i ,h i (3)

[0063] Using the established recurrent network framework, the state sequence h generated by the first loop is integrated. i The communication sequence c generated by the second loop i Compared to other recurrent network models, the dual-loop network model has a controller in the first loop that is responsible for generating actions, while the second loop is separate from the first and is used for state evaluation and feedback adjustment. This network provides the computer with a communication channel for sharing state, and the dual-loop network captures the dependencies between state and work sequence.

[0064] Step 3. Perform gradient calculation;

[0065] A dual-loop network is used to train the load policy using a deterministic policy gradient algorithm. Specifically, the second loop network is used to estimate the function value, and the first loop network is used to calculate the probability distribution of the selected behavior. For the i-th server, the parameters of its first loop network are... Let i be the policy parameter for the i-th server, and let its second cyclic network parameter be denoted as: Let K be the value function parameter for the i-th server, and let the compensation K be a negative of the total waiting time. Therefore, the loss function of the second recurrent network is:

[0066]

[0067] Where γ is the discount factor, and the state-value function Estimate the current state c i Long-term returns For the long-term reward of the next state, the bicyclic network is updated using the policy gradient algorithm, and the corresponding gradient is calculated as follows:

[0068]

[0069] Where V(c) i ) is the output of the second recurrent network, equivalent to the state-value function. Representing strategy π i The average return of all actions a, π i (c i |a) represents the probability distribution of the strategy.

[0070] The specific process for calculating the loss function and gradient in formulas (4) and (5) is as follows:

[0071] a) The first loop is based on the current state c i Select action a;

[0072] b) The environment returns reward K and the next state c. i+1 ;

[0073] c) The second loop calculates the current state value. and target value

[0074] d) Update the parameters of the second recurrent network To minimize the mean squared error loss;

[0075] e) Update the parameters of the first recurrent network. To maximize expected return;

[0076] The loss function measures the difference between the model's predicted values ​​and the actual values; it is the objective to be minimized during optimization. The gradient is the derivative of the loss function with respect to the model parameters, indicating how to adjust the parameters to minimize the loss function. Through continuous updates and iterations of the gradient and loss function, the optimal solution for the system can be found. Compared to other load balancing strategies, the model of this invention has advantages such as high stability and low system latency, because it can perfectly replace other load balancing strategies.

[0077] Step 4. Model running;

[0078] First, five computers were simulated with computing power set to [5GHz, 10GHz, 15GHz, 20GHz, 25GHz], and the maximum number of tasks was set to [6, 7, 8, 9, 10]. Uplink and downlink transmission delays were randomly sampled from [1, 25] ms. The learning rate was set to 0.01, and the discount factor to 0.96. In the dual-loop model, both the encoding module state and the task state exhibited an S-shaped growth curve. Each iteration of the dual-loop network had 64 nodes, while the action network and critical network each had a hidden layer of 128 nodes. All reinforcement learning-based methods were simulated for 200,000 cycles, and the final average latency was recorded. Three comparison methods were set up: a heuristic algorithm, a deep Q-learning-based method, and a multi-agent Q-learning method. Experiments showed that all reinforcement learning-based methods outperformed the heuristic algorithm, with the deep Q-learning-based method achieving the best results when the maximum number of tasks was 6 and 7. However, as the number of tasks increases, meaning the action space becomes larger, the average waiting time for deep Q-learning methods increases significantly. The dual-loop network model and multi-agent Q-learning-based methods achieve more satisfactory results. This phenomenon suggests that multi-agent reinforcement learning is more suitable for simulating the load strategies of large-scale computing systems. In this experiment, the results of the dual-loop network model are similar to those of other multi-agent Q-learning-based methods.

[0079] Next, the model was scaled up to simulate 10 computers with computing power set to [5GHz, 5GHz, 10GHz, 10GHz, 15GHz, 15GHz, 20GHz, 20GHz, 25GHz, 25GHz], and the maximum number of tasks set to [16, 17, 18, 19, 20]. The final average wait time was recorded. The corresponding results are as follows: Figure 3 The results show that the dual-loop network model has superior advantages. When computing systems become complex, autonomous learning can cause load strategies to fail to converge. The dual-loop network model provides a communication channel for distributed servers, ensuring coupling between different servers.

Claims

1. A computer load balancing method based on multi-agent reinforcement learning, characterized in that Comprising the following steps: Step 1. Introducing the concept of Dec-POMDP model; Due to the uncertainty of application and computing tasks, modeling as a partially observable Markov decision process, each agent regards the uncertain state transition as a part of the coupling between it and other agents, using a decentralized mechanism to simulate each independent strategy, called decentralized Markov decision process; Due to the cooperation of multiple agents in a partially observable and decentralized decision environment, the Dec-POMDP model is adopted to simulate the strong coupling of decentralized computers; at the same time, the recurrent neural network based on communication channel requires the state and work of the computer to be shared, in addition, considering the state sequence of the computer, capturing the dependence of both communication and state sequence, each computer perceives a complete environment through local observation, so a decentralized framework with double recurrent network is established on the Dec-POMDP model, so that the generated load balancing strategy can adapt to uncertain tasks; Step 2. Propose a double recurrent communication network; A system with N computers is given, each computer is deployed in a respective location, has its own processing unit, <w i , f i > represents the i-th computer, where w i represents the working state of the computer, f i is the load capacity of the computer, in units of cycles / second or megahertz; the tasks to be executed are represented as <t1, t2, ···>, and the tuple of each task is represented as <b j , d j , l j , u j , k j >, where b j is the priority for the j-th task, d j is the importance for the j-th task, l j is the time required for uplink and downlink, u j represents the workload of executing the j-th task, k j represents the waiting delay, which increases with time; the purpose is to find an overall load strategy <π1, π2, …, π N > for the minimum total delay K = ∑ i k i , since each computer has a different strategy, the task set acts on each computer to obtain the total delay k i of each computer; At each time point, the task set includes new tasks and waiting tasks. First, the first task set s is executed. t = <t1,t2,…,t M Because the number of tasks is uncertain, a maximum number of tasks M is set. When the number of tasks is less than M, s t Will be filled with 0, s t The task status is represented and transmitted to all computers, and a separate load balancing strategy is assigned to each computer; the load balancing strategy π for the i-th computer. i It is a series of task states s t and computer status s = <w i f i >To load behavior a i The mapping, load behavior a i It is divided into three states: forced execution, normal execution, and non-execution. Forced execution means that the current task is forced to wait. The load strategy is decentralized, each load behavior a i = π i (s t , s s , θ i , ρ i ), ρ i , θ i is π i The parameter ρ i Edge limited index θ i Resource allocation granularity; in order to capture the coupling between computers, a communication channel sharing message is constructed, considering the dependence of task sequence, a double loop network is developed for communication; Step 3. Gradient calculation; Double recurrent network training load strategy using deterministic policy gradient algorithm; Step 4. Model running; In the double recurrent model, the encoding module state and task state are both an S-shaped growth curve, each time there are n nodes in the double recurrent network, while the action network and the critical network have hidden layers of 2n nodes; Secondly, amplify this model to simulate computers, set its computing power to the maximum number of tasks, and record the final average waiting time.

2. The computer load balancing method based on multi-agent reinforcement learning according to claim 1, wherein: In step 2, first, the task state s t,i and the computer state s s,i are encoded by two encoders respectively, and the encoded task state and computer state are concatenated to s i ; then s i is concatenated with the previous state sequence h' i , the state sequence refers to the trajectory of the observation state of the agent in time sequence within a period of time, which reflects the complete process of the dynamic change of the environment in the multi-agent system, and is an important basis for policy learning, value function estimation and credit assignment, and the full connection layer is processed by the normalized exponential function to obtain where f denotes a fully connected layer processing for computing h i and the association strength of s i Softmax is used to normalize the value generated by f into a probability distribution; the generated is a temporary state, while the state sequence is integrated by a fully connected layer: The hyperbolic tangent function is used for nonlinear activation; The first loop is used to integrate the state sequence of each server, and the second loop is used to combine the communication sequence, which is the time sequence record of information exchange between multiple agents in the process of cooperation or competition through explicit or implicit communication, describes how agents coordinate strategies through message passing, signal sharing or environmental intermediation, and is a key mechanism to solve partial observability and achieve efficient cooperation. In Dec-POMDP, communication approximates virtual global state; the load strategy from another computer is used as the previous communication c i The input loop model generates a new communication result c i through a fully connected layer; c i = tanh [f(c i , h i )] (3) With the established recurrent network framework, the state sequence h generated by the first cycle is integrated i and the communication sequence c generated by the second cycle i Compared with other recurrent network models, the dual recurrent network model has a controller in the first cycle responsible for generating actions, and the second cycle is separated from the first cycle for state evaluation and feedback adjustment; The network provides a communication channel for computers to share states, and the double recurrent network captures the dependence of state and work sequence.

3. The computer load balancing method based on multi-agent reinforcement learning according to claim 1, wherein: In step 3, the second recurrent network is used to estimate the function value, and the first recurrent network is used to calculate the probability distribution of the selection behavior; for the i-th server, the first recurrent network parameter is The strategy parameter of the i-th server is The value function parameter of the i-th server is compensated K is set to the negative of the total waiting time, so the loss function of the second recurrent network is: where γ is a discount factor, the state value function estimating the long-term return of the current state c i the long-term return of the next state, the bi-recurrent network is updated by a policy gradient algorithm, and the corresponding gradient is calculated as​ where V(c i ) is the output of the second recurrent network, equivalent to the state value function denotes the policy π i the average return of all actions a under the policy π i (c i |a) is the probability distribution of the policy.

4. The computer load balancing method based on multi-agent reinforcement learning according to claim 3, wherein: The specific process of loss function and gradient calculation is: a) the first cycle according to the current state c i select action a; b) environment returns reward K and next state c i+1 ; c) the second loop calculates the current state value and the target value d) updating the second recurrent network parameters to minimize a mean squared error loss; e) updating the first recurrent network parameters to maximize an expected return; The loss function is used to measure the gap between the predicted value of the model and the true value, which is the target to be minimized in the optimization process, and the gradient is the derivative of the loss function to the model parameters, indicating how to adjust the parameters to minimize the loss function; Through the continuous update of gradient and loss function, the optimal solution of the system can be better found.

5. An electronic device, comprising: Comprising: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs being configured to perform the method of any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program code, which can be called and executed by the processor to perform the method of any one of claims 1-4.

Images