Load balancing method based on multi-agent reinforcement learning

Abstract

The present invention discloses a load balancing method based on multi-agent reinforcement learning, comprising the following steps: the first step: collecting network information and initializing parameters; the second step: performing multi-agent deep deterministic policy gradient training to learn the optimal user switching strategy, specifically including: initializing the current training steps; counting the current step network state; selecting the current action according to the policy network; performing actions on each user; counting the network state and reward after the current action is performed; storing the current sample in the experience replay pool; if the number of samples in the experience replay pool is greater than the batch size, updating the Q network and policy network weights of each agent; then judging whether the current training steps can divide the number of interactions per round, if so, initializing the network environment, starting a new round of training, until the cumulative reward value of each round converges to the maximum value, and obtaining n optimal policy networks. The present invention can improve the imbalance of network load and reduce the network load in a more refined way.

Description

Technical Field

[0001] The present invention belongs to the field of load balancing in wireless communications, and in particular relates to a load balancing method based on multi-agent reinforcement learning. Background Art

[0002] In wireless communication, load balancing between base stations is achieved by switching some users in a heavily loaded base station to a lightly loaded base station. Existing load balancing methods mainly determine whether a user needs to switch based on the Reference Signal Receiving Power (RSRP) of the user's serving base station. In fact, factors that affect user switching include the base station load status and the user's data rate requirements. Summary of the invention

[0003] Purpose of the invention: In order to address the deficiencies of the above-mentioned background technology, the present invention proposes a load balancing method based on multi-agent reinforcement learning, abstracts the load balancing problem into a stochastic game process (Stochastic Game), and then uses the Multi-agent Deep Deterministic Policy Gradient (MADDPG) method to solve it, which can improve the imbalance of network load and reduce the network load in a more refined way.

[0004] Technical solution: In order to achieve the above-mentioned invention object, a load balancing method based on multi-agent reinforcement learning of the present invention comprises the following steps:

[0005] Step 1: Collect network information, including the set of base stations participating in load balancing, and generate a set of agents N corresponding to users according to the users who meet the A4 event in the overloaded base stations. i ,...,u n}, where u i represents the i-th agent, n is the number of agents; initialize the reinforcement learning parameters;

[0006] Step 2: Conduct multi-agent deep deterministic policy gradient training to learn the optimal user switching strategy, including the following steps:

[0007] Step 2.1, use t to represent the current training step number, initialize t = 1; initialize all target strategy network weights and all target Q network weights;

[0008] Step 2.2: Collect the load status and reference signal received power of the service base station of all agents corresponding to the users in the agent set N. The state of the i-th agent in the current training step t is s i (t)=(ρ i (t),RSRPi (t)), where ρ i (t), RSRP i (t) are the service base station load and reference signal receiving power of the user corresponding to the ith agent respectively; the state of the entire communication network environment at the current t training step is composed of the states of all agents: total (t)=(s1(t),s2(t),...,s i (t),...,s n (t));

[0009] Step 2.3: Select the current action for all agents in the agent set N according to the policy network. The action of the i-th agent in the current training step t is where ε(t) is the Ornstein-Uhlenbeck (OU) random process, is the strategy network of the ith agent; the actions of the entire communication network environment at the current t training step are composed of the actions of all agents: a total (t)=(a1(t),a2(t),...,a i (t),...,a n (t));

[0010] Step 2.4: Execute action a of all agents total (t)=(a1(t),a2(t),...,a i (t),...,a n (t)), a i (t)∈{A0,A1}, when a i (t) = A0, the user corresponding to the i-th agent does not perform a switching operation and remains in the current serving base station. i (t) = A1, the user corresponding to the i-th agent switches to the neighboring base station with the largest reference signal receiving power. If the switching causes the neighboring base station to be overloaded, the user switches back to the serving base station. If the switching does not cause the neighboring base station to be overloaded, the user stays in the neighboring base station;

[0011] Step 2.5: The reward for all agents in the entire communication network environment is r total (t)=(r1(t),r2(t),...,r i (t),...,r n (t)), where r i (t) represents the reward of the ith agent in the current t training steps. The rewards of all agents satisfy r1(t)=...=r i (t)=...=r n (t) = r(t), r(t) is the reward value;

[0012] Step 2.6: Collect all agents in the agent set N to perform action a total (t) After that, the load state of the corresponding user's service base station and the reference signal received power are updated to update the state of the agent in the next step, that is, step t+1. The state of the i-th agent in step t+1 is s i (t+1)=(ρ i (t+1),RSRP i (t+1)), the next state of the entire communication network environment is s total (t+1)=(s1(t+1),s2(t+1),...,s i (t+1),...,s n (t+1));

[0013] Step 2.7: Set the current sample (s total (t),a total (t),r total (t),s total (t+1)) is stored in the experience replay pool D;

[0014] Step 2.8: If the number of samples stored in the current experience replay pool D is greater than the batch size batch_size, proceed to step 2.9 to complete the weight update process of the target Q network and target strategy network of all agents in the agent set N; if the number of samples stored in the current experience replay pool D is less than the batch size batch_size, proceed to step 2.10;

[0015] Step 2.9, using the gradient descent method to update the Q network weights, policy network weights, target Q network weights and target policy network weights of each agent in the agent set N;

[0016] Step 2.10, if t% steps_per_episode = 0, where % represents the modulus value, steps_per_episode is the number of interactions per round, calculate the cumulative reward value of this round of training, initialize the environment, and start a new round of training;

[0017] Step 2.11, let t = t + 1;

[0018] Step 2.12: Repeat steps 2.2 to 2.11 until the cumulative reward value of each round converges to the maximum value;

[0019] Step 2.13, exit the iteration and obtain n optimal strategy networks;

[0020] Step 2.14: According to the optimal strategy, the network can decide whether the user needs to switch in any state of the user, so as to improve the imbalance of network load and reduce the network load in a more refined way.

[0021] Furthermore, the first step of initializing the reinforcement learning parameters includes:

[0022] Learning rate lr, discount factor γ, number of neural network layers h, number of neurons in each layer p, number of interactions per round steps_per_episode, batch size batch_size, size of experience replay pool D replay_size;

[0023] Define n policy networks for the agent set N, initialize the weights of each policy network, and the policy network weight set of all agents is φ = {φ1, φ2, ..., φ i ,...,φ n}, where φ i is the policy network weight of the ith agent in the agent set N, then the policy network set of all agents is in is the strategy network of the ith agent in the agent set N; define n Q networks for the agent set N, initialize the weights of each Q network, and the weight set of all agents’ Q networks is θ={θ1,θ2,...,θ i ,...,θ n}, where θ i is the Q network weight of the i-th agent in the agent set N, then the Q network set of all agents is in is the i-th agent Q network in the agent set N.

[0024] Furthermore, in step 2.1, all target strategy network weights and all target Q network weights are initialized, specifically:

[0025] The target strategy network weight set of all agents in the agent set N is φ targ ={φ targ,1 ,φ targ ,2,...,φ targ ,i,...,φ targ,n}, where φ targ,i is the target strategy network weight of the ith agent, and the target strategy network set of all agents in the agent set N is in is the target strategy network of the i-th agent; the target Q network weight set of all agents in the agent set N is θ targ ={θ targ,1 ,θ targ ,2,...,θ targ ,i,...,θ targ,n}, where θ targ,iis the target Q network weight of the i-th agent, and the target Q network set of all agents in the agent set N is in is the target Q network of the i-th agent;

[0026] Initialize all target strategy network weights, i.e. Initialize all target Q network weights, that is

[0027] Furthermore, the reward value r(t) in step 2.5 is calculated as follows:

[0028]

[0029] Among them, Load l (t) is the load status of the lth base station in the communication network environment, and m is the number of base stations.

[0030] Furthermore, the Q network weights, policy network weights, target Q network weights, and target policy network weights of each agent in the agent set N are updated using the gradient descent method described in step 2.9, specifically:

[0031] First, randomly sample batch_size samples in the experience replay pool D. The sample set is represents the jth sample, where It refers to the state of the communication network environment at the training step corresponding to when the jth sample is stored in the experience replay pool. are the actions of all agents in this training step, is the reward for all agents in this training step, It is the state of the communication network environment corresponding to the next step of this training step;

[0032] Calculate the target value of the Q network for each sample in the sample set B. The jth sample The target value formula is:

[0033]

[0034] Among them, r i j yes The reward value corresponding to the i-th agent in , γ is the discount factor, which is generally in the range of [0,1). The specific value is determined by the operator according to the actual situation. is the target Q network of the ith agent at t training steps; a k ′ represents the target strategy network of the k-th agent when the input is The output action when in is the target policy network of the kth agent at t training steps, yes The state corresponding to the kth agent in ;

[0035] Update the Q network weight θ of the i-th agent as follows: i (t) are as follows:

[0036]

[0037]

[0038] Where |B| represents the size of the sample set B, where |B| = batch_size, It means to find the loss function L(θ of the i-th agent Q network i ) about θ i The gradient of

[0039] Update the policy network weights φ of the i-th agent using gradient descent i (t), updated as follows:

[0040]

[0041]

[0042] in yes The state corresponding to the i-th agent in i The input of the i-th agent policy network is The output action is is the i-th agent Q network in the t+1 training step, yes The action corresponding to the qth agent in , q=1,2,…,n and q≠i.

[0043] Then the weight θ of the target Q network of the i-th agent is targ,i (t) and the weights φ of the target policy network targ,i (t) Update as follows:

[0044] θ targ,i (t+1)=(1-τ)θ targ,i (t)+τθ i (t+1)

[0045] φ targ,i (t+1)=(1-τ)φ targ,i (t)+τφ i (t+1)

[0046] Wherein, τ is the target network update coefficient, which takes a value between [0,1]. The specific value is determined by the operator according to the actual situation.

[0047] Furthermore, the cumulative reward value r described in step 2.10 * (t) is calculated as follows:

[0048]

[0049] Beneficial effects: Compared with the prior art, the present invention has the following beneficial effects:

[0050] The load balancing method based on multi-agent reinforcement learning of the present invention regards the load balancing problem as a multi-agent cooperation problem, models it as a random game process, and uses a multi-agent deep deterministic policy gradient to solve it. Through the interaction between the operation and maintenance management system and the network environment, it continuously learns whether the user needs to switch to improve the network load imbalance. Compared with the existing load balancing method, the method of the present invention does not only decide on the switching user set based on the user's service base station RSRP, but reasonably combines the switching users with the optimization goal of reducing the network load, which can improve the network load imbalance and reduce the network load in a more refined way. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of the present invention. Specific embodiments

[0052] In order to explain the technical solution of the present invention in detail, further elaboration is given below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0053] The load balancing method based on multi-agent reinforcement learning of the present invention first abstracts the load balancing problem into a random game process, and then uses a model-free multi-agent reinforcement learning method - Multi-agent Deep Deterministic Policy Gradient (MADDPG) method to solve it. Through the interaction between the network management layer of the Operation Administration and Maintenance (OAM) system and the network environment, the multi-agent reinforcement learning strategy network is continuously trained to obtain the switching actions of each user under each network load state.

[0054] Stochastic games include five elements, which can be expressed as<N,S,A,P,R> . Where N is the set of agents, S is the state space (State Space), which is used to describe the state set of the network environment, A is the action space (Action Space), which represents the action set of the agent, P represents the state transition probability, and R is the reward function (Reward Function), which represents the feedback of the environment to the selected action. In the present invention, the four elements of the agent set N, the state space S, the action space A and the reward function R are specifically defined as follows:

[0055] 1. Agent set: The agent is the object participating in the random game. In the present invention, the agent realizes load balancing by deciding user switching. Therefore, each agent corresponds to a candidate switching user. The user who meets the A4 event in the overloaded base station is defined as the candidate switching user. The user who meets the A4 event is called the edge user, and the user who does not meet the A4 event is called the non-edge user. Once the non-edge user switches the base station, the user communication link will fail or the neighboring base station will be overloaded. Therefore, the present invention first determines the agent through the A4 event. The agent set is denoted as N = {u1,u2,...,u i ,...,u n}, the set N consists of n agents, corresponding to n users who satisfy event A4, where u i represents the i-th agent.

[0056] 2. State space: In the present invention, the state of each agent is defined as the service base station load state and the service base station RSRP of the corresponding user. The service base station load of the ith agent corresponding to the user in the tth training step is ρ i (t), the serving base station RSRP is RSRP i (t), then its state is s i (t)=(ρ i (t),RSRP i (t)). The state of the entire communication network environment is composed of the states of all agents, denoted as

[0057] s total (t)=(s1(t),s2(t),...,s i (t),...,s n (t)). (1)

[0058] 3. Action space: The present invention defines two actions for each agent, namely, staying in the serving base station and switching to the neighboring base station with the largest RSRP. The action of the ith agent in training step t is denoted as a i(t)∈{A0,A1}, where action A0 means that the user corresponding to the agent remains in the serving base station, and action A1 means that the user corresponding to the agent switches to the neighboring base station with the largest RSRP. The actions of the entire communication network environment are composed of the actions of all agents, recorded as

[0059] a total (t)=(a1(t),a2(t),...,a i (t),...,a n (t)). (2)

[0060] 4. Reward function: Multi-agent reinforcement learning reaches the Nash equilibrium state through continuous training. The strategy at this time is the optimal strategy to maximize the cumulative discounted rewards of all agents. The present invention models the load balancing problem as a multi-agent cooperation problem, that is, a user cooperation problem, so each agent uses the same reward function. The reward function is defined as the inverse value of the maximum load in the network, with the purpose of reducing the maximum load in the network.

[0061] like Figure 1 As shown, the following steps are included:

[0062] Step 1: Collect network information and initialize parameters. This step includes the following process:

[0063] Step 1.1, collect network information. The network information includes the set of base stations participating in load balancing M (7 base stations in this example), and the set of intelligent agents N (20 intelligent agents in this example) corresponding to users in the overloaded base stations that meet the A4 event. These values ​​can be determined by the operator according to the actual situation;

[0064] Step 1.2, initialize reinforcement learning parameters. Initialized reinforcement learning parameters include learning rate lr = 0.001, discount factor γ = 0.5, number of neural network layers h = 2, number of neurons in each layer p = 64, number of interactions per round steps_per_episode = 800, batch size batch_size = 512, and size of experience replay pool D replay_size = 8000. Define n policy networks for the agent set N, initialize the weights of each policy network (in this example, use kaiming uniform distribution initialization), and record the policy network weight set of all agents in the set N as φ = {φ1, φ2, ..., φ i ,...,φ n},φ i That is, the strategy network weight of the ith agent in set N, then the strategy network set of all agents in set N is That is, the strategy network of the ith agent in the set N. For the agent set N, n Q networks are defined, and the weights of each Q network are initialized (in this example, Kaiming uniform distribution is used for initialization). The weight set of Q networks of all agents in the set N is θ = {θ1, θ2, ..., θ i ,...,θ n}, θ i That is, the Q network weight of the i-th agent in the set N, then the Q network set of all agents in the set N is That is, the i-th intelligent agent Q network in the set N;

[0065] Step 2: Perform MADDPG training to learn the optimal user switching strategy. This step includes the following process:

[0066] Step 2.1: Let t represent the current training step number, and the target strategy network weight set of all agents in the set N is denoted as φ targ ={φ targ,1 ,φ targ,2 ,...,φ targ,i ,...,φ targ,n},φ targ,i That is, the target strategy network weight of the i-th agent in set N. The target strategy network set of all agents in set N is That is, the target strategy network of the i-th agent in the set N, and the target Q network weight set of all agents in the set N is recorded as θ targ ={θ targ,1 ,θ targ,2 ,...,θ targ,i ,...,θ targ,n}, θ targ,i That is, the target Q network weight of the i-th agent in set N. The target Q network set of all agents in set N is That is, the target Q network of the i-th agent in the set N;

[0067] Initialize all target strategy network weights, i.e. Initialize all target Q network weights, that is Initialize t=1;

[0068] Step 2.2: Collect the load status and RSRP of the service base stations corresponding to the users of all agents in the set N. The state of the i-th agent in the t training step is s i (t)=(ρ i (t),RSRP i (t)), the state of the entire communication network environment s total(t)=(s1(t),s2(t),...,s i (t),...,s n (t));

[0069] Step 2.3: Generate current actions for all agents in the set N. The action of the i-th agent is where ε(t) is the Ornstein-Uhlenbeck (OU) random process, and the action a of the entire communication network environment total (t)=(a1(t),a2(t),...,a i (t),...,a n (t));

[0070] Step 2.4: Execute action a of all agents in set N total (t)=(a1(t),a2(t),...,a i (t),...,a n (t)), that is, when a i (t) = A0, the user corresponding to the i-th agent does not perform a switching operation and remains in the current serving base station. i (t) = A1, the user corresponding to the i-th agent switches to the neighboring base station with the largest RSRP. If the switching causes the neighboring base station to be overloaded, the user switches back to the serving base station. If the switching does not cause the neighboring base station to be overloaded, the user stays at the neighboring base station.

[0071] Step 2.5: Collect the load states of m base stations in the communication network environment and record them as Load1(t), Load2(t), ..., Load m (t), the reward value is calculated as

[0072]

[0073] Among them, Load l (t) is the load status of the lth base station in the communication network environment, and m is the number of base stations.

[0074] Then the reward values ​​of all agents in the set N are r1(t)=r2(t)=...=r n (t) = r(t), where r i (t) represents the reward of the ith agent. For the convenience of explanation, the rewards of all agents in the entire communication network environment are recorded as r total (t)=(r1(t),r2(t),...,r i (t),...,r n (t));

[0075] Step 2.6: Collect all agents in set N to perform action a total (t) After the corresponding user's service base station load status and RSRP, the state of the agent in the next step (i.e., step t+1) is updated. The state of the i-th agent in step t+1 is s i (t+1)=(ρ i (t+1),RSRP i (t+1)), the next state of the entire communication network environment is s total (t+1)=(s1(t+1),s2(t+1),...,s i (t+1),...,s n (t+1));

[0076] Step 2.7: Set the current sample (s total (t),a total (t),r total (t),s total (t+1)) is stored in the experience replay pool D, which is expanded into (s1(t), s2(t), ..., s n (t),a1(t),a2(t),...,a n (t),r1(t),r2(t),...,r n (t),s1(t+1),s2(t+1),...,s n (t+1));

[0077] Step 2.8: If the number of samples stored in the current experience replay pool D is greater than the batch size batch_size, proceed to step 2.9 to complete the Q network and policy network weight update process for all agents in the set N. If the number of samples stored in the current experience replay pool D is less than the batch size batch_size, proceed to step 2.10.

[0078] Step 2.9: Update the Q network and policy network weights of each agent in the set N as follows. Take the i-th agent as an example: First, randomly sample batch_size samples in the experience replay pool D. The sample set is recorded as represents the jth sample, where It refers to the communication network environment state of the corresponding training step when the jth sample is stored in the experience replay pool. are the actions of all agents in this training step, is the reward for all agents in this training step, is the communication network environment state corresponding to the next step of this training step; the target value of the Q network is calculated for each sample in the sample set B, such as for the jth sample The target value formula is:

[0079]

[0080] Among them, r i j yes The reward value corresponding to the i-th agent in , γ is the discount factor, which is generally in the range of [0,1). The specific value is determined by the operator according to the actual situation. In this example, γ = 0.5. is the target Q network of the ith agent at t training steps, is the target policy network of the kth agent at t training steps, yes The state corresponding to the kth agent in a k ′ represents the target strategy network of the k-th agent when the input is The output action when Then the loss function of the i-th agent Q network is as follows:

[0081]

[0082] Where |B| represents the size of the sample set B, where |B| = batch_size = 512, and updates the network weight θ of the i-th agent Q i (t) are as follows:

[0083]

[0084] in It means to find the loss function L(θ i ) about θ i gradient.

[0085] Then use the gradient descent method to update the weight φ of the i-th agent strategy network i (t), updated as follows:

[0086]

[0087]

[0088] in yes The state corresponding to the i-th agent in i The input of the i-th agent policy network is The output action is yes The action corresponding to the qth agent in the network, q = 1, 2, ..., n and q ≠ i. Then the weight θ of the target Q network of the i-th agent targ,i(t) and the weights φ of the target policy network targ,i (t) is updated as follows:

[0089] θ targ,i (t+1)=(1-τ)θ targ,i (t)+τθ i (t+1)

[0090] φ targ,i (t+1)=(1-τ)φ targ,i (t)+τφ i (t+1) (8)

[0091] Where τ is the target network refinement coefficient, which takes a value in [0,1] and is close to 0. In this example, τ = 0.01. The specific value is determined by the operator based on actual conditions.

[0092] Step 2.10, if t% steps_per_episode = 0, where % represents the modulus value, calculate the cumulative reward for this round of training as Initialize the environment and start a new round of training;

[0093] Step 2.11, let t = t + 1;

[0094] Step 2.12: Repeat steps 2.2 to 2.11 until the cumulative reward value of each round converges to the maximum value;

[0095] Step 2.13, exit the iteration and obtain n optimal strategy networks;

[0096] Step 2.14: According to the optimal strategy, the network can decide whether the user needs to switch in any state of the user, so as to improve the imbalance of network load and reduce the network load in a more refined way.

[0097] The specific embodiments described herein are merely examples of the spirit of the present invention. Those skilled in the art may make various modifications or additions to the specific embodiments described or replace them in similar ways, but they will not deviate from the spirit of the present invention or exceed the scope defined by the appended claims.

Claims

1. A load balancing method based on multi-agent reinforcement learning, It is characterized in that The following steps are involved: Step 1: Collect network information, including the set of base stations participating in load balancing M, and generate a set of intelligent agents N corresponding to users according to the users who meet the A4 event in the overloaded base stations. 1 ,u 2 ,...,u i ,...,u n }, where u i represents the i-th agent, n is the number of agents; initialize the reinforcement learning parameters; Step 2: Conduct multi-agent deep deterministic policy gradient training to learn the optimal user switching strategy, including the following steps: Step 2.1, use t to represent the current training step number, initialize t = 1; initialize all target strategy network weights and all target Q network weights; Step 2.2: Collect the load status and reference signal received power of the service base station of all agents corresponding to the users in the agent set N. The state of the i-th agent in the current training step t is s i (t)=(ρ i (t),RSRP i (t)), where ρ i (t), RSRP i (t) are the service base station load and reference signal receiving power of the user corresponding to the i-th agent respectively; The state of the entire communication network environment at the current t training step consists of the states of all agents: s total (t)=(s 1 (t),s 2 (t),...,s i (t),...,s n (t)); Step 2.3: Select the current action for all agents in the agent set N according to the policy network. The action of the i-th agent in the current training step t is where ε(t) is the Ornstein-Uhlenbeck (OU) random process, is the strategy network of the ith agent; the actions of the entire communication network environment at the current t training step are composed of the actions of all agents: a total (t)=(a 1 (t),a 2 (t),...,a i (t),...,a n (t)); Step 2.4: Execute action a of all agents total (t)=(a 1 (t),a 2 (t),...,a i (t),...,a n (t)), a i (t)∈{A 0 ,A 1 }, when a i (t) = A 0 , the user corresponding to the i-th agent does not switch and remains in the current service base station. i (t) = A 1 , the user corresponding to the i-th agent switches to the neighboring base station with the largest reference signal receiving power. If the switching causes the neighboring base station to be overloaded, the user switches back to the serving base station. If the switching does not cause the neighboring base station to be overloaded, the user stays in the neighboring base station; Step 2.5: The reward for all agents in the entire communication network environment is r total (t)=(r 1 (t),r 2 (t),...,r i (t),...,r n (t)), where r i (t) represents the reward of the i-th agent in the current t training steps, and the rewards of all agents satisfy r 1 (t)=...=r i (t)=...=r n (t) = r(t), r(t) is the reward value; Step 2.6: Collect all agents in the agent set N to perform action a total (t) After that, the load state of the corresponding user's service base station and the reference signal received power are updated to update the state of the agent in the next step, that is, step t+1. The state of the i-th agent in step t+1 is s i (t+1)=(ρ i (t+1),RSRP i (t+1)), the next state of the entire communication network environment is s total (t+1)=(s 1 (t+1),s 2 (t+1),...,s i (t+1),...,s n (t+1)); Step 2.7: Set the current sample (s total (t),a total (t),r total (t),s total (t+1)) is stored in the experience replay pool D; Step 2.8: If the number of samples stored in the current experience replay pool D is greater than the batch size batch_size, proceed to step 2.9 to complete the weight update process of the target Q network and target strategy network of all agents in the agent set N; if the number of samples stored in the current experience replay pool D is less than the batch size batch_size, proceed to step 2.10; Step 2.9, using the gradient descent method to update the Q network weights, policy network weights, target Q network weights and target policy network weights of each agent in the agent set N; Step 2.10, if t% steps_per_episode = 0, where % represents the modulus value, steps_per_episode is the number of interactions per round, calculate the cumulative reward value of this round of training, initialize the environment, and start a new round of training; Step 2.11, let t = t + 1; Step 2.12: Repeat steps 2.2 to 2.11 until the cumulative reward value of each round converges to the maximum value; Step 2.13, exit the iteration and obtain n optimal strategy networks; Step 2.14: According to the optimal strategy, the network can decide whether the user needs to switch in any state of the user; in, The first step is to initialize the reinforcement learning parameters, including: Learning rate lr, discount factor γ, number of neural network layers h, number of neurons in each layer p, number of interactions per round steps_per_episode, batch size batch_size, size of experience replay pool D replay_size; Define n policy networks for the agent set N, initialize the weights of each policy network, and the policy network weight set of all agents is φ = {φ 1 ,φ 2 ,...,φ i ,...,φ n }, where φ i is the policy network weight of the ith agent in the agent set N, then the policy network set of all agents is in is the strategy network of the ith agent in the agent set N; define n Q networks for the agent set N, initialize the weights of each Q network, and the weight set of all agents’ Q networks is θ={θ 1 ,θ 2 ,...,θ i ,...,θ n }, where θ i is the Q network weight of the i-th agent in the agent set N, then the Q network set of all agents is in is the i-th agent Q network in the agent set N; Step 2.1 initializes all target strategy network weights and all target Q network weights, specifically: The target strategy network weight set of all agents in the agent set N is φ targ ={φ targ,1 ,φ targ,2 ,...,φ targ,i ,...,φ targ,n }, where φ targ,i is the target strategy network weight of the ith agent, and the target strategy network set of all agents in the agent set N is in is the target strategy network of the i-th agent; the target Q network weight set of all agents in the agent set N is θ targ ={θ targ,1 ,θ targ,2 ,...,θ targ,i ,...,θ targ,n }, where θ targ,i is the target Q network weight of the i-th agent, and the target Q network set of all agents in the agent set N is in is the target Q network of the i-th agent; Initialize all target strategy network weights, i.e. Initialize all target Q network weights, that is Step 2.9 uses the gradient descent method to update the Q network weights, policy network weights, target Q network weights, and target policy network weights of each agent in the agent set N, specifically: First, randomly sample batch_size samples in the experience replay pool D. The sample set is represents the jth sample, where It refers to the state of the communication network environment at the training step corresponding to when the jth sample is stored in the experience replay pool. are the actions of all agents in this training step, is the reward for all agents in this training step, It is the state of the communication network environment corresponding to the next step of this training step; Calculate the target value of the Q network for each sample in the sample set B. The jth sample The target value formula is: Among them, r i j yes The reward value corresponding to the i-th agent in , γ is the discount factor, which takes a value in [0,1), is the target Q network of the ith agent at the tth training step; a′ k It means that the k-th agent target policy network has the input The output action when in is the target policy network of the kth agent at t training steps, yes The state corresponding to the kth agent in ; Update the Q network weight θ of the i-th agent as follows: i (t) are as follows: Where |B| represents the size of the sample set B, where |B| = batch_size, It means to find the loss function L(θ of the i-th agent Q network i ) about θ i The gradient of Update the policy network weights φ of the i-th agent using gradient descent i (t), updated as follows: in yes The state corresponding to the i-th agent in i The input of the i-th agent policy network is The output action is is the i-th agent Q network in the t+1 training step, yes The action corresponding to the qth agent in , q=1,2,...,n and q≠i; Then the weight θ of the target Q network of the i-th agent is targ,i (t) and the weights φ of the target policy network targ,i (t) Update as follows: i targ,i (t+1)=(1-τ)θ targ,i (t)+τθ i (t+1) f targ,i (t+1)=(1-τ)φ targ,i (t)+tφ i (t+1) Among them, τ is the target network update coefficient, which ranges from [0,1].

2. According to the load balancing method based on multi-agent reinforcement learning according to claim 1, It is characterized in that The reward value r(t) described in step 2.5 is calculated as follows: Among them, Load l (t) is the load status of the lth base station in the communication network environment, and m is the number of base stations.

3. The load balancing method based on multi-agent reinforcement learning according to claim 1, It is characterized in that The accumulated reward value r described in step 2.10 * (t) is calculated as follows:

Images