Control method for efficient utilization of energy in green station area for micro-grid group

Abstract

The invention discloses a control method for efficient utilization of energy in a green station area for a micro-grid group. The method comprises the following steps: constructing an active and reactive power coordinated optimization scheduling model of the micro-grid group in the green station area, and taking the minimum network loss in a scheduling period as an objective function of the model; taking constraints of power flow, photovoltaic output, a static var compensator, branch transmission power, voltage amplitude and energy storage, and limits of action times of an on-load tap-changer and a circuit breaker as constraint conditions of the model; the method comprises the following steps: defining an agent reward function by determining an agent observation space and an agent action space, and describing an active and reactive coordinated optimization scheduling model of a microgrid group in a green radio station area as a Markov game problem; and an improved distributed multi-agent soft actor-commentator algorithm and a migration reinforcement learning method are used to train the green station area microgrid group active and reactive coordination optimization scheduling model converted into the Markov game problem. According to the method, the intelligent agent can be trained to learn the optimal scheduling strategy in the process of interaction with the environment, and the dependence on the model is overcome.

Description

Technical Field

[0001] The present invention relates to a control method for efficient utilization of energy in a green station area, and in particular to a control method for efficient utilization of energy in a green station area for a microgrid group. Background Art

[0002] In the microgrid, the photovoltaic and energy storage microgrid is a small local area network, which has a connection point with the large power grid and can choose to operate independently or connect to the large power grid according to external conditions. The photovoltaic and energy storage microgrid itself is composed of distributed photovoltaics, energy storage devices, and local loads. It has the advantages of cleanliness, intelligence, autonomy, reliability, flexibility, and interaction, and can adjust its own operation strategy according to the external environment of the rural power grid.

[0003] From the current stage of development of photovoltaic and energy storage microgrids, there are still some problems. First, due to the low penetration rate of photovoltaics, when photovoltaic and energy storage microgrids generate electricity, it is very easy to encounter difficulties in absorption and grid connection, and the utilization rate of light energy is low. Second, photovoltaic power generation is affected by other uncontrollable factors such as weather, which makes it highly unstable, intermittent and volatile, and it will impact the large power grid when it is connected. Third, the behavior of the user side has an impact on the photovoltaic and energy storage microgrid. At the same time, the peak-to-valley difference of the grid load is large, and the utilization efficiency of the grid facilities is low. In summary, the study of the complementary operation strategy of photovoltaic and energy storage in rural microgrids can better call on energy to improve the stability of the grid. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a control method for efficient utilization of green station energy in microgrid groups, which can enable a trainable intelligent agent to learn the optimal scheduling strategy in the process of interacting with the environment and overcome the dependence of traditional optimization methods on models.

[0005] The technical solution adopted by the present invention is: a control method for efficient utilization of green station area energy in a microgrid group, comprising the following steps:

[0006] 1) Construct an active and reactive coordinated optimization dispatch model for microgrids in green station areas, take the minimum network loss within the dispatch period as the objective function of the model, and take power flow constraints, photovoltaic output constraints, static VAR compensator constraints, branch transmission power constraints, voltage amplitude constraints, energy storage constraints, and the number of operations of on-load tap changers and circuit breakers as constraints of the model;

[0007] 2) By determining the agent observation space and the agent action space, and defining the agent reward function, the active and reactive coordinated optimization scheduling model of the green station area microgrid group is described as a Markov game problem;

[0008] 3) The improved distributed multi-agent soft actor-critic algorithm and transfer reinforcement learning method are used to train the active and reactive coordinated optimization scheduling model of the green station area microgrid group converted into a Markov game problem.

[0009] The present invention provides a control method for efficient utilization of green station area energy in a microgrid group, which can train the intelligent agent to learn the optimal scheduling strategy in the process of interacting with the environment and overcome the dependence of traditional optimization methods on models. Compared with the existing multi-agent deep reinforcement learning method based on the "centralized training" framework, this method does not need to collect global information, can better protect the privacy of each sub-microgrid information, reduce communication pressure, and show robustness to communication failures in both training and execution stages. Considering that there are topological changes in microgrid groups and the generalization ability of reinforcement learning models is poor, a transfer reinforcement learning method is proposed to use the existing intelligent agent knowledge to efficiently train a group of intelligent agents for each topology. Finally, the effectiveness of the proposed method in solving the problem of coordinated optimization and scheduling of distributed active and reactive power in microgrid groups is verified. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is the IEEE33 node system topology diagram of the present invention;

[0011] Figure 2a It is a comparison chart of the global reward results of each round of training of DMASAC of the present invention and centralized learning;

[0012] Figure 2b It is a comparison chart of the global reward results of each round of training of DMASAC of the present invention and centralized learning;

[0013] Figure 3 It is a schematic diagram of the active output of energy storage connected to the node 18 of the present invention;

[0014] Figure 4 It is a schematic diagram of the reactive power output of the node SVC of the present invention. DETAILED DESCRIPTION

[0015] A control method for efficient utilization of green station area energy in a microgrid group of the present invention is described in detail below in conjunction with embodiments and drawings.

[0016] according to Figure 1 The IEEE 33 node system topology diagram shown in the figure, a control method for efficient utilization of green station area energy of a microgrid group of the present invention comprises the following steps:

[0017] 1) Construct the active and reactive coordinated optimization dispatch model of the microgrid group in the green station area, take the minimum network loss within the dispatch period as the objective function of the model, and take the power flow constraint, photovoltaic output constraint, static VAR compensator (SVC) constraint, branch transmission power constraint, voltage amplitude constraint, energy storage constraint, on-load tap changer (OLTC) and circuit breaker (CB) action number limit as the constraint conditions of the model;

[0018] The objective function is expressed as:

[0019]

[0020] Where n is the number of microgrids; T is the total number of dispatching periods in a day; R ij is the line resistance of branch ij; i 2 ij is the square of the current in branch ij; i is the set of branches that form a branch with node i in the sub-microgrid;

[0021] The constraints described are:

[0022] (1) The power flow constraint is:

[0023]

[0024] In the formula, and are the active and reactive powers injected into node i respectively; and are the active and reactive loads of node i respectively; and are the sum of the branch active powers flowing from the adjacent node m into the node i and flowing from the node i to the adjacent node j respectively; and are the sum of the branch reactive powers flowing from the adjacent node m into the node i and flowing from the node i to the adjacent node j; V i and V j are the voltages of nodes i and j respectively; P ij and Q ij are the active and reactive powers of the branch flowing from node i to node j respectively; θ ij is the phase difference between branches ij; G ij is the conductance of branch ij; B ij is the susceptance of branch ij.

[0025] (2) Photovoltaic output constraints

[0026] In order to improve economic efficiency, the photovoltaic system is required to generate electricity at maximum power, and only reactive power can be adjusted, that is,

[0027]

[0028] In the formula, and are the active and reactive powers generated by the g-th PV inverter at time t respectively; is the active power of the g-th PV inverter working in the maximum power point tracking mode at time t; pv A photovoltaic inverter assembly; and are the upper and lower limits of reactive power of the g-th photovoltaic inverter respectively; is the apparent power of the g-th PV inverter;

[0029] (3) Static VAR Compensator (SVC) Constraints:

[0030]

[0031] In the formula, is the reactive power generated by the cth static VAR compensator at time t; and are the upper and lower limits of reactive power of the cth static VAR compensator respectively; svc is a static VAR compensator assembly.

[0032] (4) Branch transmission power constraints

[0033]

[0034] Where P ij，t is the active power transmitted by branch ij at time t; is the maximum limit of active power transmitted by branch ij; ε is the branch set.

[0035] (5) Voltage amplitude constraint

[0036]

[0037] In the formula, and V j are the upper and lower limits of the voltage amplitude at node j, respectively; V j，t is the voltage amplitude of node j at time t; N is the number of nodes;

[0038] (6) Energy storage constraints:

[0039]

[0040] In the formula, is the power of the oth energy storage at time t; and are the discharge and charge powers of the oth energy storage at time t respectively; is the charging and discharging efficiency of the oth energy storage; is the upper limit of the charging and discharging power of the oth energy storage, where discharging is positive and charging is negative; DESS For energy storage collection; is the initial capacity of the oth energy storage; and are the upper and lower limits of the capacity of the oth energy storage, respectively; Δt is the duration of a scheduling period;

[0041] (7) Limitation on the number of operations of on-load tap changer (OLTC) and circuit breaker (CB):

[0042]

[0043] In the formula, is the number of operations of the lth on-load tapchanger at time t; is the upper limit of the number of operations of the lth on-load tapchanger; OLTC Assemble on-load tap-changers; is the number of operations of the kth circuit breaker at time t; is the upper limit of the number of operations of the kth circuit breaker; CB for the circuit breaker collection; is a 0-1 variable indicating whether the gear position of the lth on-load tapchanger has changed at time t′; is the gear position of the lth on-load tapchanger at time t′; is a 0-1 variable indicating whether the circuit breaker position has changed at time t′; is the gear position of the kth circuit breaker at time t′.

[0044] 2) By determining the agent observation space and the agent action space, and defining the agent reward function, the active and reactive coordinated optimization dispatch model of the green station area microgrid group is described as a Markov game problem;

[0045] The MADRL method is to let the agent interact with the environment for multiple rounds and learn the optimal strategy through "trial and error". Each round of interaction starts from the initial state of the environment. Each agent will first observe the local environment state, then select actions based on the observations according to its own strategy, and make the environment enter the next state, and then obtain the reward given by the environment, so as to optimize its own strategy. Repeat the above process until the environment state reaches the terminal state to complete a round of training. By gradually optimizing its own strategy during the interaction process, the cumulative reward obtained by the agent will gradually increase until it stabilizes.

[0046] The above interaction process is often quantitatively described by Markov game, which includes 6 elements, using the tuple<S,Oi,Ai,P,R,γ> In which S represents the set of all states of the public environment, and the state at time t is st ∈S;O i is the local observation set of agent i, and the local observation of agent i at time t is o i , t∈O i , the observations of n agents are combined to form the joint observation O = O1O2…O n ; A i is the action set of agent i, and the local action of agent i at time t is a i , t∈A i , the actions of n agents are combined to form a joint action A = A1A2…A n ; P is the state transition probability, indicating that in state s t Perform joint action a t Then the environment enters the next state s t +1 probability; R is the reward function, indicating that in state s t The next group of agents performs a joint action t After that, the feedback reward given by the environment satisfies s t a t →r t , r t ∈R; γ is the discount factor.

[0047] The key to solving specific problems with MADRL is to design the above Markov game. In the context of coordinated optimization of active and reactive power dispatch of microgrid groups, the specific settings of observation, action space and reward function of each intelligent agent are given as follows.

[0048] (1) Agent Observation Space

[0049] The observation space of the agent is the agent's perception of the local environment. Here, the local observation of the agent ag is set as the local node load power, renewable energy generation power, energy storage power, OLTC and CB gears and time t at each time step t. In order to consider the action number constraints of discrete devices, the number of times the on-load tap changer and circuit breaker have been operated is added to the observation space, as shown in the following formula:

[0050]

[0051] In the formula, and are the active and reactive loads in the intelligent agent ag at time t respectively; is the active power generated by the photovoltaic system in the intelligent agent ag at time t; and are the gear positions of the on-load tap-changer and circuit breaker in the intelligent agent ag at time t respectively; and are the number of times the on-load tap changer and circuit breaker in the intelligent agent ag have been operated at time t; is the power of the energy storage device in the intelligent agent ag at time t;

[0052] (2) Agent Action Space

[0053] Since there are both discrete and continuous adjustment devices locally, the action space of the agent includes discrete action space and continuous action space, that is,

[0054]

[0055] In the formula, is the discrete action space of agent ag at time t; is the continuous action space of agent ag at time t; and are the reactive power generated by the photovoltaic inverter and static VAR compensator in the intelligent agent ag at time t;

[0056] (3) Agent Reward Function

[0057] In order to ensure that the decision results of the intelligent agent meet the requirements of the stability of the microgrid system operation and to minimize the network loss within the scheduling period, the negative value of the network loss is used as the reward value of the intelligent agent. Use node voltage and branch power exceeding the limit as penalty Add to the reward function; the reward r of each agent per period i,t as follows:

[0058]

[0059] In the formula, is the penalty coefficient for exceeding the limit; and are the voltage over-limit function and branch power over-limit function at node i at time t respectively; N ag is the number of nodes of agent ag; represents max(x, 0), x represents the number of cutoffs; R ij is the line resistance of branch ij; i ij is the square of the current in branch ij; and V j are the upper and lower limits of the voltage amplitude at node j, respectively; V j，t is the voltage amplitude of node j at time t; N is the number of nodes; P ij，t is the active power transmitted by branch ij at time t; is the maximum limit of active power transmitted by branch ij.

[0060] It should be noted that since there is no coordination center, the total network loss and total over-limit penalty of the system cannot be directly obtained during strategy optimization. The above formula can be decomposed into the local network loss and over-limit penalty of each sub-microgrid and then accumulated, so it can be obtained through communication between intelligent agents. Since it does not involve the privacy information such as the local source and load output of each sub-microgrid, equipment scheduling actions, network parameters, and operating costs, it does not infringe privacy.

[0061] The number of actions of discrete devices is limited. Forcibly adding a penalty after the number of actions exceeds the limit can easily lead to divergence of training results. Considering that the action set of discrete agents is limited, Mask is used here to constrain the actions of discrete agents.

[0062] 3) The improved distributed multi-agent softactor critic (DMASAC) algorithm and transfer reinforcement learning method are used to train the active and reactive coordination optimization dispatch model of the green station area microgrid group converted into a Markov game problem;

[0063] The MASAC algorithm is only applicable to continuous action spaces. In order to adapt it to discrete-continuous mixed action spaces, it needs to be improved. An output layer is added to the Critic network of each agent, with a dimension of where h dim is the number of neurons in the hidden layer, is the maximum action gear of the local discrete device, which is used to output the Q value corresponding to each discrete gear. Correspondingly, the Actor network also adds an output layer with the same dimension of After the output is processed by the Softmax function, the probability corresponding to each discrete gear is obtained. This step specifically includes:

[0064] The training goal of the improved distributed multi-agent soft actor-critic algorithm is to find the optimal strategy π* for the agent ag i , as shown below;

[0065]

[0066] In the formula, E{·} represents the expected function; (s t , a t )~ρ π represents the state-action trajectory distribution induced by strategy π; H(π ag (·|o ag，t )) is the intelligent agent ag in the observation amount o ag，t According to the strategy π ag The entropy of taking an action; α is the temperature coefficient used to adjust the importance of action entropy relative to the reward;

[0067] The above optimization process is carried out during the interaction between the agent and the environment, which can be simply described as the iterative update of the strategy. 0 At the beginning, the Actor network selects actions through strategies, and the Critic network evaluates the value of the actions selected by the Actor and guides the Actor to update the strategy parameters according to the maximum evaluation value to obtain the new strategy π 1 , and then repeat the above steps to gradually improve the strategy.

[0068] Scheduling control under multiple topologies requires training a set of corresponding agents for each network topology. To improve the training efficiency, a transfer learning mechanism is introduced based on the improved distributed multi-agent soft actor-critic algorithm.

[0069] First, the parameters of the Critic network of the trained agent are transferred to the new agent, and the action value function of the new agent is modified to ensure that the Critic network of the new agent can accurately evaluate the quality of the action in the initial stage. Then, the Actor network of the trained agent is used as the teacher model, and a "gap" with the teacher model is added to the loss function of the Actor network of the new agent. It is expected that by narrowing the "gap", the teacher model can be used to guide the training of the new agent. Describing the "gap" as the difference between the values ​​of the two actions obtained by sampling actions according to the teacher strategy and the current strategy of the new agent in the same state, it not only better reflects the use of the teacher model to guide the new agent to learn in the direction of higher action value, but also applies to both continuous and discrete action spaces, and can be easily extended to other algorithms based on the Actor-Critic framework. The details are as follows:

[0070]

[0071] In the formula, Q S and Q T are the action value functions of the existing agent and the new agent respectively; Modified loss function for the new agent Actor network; is the supplementary term of the new agent Actor network loss function; L Actor is the original loss function of the new agent Actor network; O is the observation volume of the agent; a T and a S are the actions given by the strategies of the new agent and the existing agent respectively; β and Δβ are the weight coefficient and the decay value of the agent respectively, which are used to prevent negative transfer when the teacher model does not match the new task.

[0072] See also Figure 2a , Figure 2bTo verify the convergence of the method of the present invention, the historical data of load and photovoltaic output are used as training samples. The DMASAC algorithm and the MASAC algorithm of the "centralized learning, decentralized execution" framework are used to train the agent group. The cumulative rewards obtained by the agent group in each scheduling cycle are counted, and the reward curve is generated. It can be observed that both converge after about 100 trainings. After about 70 trainings, the constraint violation penalty imposed on the agent tends to 0, indicating that the agent's actions are restricted to a reasonable range. The use of DMASAC training does not require the collection of global information, which ensures the privacy of each sub-microgrid information. It can be seen that a limited time of communication interruption does not have a great impact on the convergence of the method. This is because after the communication interruption, the affected agent does not update the parameters and retains the current strategy. When the communication is restored, the agent will continue to train. As long as the number of training times is sufficient, it can converge to the result equivalent to that without communication interruption. Similarly, data packet loss only stops the update and optimization of the strategy, and does not affect the execution of the current strategy. In summary, the method of the present invention has good robustness to problems such as communication interruption and data packet loss.

[0073] See also Figure 3 , select the source-load data of a certain day for testing, and use the trained agent group for scheduling. It can be seen that starting from 09:00, with the increase of photovoltaic output, the energy storage charging reaches the upper limit of capacity, so as to consume photovoltaic power locally and reduce power reverse transmission; and after 16:00, the photovoltaic output is not enough to supply the local load, and the energy storage begins to discharge until it reaches the lower limit of capacity at 21:00 and maintains it, so as to maintain the state of energy storage unchanged within a scheduling cycle. Through the above scheduling method, peak shaving and valley filling are achieved to a certain extent, reducing the interference of photovoltaic volatility to the system.

[0074] See also Figure 4 ,The SVC connected to nodes 29 and 23 also emits and absorbs inductive reactive power according to the voltage amplitude of the nodes, thereby smoothing the voltage fluctuation.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made to the present invention shall be included in the protection scope of the present invention.

Claims

1. A control method for efficient utilization of green station area energy in a microgrid group, characterized in that: The steps include: 1) Construct an active and reactive coordinated optimization dispatch model for microgrids in green station areas, take the minimum network loss within the dispatch period as the objective function of the model, and take power flow constraints, photovoltaic output constraints, static VAR compensator constraints, branch transmission power constraints, voltage amplitude constraints, energy storage constraints, and the number of operations of on-load tap changers and circuit breakers as constraints of the model; 2) By determining the agent observation space and the agent action space, and defining the agent reward function, the active and reactive coordinated optimization scheduling model of the green station area microgrid group is described as a Markov game problem; 3) The improved distributed multi-agent soft actor-critic algorithm and transfer reinforcement learning method are used to train the active and reactive coordinated optimization scheduling model of the green station area microgrid group converted into a Markov game problem.

2. A control method for efficient utilization of green station area energy for a microgrid group according to claim 1, characterized in that: The objective function described in step 1) is expressed as: In the formula, n is the number of sub-microgrids; T is the total number of scheduling periods in a day; R ij is the line resistance of branch ij; i 2 ij is the square of the current in branch ij; i is the set of branches that form a branch with node i in the sub-microgrid.

3. A control method for efficient utilization of green station area energy for a microgrid group according to claim 1, characterized in that: Step 1): (1) The power flow constraint is: Where P i g and are the active and reactive powers injected into node i respectively; P i load and are the active and reactive loads of node i respectively; and are the sum of the branch active powers flowing from the adjacent node m into the node i and flowing from the node i to the adjacent node j respectively; and are the sum of the branch reactive powers flowing from the adjacent node m into the node i and flowing from the node i to the adjacent node j; V i and V j are the voltages of nodes i and j respectively; P ij and Q ij are the branch active and reactive powers flowing from node i to node j respectively; θ ij is the phase angle difference between branches ij; G ij is the conductance of branch ij; B ij is the susceptance of branch ij; (2) Photovoltaic output constraints In order to improve economic efficiency, the photovoltaic system is required to generate electricity at maximum power, and only reactive power can be adjusted, that is, In the formula, and are the active and reactive powers generated by the g-th photovoltaic inverter at time t respectively; is the active power of the g-th PV inverter working in the maximum power point tracking mode at time t; pv A photovoltaic inverter assembly; and are the upper and lower limits of reactive power of the g-th photovoltaic inverter respectively; is the apparent power of the g-th PV inverter; (3) Static VAR compensator constraints: In the formula, is the reactive power generated by the cth static VAR compensator at time t; and are the upper and lower limits of reactive power of the cth static VAR compensator respectively; svc is a static VAR compensator set; (4) Branch transmission power constraints Where P ij，t is the active power transmitted by branch ij at time t; is the maximum limit of active power transmitted by branch ij; ε is the branch set; (5) Voltage amplitude constraint In the formula, and V j are the upper and lower limits of the voltage amplitude at node j, respectively; V j，t is the voltage amplitude of node j at time t; N is the number of nodes; (6) Energy storage constraints: In the formula, is the power of the oth energy storage at time t; and are the discharge and charge powers of the oth energy storage at time t respectively; is the charging and discharging efficiency of the oth energy storage; is the upper limit of the charging and discharging power of the oth energy storage, where discharging is positive and charging is negative; DESS For energy storage collection; is the initial capacity of the oth energy storage; and are the upper and lower limits of the capacity of the oth energy storage, respectively; Δt is the duration of a scheduling period; (7) Limitation on the number of operations of on-load tap changers and circuit breakers: In the formula, is the number of operations of the lth on-load tapchanger at time t; is the upper limit of the number of operations of the lth on-load tapchanger; OLTC Assemble on-load tap-changers; is the number of operations of the kth circuit breaker at time t; is the upper limit of the number of operations of the kth circuit breaker; CB for the circuit breaker collection; is a 0-1 variable indicating whether the gear position of the lth on-load tapchanger has changed at time t′; is the gear position of the lth on-load tapchanger at time t′; is a 0-1 variable indicating whether the circuit breaker position has changed at time t′; is the gear position of the kth circuit breaker at time t′.

4. A control method for efficient utilization of green station area energy for a microgrid group according to claim 1, characterized in that: In step 2): Agent Observation Space In order to consider the action times constraint of discrete devices, the times of on-load tap changer and circuit breaker actions are added to the observation space as follows: In the formula, and are the active and reactive loads in the intelligent agent ag at time t respectively; is the active power generated by the photovoltaic system in the intelligent agent ag at time t; and are the gear positions of the on-load tap-changer and circuit breaker in the intelligent agent ag at time t respectively; and are the number of times the on-load tap changer and circuit breaker in the intelligent agent ag have been operated at time t; is the power of the energy storage device in the intelligent agent ag at time t; Agent action space Since there are both discrete and continuous adjustment devices locally, the action space of the agent includes discrete action space and continuous action space, that is, In the formula, is the discrete action space of agent ag at time t; is the continuous action space of agent ag at time t; and are the reactive power generated by the photovoltaic inverter and static VAR compensator in the intelligent agent ag at time t; Agent Reward Function In order to ensure that the decision results of the intelligent agent meet the requirements of the stability of the microgrid system operation and to minimize the network loss within the scheduling period, the negative value of the network loss is used as the reward value of the intelligent agent. Use node voltage and branch power exceeding the limit as penalty Add to the reward function; the reward r of each agent per period i,t as follows: In the formula, is the penalty coefficient for exceeding the limit; and are the voltage over-limit function and branch power over-limit function at node i at time t respectively; N ag is the number of nodes of agent ag; It means max(x, 0), x means the number of cutoffs; R ij is the line resistance of branch ij; i ij is the square of the current in branch ij; and V j are the upper and lower limits of the voltage amplitude at node j, respectively; V j，t is the voltage amplitude of node j at time t; N is the number of nodes; P ij，t is the active power transmitted by branch ij at time t; is the maximum limit of active power transmitted by branch ij.

5. A control method for efficient utilization of green station area energy for a microgrid group according to claim 1, characterized in that: Step 3) includes: The training goal of the improved distributed multi-agent soft actor-critic algorithm is to find the optimal strategy π for the agent ag i * , as shown below; In the formula, E{·} represents the expected function; (s t , a t )~ρ π represents the state-action trajectory distribution induced by strategy π; H(π ag( ·|o ag , t )) is the intelligent agent ag in the observation amount o ag，t According to the strategy π ag The entropy of taking an action; α is the temperature coefficient used to adjust the importance of action entropy relative to the reward; Scheduling control under multiple topologies requires training a set of corresponding agents for each network topology. To improve training efficiency, a transfer learning mechanism is introduced based on the improved distributed multi-agent soft actor-critic algorithm: Q T =βQ S +(1-β)Q T (27) β=β-Δβ (30) In the formula, Q S and Q T are the action value functions of the existing agent and the new agent respectively; Modified loss function for the new agent Actor network; is the supplementary term of the new agent Actor network loss function; L Actor is the original loss function of the new agent Actor network; O is the observation volume of the agent; a T and a S are the actions given by the strategies of the new agent and the existing agent respectively; β and Δβ are the weight coefficient and the decay value of the agent respectively, which are used to prevent negative transfer when the teacher model does not match the new task.

Images