Voltage and reactive power optimization method based on two-layer reinforcement learning for power grid-user collaboration

Abstract

The present invention discloses a voltage-reactive power optimization method based on double-layer reinforcement learning and grid-user collaboration. In order to effectively coordinate the two types of resources with different ownership, namely, the grid-side equipment and the user-side equipment of the distribution network, the present invention proposes a double-layer reactive power and voltage optimization framework from the perspective of asymmetric Markov game. Among them, the distribution operator (DSO) on the grid side is regarded as a leader who makes decisions at the upper level, with the aim of minimizing the network loss of the system; the user is regarded as a follower who makes decisions at the lower level, with the aim of alleviating the voltage deviation of the node. In addition, the present invention also proposes a model-free Bi-level Actor-Critic (Bi-AC) algorithm to solve the above-mentioned asymmetric Markov game model, which formulates unique strategy (actor) and value (critic) networks for the leader and follower agents, and also defines the decision priority of the leader agent. Finally, the present invention verifies the effectiveness of the proposed scheme through a standard IEEE33 node test system and actual grid operation data.

Description

Technical Field

[0001] The present invention belongs to the fields of power system automation technology and computer technology, and specifically relates to a voltage-reactive power optimization method based on double-layer reinforcement learning power grid-user collaboration. Background Art

[0002] According to data from the International Energy Agency, by 2022, the total installed capacity of global solar photovoltaic power generation (PV) will gradually increase to 240GW, a year-on-year increase of 41%. With the continuous maturity of new energy technologies, more and more photovoltaic power sources are also installed on the user side. Among the newly added PV, about 43% come from residential, commercial and industrial projects. The high penetration rate of intermittent PV has changed the operation mode of the distribution network and caused frequent node voltage violations. The inverter attached to PV is a fast-time-scale continuous device and is usually used to mitigate the voltage problems caused by PV. However, unlike other voltage regulation resources on the grid side, distribution operators (DSOs) are usually unable to effectively observe and control PV on the user side. Therefore, it is necessary to invent an effective distribution network reactive power optimization solution to deal with the above problems.

[0003] At present, the methods used by researchers to solve the problem of reactive voltage optimization (VVO) in distribution networks can be roughly divided into two categories: model-based methods and model-free methods. Among them, the model-based method describes VVO as a mathematical programming problem for solution, but this method requires precise parameters to model the problem. However, current power grid companies often cannot provide accurate physical models of power grids. At the same time, due to the intermittent nature of PV, some control instructions need to be issued to the equipment quickly. The model-based method has a large computational burden and cannot meet this requirement. Therefore, the model-free method based on deep reinforcement learning (DRL) is a good alternative. Its solution idea is to convert the model of the problem into a Markov decision process (MDP), and then let the DRL agent learn the best strategy by continuously interacting with the distribution network environment by setting the state space, action space and reward function. At the same time, the trained DRL agent has no computational burden and can make real-time decisions.

[0004] Nowadays, many DRL algorithms have been used to solve the VVO problem of distribution networks. Among them, the multi-agent reinforcement learning (MADRL) method (such as MADQN, MADDPG, MASAC and other algorithms) has been favored by many scholars and has also achieved good results in the field of reactive power optimization of distribution networks. For example, some studies decompose the distribution network into several sub-networks and regard each sub-network as an agent, or regard voltage-regulating devices such as on-load tap-changing transformers (OLTCs), switched capacitor banks (SCBs), static VAR compensators (SVCs) and DG inverters as an agent, and then use the MADRL algorithm to train each agent through a centralized training and decentralized execution framework to solve the optimal strategy. Another study proposed a dual-time-scale voltage-reactive power control method, using a single-agent reinforcement learning algorithm to centrally control slow-responding voltage-regulating devices, and using a multi-agent reinforcement learning algorithm to decentralized control fast-responding devices. The main idea of ​​the above MADRL methods is to model the VVO problem as a multi-agent Markov Game (MG) model. This MG is a symmetric cooperative game whose purpose is to find a Nash equilibrium (NE). However, none of the above multi-agent reinforcement learning (MADRL) methods can ensure convergence to a specific Nash equilibrium, resulting in uncertainty and suboptimality in the solution. At the same time, the existing MADRL scheme assumes that the distribution operator (DSO) can monitor and control all equipment, but due to data security and privacy, the DSO cannot effectively dispatch the equipment on the user side. Summary of the invention

[0005] To solve the above problems, the present invention discloses a voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration, reconsiders the VVO problem of the distribution network from an asymmetric perspective, proposes an asymmetric Markov game model (Asymmetric Markov games, ASMG), and invents a multi-objective double-level reactive voltage optimization (bi-level VVO, Bi-VVO) framework. In addition, the present invention also proposes a model-free Bi-level Actor-Critic (Bi-AC) algorithm to solve the above asymmetric Markov game model. The algorithm formulates a unique strategy (actor) and value (critic) network for the leader and follower agents, and also defines the decision priority of the leader agent so that the follower agent always executes the best response strategy, and then divides the entire solution process into two stages: offline training and online execution. Finally, the present invention verifies the effectiveness of the proposed scheme through an improved IEEE 33-node test system and actual grid operation data.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] The voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration includes the following steps:

[0008] Step 1: Construct a two-layer voltage-reactive power optimization (Bi-VVO) model for the distribution network. On the grid side, the upper optimization model is constructed with the distribution operator (DSO) as the control subject, and its goal is to reduce the network loss of the entire system. On the user side, the lower optimization model is constructed with photovoltaic (PV) users as the control subject, and its purpose is to reduce the voltage deviation of the distribution network nodes.

[0009] Step 2: Consider the upper-layer DSO in the two-layer optimization model in step 1 as the leader and the lower-layer PV users as followers, and define their decision priorities to construct an asymmetric Markov game (ASMG) model.

[0010] Step 3: In order to solve the method proposed in steps 1 and 2, a two-level reinforcement learning algorithm Bi-level Actor-Critic (Bi-AC) is used to develop unique policy (Actor) and value (Critic) networks for the leader and follower agents.

[0011] Step 4: For the Bi-AC algorithm in step 3, each intelligent agent is trained offline. During this process, the intelligent agent continuously interacts with the simulation environment through the real historical operation data of the distribution network to learn the best optimization strategy.

[0012] Step 5: After the training in step 4 is completed, deploy the Actor network of each agent to the controller, and input observation information into the controller to obtain the optimization strategy in real time.

[0013] Since Stackelberg game is an asymmetric dynamic game, it is usually used to study multi-level decision-making problems with different priorities. Each level finds Stackelberg equilibrium (SE) through interactive games. Although SE discriminates against agents with low priority decisions, SE is better than NE in a wide range of environments, especially in cooperative games, SE is Pareto optimal. If the various reactive optimization devices in the distribution network are regarded as game subjects, the VVO of the distribution network is a cooperative collaborative game process.

[0014] On this basis, the present invention proposes a two-layer voltage and reactive power optimization (Bi-VVO) framework for PV high penetration distribution network with coordinated grid-user side resources. Among them, the upper layer is controlled by the DSO on the grid side, and its purpose is to reduce the power loss of the entire system by controlling the equipment on the grid side. The lower layer is controlled by the user side, and its purpose is to alleviate the overvoltage problem caused by PV by scheduling distributed resources on the user side. We describe the Bi-VVO problem as an ASMG problem, and define the agents of the upper and lower layers as leader agents and follower agents respectively, and define the decision priority, that is, the leader makes a decision first, and then only needs to pass the decision information to the follower, and the follower makes a decision based on this, that is, the lower layer optimization is performed on the basis of the upper layer optimization. Finally, the present invention proposes a Bi-AC algorithm to construct unique action (actor) and value (critic) networks for the leader and follower agents to train each agent to learn the best strategy.

[0015] Beneficial effects of the present invention:

[0016] The Bi-VVO method proposed in the present invention is a multi-objective optimization method. The upper-level optimization can reduce the power loss of the entire system, and the lower-level optimization can reduce the voltage deviation caused by PV intermittency. This method makes full use of the optimization equipment on the grid side and the user side, and can dispatch distributed resources on the user side under limited communication conditions.

[0017] At the same time, the ASMG model we proposed defines the decision priority of the leader. After the optimization is completed, the leader passes the best decision to the follower, so the follower can always execute the best response strategy. The proposed ASMG model not only ensures the privacy of grid-side data but also ensures that the user side can make better decisions using its own equipment.

[0018] At the same time, the Bi-AC algorithm proposed in the present invention does not rely on accurate distribution network models and prior statistical information, and divides the entire process into two stages: offline training and online execution. The algorithm adopts a centralized training and decentralized execution framework in the offline training stage, and learns the dynamic characteristics of the distribution system through real historical operation data of the distribution network and the continuous activities of the power grid environment, while also capturing the uncertainty of PV. In the online execution stage, the trained intelligent agent can calculate the optimal strategy in a relatively short time, and can respond to the intermittency and volatility of PV in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flow chart of the present invention.

[0020] Figure 2 This is the framework diagram of Bi-VVO based on ASMG.

[0021] Figure 3 It is the online execution process after the offline training of each agent is completed.

[0022] Figure 4 This is the IEEE33 node test system diagram.

[0023] Figure 5 PV active output curve and load active power on typical days in summer and winter.

[0024] Figure 6 Cumulative reward convergence of the leader and node 17’s follower training process. DETAILED DESCRIPTION

[0025] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0026] As shown in the figure, the voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration of the present invention includes the following steps:

[0027] Step 1: Construct the Bi-VVO model of the distribution network.

[0028] The upper control subject is the DSO on the grid side. In order to cope with the intermittent nature of PV, the reactive power optimization device we choose in the specific implementation is the static VAR compensator (SVC). The upper optimization goal is to minimize the active power loss of the distribution system. Formula (1) is the precise active power loss calculation formula:

[0029]

[0030] Where N is the total number of nodes in the distribution network, P i ,Q i and P j ,Q j Represents the net injection of active power and reactive power at nodes i and j respectively. Coefficient α ij and β ij It can be calculated by the following formula:

[0031]

[0032] In the formula, δ i 、V i and δ j 、V j Represents the voltage phase angle and amplitude r of node i and node j respectively ij is the line resistance between the two nodes. The control subject at the lower level is the user who installs the PV. The inverter attached to the PV is used to optimize the reactive power of the node to alleviate the voltage deviation caused by the intermittent nature of the PV. The optimization target is as shown in formula (4).

[0033] V i dev (t)=|V i -V set |(4)

[0034] Where V i is the voltage amplitude of node i, V set is the set value of the standard voltage, which is generally set to 1. The constraints of equality and inequality in the VVO problem include power flow balance constraints of the power system, node voltage constraints and variable constraints of the control equipment, and its mathematical model is given in the following formula.

[0035]

[0036] V min ≤V i t ≤V max (7)

[0037]

[0038] In the above formula, T is the set of times t N is the set of distribution network nodes G ij and B ij is the real and imaginary part of the ijth element of the power system admittance matrix, V min and V max It is the upper and lower limits of the node voltage amplitude. SVC is the number of installed SVC nodes, N PV is the number of installed PV nodes, Q t PV,i represents the reactive power output of the i-th PV node PV at time t, S i is the installed capacity of the i-th PV, P t PV,i is the active power output of the i-th PV node PV at time t.

[0039] Step 2: For the Bi-VVO framework of step 1, the present invention proposes the concept of asymmetric Markov game (ASMG), regards the upper-layer DSO as the leader agent and the lower-layer PV user as the follower agent, and then describes the state space, action space, reward function and state transition probability of ASMG. The Bi-VVO framework based on ASMG is as follows: Figure 2 shown.

[0040] In an N-agent Markov game (MG) 〈S,A i ,P,R i ,γ〉, S represents the state space, A represents the joint action space, and Ri is the reward function of the ith agent, where γ is the discount factor, and the action a of agent i at time t i t ∈A i To get the action, set a policy function for each agent i Then according to the state transition probability P:S×A1×…×A n ×S to generate the state of the next moment.

[0041] For a Markov game, the optimal state value V i * (s) and action state-action value Q i * (s,a i ) is determined according to the optimal Bellman equation, which is expressed as follows:

[0042]

[0043] In the formula, s t+1 represents the state of the agent at the next moment, P(s t+1 ∣s t ,a t ) indicates that the agent changes from state s after taking action a at time t t Transfer to t+1 The following describes the state space, action space, reward function and state transition process of the leader and follower in ASMG in combination with the Bi-VVO problem.

[0044] 1) State space: The state space of the leader is S l =(P,Q,V), where P and Q represent the net amount of active power and reactive power injected into the node P i and is a collection of nodes, V is the voltage amplitude of the node The state space of the follower is S f =(P,Q,V,A), the follower receives the leader's decision information and inputs it into the strategy function as its own state, so A is the leader's action A l ,In addition, P, Q, and V represent the information of the node where the PV user is installed.

[0045] 2) Action space: The leader’s action space A l is the reactive power output set of each SVC, a l,i t ∈A l is the reactive power output Q of the i-th SVC at time t t svc,i , where Q tsvc,i It is defined in equation (8). The follower’s action space A f is the reactive power output of each PV inverter, a f,i t ∈A f is the reactive output Q of the i-th PV at time t t PV,i , where Q t PV,i It is defined in formula (9).

[0046] 3) Reward function: The leader’s reward function is r l t =-P loss (t)-η, where P loss (t) is defined in formula (1). The reward function of the follower is r f t =-V dev (t)-η, where V dev (t) is defined in formula (4), η is the penalty factor in the reward function, and η=0 when the voltage amplitudes of all nodes satisfy formula (7).

[0047] 4) State transfer: At each time t, the leader obtains the state of observing itself as s l t =(P t ,Q t ,V t ), and take corresponding action a according to the state l t , and pass it to the follower. The state of the follower at time t is s f t =(P i t ,Q i t ,V i t ,a l t ), which includes its current state and the leader's action, and then makes corresponding actions a f t . Then the agents get their respective rewards r i t , the state is based on the probability model P(s i t+1 |s i t ,a i t ) transfer to the next time s i t+1The goal of each agent is to find a solution that maximizes its cumulative discounted reward R i =∑ Τ t＝0 γ t r i t strategy.

[0048] Step 3: According to the models proposed in steps 1 and 2, the present invention proposes a Bi-AC algorithm.

[0049] Considering that the actions of SVC and PV are continuous, we formulate an Actor model and a Critic model for each agent and adopt a deterministic policy function To represent the actor model, and use the function Q i (s i ,a i |θ i ) represents the critic model, where and θ i is the weight parameter of the neural network. The following introduces the parameter update principle of each model in the optimization process. The present invention assumes that there is a leader and M followers in the ASMG model. We construct a joint strategy set {μ l ,μ f,1 ,…,μ f,M}. The actions output by the leader actor Actions output by the follower actor The critic model used by followers and leaders to judge actors is Q i (s l t ,a l t ,a f,k t |θ i ). The gradient of the cumulative expected return of agent i is J(μ i )=E[R i ], the following is the strategy gradient update formula for the leader and follower:

[0050]

[0051] Where, X t =(s l t ,s f,1 t ,…,s f,M t ), Represents the experience pool (replay buffer), which contains (X t ,Xt+1 ,a l t ,a f,1 t ,…,a f,M t ,r l t ,r f,1 t ,…,r f,M t ). At the same time, the value function Q can be updated by minimizing the loss between the predicted value and the target value. i (s l t ,a l t ,a f,k t |θ i ). The update formula is shown in (16) and (17):

[0052]

[0053] In the formula, E represents expectation, i can represent l,f 1…M , during the update process we design the target actor μ' i and target critic Q' i This prevents overestimation of values ​​and increases the stability of the model. and θ' i are the weight parameters of the target actor and target critic neural networks, respectively, and the target network is used to replace y in Q i t The following formula is obtained:

[0054]

[0055] The weight update formula of the target network is: θ'←τθ+(1-τ)θ', This means that the target value can only change slowly, which greatly improves the stability of learning.

[0056] Step 4: For the models proposed in steps 1, 2, and 3, the present invention uses a centralized training and decentralized execution framework for offline training.

[0057] The present invention uses a data-driven approach to centrally train all models. Each intelligent agent uses the real historical operation data of the distribution network to learn the dynamic characteristics of the distribution network through continuous interaction with the simulation environment. During the interaction, the variables of the distribution network are the active output of the PV and the active power and reactive power of each node load. The time interval of the training data we use is 5 minutes, so we set each day as an episode, and each episode includes 288 steps t (step), that is, T = 288t. At the beginning of training, the parameters of the actor and critic neural network models of agent i are randomly initialized. and θ i , and then set the number of random exploration steps. In the exploration phase, the actions taken by the agent are random. In a step, the leader makes a decision based on the current observed state s l t Calculate the action a that needs to be done l t , the follower follows its local observation state s f t and the leader's actions l t Calculate the actions you need to take f t , and then transmit the joint action to the power system simulation environment for power flow calculation. Finally, the rewards r of the leader and follower at time t are obtained from the calculation results of the simulation environment. l t and r f t , and the state s at time t+1 l t and f t The obtained information is then stored in the experience pool (replay buffer). When the amount of information stored in the experience pool reaches a certain amount, the model is updated. Each time a mini-batch is taken out from the replay buffer, the parameters of the actor and critic neural network models are updated using equations (19), (20), and (21).

[0058]

[0059] In the formula, σ is the learning rate of the neural network during gradient update. So far, as the training rounds progress, the strategies learned by the agent are getting better and better. After the training is completed, the parameters of each neural network are saved. The training process is shown in the following table:

[0060] Table 1 Offline training process of Bi-AC algorithm

[0061]

[0062] Step 5: After the training in step 4 is completed, the Actor network of each agent is deployed to the controller, and the optimization strategy can be obtained in real time by inputting observation information to the controller. The online execution process is as follows Figure 3 shown.

[0063] We used the standard IEEE 33-node distribution network test system and the real PV and load data in a pilot project in Jiangsu Province to verify the effectiveness of the proposed method. At the same time, we compared the proposed Bi-AC algorithm with the SoftActor-Critic (SAC) algorithm for a single-agent model and the MADDPG algorithm for a multi-agent model, where the formulation and training process of each agent model was completed by "TensorFlow" on a computer equipped with 16GB memory and 2.50GHz Intel(R) Core(TM) i7-11700. Below we describe the scenario design of the simulation experiment.

[0064] In the 33-node test system, we embedded three photovoltaic power sources as voltage regulation resources on the user side, which were installed at nodes 17, 22, and 32, with installed capacities of 2MVA, 0.9MVA, and 1MVA respectively. We embedded two SVCs as voltage regulation resources on the grid side, which were installed at nodes 5 and 12, with reactive capacities of 600kVar. Figure 4 We selected the active output of photovoltaic power sources of three large enterprises in Zhejiang Province as photovoltaic output, and the load data came from a pilot project in Zhejiang Province. Figure 5 The PV active output curve and load active power of typical days in summer and winter. We divide the data into training set and test set, and use 500 days of historical PV and load data with a resolution of 5 minutes in the offline training process. These 144,000 training samples can better describe the uncertainty problem and fit the real environment.

[0065] On the grid side, a leader agent controls the two SVCs. The input of the leader is the node information of the entire system, and the output is the reactive power of the two SVCs. On the user side, three follower agents (Follower 1, Follower 2, Follower 3) control the PV inverters of nodes 17, 22, and 32 respectively. The input of the followers is the node information of the nodes where each photovoltaic power source is located, and the output is the reactive power of the inverter. In the offline training phase, the agent model is trained for 500 rounds, each round contains 288 steps, and the voltage limit is set to [0.95, 1.05]. If the voltage exceeds the limit, the agent is punished accordingly, and the penalty factor η is set to 2. If the power flow does not converge during the training process, the termination command is returned. The environment is initialized at the end of the round or when the termination occurs. Each agent has an Actor model and a Critic model, and the neural network architecture of all agents is the same. The hyperparameters are shown in the table below.

[0066] Table 2 Algorithm hyperparameter settings

[0067]

[0068] In order to compare the performance of the method proposed in the present invention, we use the MADDPG algorithm as the multi-agent RL baseline. In order to better compare the advantages of SE over NE, one agent of MADDPG is regarded as the leader of Bi-AC, and the other three agents are regarded as followers. The difference is that they cannot receive the action information of the leader. The hyperparameter settings of each agent model of MADDPG are the same as those of the Bi-AC algorithm. At the same time, the SAC algorithm is used as a centralized RL baseline. Without considering communication, all control devices are regarded as a SAC agent, and the agent can make optimization decisions based on global information. Due to the randomness of the DRL algorithm, we use different random seeds to train each algorithm three times. The average value and error bound of the training results are represented by solid lines and filled areas in the figure. The average cumulative rewards of the leader and followers of node 17 during training are shown as follows. Figure 6 shown.

[0069] During the training process, since the exploration capabilities of MADDPG and the algorithm proposed in the present invention are relatively weak compared to the SAC algorithm with an entropy term, the number of exploration steps of the former two is increased. The actions generated by the agent in the exploration stage are random, so each agent receives a lower reward. Since the followers of the method proposed in this article make decisions based on the leader's decision, the average cumulative rewards of the two converge slowly and have large fluctuations. After fully understanding the changes in the environment, each agent has learned good decisions, and the average cumulative reward per round tends to be stable. After the training of each algorithm is stable, the number of voltage violations in the distribution network is close to 0, which also confirms the good performance of the RL method.

[0070] Table 3 Performance comparison of each algorithm in the last 100 rounds

[0071]

[0072] like Figure 6 As shown in the figure, since the centralized algorithm SAC controls all devices based on global information, its cumulative reward is higher than that of the other two algorithms. Since there is communication between the agents of the proposed method, the optimization performance of global active power loss is better than that of MADDPG. The followers of Bi-AC are optimized based on the optimal decision of the leader, so the rewards of the followers on the user side are much higher than those of MADDPG. Table 3 shows the active power loss of each algorithm in the last 100 rounds, the voltage deviation of the three photovoltaic users, and the average voltage violation rate of the entire system.

[0073] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration, characterized in that: The following steps are involved: Step 1: Construct the Bi-VVO model of the distribution network. On the grid side, the upper optimization model is constructed with the distribution operator DSO as the control subject, and its goal is to reduce the network loss of the entire system. On the user side, the lower optimization model is constructed with the solar photovoltaic power generation PV user as the control subject, and its purpose is to reduce the voltage deviation of the distribution network node. Step 2: Consider the upper-layer DSO in the two-layer optimization model in step 1 as the leader and the lower-layer PV users as followers, and define their decision priorities to build the ASMG model; In the ASMG model, the upper-level DSO is regarded as the leader and the lower-level PV users are regarded as followers, and their decision priorities are defined. At the same time, the state space, action space, reward function and state transition probability of ASMG are described; 1) State space: The state space of the leader is S l =(P,Q,V), where P and Q represent the net amount of active power and reactive power injected into the node P i and Q i A collection of V is the voltage amplitude of the node V i ,in The state space of the follower is S f =(P,Q,V,A), the follower receives the leader's decision information and inputs it into the strategy function as its own state, so A is the leader's action A l ,In addition, P, Q, and V represent the information of the node where the PV user is installed; 2) Action space: The leader’s action space A l is the reactive power output set of each SVC, a l,i t ∈A l is the reactive power output Q of the i-th SVC at time t t svc,i , the follower’s action space A f is the reactive power output of each PV inverter, a f,i t ∈A f is the reactive output Q of the i-th PV at time t t PV,i ; 3) Reward function: The leader’s reward function is r l t =-P loss (t)-η, the follower’s reward function is r f t =-V dev (t)-η, η is the penalty factor in the reward function, and η = 0 when the voltage amplitudes of all nodes meet the specified range; 4) State transfer: At each time t, the leader obtains the state of observing itself as s l t =(P t ,Q t ,V t ), and take corresponding action a according to the state l t , and pass it to the follower; the state of the follower at time t is s f t =(P i t ,Q i t ,V i t ,a l t ), which includes its current state and the leader's action, and then takes corresponding action a f t ; Then the agents get their respective rewards r i t , the state is based on the probability model P(s i t+1 |s i t ,a i t ) transfer to the next time s i t+1 ; The goal of each agent is to find a solution that maximizes its cumulative discounted reward R i =∑ Τ t＝0 γ t r i t strategy; Step 3: To solve the methods proposed in steps 1 and 2, a two-layer reinforcement learning algorithm Bi-AC is used to formulate unique Actor and Critic networks for the leader and follower agents; The Bi-AC algorithm described above develops an Actor model and a Critic model for each agent, using a deterministic strategy function To represent the actor model, use the function Q i (s i ,a i |θ i ) represents the critic model, where and θ i is the weight parameter of the neural network; the following is the parameter update principle of each model during the optimization process; Assume that there is a leader and M followers in the ASMG model, and construct a joint strategy set {μ l ,μ f,1 ,…,μ f,M }; Action output by the leader actor Actions output by the follower actor in The critic model used by followers and leaders to judge actors is Q i (s l t ,a l t ,a f,k t |θ i ); the gradient of the cumulative expected return of agent i is J(μ i )=E[R i ], the following is the strategy gradient update formula for the leader and follower: Where, X t =(s l t ,s f,1 t ,…,s f,M t ), represents the experience pool, which contains (X t ,X t+1 ,a l t ,a f,1 t ,…,a f,M t ,r l t ,r f,1 t ,…,r f,M t ); at the same time, the value function Q is updated by minimizing the loss between the predicted value and the target value i (s l t ,a l t ,a f,k t |θ i ); the update formula is as follows: In the formula, E represents expectation, i represents l, f 1…M , the target actor μ' is designed during the update process i and target critic Q' i To prevent overestimation of values ​​and increase the stability of the model; and θ' i are the weight parameters of the target actor and target critic neural networks, respectively, and the target network is used to replace y in Q i t The following formula is obtained: The weight update formula of the target network is: θ'←τθ+(1-τ)θ', This means that the target value can only change slowly, which improves the stability of learning; Step 4: For the Bi-AC algorithm in step 3, each agent is trained offline. During this process, the agent continuously interacts with the simulation environment through the real historical operation data of the distribution network to learn the best optimization strategy; Step 5: After the training in step 4 is completed, the Actor network of each agent is deployed to the controller, and the observation information is input into the controller to obtain the optimization strategy in real time.

2. The voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration according to claim 1 is characterized in that: In the Bi-VVO model of the distribution network described in step 1, the upper control subject is the DSO on the grid side. In order to cope with the intermittent nature of PV, the reactive power optimization device selected is the static VAR compensator; the upper optimization goal is to minimize the active power loss of the distribution system. The following formula is the accurate active power loss calculation formula: Where N is the total number of nodes in the distribution network, P i , Q i and P j , Q j Represents the net injection of active power and reactive power of nodes i and j respectively; coefficient α ij and β ij Calculated by the following formula: In the formula, δ i 、V i and δ j 、V j Represents the voltage phase angle and amplitude r of node i and node j respectively ij is the line resistance between two nodes; The control subject at the lower level is the user who installs PV. The inverter attached to the PV is used to optimize the reactive power of the node to alleviate the voltage deviation caused by the intermittent PV. The optimization target is as follows: V i dev (t)=|V i -V set | Where V i is the voltage amplitude of the ith PV installation node, V set This is the set value for the standard voltage, set to 1.

3. The voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration according to claim 1 is characterized in that: In step 4, each intelligent agent is trained in an offline manner. During this process, the intelligent agent continuously interacts with the simulation environment through the real historical operation data of the distribution network to learn the best optimization strategy.

4. The voltage and reactive power optimization method based on double-layer reinforcement learning grid-user collaboration according to claim 1 is characterized by: Step 5 is as follows: The trained Actor models are saved and deployed to the DSO and the user's controller. The DSO inputs its own observation information to obtain the optimal strategy, which is then passed to the user. The user can obtain the optimal strategy based on this information and local observations; this realizes real-time distributed execution.

Images