A power distribution network voltage reactive power optimization method based on graph reinforcement learning

Abstract

The disclosure belongs to the field of power system automation and computer technology, and discloses a power distribution network voltage reactive power optimization method based on graph reinforcement learning, first modeling a reactive power voltage optimization problem as a Markov decision process, and using a model-free deep reinforcement learning method to solve; then a proximal policy optimization algorithm based on a graph neural network is invented, the algorithm improves the perception ability of a reinforcement learning intelligent agent to power distribution network graph data by embedding a graph convolution network; finally, the invention carries out example analysis on an improved IEEE33 test system, verifies the effectiveness of the proposed method and the advantages compared with other methods, and the results show that the reinforcement learning intelligent agent trained based on the graph neural network still shows good performance when the power distribution network topology changes.

Description

Technical Field

[0001] This disclosure belongs to the fields of power system automation and computer technology, and specifically relates to a distribution network voltage reactive power optimization method based on graph reinforcement learning. Background Technology

[0002] According to statistics from the National Energy Administration, by the end of 2021, the installed capacity of distributed photovoltaic (PV) power reached 107.5 million kilowatts, accounting for approximately one-third of the total grid-connected PV capacity. The high penetration of PV poses certain challenges to the stable operation of the distribution network, specifically manifested in increased active power losses, node voltage exceeding limits, and reduced lifespan due to frequent tap changes in discrete voltage regulators. Furthermore, the intermittent and fluctuating nature of weather-dependent PV power generation adds complexity and uncertainty to the problem-solving process. Therefore, an effective voltage-reactive power optimization (VVO) method is needed to address these challenges.

[0003] Typically, VVO (Voltage-Voltage-Transfer) tasks can be described as mixed-integer nonlinear programming problems, where variables generally represent the discrete actions of voltage regulation equipment and the continuous actions of reactive power compensation equipment. Current research focuses on characterizing the uncertainty of PV (Power Generation) through prediction or the use of stochastic probability functions. For example, model predictive control methods are used to solve VVO problems with high penetration rates of distributed generation; or stochastic programming models are used, which are then transformed into deterministic mixed-integer quadratic programming models and solved using commercial solvers. Unlike stochastic programming, robust optimization methods construct a set of uncertainties and obtain solutions in the worst case, with results that are usually conservative. Some studies have used such robust optimization models to formulate VVO tasks and then solved them using a multiplier alternating direction method. However, the above model-based prediction methods are inaccurate. Although traditional mathematical optimization methods can obtain high-quality solutions, their computational burden is too high, and the optimal instructions cannot be issued to the equipment within the specified time. These characteristics hinder the development and implementation of model-based methods in the future operation of power distribution systems.

[0004] A growing body of research recommends using the DRL method to solve the VVO problem in distribution networks, demonstrating the superiority of this model-free approach. Some studies employ single-agent DRL models, modeling the reactive power optimization problem of distribution networks as a Markov decision process (MDP). For example, an Actor-Critic (AC) algorithm is used to minimize network losses and equipment operating costs as optimization objectives, with the switching commands of discrete reactive power regulators as control variables to solve for VVO. Alternatively, static var compensators (SVCs) and inverters attached to PV systems are used for reactive power compensation in the distribution network, and a soft Actor-Critic (SAC) algorithm is used to assign unique value and policy models to the agents. Other studies describe the VVO problem as a Markov game model and use multi-agent reinforcement learning algorithms for solution. However, in most existing research on DRL for solving the VVO problem, electrical quantities in the power system are input as vectors into a neural network to train the DRL agent. However, the steady-state data in the power system is a graph data structure that includes the electrical quantities of nodes and edges and the topological connections. Inputting such graph data into the neural network in the form of vectors for training will greatly reduce the DRL agent's ability to perceive the distribution network structure. The performance of the trained agent will be greatly reduced when the distribution network topology changes, and it will not be able to be extended to other distribution networks. Summary of the Invention

[0005] To enhance the DRL agent's ability to perceive distribution network graph data, this invention proposes a framework that combines DRL and GCN to solve the voltage control problem in PV high-penetration distribution networks, and proves that the DRL model trained based on graph neural networks has strong scalability.

[0006] The objective of this disclosure can be achieved through the following technical solution: a power distribution network voltage reactive power optimization method based on graph reinforcement learning, comprising the following steps: A mathematical model for voltage reactive power optimization (VVO) of the distribution network is constructed. The mathematical model of VVO includes the objective function, equality constraints for power flow calculation, and inequality constraints during operation. The constructed VVO mathematical model is transformed into a Markov Decision Process (MDP) model; A graph convolutional neural network-based near-end policy optimization (GCN-PPO) algorithm is proposed to solve the MDP model. The GCN-PPO algorithm defines a unique policy (Actor) and value (Critic) network for the reinforcement learning agent. The GCN-PPO algorithm integrates the graph convolutional layer into the hidden layer composed of a multilayer perceptron neural network to perceive the graph data structure composed of the electrical quantities and topological connections of nodes and edges in the distribution network. The agent of the GCN-PPO algorithm is trained offline using real historical operating data of the distribution network. During this process, the reinforcement learning agent continuously interacts with the simulation environment to capture the intermittency and volatility of photovoltaic (PV) and load in the historical data. After offline training is completed, the trained agent model is saved, and then the effectiveness of the agent model and its scalability under changes in the distribution network topology are verified.

[0007] Preferably, in the VVO mathematical model for constructing the distribution network, the uncertainties in the PV high-penetration distribution network include the active power output of PV. Active power of the load and the reactive power of the load The control variable during OPF execution is the reactive power output of SVC. and the reactive power output of PV inverters The goal of reactive power optimization is to minimize the active power loss of the distribution network, as shown in the following formula. t Optimization goals for each moment: In the formula, T It refers to the time period being optimized. N It is the number of nodes in the distribution network; F During the time period T Total network loss cost within; It is a power distribution network t Network loss at any time; It is the network loss cost coefficient. It is the real part of the admittance matrix; and yes Within a certain time i The real and imaginary parts of the node voltage; and yes Within a certain time The real and imaginary parts of the node voltage; Preferably, the mathematical model constraints for VVO include: equality and inequality constraints for power flow calculation and operating capacity constraints for voltage regulating equipment, the expressions of which are as follows: In the above formula, It is the real part of the admittance matrix; It is the imaginary part of the admittance matrix; and yes t Time Node i The net injection of active and reactive power; and yes t Within a certain time i The real and imaginary parts of the node voltage; and yes t Within a certain time j The real and imaginary parts of the node voltage; yes t Voltage amplitude at time node; and These are the upper and lower limits of the node voltage, set to 1.05 and 0.95 respectively; and These are the upper and lower limits of SVC capacity. and for t The upper and lower limits of reactive power capacity of PV at any given time; and for t The reactive power output values ​​of SVC and PV at any given time; The PV inverter has redundant rated capacity and operates in maximum power point tracking mode; The reactive power control range of the PV inverter is determined by the rated installed capacity of the PV. and current active power output The scope of the decision is as follows: .

[0008] Preferably, the VVO mathematical model is transformed into an MDP model. Under the GCN-PPO framework, the state space, action space, reward function, and state transition process of the VVO are represented as follows: State space: the vector As t Time Node i The eigenvalues ​​of the nodes are respectively.i The net injection of active and reactive power and voltage amplitude are calculated, and it is assumed that the features of each node are observed; in the GCN-PPO framework, the feature matrix and connection matrix are used as state inputs into the graph convolutional neural network. t The state-space representation at time t is: X is the feature matrix composed of all node features, and A is the connection relationship of all lines in the power distribution system. Action Space: The actions of the agent include the reactive power output of all reactive power regulation equipment in the power distribution network, assuming there is... m SVC and n One PV inverter, then ; Reward function: Voltage exceeding the limit is added as a penalty to the reward function to ensure that the agent executes voltage safety constraints during the learning process. t The reward function at time step is: ,in This is the penalty coefficient for voltage exceeding the limit; State transition process: at each time step t Inside, the agent observes the current state. Then according to Make the current action a t Finally, a reward value is obtained. And according to P Get the state at the next moment The goal of the intelligent agent is to find a way to maximize the cumulative expected return through the above process. The strategy.

[0009] Preferably, the GCN-PPO algorithm is an improvement on Trust Domain Policy Optimization (TRPO). The TRPO algorithm uses a Kolb-Leibran divergence-constrained policy network, and the optimization objective and constraints are shown in the following equation: In the formula, This represents the ratio of the old to the new strategies. The old strategy before the update. For policy parameters; KL divergence, also known as relative entropy, is used to measure the difference between probability distributions. This represents the confidence level, used to limit the magnitude of policy updates; Let be the expectation, and let represent the empirical average on a finite sample. Indicating in strategy Down tThe advantage function estimate for the decision step; since the computational cost of calculating the KL divergence in each policy update is very high, the PPO algorithm uses a truncation function instead of the KL divergence constraint, which ensures the stability of the TRPO algorithm and reduces the computational cost; the objective function of PPO using the truncation function is expressed as: In the formula, This is a truncation function that controls the changes between the old and new strategies within a certain range. Inside, The truncation constant is used to set the range of policy updates.

[0010] Preferably, the GCN-PPO algorithm adds a graph convolutional layer before the multilayer perceptron. The policy (Actor) network architecture consists of two graph convolutional layers and three multilayer perceptron layers. Each layer is accompanied by a ReLU activation function, and a summation pooling function is used to aggregate the output of the graph convolutional layer on adjacent nodes before passing it to the multilayer perceptron layer to output a policy. The architecture of the value (Critic) network, which defines the value function, is roughly the same as that of the Actor network. The main difference is that a global summation pooling function is added after the graph convolutional layer in the Critic network architecture.

[0011] Preferably, the agent proposing the GCN-PPO algorithm is trained using real historical operating data of the distribution network. During this process, the parameters of the Actor network and the Critic network are continuously updated. After each training round, the network parameters are saved, and then VVO is executed in real time based on the trained Actor model.

[0012] An apparatus comprising: One or more processors; Memory, used to store one or more programs; When one or more of the programs are executed by one or more of the processors, the one or more processors implement the power distribution network voltage reactive power optimization method based on graph reinforcement learning as described above.

[0013] A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the graph reinforcement learning-based power distribution network voltage reactive power optimization method as described above.

[0014] The beneficial effects of this disclosure are: Compared to traditional model-based mathematical optimization methods, the graph reinforcement learning method proposed in this invention does not require an accurate distribution network model. It can capture the dynamic characteristics of the system, as well as the intermittency and volatility of PV and load, from historical data. At the same time, the trained DRL agent can make decisions in real time to deal with rapid fluctuations in PV, greatly reducing the computational burden.

[0015] Compared to the basic DRL algorithm, the GCN-PPO algorithm proposed in this invention incorporates a graph convolutional neural network in front of the multilayer perceptron of the Actor and Critic networks. This aggregates information and connection relationships between adjacent distribution network nodes, enabling the trained agent to maintain good performance even when the distribution network topology changes. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the present invention; Figure 2 Here is a flowchart of the VVO based on the GCN-PPO algorithm; Figure 3 The output curves for PV and load on the summer and winter solstices are shown. Figure 4 The reward value and the number of voltage exceedances during the training process; Figure 5 A comparison of the network loss of the three methods on the test set; Figure 6 A comparison of the number of voltage overruns under three topology transformation scenarios. Detailed Implementation

[0018] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] A voltage reactive power optimization method for distribution networks based on graph reinforcement learning. Step 1: Construct a mathematical model of VVO for the distribution network The VVO of a distribution network can be described as an optimal power flow (OPF) problem, that is, optimizing the target value in the power flow under certain constraints. Distribution systems typically exhibit a radial network topology during operation; this invention represents it as a tree diagram. G ,here , and These represent the sets of all nodes and lines in the distribution network, respectively. N For a distribution network with individual nodes, the nodes The uncertainties considered in the high PV penetration distribution network of this invention include the active power output of PV. Active power of the load and the reactive power of the load The control variable during OPF execution is the reactive power output of SVC. and the reactive power output of PV inverters The optimization objectives and constraints of VVO will be explained below.

[0020] The objective of reactive power optimization in this invention is to minimize the active power loss of the distribution network, as shown in equation (1). t The optimization objective at any given time is calculated as shown in equation (2).

[0021] (1) (2) In the formula, T It refers to the time period being optimized. N It is the number of nodes in the distribution network; F During the time period T Total network loss cost within; It is a power distribution network t Network loss at any time; It is the network loss cost coefficient. It is the real part of the admittance matrix; and yes Within a certain time i The real and imaginary parts of the node voltage; and yes Within a certain time The real and imaginary parts of the node voltage; The constraints of VVO in this invention can be roughly divided into three categories: equality constraints and inequality constraints for power flow calculation, and operating capacity constraints for voltage regulating equipment, the expressions of which are as follows: (3) (4) (5) (6) (7) In the above formula, It is the real part of the admittance matrix; It is the imaginary part of the admittance matrix; and yes t Time Node i The net injection of active and reactive power; and yes t Within a certain time i The real and imaginary parts of the node voltage; and yes t Within a certain time j The real and imaginary parts of the node voltage; yes t Voltage amplitude at time node; and These are the upper and lower limits of the node voltage, set to 1.05 and 0.95 respectively; (8) (9) Formulas (8) and (9) represent the adjustment ranges for reactive power regulation of SVC and PV inverters. and The upper and lower limits of the SVC capacity are defined by these limits. For safety reasons, PV inverters typically have redundant rated capacity and operate in maximum power point tracking (MPPT) mode. Therefore, the reactive power control range of the inverter can be determined by the rated installed capacity of the PV. S PV and current active power output The scope of the decision is as follows: .

[0022] Step 2: We transform the VVO model from Step 1 into a Markov decision process. The state space, action space, reward function, and state transition process of the VVO under the GCN-PPO framework of this invention can be represented as follows: 1) State Space We will vector As t Time Node i The eigenvalues ​​of the nodes are respectively. iThe net injection of active and reactive power, as well as the voltage amplitude, are assumed to be observable for each node. In the GCN-PPO framework of this invention, the feature matrix and connection matrix are used as state variables input into the graph convolutional neural network. t The state space at time t can be represented as: X is the feature matrix composed of the features of all nodes (X has a dimension of 1). A represents the connection relationship of all lines in the power distribution system.

[0023] 2) Action Space The actions of the intelligent agent include the reactive power output of all reactive power regulation equipment in the power distribution network, assuming there are m SVC and n One PV inverter, then .

[0024] 3) Reward Function This invention incorporates voltage exceeding limits as a penalty in the reward function to incentivize the agent to enforce voltage safety constraints during the learning process. t The reward function at time step is: ,in This is the penalty coefficient for voltage exceeding the limit.

[0025] 4) State transition process At each time step t Inside, the agent observes the current state. Then according to Make the current action Finally, a reward value is obtained. And according to P Get the state at the next moment The goal of the intelligent agent is to find a way to maximize the cumulative expected return through the above process. The strategy.

[0026] Step 3: For the models proposed in Steps 1 and 2, this invention proposes a GCN-PPO algorithm.

[0027] The proximal policy optimization (PPO) algorithm is an improvement on trust domain policy optimization. The PPO algorithm uses a Kullback-Leibler (KL) divergence-constrained policy network to make the updated policy close to the old policy. Its optimization objective and constraints are shown in equations (10) and (11): (10) (11) In the formula, This represents the ratio of the old to the new strategies. The old strategy before the update. For policy parameters; KL divergence, also known as relative entropy, is used to measure the difference between probability distributions. This represents the confidence level, used to limit the magnitude of policy updates; Let be the expectation, and let represent the empirical average on a finite sample. Indicating in strategy Down t The advantage function estimate for the decision step. Since calculating the KL divergence in each policy update is computationally very expensive, the PPO algorithm uses a truncation function instead of the KL divergence constraint, which ensures the stability of the TRPO algorithm while reducing computational cost. The objective function of PPO using the truncation function can be expressed as: (12) In the formula, This is a truncation function that controls the changes between the old and new strategies within a certain range. Inside, The truncation constant is used to set the range of policy updates.

[0028] The proposed GCN-PPO algorithm adds a graph convolutional layer before the Multi-Layer Perception (MLP) neural network, improving the PPO agent's ability to perceive graph data. The Actor network architecture consists of two graph convolutional layers and three MLP layers. Each layer has a ReLU activation function, and a pooling function is used to aggregate the outputs of the graph convolutional layers across neighboring nodes before passing them to the MLP layers to output a policy. The Critic network, which defines the value function, has a similar architecture to the Actor network, the main difference being the addition of a global pooling function after its graph convolutional layers. This allows the value function to aggregate information from all nodes in the graph, thereby calculating the estimated value of the entire network.

[0029] Step 4: Train the agent using the GCN-PPO algorithm proposed in Step 3 using real historical operating data of the distribution network. The entire training process is as follows: Figure 2 As shown. During this process, the parameters of the Actor network and the Critic network are continuously updated. After each training round, the network parameters are saved, and then VVO is executed in real time based on the trained Actor model.

[0030] Actor networks are used to represent states. s t Mapping to Action The policy function, whose parameters Typically, parameters are updated based on gradient descent. To improve data efficiency and prevent excessive policy changes, we introduce a truncation function and use equation (12) to update the parameters. and In equation (12) The dominant function is expressed as equation (13): (13) Equation (13) also represents the timing difference error, which is expressed as the error in the state. Next action The advantage outweighs the expected reward value of all actions. Because... It is an immediate reward, and equation (13) can be parameterized into a Critic network to progressively update its parameters, so the parameters of the advantage function... This can be achieved by minimizing equation (14) Update: (14) (15) The specific training and update process is shown in Table 1. Its main process can be described as follows: First, the parameters of the Actor network... and and the parameters of the Critic network The parameters of the old policy are randomly initialized. It was copied from the new strategy. During training, we treated one day of interaction between the agent and the power system environment as a round. T Since the historical data we use is at 3-minute intervals, each interaction is treated as a time step. t ( T =480 t In each round T Internally, the agent first interacts with the environment for 480 steps to form a set of old strategies, and then updates them in each step. t Inside, the Actor, based on the current state Make the corresponding actions Then you get a reward and transition the state Then, the advantage estimate is calculated using equation (13) when the Actor completes... T During the interaction of steps, the parameters of the Actor network Update using equation (16).

[0031] (16) In the formula, It is the learning rate of the Actor network; and we can use it multiple times. TUpdate parameters based on historical experience collected within the step. Similarly, the parameters of the Critic network can be updated using equation (17).

[0032] (17) In the formula, This is the learning rate of the Critic network. In each round... T After the update is complete, assign the parameters of the policy network to the old policy: .

[0033] Table 1 Training process of GCN-PPO algorithm Step 5: After training is completed in step 4, save the trained agent model and then verify its effectiveness and scalability when the distribution network topology changes.

[0034] This invention uses Figure 2 The effectiveness of the proposed method was verified by the IEEE 33-node test system, which has 33 nodes and 37 lines (including 32 normally closed lines and 5 normally open lines). During the simulation phase, the test system used the PyPower toolkit in Python for power flow calculations. The design and training of all DRL agents were completed using the PyTorch toolkit on a computer equipped with 16GB of memory and a 2.50GHz Intel(R) Core(TM) i7-11700 processor.

[0035] The test system consists of four PV units installed at nodes 13, 22, 25, and 33, with installed capacities of 0.8MW, 1.2MW, 1.5MW, and 0.5MW, respectively. Two SVC units are installed at nodes 7 and 18, each with a reactive power compensation capacity of 1Mvar. This invention uses two years of measured distributed PV output and load data from a region in Yunnan Province to train the DRL agent. The output interval between PV and load data is 3 minutes; therefore, the agent will start from 240000 ( The dataset captures the uncertainty of the problem and the dynamic characteristics of the environment. Data from the summer solstice and winter solstice are also selected as the test set to observe the training effect. The PV and load data for these two typical days are as follows: Figure 3 As shown.

[0036] During the offline training phase, the DRL agent was trained for 500 epochs, with 480 steps per epoch. The voltage limit was set to 0.95~1.05, and penalties were imposed for exceeding the voltage limit. If the power flow did not converge during training, a termination instruction was returned. The hyperparameters of the Actor network and Critic network, as well as the weights of the reward function, are shown in Table 2.

[0037] Table 2 Hyperparameter settings of the algorithm To verify the effectiveness of the proposed GCN-PPO algorithm, it was compared with a PPO algorithm that does not use graph neural networks. The Actor and Critic networks of both algorithms used the same design and hyperparameters, and were trained for 500 epochs on the same IEEE-33 node test system. The cumulative reward value and voltage overrun counts for each training epoch are as follows: Figure 4 As shown. Due to the randomness of the DRL algorithm, we conducted three simulation experiments for each algorithm using different random seeds. The average value and error bound of the training results are within... Figure 4 The middle section is represented by a solid line and a filled area.

[0038] At the beginning of training, the randomly initialized parameters caused the agent to exhibit low performance, specifically severe voltage overshooting, resulting in incorrect actions. As training progressed, guided by reward values, the agent began to learn correct strategies from experience. Simultaneously, based on the gradient provided by the loss function, the parameters of the Critic network were continuously optimized, and the loss decreased. After 200 training iterations, the agent could perform correct actions, at which point the reward values ​​and loss began to converge, and the number of voltage overshoots per iteration approached zero. Figure 4 As can be seen, the performance of GCN-PPO after convergence is slightly better than that of PPO algorithm. However, the reward value of GCN-PPO algorithm fluctuates more during training and is not as stable as PPO algorithm.

[0039] To further demonstrate the advantages of the proposed GCN-PPO algorithm in VVO, the trained model was tested on a test set and compared with the centralized control OPF algorithm and the heuristic particle swarm optimization (PSO) algorithm. Figure 5 This paper presents a comparison of active power loss for three algorithms on test sets during the summer and winter solstices. The centralized OPF algorithm, optimized through extensive calculations using precise grid parameters and data, exhibits the best VVO performance. The GCN-PPO algorithm presented in this paper demonstrates similar performance to the OPF algorithm, and under voltage constraints, the GCN-PPO algorithm reduces average grid loss by 37kW and 16.2kW on the summer and winter solstices test sets, respectively, compared to the PSO algorithm.

[0040] To verify the scalability of the proposed algorithm when the distribution network topology changes, this invention saves the trained GCN-PPO and PPO agent models and randomly selects 10 days of PV and load operation data for testing in three designed scenarios. To simulate fault maintenance scenarios, we consider closing tie switches and opening sectionalizing switches as a set of topology transformation methods. Three scenarios are designed based on the degree of topology transformation: 1) Scenario 1 involves closing the tie switch between nodes 12 and 22 and opening the sectionalizing switch between nodes 10 and 11; 2) Scenario 2 involves closing the tie switches between nodes 12 and 22, 8 and 21, and 25 and 29, and then opening the sectionalizing switches between nodes 10 and 11, 4 and 5, and 6 and 26; 3) Close the tie switches between nodes 12 and 22, 8 and 21, 25 and 29, 9 and 15, and 18 and 33, and then open the sectionalizing switches between nodes 10 and 11, 4 and 5, 6 and 26, 8 and 9, and 30 and 31.

[0041] Figure 6 The figure shows the number of voltage violation rates of the GCN-PPO and PPO algorithms over 10 days in three scenarios. As can be seen from the figure, the ordinary PPO agent, due to its lack of consideration for topology connectivity during training, exhibits a continuously increasing voltage violation rate as the distribution network topology changes. In contrast, the proposed GCN-PPO agent maintains good voltage control performance even under significant topology changes.

[0042] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0043] The foregoing has shown and described the basic principles, main features, and advantages of this disclosure. Those skilled in the art should understand that this disclosure is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this disclosure. Various changes and modifications can be made to this disclosure without departing from its spirit and scope, and all such changes and modifications fall within the scope of this disclosure as claimed.

Claims

1. A method for optimizing voltage and reactive power in distribution networks based on graph reinforcement learning, characterized in that, Includes the following steps: A mathematical model for voltage reactive power optimization (VVO) of the distribution network is constructed. The mathematical model of VVO includes the objective function, equality constraints for power flow calculation, and inequality constraints during operation. The constructed VVO mathematical model is transformed into a Markov Decision Process (MDP) model. The state space, action space, reward function, and state transition process of the VVO under the GCN-PPO framework are represented as follows: State space: a vector As t Time Node i The eigenvalues ​​of the nodes are respectively. i The net injection of active and reactive power and voltage amplitude are calculated, and it is assumed that the characteristics of each node are observed. t The state-space representation at time t is: , It is a feature matrix composed of the features of all nodes. It refers to the connection relationships of all lines in the power distribution system; Action Space: The actions of the agent include the reactive power output of all reactive power regulation equipment in the power distribution network, assuming there is... m SVC and n One PV inverter, then ; and for t The reactive power output values ​​of SVC and PV at any given time; Reward function: t The reward function at time step is: ,in This is the voltage over-limit penalty coefficient. It is a power distribution network t Network loss over time It is the network loss cost coefficient; State transition process: at each time step t Inside, the agent observes the current state. Then according to Make the current action Finally, a reward value is obtained. And according to P Get the state at the next moment The goal of an intelligent agent is to find a value that maximizes the cumulative expected return. Strategies; The GCN-PPO algorithm, based on graph convolutional neural networks, is used to solve the MDP model. The GCN-PPO algorithm adds a graph convolutional layer before a multilayer perceptron. The policy Actor network architecture consists of two graph convolutional layers and three multilayer perceptron layers. Each layer has a ReLU activation function, and a summation pooling function is used to aggregate the outputs of the graph convolutional layers on adjacent nodes before passing them to the multilayer perceptron layers to output a policy. The value function is defined in the Critic network architecture, which is similar to the Actor network, but the main difference is that the Critic network architecture adds a global summation pooling function after the graph convolutional layers. The agent of the GCN-PPO algorithm is trained offline using real historical operation data of the distribution network. During this process, the reinforcement learning agent continuously interacts with the simulation environment to capture the intermittency and volatility of photovoltaic PV and load in the historical data. After offline training is completed, the trained agent model is saved, and then the effectiveness of the agent model and its scalability under changes in the distribution network topology are verified.

2. The distribution network voltage reactive power optimization method based on graph reinforcement learning according to claim 1, characterized in that: The VVO mathematical model for constructing the distribution network includes uncertainties in the high PV penetration distribution network, such as the active power output of PV. Active power of the load and the reactive power of the load The control variable during OPF execution is the reactive power output of SVC. and the reactive power output of PV inverters The goal of reactive power optimization is to minimize the active power loss of the distribution network, as shown in the following formula. t Optimization goals for each moment: In the formula, T It refers to the time period being optimized. N It is the number of nodes in the distribution network; F During the time period T Total network loss cost within; It is the real part of the admittance matrix; and yes Within a certain time i The real and imaginary parts of the node voltage; and yes Within a certain time The real and imaginary parts of the node voltage.

3. The distribution network voltage reactive power optimization method based on graph reinforcement learning according to claim 2, characterized in that: The mathematical model constraints of the VVO include: equality and inequality constraints for power flow calculation and operating capacity constraints for voltage regulating equipment, the expressions of which are as follows: In the above formula, It is the real part of the admittance matrix; It is the imaginary part of the admittance matrix; and yes t Time Node i The net injection of active and reactive power; and yes t Within a certain time i The real and imaginary parts of the node voltage; and yes t Within a certain time j The real and imaginary parts of the node voltage; yes t Voltage amplitude at time node and These are the upper and lower limits of the node voltage, set to 1.05 and 0.95 respectively.

4. The distribution network voltage reactive power optimization method based on graph reinforcement learning according to claim 3, characterized in that: and These are the upper and lower limits of SVC capacity. and for t The upper and lower limits of reactive power capacity of PV at any given time; The PV inverter has redundant rated capacity and operates in maximum power point tracking mode; The reactive power control range of the PV inverter is determined by the rated installed capacity of the PV. and current active power output The scope of the decision is as follows: .

5. The distribution network voltage reactive power optimization method based on graph reinforcement learning according to claim 1, characterized in that: The GCN-PPO algorithm is an improvement on the Trust Domain Policy Optimization (TRPO) algorithm. The Trust Domain Policy Optimization (TRPO) algorithm uses a Kolb-Leibran divergence-constrained policy network, and the optimization objective and constraints are shown in the following equations: In the formula, This represents the ratio of the old to the new strategies. The old strategy before the update. For policy parameters; KL divergence, also known as relative entropy, is used to measure the difference between probability distributions. This represents the confidence level, used to limit the magnitude of policy updates; Let be the expectation, and let represent the empirical average on a finite sample. Indicating in strategy Down t The advantage function estimate for the decision step; since the computational cost of calculating the KL divergence in each policy update is very high, the PPO algorithm uses a truncation function instead of the KL divergence constraint, which ensures the stability of the TRPO algorithm and reduces the computational cost; the objective function of PPO using the truncation function is expressed as: In the formula, This is a truncation function that controls the changes between the old and new strategies within a certain range. Inside, The truncation constant is used to set the range of policy updates.

6. The distribution network voltage reactive power optimization method based on graph reinforcement learning according to claim 1, characterized in that: The agent proposing the GCN-PPO algorithm is trained using real historical operating data of the power distribution network. During this process, the parameters of the Actor network and the Critic network are continuously updated. After each training round, the network parameters are saved, and then VVO is executed in real time based on the trained Actor model.

7. An electronic device, characterized in that, include: One or more processors; A memory for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to perform the graph reinforcement learning-based power distribution network voltage reactive power optimization method as described in any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the power distribution network voltage reactive power optimization method based on graph reinforcement learning as described in any one of claims 1-6.

Images