Voltage and reactive power optimization method and system based on time-space aware safety reinforcement learning

Abstract

The present invention discloses a voltage-reactive optimization method and system based on time-space-aware safety reinforcement learning, constructs a PE-VVO framework taking into account photovoltaic PV and electric vehicle charging station EV, and converts it into a constrained Markov decision process; the PE-VVO framework includes an objective function, constraints, and decision variables, and the Markov decision process includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a PDN environment; designs a TGCN-IPO algorithm for solving CMDP, including the algorithm taking into account the goal of the obstacle function and the Actor and Critic network taking into account TGCN; designs the training rules of the TGCN-IPO algorithm, and uses real historical operating data and a PDN simulation environment to train the TGCN-IPO agent; uses a test set to test the trained TGCN-IPO agent, and deploys it online to complete the online voltage-reactive control of the distribution network. This technical solution can solve the problems of increased operating costs and voltage over-limit caused by high-penetration PV and EV to the distribution network.

Description

Technical Field

[0001] The present invention belongs to the fields of intelligent reinforcement learning technology and computer technology, and specifically relates to a voltage-reactive power optimization method and system based on time-space perception safety reinforcement learning. Background Art

[0002] The integration of large-scale distributed photovoltaic (PV) and electric vehicles (EV) will change the operation mode of the distribution network (PDN), resulting in frequent node voltage violations in the PDN and increasing the power loss of the network. Studies have shown that 50% EV or PV penetration will cause about 50% of the distribution transformers to be overloaded and cause more than 25% energy loss to the feeder. In order to cope with this situation, it is crucial to develop an efficient voltage and reactive power optimization (VVO) method. Due to the influence of weather factors, PV power generation exhibits strong intermittency and rapid fluctuations. At the same time, unlike traditional loads, EV is a flexible load and the user's charging and discharging behavior is highly uncertain. The aggregation of large-scale EVs may cause the PDN power to increase or decrease sharply. The frequent voltage violations caused by PV and EV increase the operating burden of traditional mechanical voltage regulation equipment, such as on-load tap changers (OLTCs) and shunt capacitors (SCBs), which in turn reduces their lifespan. Therefore, more and more research focuses on fast reactive power compensation based on power electronic devices, and VVO tasks are completed by dispatching PV and electric vehicle charging station (EVS) smart inverters. The present invention calls it PE-VVO, which has the advantages of fast dynamic response, high control accuracy, and integration with distributed resources. When there is no EV access, EVS can still interact with the grid through the inverter. Therefore, the main purpose of the present invention is to complete the proposed PE-VVO task while fully considering the uncertainty of PV and EVS.

[0003] Different from this deterministic optimization task, the PE-VVO task of a high-penetration distribution network of distributed PV and EV has three challenges: 1) building a probability distribution model that can describe the travel behavior and charging and discharging scheduling of EV users; 2) building a probability distribution model that can predict the uncertainty of PV, EVS, and load generation / consumption in the distribution network; 3) building a real-time decision model that can quickly respond to PV and EV fluctuations. Model-free deep reinforcement learning (DRL) methods have received widespread attention in recent years. Unlike model predictive control that is highly dependent on accurate models, model-free DRL can continuously learn and adjust strategies directly in the process of interacting with the environment, and use parameterized deep neural networks to approximate high-dimensional complex nonlinear systems. Compared with stochastic programming and robust optimization methods, it can better adapt to changes and uncertainties in the environment. Summary of the invention

[0004] The purpose of the present invention is to provide a voltage-reactive power optimization method and system based on spatiotemporal-aware safety reinforcement learning, which can solve the problems of increased operating costs and voltage over-limit caused by high penetration PV and EV to the distribution network.

[0005] In order to achieve the above object, the solution of the present invention is:

[0006] A voltage-reactive power optimization method based on time-space-aware safety reinforcement learning includes the following steps:

[0007] Step 1, construct a distribution network voltage reactive power optimization framework taking into account photovoltaic PV and electric vehicle charging station EV, that is, PE-VVO framework, and convert the PE-VVO problem into a constrained Markov decision process CMDP; wherein the PE-VVO framework includes an objective function, constraints, and decision variables, and the Markov decision process includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a distribution network environment;

[0008] Step 2, design the TGCN-IPO algorithm for solving CMDP, including the algorithm's goal of taking into account the obstacle function and the Actor and Critic networks taking into account the spatiotemporal graph convolutional neural network TGCN;

[0009] Step 3: Design the training rules of the TGCN-IPO algorithm and use real historical operation data and the PDN simulation environment to train the TGCN-IPO agent;

[0010] Step 4: Use the test set to test the trained TGCN-IPO agent and deploy it online to complete the online voltage and reactive power control of the distribution network.

[0011] In the above step 1, the decision variables of the PE-VVO framework are the reactive power output of PV and EV, the objective function of the PE-VVO framework is to minimize the active power loss of PDN, and the constraints of the PE-VVO framework are power balance constraints and node voltage constraints.

[0012] In step 1 above, the state space of CMDP is Status t Reflecting the spatiotemporal perception of the deep reinforcement learning DRL agent on the distribution network operation status at time t, let X t =(P t ,Q t ,V t ,A) is the perceived spatial information, P t ,Q t and V tare the active power net injection vector, reactive power net injection vector and node voltage amplitude vector respectively, A is the connection matrix representing the connection relationship of each node in the distribution network; X composed of the past w steps is used t Move the window to represent s t , according to s t =(X t ,X t-1 ,…,X t-w+1 ) can infer the future trends of various electrical quantities and thus describe the timing characteristics of the distribution network.

[0013] In step 1 above, the action space of CMDP is represents the reactive output action of the DRL agent at time t, where m represents the number of PVs and n represents the number of EVSs; is the reactive power output of PV at node i at time t, is the reactive power output of EV at node i at time t, and their output range is: and In the formula is the capacity of PV at node i at time t, is the active power output of PV at node i at time t, is the capacity of EV at node i at time t, is the active power output of EV at node i at time t.

[0014] In step 1 above, the reward function of CMDP is in, is the state space, is the action space, Indicates taking action The reward gained from moving from one state to another; represents the reward value of the DRL agent at time t, where l ij,t is the square of the branch current between nodes i and j at time t, r ij is the branch resistance between nodes i and j, ε represents the number of branches, C p is the cost of active power loss.

[0015] In step 1 above, the constraints of CMDP are Represents the set of all nodes in the distribution network; V max ,V min are the upper and lower limits of the node voltage respectively.

[0016] In step 1 above, the state transition probability function of CMDP is defined as in, represents the DRL agent taking action a at time t t When the state changes from s t Transfer to t+1 The probability here represents power flow calculation; is the state space, For the action space.

[0017] Among them, the TGCN-IPO algorithm for solving CMDP is designed, including the algorithm's objective of taking into account the obstacle function and the T-GCN network.

[0018] In step 2 above, the goal of CMDP is to find a strategy π that maximizes the discounted cumulative reward while satisfying the cumulative constraint. θ , expressed as:

[0019]

[0020]

[0021] Among them, τ~π θ Indicates that the distribution on the experience trajectory follows the strategy π θ , Represents strategy π θ The feasible domain of , T is the length of the control horizon, represents the cumulative discounted cost of the policy π with respect to the constraints ξ represents the limit of the constraint, the expectation It is for all random sources;

[0022] The interior point strategy optimization algorithm is used to solve the CMDP framework. For those with In this paper, we expand an indicator function I(·) in the objective function to transform it into an unconstrained optimization problem. It is expressed as:

[0023]

[0024] That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when the constraints are violated, a negative value -∞ is introduced into the objective function, and the strategy is adjusted to meet the constraints;

[0025] We approximate I(·) using a logarithmic barrier function φ(·), which is defined as:

[0026]

[0027] Among them, q>0 is a hyperparameter; Based on the above description, the goal of IPO is summarized as:

[0028]

[0029] It consists of two parts: the proxy objective L pruned by PPO PPO and the logarithmic barrier function φ(·);

[0030] The T-GCN network is used as the neural network. T-GCN combines the graph convolutional network GCN and the gated recurrent unit GRU. GCN is used to learn the topological structure of the distribution network and the complex spatial distribution of power flow to capture spatial dependencies. GRU is used to learn the time series change characteristics of PV, EV, and basic load power generation / consumption data to capture time dependencies.

[0031] The actor and critic networks of the TGCN-IPO agent are composed of a T-GCN network and a fully connected network. The agent captures the graph data with time series characteristics of the distribution network through T-GCN and inputs it into the neural network; the T-GCN network contains several T-GCN units, each T-GCN unit includes an input layer, a spatial feature extraction layer, a temporal feature extraction layer and an output layer, the fully connected network includes several hidden layers and an output layer, and the output layer of the T-GCN network is the input layer of the fully connected network.

[0032] A voltage and reactive power optimization system based on time-space perception safety reinforcement learning, comprising:

[0033] The CMDP design module of PE-VVO is configured to construct a voltage and reactive power optimization framework of a distribution network taking into account photovoltaic PV and electric vehicle charging stations EV, namely, a PE-VVO framework, and convert the PE-VVO problem into a constrained Markov decision process; wherein the PE-VVO framework includes an objective function, constraints, and decision variables, and the Markov decision process includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a PDN environment;

[0034] The secure reinforcement learning algorithm TCGN-IPO module is configured to design the TGCN-IPO algorithm for solving CMDP, including the algorithm's objective taking into account the obstacle function and the actor and critic network taking into account the TGCN;

[0035] An agent training module, configured to design training rules for the TGCN-IPO algorithm, and train the TGCN-IPO agent using real historical operation data and a PDN simulation environment; and,

[0036] The online decision-making module is configured to test the trained TGCN-IPO agent using the test set and deploy it online to complete the online voltage and reactive power control of the distribution network.

[0037] After adopting the above scheme, the present invention proposes a PE-VVO method based on model-free safety reinforcement learning (SDRL) to solve the research gap in the reactive power optimization of distribution networks based on the DRL algorithm. The method adopts an advanced interior point policy optimization (IPO) algorithm, and on this basis, proposes a spatiotemporal graph convolutional neural network (T-GCN) architecture to increase the perception of the spatiotemporal distribution characteristics of the distribution network, called TGCN-IPO. The present invention combines the spatiotemporal perception type safety reinforcement learning algorithm with the voltage reactive power control of the distribution network, and realizes the online voltage control of the distribution network through the reinforcement learning algorithm. Compared with other methods, the proposed scheme can capture the multiple uncertainties of the distribution network from the historical operation data; the proposed TGCN-IPO algorithm can realize the constraints related to the behavior without relying on the specified penalty terms and adjusting the related penalty factors. This method simplifies the optimization process, reduces computational costs, and improves the performance and efficiency of the algorithm. At the same time, the network structure of the algorithm combines graph convolutional neural networks and gated recurrent units. This structure enables the intelligent agent to more effectively perceive the spatiotemporal characteristics of the distribution network, including the spatial topological structure and the timing characteristics of PV, EV and load, thereby improving the robustness and scalability of the intelligent agent and providing more accurate decision support for distribution network optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic flow chart of a voltage-reactive power optimization method based on time-space-aware safety reinforcement learning provided by an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of the structure of a TGCN-IPO algorithm provided by an embodiment of the present invention;

[0040] Figure 3 is a schematic diagram of an IEEE 33-bus test system provided by an embodiment of the present invention;

[0041] Figure 4 The average network loss cost and the cumulative number of voltage-exceeding nodes in each round of the training process provided by an embodiment of the present invention;

[0042] Figure 5 The voltage amplitude of the bus 17 and the active power loss of the PDN in one day under different schemes provided by an embodiment of the present invention;

[0043] Figure 6 It is a flow chart of a voltage and reactive power optimization system based on time-space awareness safety reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0047] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0048] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0049] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0050] Example 1

[0051] Reference Figure 1-2 , which is the first embodiment of the present invention, and provides a voltage-reactive power optimization method based on time-space-aware safety reinforcement learning, including:

[0052] S1: Construct a distribution network voltage and reactive power optimization framework taking into account PV and EV, namely the PE-VVO framework, and convert the PE-VVO problem into a constrained Markov decision process (CMDP); wherein: the PE-VVO framework includes an objective function, constraints, and decision variables, and the CMDP includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a PDN environment;

[0053] The distribution network usually presents a radial network topology during operation. To represent a distribution network, and ε represent the set of all nodes and lines in the distribution network respectively. i Indicates bus voltage, P i / Q i Indicates the active / reactive power injected into the bus. In a PDN with high penetration of PV and EV, the power injected into the node includes the active / reactive power of PV, EV and basic load, which are represented by P PV / Q PV ,P EV / Q EV ,P L / Q L Please note that the EV here refers to the EV charging station behind the electric vehicle cluster with inverter.

[0054] The decision variables of the PE-VVO framework are the reactive power output of PV and EV. The reactive power output of node i at time t is expressed as and Indicates that and The range is calculated based on the active power output and capacity of PV and EV:

[0055]

[0056]

[0057] In the formula, and is the capacity of PV and EV, and is the active power output of node i at time t;

[0058] The objective function of the PE-VVO framework is to minimize the PDN active power loss, as shown below:

[0059]

[0060] In the formula, l ij,t is the square of the branch current between nodes i and j at time t, r ij is the branch resistance, and They represent the reactive power output of PV and EV at node j at time t, T represents the optimized time period, and ε represents the number of branches.

[0061] The constraints of the PE-VVO framework are power balance constraints and node voltage constraints:

[0062]

[0063]

[0064]

[0065] Where P PV ,P EV ,P L and Q PV ,Q EV ,Q L Represent the active and reactive power outputs of PV, EV, and load injected into the node, respectively, v i,t represents the square of the node voltage amplitude at node i at time t; x and r represent the branch reactance and resistance, where the subscript jk represents the branch between node j and node k; P ij and Q ij Represents the active and reactive power output on branch ij, V min and V max Represents the upper and lower limits of the node voltage;

[0066] In this embodiment, DSO is regarded as a DRL agent, and the above VVO task is expressed as a Constrained Markov Decision Process (CMDP). In CMDP, is the state space, is the action space, is the reward function, Indicates taking action The reward obtained from moving from one state to another; is the constraint function, is the state transition probability distribution, γ∈[0,1] is the discount factor of future rewards, which represents the attention to future rewards.

[0067] State space: In the problem under study, the state s t It reflects the spatiotemporal perception of the DRL agent on the operating status of the distribution network at time t. Let Xt =(P t ,Q t ,V t ,A) is the perceived spatial information, where P t ,Q t and V t are respectively the active power net injection vector, the reactive power net injection vector, the node voltage amplitude vector, and A is the connection matrix representing the connection relationship (topological structure) of each node in the distribution network. We use the X composed of the past w steps t Move the window to represent s t , therefore, s t =(X t ,X t-1 ,…,X t-w+1 ) can infer the future trends of various electrical quantities, thereby better describing the timing characteristics of the distribution network.

[0068] Action space: The action of the agent includes the reactive output of all PVs and EVSs in the distribution network. Assume there are m PVs and n EVSs.

[0069] Reward function: The objective function is the same as that of PE-VVO. The goal is to minimize the network loss during operation. Therefore, the reward function at time t is: here C p is the cost of active power loss.

[0070] Constraints: The constraints of the agent are to ensure that the voltage does not violate the rules during operation. Here we set the upper and lower limits of the voltage to [0.95, 1.05].

[0071] State transition probability distribution: In an MDP, the state transition probability function is usually defined as in, Represents the agent taking action a at time t t When the state changes from s t Transfer to t+1 The probability of The state of the defined DRL agent includes the power generation / consumption behavior of PV, EV and load, and their changes are random (theoretically, they can be modeled as any probability distribution), while the state transition probability distribution is used in MDP to characterize this uncertainty.

[0072] Environment: The distribution network environment is built using the pypower toolkit. At each decision step, the distribution network will inject variables (These electrical quantities can be obtained through supervisory control and data acquisition (SCADA)). These variables also lead to changes in node voltage and active network loss. The DRL agent is based on the state s at time t. t Take corresponding actions t Input into the distribution network simulation environment, and then perform power flow calculation to obtain s t+1 and new power flow distribution results.

[0073] S2: Design the TGCN-IPO algorithm for solving CMDP, including the algorithm's goal of taking into account the obstacle function and the actor and critic network taking into account the TGCN;

[0074] For CMDP, the goal is to find a strategy π that maximizes the discounted cumulative reward while satisfying the cumulative constraint θ . It is expressed as:

[0075]

[0076]

[0077] In the formula, τ~π θ Denotes that the distribution on the experience trajectory follows the strategy π θ , Represents strategy π θ The feasible domain of , T is the length of the control horizon, represents the cumulative discounted cost of the policy π with respect to the constraints ξ represents the limit of the constraint, the expectation It is for all random sources;

[0078] The common policy-based DRL algorithm directly optimizes in the policy space and updates the policy by estimating the gradient of the cumulative reward relative to the policy parameters. The process can be described by the following formula:

[0079]

[0080] A t =δ t +γδ t+1 +…+γ T-t+1 δ T-1

[0081]

[0082] In the above formula, π θ is the current strategy under parameter θ, A t is the advantage function at time step t, δ t is the differential error, V μ (s t) is the value function, and here we use a parameterized Critic network to express the value function;

[0083] The popular PPO algorithm uses a clipped objective function to limit policy updates, ensuring training stability and improving computational efficiency. The objective function of the PPO algorithm is as follows:

[0084]

[0085] In the formula, represents the ratio of the new and old strategies, clip(·) is the cutoff function, which controls the change of the new and old strategies within [1-ε,1+ε], and ε is the cutoff constant used to set the range of strategy update;

[0086] In order to solve the CMDP framework and ensure the security during the policy update process, the present invention uses an interior point policy optimization algorithm, which uses the PPO objective instead of the CMDP objective and uses the logarithmic barrier function in the interior point method to handle constraints. For those with In order to solve the problem of constraints, an indicator function I(·) is added to the objective function to transform it into an unconstrained optimization problem. can be expressed as:

[0087]

[0088] That is, when the constraints are satisfied, the problem is solved as an unconstrained policy optimization problem that only considers rewards; when the constraints are violated, a negative value -∞ (i.e., penalty) is introduced into the objective function, and the policy must be adjusted to satisfy the constraints. In order to better implement the above indicator function in the optimization objective, we use the logarithmic barrier function φ(·) to approximate I(·), which is defined as:

[0089]

[0090] Among them, q>0 is a hyperparameter. It can be seen that as q increases, φ(·) becomes closer to I(·). In addition, the advantage of using the logarithmic barrier function in the above formula is that it is first-order differentiable (while I(·) is not differentiable at the origin). Based on the above description, the goal of IPO can be summarized as:

[0091]

[0092] It consists of two parts: the proxy objective L pruned by PPO PPO and the logarithmic barrier function φ(·).

[0093] In order to enhance the IPO algorithm's perception of the spatiotemporal characteristics of the distribution network environment, a T-GCN network is proposed to replace the original fully connected layer neural network. T-GCN combines the graph convolutional network (GCN) and the gated recurrent unit (GRU). GCN is used to learn the topological structure and complex spatial distribution of the distribution network to capture spatial dependencies; GRU is used to learn the temporal variation characteristics of PV, EV, and basic load power generation / consumption data to capture temporal dependencies. The working principles of GCN, GRU, and T-GCN are introduced below.

[0094] At present, most graph neural network models can be regarded as a method of learning a "permutation invariant function", whose input is a feature matrix X and a connection matrix A. X describes the characteristics of each node, and A represents the connection relationship between nodes. The present invention uses a function f(X, A) described by a graph convolution operator for effective information transfer. The following formula describes the information transfer rule of GCN.

[0095]

[0096] In the formula, (I is the identity matrix), yes The diagonal matrix of , σ(·) represents the activation function, and W is the parameter matrix of the fully connected layer. In summary, GCN describes a permutation-invariant propagation rule that updates the features of a node by aggregating information from neighboring nodes.

[0097] GRU uses a gating mechanism to memorize long-term information, which can solve the gradient explosion and gradient vanishing problems in recurrent neural networks (RNNs). GRU has a relatively simple structure, smaller parameters, and is easier to train than another variant of RNN, the long short-term memory neural network. GRU introduces a gate mechanism to control the flow of information, thereby better learning and memorizing long-range dependencies. t The degree of forgetting the previous state information can be controlled by updating the gate z t It is used to control the extent to which the state information of the previous moment is brought into the current state, c t is the memory content stored at time t;

[0098] The present invention integrates the above-mentioned T-GCN network into the IPO algorithm. The actor and critic networks of the TGCN-IPO agent are composed of T-GCN units and fully connected layers, which enables the agent to capture the graph data with time series characteristics of the distribution network through T-GCN and input it into the neural network, rather than simply inputting the electrical quantity in the form of a one-dimensional vector into the neural network to train the agent. Figure 2The overall architecture of the TGCN-IPO algorithm is shown. From the figure, it can be seen that the neural network of the TGCN-IPO agent consists of two parts. The front part (the upper part in the figure) is composed of the T-GCN network, which is composed of several T-GCN units, including the input layer, spatial feature extraction layer, temporal feature extraction layer and output layer. The back part (the lower part in the figure) is composed of ordinary fully connected layers, including several hidden layers and output layers. The output layer of the T-GCN network is the input layer of the fully connected network.

[0099] S3: Design the training rules of the TGCN-IPO algorithm and use real historical operation data and PDN simulation environment to train the TGCN-IPO agent;

[0100] The process of solving PE-VVO using the proposed TGCN-IPO method can be described by Algorithm 1 in the following table. In the exploration phase (Algorithm 1: 4-8), with strategy π θ The agent continuously interacts with historical operation data and the distribution network environment to generate experience trajectories. In the training phase (Algorithm 1:9-12), the TGCN-IPO agent updates the parameters of the actor and critic networks based on the experience trajectories. This is also the process of capturing system uncertainty and learning the best strategy from the experience trajectories, and finally fitting this strategy into a parameterized neural network.

[0101]

[0102] S4: Use the test set to test the trained TGCN-IPO agent and deploy it online to complete the online voltage and reactive power control of the distribution network;

[0103] Example 2

[0104] Reference Figure 3-5 , which is an embodiment of the present invention, provides a voltage-reactive power optimization method based on time-space perception safety reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0105] use Figure 3 The proposed TGCN-PPO agent is trained based on the IEEE33-node test system and the real PV, EV and load data of a certain area. In the IEEE33-node system, three EVSs are connected to bus 13, 22, and 24, and three PVs are connected to bus 17, 25, and 30. A certain load is aggregated on each bus, including residential users and industrial users. 365 days of historical operating data with a resolution of 3 minutes are selected to train and test the DRL agent.

[0106] In this embodiment, the DRL agent is trained for 3000 rounds. Each round randomly selects one day of data from the training set for interaction, including 480 decision steps. The timestamp of each decision step is 3 minutes, corresponding to the data resolution. During the training process, the limit of each bus voltage is set to [0.95, 1.05]. Due to the presence of large industrial loads and EVS in the PDN, severe undervoltage is prone to occur. We increase the voltage of the head-end transformer node to 1.025pu. The hyperparameter settings of the proposed algorithm are shown in Table 1.

[0107] Table 1 Algorithm hyperparameter settings

[0108]

[0109] In this example, TGCN-IPO, IPO and Proximal Policy Optimization (PPO) are trained on the same dataset to compare their effects. For fairness, the same parameters are set for the actor and critic networks of the three algorithms. The difference is that a penalty term is added to the reward function of the PPO algorithm. σ(·) is a 0-1 function, which is 1 if the voltage of node i does not satisfy the constraint, and the penalty factor η is set to -20. Figure 4 The average network loss cost and the cumulative number of voltage-exceeding nodes in each round during the training process are shown, where the shaded part is the fluctuation of the round return value and the solid line is the average sliding value.

[0110] like Figure 4 As shown in the figure, the rewards of the DRL agent are low and unstable in the early stage of training. This is the result of the random initialization of the neural network parameters, which leads to poor actions of the agent. In each round, the agent updates its parameters according to the experience trajectory. Due to the introduction of the barrier function and the penalty term, the agent gradually learns the strategy to meet the constraints, and continuously learns the best strategy under the feedback of the average cumulative discounted return. After the training becomes stable, the proposed TGCN-IPO algorithm has a higher average reward per round and a lower voltage violation rate, but due to its more complex network structure, the convergence speed is slower than the IPO algorithm. At the same time, the algorithm based on the CMDP framework is more stable in satisfying constraints. This is because the PPO agent has to constantly weigh the reward and penalty terms during the training process, so it cannot focus on learning the best strategy.

[0111] This example selects a typical day to verify the effect of the proposed method and visualize the test results. In order to compare the advantages of the TGCN-IPO algorithm, the IPO and PPO models trained in the previous section are also tested. At the same time, a solution based on the second-order cone programming method (a model-based method) is implemented. This solution assumes that all PV, EVS, power generation and power consumption are known in advance, and uses a commercial solver to continuously iterate and solve, so this solution can be regarded as an optimal solution model. Figure 5 The voltage amplitude of bus 17 and the active power loss of PDN in one day under four schemes and without any control measures are shown.

[0112] like Figure 5 As shown in (a), since bus 17 is an industrial load and is affected by PV and EVS, serious voltage violations will occur at noon and at night. The solution of SOCP strictly follows the defined constraints. The proposed TGCN-IPO algorithm shows similar performance to SOCP and can adjust the voltage within the limit. Figure 5 (b) shows the network loss under four schemes. The proposed TGCN-IPO method optimizes the active power loss by 28.69% better than the IPO method and is very close to the optimal solution of SOCP. As can be seen from the figure, the PPO method optimizes the voltage to a position farther away from the limit, but the excessive reactive power injection also increases the active power loss. This is because the actions taken are more inclined to satisfy the voltage constraint and ignore the active power loss.

[0113] Example 3

[0114] The third embodiment of the present invention is different from the first two embodiments in that:

[0115] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0116] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0117] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0118] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] Example 4

[0120] Reference Figure 6 , which is the fourth embodiment of the present invention, and provides a voltage-reactive power optimization system based on time-space perception safety reinforcement learning, which can improve the voltage stability of the distribution network, reduce energy loss and equipment overload operation, and also make the voltage-reactive power control of the distribution network more intelligent and adaptive, including, PE-VVO CMDP design module, safety reinforcement learning algorithm TCGN-IPO module, intelligent agent training module, online decision module;

[0121] The CMDP design module of the PE-VVO is responsible for constructing a voltage and reactive power control model suitable for a specific distribution network according to the topological structure, equipment parameters, and load information of the distribution network, including design objective functions, decision variables, and constraints; the CMDP model includes designing state space, action space, reward function, constraints, state transition probability distribution, and intelligent agent training environment;

[0122] The safety reinforcement learning algorithm TCGN-IPO module is responsible for designing the TCGN-IPO algorithm for solving according to the voltage and reactive power control model and CMDP model of the distribution network, mainly including the design of the IPO algorithm based on the logarithmic barrier function and the TGCN network;

[0123] The intelligent agent training module includes designing training rules for the intelligent agent and collecting historical operation data of the distribution network to train the intelligent agent;

[0124] The online decision-making module includes testing the trained TGCN-IPO agent using a test set, and then deploying it online. By inputting the observed state of the distribution network, including voltage, power, and current information, a decision instruction for voltage and reactive power control of the distribution network is obtained, thereby realizing online voltage and reactive power control of the distribution network.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A voltage and reactive power optimization method based on time-space perception safety reinforcement learning, characterized in that The steps include: Step 1, construct a distribution network voltage reactive power optimization framework taking into account photovoltaic PV and electric vehicle charging station EV, that is, PE-VVO framework, and convert the PE-VVO problem into a constrained Markov decision process CMDP; wherein the PE-VVO framework includes an objective function, constraints, and decision variables, and the Markov decision process includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a distribution network environment; Step 2, design the TGCN-IPO algorithm for solving CMDP, including the algorithm's goal of taking into account the obstacle function and the Actor and Critic networks taking into account the spatiotemporal graph convolutional neural network TGCN; Among them, the TGCN-IPO algorithm for solving CMDP is designed, including the algorithm's goal of taking into account the obstacle function and the T-GCN network; In step 2, the goal of CMDP is to find a strategy π that maximizes the discounted cumulative reward while satisfying the cumulative constraint. θ , expressed as: Among them, t~π θ Denotes that the distribution on the experience trajectory follows the strategy π θ , P c Represents strategy π θ The feasible domain of T is the length of the control horizon, J c (π θ ) represents the cumulative discounted cost of strategy π with respect to the constraints x represents the limit of the constraint, the expectation is for all random sources, γ∈[0,1] is the discount factor for future rewards; The interior point strategy optimization algorithm is used to solve the CMDP framework. For those with constrained problem, an indicator function I(·) is expanded in the objective function to transform it into an unconstrained optimization problem; the indicator function It is expressed as: That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when the constraints are violated, a negative value -∞ is introduced into the objective function, and the strategy is adjusted to meet the constraints; We approximate I(·) using a logarithmic barrier function φ(·), which is defined as: Among them, q>0 is a hyperparameter; Based on the above description, the goal of IPO is summarized as: It consists of two parts: the proxy objective L pruned by PPO PPO and the logarithmic barrier function φ(·); The T-GCN network is used as the neural network. T-GCN combines the graph convolutional network GCN and the gated recurrent unit GRU. GCN is used to learn the topological structure of the distribution network and the complex spatial distribution of power flow to capture spatial dependencies. GRU is used to learn the time series change characteristics of PV, EV, and basic load power generation / consumption data to capture time dependencies. The actor and critic networks of the TGCN-IPO agent are composed of a T-GCN network and a fully connected network. The agent captures the graph data with time series characteristics of the distribution network through T-GCN and inputs it into the neural network; the T-GCN network contains several T-GCN units, each of which includes an input layer, a spatial feature extraction layer, a temporal feature extraction layer and an output layer. The fully connected network includes several hidden layers and an output layer, and the output layer of the T-GCN network is the input layer of the fully connected network; Step 3: Design the training rules of the TGCN-IPO algorithm and use real historical operation data and the PDN simulation environment to train the TGCN-IPO agent; Step 4: Use the test set to test the trained TGCN-IPO agent and deploy it online to complete the online voltage and reactive power control of the distribution network.

2. The method according to claim 1, characterized in that: In step 1, the decision variables of the PE-VVO framework are the reactive power output of PV and EV, the objective function of the PE-VVO framework is to minimize the active power loss of PDN, and the constraints of the PE-VVO framework are power balance constraints and node voltage constraints.

3. The method according to claim 1, characterized in that: In step 1, the state space of CMDP is Status t Reflecting the spatiotemporal perception of the deep reinforcement learning DRL agent on the distribution network operation status at time t, let X t =(P t ,Q t ,V t ,A) is the perceived spatial information, P t ,Q t and V t are the active power net injection vector, reactive power net injection vector and node voltage amplitude vector respectively, A is the connection matrix representing the connection relationship of each node in the distribution network; X composed of the past w steps is used t Move the window to represent s t , according to s t =(X t ,X t-1 ,…,X t-w+1 ) can infer the future trends of various electrical quantities and thus describe the timing characteristics of the distribution network.

4. The method according to claim 1, characterized in that: In step 1, the action space of CMDP is i∈n,j∈m represents the reactive output action of the DRL agent at time t, where m represents the number of PVs and n represents the number of EVSs; is the reactive power output of PV at node i at time t, is the reactive power output of EV at node i at time t, and their output range is: and In the formula is the capacity of PV at node i at time t, is the active power output of PV at node i at time t, is the capacity of EV at node i at time t, is the active power output of EV at node i at time t.

5. The method according to claim 1, characterized in that: In step 1, the reward function of CMDP is in, is the state space, is the action space, Indicates taking action The reward obtained from moving from one state to another; represents the reward value of the DRL agent at time t, where is the square of the branch current between nodes i and j at time t, r ij is the branch resistance between nodes i and j, ε represents the number of branches, C p is the cost of active power loss.

6. The method according to claim 1, characterized in that: In step 1, the constraint of CMDP is V min ≤V i ≤V max , Represents the set of all nodes in the distribution network; V max ,V min are the upper and lower limits of the node voltage respectively.

7. The method according to claim 1, characterized in that: In step 1, the state transition probability function of CMDP is defined as in, represents the DRL agent taking action a at time t t When the state changes from s t Transfer to t+1 The probability here ands t , represents power flow calculation; is the state space, For the action space.

8. A voltage and reactive power optimization system based on time-space perception safety reinforcement learning, characterized by: include, The CMDP design module of PE-VVO is configured to construct a voltage and reactive power optimization framework of a distribution network taking into account photovoltaic PV and electric vehicle charging stations EV, namely, a PE-VVO framework, and convert the PE-VVO problem into a constrained Markov decision process; wherein the PE-VVO framework includes an objective function, constraints, and decision variables, and the Markov decision process includes a state space, an action space, a reward function, constraints, a state transition probability distribution, and a PDN environment; The secure reinforcement learning algorithm TCGN-IPO module is configured to design the TGCN-IPO algorithm for solving CMDP, including the algorithm's objective taking into account the obstacle function and the actor and critic network taking into account the TGCN; An agent training module, configured to design training rules for the TGCN-IPO algorithm, and train the TGCN-IPO agent using real historical operation data and a PDN simulation environment; and, The online decision-making module is configured to test the trained TGCN-IPO agent using the test set and deploy it online to complete the online voltage and reactive power control of the distribution network; Among them, the secure reinforcement learning algorithm TCGN-IPO module designs the TGCN-IPO algorithm for solving CMDP, including the algorithm's objectives taking into account the obstacle function and the T-GCN network; The goal of CMDP is to find a strategy π that maximizes the discounted cumulative reward while satisfying the cumulative constraint. θ , expressed as: Among them, t~π θ Denotes that the distribution on the experience trajectory follows the strategy π θ , P c Represents strategy π θ The feasible domain of T is the length of the control horizon, J c (π θ ) represents the cumulative discounted cost of strategy π with respect to the constraints x represents the limit of the constraint, the expectation is for all random sources, γ∈[0,1] is the discount factor for future rewards; The interior point strategy optimization algorithm is used to solve the CMDP framework. For those with constrained problem, an indicator function I(·) is expanded in the objective function to transform it into an unconstrained optimization problem; the indicator function It is expressed as: That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when the constraints are violated, a negative value -∞ is introduced into the objective function, and the strategy is adjusted to meet the constraints; We approximate I(·) using a logarithmic barrier function φ(·), which is defined as: Among them, q>0 is a hyperparameter; Based on the above description, the goal of IPO is summarized as: It consists of two parts: the proxy objective L pruned by PPO PPO and the logarithmic barrier function φ(·); The T-GCN network is used as the neural network. T-GCN combines the graph convolutional network GCN and the gated recurrent unit GRU. GCN is used to learn the topological structure of the distribution network and the complex spatial distribution of power flow to capture spatial dependencies. GRU is used to learn the time series change characteristics of PV, EV, and basic load power generation / consumption data to capture time dependencies. The actor and critic networks of the TGCN-IPO agent are composed of a T-GCN network and a fully connected network. The agent captures the graph data with time series characteristics of the distribution network through T-GCN and inputs it into the neural network; the T-GCN network contains several T-GCN units, each T-GCN unit includes an input layer, a spatial feature extraction layer, a temporal feature extraction layer and an output layer, the fully connected network includes several hidden layers and an output layer, and the output layer of the T-GCN network is the input layer of the fully connected network.

Images