Power distribution network dynamic network reconstruction method based on constraint enhanced safety reinforcement learning

Abstract

The invention provides a power distribution network dynamic network reconstruction method based on constraint enhanced safety reinforcement learning, and the method comprises the steps: describing a DDNR task of a power distribution network as a constraint type Markov decision process (CMDP); then, soft constraints (node voltage and line current constraints) and hard constraints (radial topology constraints) in the DDNR process are processed by adopting an interior point strategy optimization (IPO) algorithm and a mode of adding a safety layer based on a branch switching method respectively; meanwhile, the invention provides a space-time perception neural network model to increase the dynamic space-time change characteristics of the reinforcement learning agent to the power flow of the power distribution network in the DDNR process. The method plays an important role in reducing the energy loss of the power distribution network, optimizing the voltage distribution and increasing the bearing capacity of new energy.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent reinforcement learning and the technical field of new power systems, and specifically relates to a distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning. Background Art

[0002] The power of distributed renewable energy (DRE) such as photovoltaic (PV) and wind power (WP) is highly intermittent and volatile. Their high proportion of access increases the uncertainty of power flow state transfer in distribution network (PDN) and the complexity of optimization problems. Network reconstruction technology can achieve spatial energy transfer by changing the state of segment switches and tie switches in PDN. With the increasing penetration of remote control switches and DRE, dynamic distribution network reconstruction (DDNR) plays an important role in improving the carrying capacity of DRE, reducing PDN energy loss, and improving voltage distribution.

[0003] DDNR can be described as a mixed integer nonlinear programming problem, where the integer variables represent the state of the remote control switch. There are two challenges in the current research on the DDNR problem. First, when solving the DDNR problem, a random probability distribution that can accurately describe the uncertainty of DRE power and PDN power flow state transition is constructed; second, DDRN is a typical sequential decision problem. The distribution operator (DSO) needs to solve the switch state of multiple decision steps while predicting future uncertainties, which greatly increases the computational burden of the problem.

[0004] At present, researchers focus on the study of three types of methods: mathematical programming methods, heuristic methods, and learning-based methods. In order to deal with the uncertainty of DRE, the random programming method first constructs the uncertainty scenario, and then constructs the DDNR as a mixed integer second-order cone programming model for solution. The robust optimization method achieves the minimum network loss reconstruction of DRE high-penetration PDN by constructing an uncertainty set. The heuristic method uses a simple step-by-step search process to analyze possible solutions and logically selects high-quality solutions. Among them, branch switching is a common heuristic method. The branch switching strategy is used to minimize the active power loss of the loop to find the best switch exchange pair. On this basis, a study proposed a heuristic method of switch disconnection and exchange to solve the random distribution network reconstruction of multiple time steps. At the same time, meta-heuristic intelligent optimization algorithms such as ant colony optimization algorithm, genetic algorithm and particle swarm optimization algorithm are also used to solve the more complex DDNR problem.

[0005] However, as the uncertainty of PDN increases, the scenario scale of the random programming method grows exponentially, the computational burden increases greatly, and the solutions of the robust optimization method are usually conservative, which reduces the performance of the mathematical programming method. Although heuristic methods reduce the computational burden to a certain extent, they cannot guarantee the optimality of the solution and cannot handle the real-time uncertainty of the power flow state during PDN operation.

[0006] Deep reinforcement learning (DRL) methods can capture the uncertainty of DRE from historical data, and can continuously learn the dynamic characteristics of high-dimensional nonlinear systems in the process of interacting with the environment, showing good performance in solving the sequential decision-making problem of DDNR. Some studies have used Deep Q-learning algorithms to solve the optimal network topology of high-penetration PDNs of new energy, or proposed new algorithms such as Deep Q-Network and SAC to solve the optimal network topology. However, with the increase of DRE in PDN, the existing DRL-based solutions have shown two new challenges. First, there are a large number of safety constraints in the DDNR problem, including radial topology constraints (hard constraints) and node voltage and line current constraints (soft constraints). The existing DRL method satisfies the constraints in the optimization process by simplifying the PDN structure and adding penalty terms in the reward function. However, this method requires a cumbersome process to design the reconstruction rules and penalty factors of the PDN, and inappropriate design may reduce the performance of the DRL algorithm. Secondly, in order to capture the evolutionary characteristics of the PDN power flow state, the DRL agent needs to perceive the timing characteristics of source-load power generation and consumption and the dynamic spatial characteristics of PND topology changes when performing DDNR, rather than simply inputting the state quantity into the agent's neural network in the form of a one-dimensional vector. Summary of the invention

[0007] The present invention proposes a distribution network dynamic network reconstruction (DDNR) method based on constraint-enhanced safety reinforcement learning. The DDNR task of the distribution network is described as a constrained Markov decision process (CMDP). Then, the interior point policy optimization (IPO) algorithm and the method of increasing the safety layer are used to handle the soft constraints (node ​​voltage and line current constraints) and hard constraints (radial topology constraints) in the DDNR process. At the same time, the present invention proposes a spatiotemporal perception neural network model to increase the dynamic spatiotemporal variation characteristics of the distribution network flow in the DDNR process by the reinforcement learning agent.

[0008] The purpose of this section is to summarize some aspects of embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the specification abstract and the invention title of this application to avoid blurring the purpose of this section, the specification abstract and the invention title, and such simplifications or omissions cannot be used to limit the scope of the present invention.

[0009] Therefore, the present invention provides a distribution network dynamic network reconstruction method based on constraint-enhanced safety reinforcement learning, which plays an important role in reducing power distribution network (PDN) energy loss, optimizing voltage distribution, and increasing new energy carrying capacity. To solve the above technical problems, the present invention provides the following technical solutions, a distribution network dynamic network reconstruction method based on constraint-enhanced safety reinforcement learning, comprising:

[0010] Step S1: The DDNR task of the active distribution network with high penetration of photovoltaic and wind power generation is regarded as a CMDP model, and a secure reinforcement learning solution framework based on the IPO algorithm is constructed. The logarithmic barrier function of IPO can enable the agent to meet the soft constraints of the DDNR task when learning the strategy;

[0011] Step S2: Design a security layer based on the action mask in combination with the branch switching mechanism of the distribution network, and apply the security layer to the output layer of the IPO strategy network, so that the strategy output by the agent always meets the hard constraints of the DDNR task;

[0012] Step S3: construct a spatiotemporal perception neural network model, and apply the spatiotemporal perception neural network to the input layer of the IPO strategy network and the value network, so that the intelligent agent can better perceive the spatiotemporal variation characteristics of the distribution network flow in the DDNR process;

[0013] Step S4: Design a specific CMDP model for the DDNR task for the improved IPO solution framework, including the specific forms of the state space, action space, reward function, constraint function, and state transfer function;

[0014] Step S5: Use the improved IPO algorithm constructed in steps S1, S2, and S3 to solve the CMDP model of step S4, design offline training rules to train the agent to learn the optimal DDNR strategy, then save the trained agent's strategy and apply the strategy to the online execution process of the distribution network.

[0015] More specifically, in the above step S1, the CMDP includes 6 elements, including the state space Action Space Reward Function Constraint Function State transition probability distribution Discount factor γ∈[0,1]; The DRL agent selects an action at based on the current state st, which is determined by a strategy π. The goal of the DRL agent is to find a strategy π that maximizes the discounted cumulative reward while satisfying the constraints. The expression is as follows:

[0016]

[0017]

[0018] In the formula, Representative strategy π θ The cumulative discounted cost relative to the constraint, represents the feasible domain of strategy π, γ t is the discount factor, is the reward function at time t, d is the constraint limit, represents the random source expectation, represents the control horizon of the agent; the goal of CMDP is to select a strategy π that satisfies the constraint θ To maximize the cumulative discounted return

[0019] More specifically, in the above step S1, the policy gradient algorithm uses the policy gradient theory to optimize the target J(π θ ), as follows:

[0020]

[0021] In the above formula, θ is the parameter of the policy network, π θ (a t |s t ) represents the current policy function according to s t Select action a t The probability of N is the length of a control horizon, and the state action value function Q(s t ,a t ) is represented by a neural network with a parameter μ, and its calculation formula is as follows:

[0022]

[0023] In the above formula, r(s t' ,a t' ) is in state s t' Next select action a t' The reward obtained, V(s t ) is the state value function;

[0024] The policy gradient algorithm is a proximal policy optimization algorithm, and its objective function is expressed as:

[0025]

[0026]

[0027] in:

[0028]

[0029] In the above formula, is the expectation, which represents the empirical average over a finite sample; In the strategy π θ The estimated value of the advantage function for the next t decision steps; Indicates the ratio of the new and old strategies; is the old strategy before the update; θ is the strategy parameter; clip(·) is the cutoff function, which controls the change between the old and new strategies within [1-ε,1+ε], and ε is the cutoff constant used to set the range of strategy update.

[0030] More specifically, in the IPO algorithm of step S1 above, for each constraint satisfaction problem, there is an indicator function Satisfies the following expression:

[0031]

[0032] That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when any constraint is violated, the strategy needs to be adjusted to meet the constraint first;

[0033] The logarithmic barrier function is defined as:

[0034]

[0035] Among them, k is a hyperparameter;

[0036] At this time, the target is expanded by the logarithmic barrier function, and the final objective function of the IPO algorithm is:

[0037]

[0038] More specifically, in the above step S2, the principle of the distribution network reconstruction based on the branch exchange mechanism is: close a tie switch i to form a loop, and then open a segment switch j in the loop to restore the distribution network to a radial shape, and regard this action as a switch pair SP(i,j);

[0039] All feasible switch pairs SP(i,j) at time t are stored in the binary matrix Mt at time t. If the switch pair SP is feasible, Mt(i,j)=1; otherwise, it is 0. At each decision step, the exchange pairs with Mt(i,j)=1 are selected from the binary mask matrix M for reconstruction operation.

[0040] The output of the IPO strategy network is set to a two-dimensional array h ij (s t ), whose dimension is the same as M. In order to output a feasible switch pair SP, a security layer based on the feasible action mask is added to the output layer of the policy network, which is expressed as:

[0041]

[0042] In the formula, πij (s t ) is a 2-dimensional array representing the probability distribution of SP(i,j). If Mt(i,j)=0, then π ij (s t )=0; represents the state value of the switch pair (i, j) at time t, k and l represent the number of all switches that can be closed and opened, and h kl (s t ) represents the hidden layer state value of all actions output by the action network at time t, Represents the state value of all switch pairs at time t.

[0043] More specifically, in the above step S3, the spatiotemporal perception neural network model includes a GCN network and a GRU network, wherein the GCN network performs a convolution operation on the graph structure data by defining an invariant convolution kernel; the GRU network regulates the transmission and memory of information through a gating mechanism, and the structure of the GRU network includes a reset gate and an update gate;

[0044] The updating process of the spatiotemporal-aware neural network is expressed as the following equation:

[0045]

[0046] r t =σ(W r [f(X t ,A),h t-1 ]+b r )

[0047] z t =σ(W z [f(X t ,A),h t-1 ]+b z )

[0048] c t =tanh(W c [f(X t ,A),(r t *h t-1 )]+b c )

[0049] h t =z t *h t-1 +(1-z t )*c t

[0050] In the above formula, X is the characteristic matrix of PDN, A is the adjacency matrix of PDN, represents the adjacency matrix plus self-loops, is the degree matrix of the adjacency matrix, σ is the nonlinear activation function, W is the learnable weight matrix, X′ is the new feature representation after GCN, r t is the reset gate, z t is the update gate, h t and h t-1 are the hidden states of the current moment and the previous moment respectively, W r , W z and W c are weight matrices, b r 、b z and b c is the bias term, c t is the candidate hidden state in the GRU network, X t Represents the characteristic vector of the distribution network at time t.

[0051] More specifically, in the above step S4, the CMDP model includes:

[0052] State space: The state space contains spatial topology information and electrical characteristics representing multiple time periods. Let the PDN state at the current time t be X t =(P t ,Q t ,V t ,I t ,P t loss ,A), where P t is the net active power injection amount of the node, Q t is the net reactive power injection amount of the node, V t is the node voltage amplitude, I t is the branch current amplitude, P t loss is the line active power loss, so s t =(X t ,X t-1 ,…,X t-w+1 ), using a moving window of Xt consisting of the past w steps to represent s t ;

[0053] Action space: The action of DDNR is set as a switch pair SP. If the closed switch and the open switch are the same, it is considered that the PDN does not perform topology reconstruction at the current moment and does not incur switching costs.

[0054] Reward function: The reward function is the network loss cost and switch action cost of PDN. The reward function at time t is Among them C p is the network loss cost, C s is the switching operation cost, is the active power loss on branch ij at time t;

[0055] Constraint function: Node voltage V i Limits: V min ≤V i ≤V max , Branch current I ij Limits to: I ij ≤I max ,ij∈ε,V min is the voltage lower limit, V max is the voltage upper limit, N is the number of distribution network node voltages, I max is the upper limit of branch current, ε is the number of distribution network branches;

[0056] State transition probability distribution: State transition probability distribution Represents the DRL agent taking action a t When the state changes from s t Transfer to t+1 The probability of represents the action of the agent at time t, is a collection of actions, represents the state at time t and t+1, is the state set, Represents the power flow calculation process.

[0057] More specifically, the process is divided into exploration, training, and execution phases;

[0058] In the exploration phase, the IPO algorithm interacts with the PDN environment based on the current strategy πθ;

[0059] In the training phase, the parameters θ and μ of the actor network and critic network of the IPO agent are updated according to the current experience trajectory. At the same time, the IPO agent captures the uncertainty of the PDN environment and the spatiotemporal dynamic characteristics of the DDNR problem from the experience trajectory.

[0060] In the execution phase, the trained policy network is saved, the spatiotemporal features of the environment are extracted through the spatiotemporal graph convolutional network, and the spatiotemporal features are input into the policy network with actionable action masks to generate a safe DDNR policy.

[0061] The present invention also provides a device, which includes: one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning.

[0062] The present invention also provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning is implemented.

[0063] Beneficial effects of the present invention: The present invention realizes the dynamic reconstruction of active distribution networks with high penetration of photovoltaic (PV) and wind power (WP) power generation through a secure reinforcement learning method, solves the problems of increased active power loss, voltage and current limit violations caused by the access of new energy, and the proposed method can face the uncertainty of PV and WP in real time online calculation and execution, and the obtained strategy has security guarantee. The method proposed by the present invention has the following three advantages over the existing technology:

[0064] The IPO algorithm used introduces a logarithmic barrier function. This design allows the algorithm to filter out strategies that do not meet the constraints during the training process, thereby taking into account the soft constraints in the DDNR process. Compared with existing reinforcement learning methods, IPO does not require specified penalty terms and adjustment of cumbersome penalty factors, and is easier to calculate.

[0065] When performing DDNR, the hard constraints of the radial topology of the distribution network need to be met at all times (i.e., open-loop operation). Therefore, the present invention combines the idea of ​​the branch exchange method, designs a feasible action mask according to the state of all switches in the distribution network at each decision step, and applies the mask to the output layer of the IPO strategy network, which not only ensures the hard constraints of the radial topology of the distribution network but also reduces the dimension of the action space.

[0066] The spatiotemporal perception neural network model proposed in the present invention integrates a graph convolutional network (GCN) and a gated recurrent unit neural network (GRU). GCN is used to obtain the spatial topological structure and power flow distribution characteristics of the DDNR process, and GRU is used to capture the timing characteristics of power generation and consumption of PV, WP, and basic load. Therefore, with the support of the spatiotemporal perception neural network model, the reinforcement learning agent can better perceive the strong spatiotemporal uncertainty of PDN in the DDNR process. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0068] in:

[0069] Figure 1 A schematic flow chart of a method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning provided by an embodiment of the present invention.

[0070] Figure 2 A schematic diagram of an IEEE 14-node distribution network provided for an embodiment of the present invention.

[0071] Figure 3 A schematic diagram of a distribution network dynamic network reconstruction framework based on constraint-enhanced security reinforcement learning provided for an embodiment of the present invention.

[0072] Figure 4 A schematic diagram of an IEEE 33-node distribution network provided by an embodiment of the present invention.

[0073] Figure 5 The training situation of the proposed method on two test systems is provided for one embodiment of the present invention.

[0074] Figure 6 A comparison chart of PDN operating costs in a typical test day under different solutions provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0076] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0077] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0078] The present invention is described in detail with reference to schematic diagrams. When describing the embodiments of the present invention, for the sake of convenience, the cross-sectional diagrams showing the device structure will not be partially enlarged according to the general scale, and the schematic diagrams are only examples, which should not limit the scope of protection of the present invention. In addition, in actual production, the three-dimensional dimensions of length, width and depth should be included.

[0079] At the same time, in the description of the present invention, it should be noted that the directions or positional relationships indicated by the terms "upper, lower, inner and outer" are based on the directions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention. In addition, the terms "first, second or third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0080] In the present invention, unless otherwise clearly specified and limited, the terms "install, connect, connect" should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection or an integral connection; it can also be a mechanical connection, an electrical connection or a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0081] Reference Figure 1-3 , which is the first embodiment of the present invention, and provides a distribution network dynamic network reconstruction method based on constraint enhanced security reinforcement learning, comprising:

[0082] Step S1: The DDNR task of the active distribution network with high penetration of photovoltaic and wind power generation is regarded as a CMDP model, and a secure reinforcement learning solution framework based on the IPO algorithm is constructed. The logarithmic barrier function of IPO can enable the agent to meet the soft constraints of the DDNR task when learning strategies.

[0083] A general CMDP consists of 6 elements, including the state space Action Space Reward Function Constraint Function State transition probability distribution Discount factor γ∈[0,1]. The DRL agent is based on the current state s t The selection of action at is determined by a strategy π, which can be represented by a neural network with parameters. The goal of the DRL agent is to find a strategy π that maximizes the discounted cumulative reward while satisfying the constraints. The expression is as follows:

[0084]

[0085] In the formula, Representative strategy π θ Relative to the cumulative discounted cost of the constraint, Π c represents the feasible domain of strategy π, γ t is the discount factor, is the reward function at time t, d is the constraint limit, represents the random source expectation, represents the control horizon of the agent. The goal of CMDP is to select a strategy π that satisfies the constraint θ To maximize the cumulative discounted return The common policy gradient algorithm uses policy gradient theory to optimize the target J(π θ ), as follows:

[0086]

[0087] In the above formula, θ is the parameter of the policy network, π θ (a t |s t ) represents the current policy function according to s t Select action a t The probability of N is the length of a control horizon, and the state action value function Q(s t ,a t ) is represented by a neural network with a parameter μ, and its calculation formula is as follows:

[0088]

[0089] In the above formula, r(s t' ,a t' ) is in state s t' Next select action a t' The reward obtained, V(s t ) is the state value function. The proximal policy optimization (PPO) algorithm, as a common policy gradient algorithm, uses a truncation function to make the gradient update in formula (2) more stable and efficient. Its objective function can be expressed as:

[0090]

[0091] in:

[0092]

[0093] In the above formula, is the expectation, which represents the empirical average over a finite sample; In the strategy π θ The estimated value of the advantage function for the next t decision steps; Indicates the ratio of the new and old strategies; is the old strategy before the update; θ is the strategy parameter; clip(·) is the cutoff function, which controls the change between the old and new strategies within [1-ε,1+ε], and ε is the cutoff constant used to set the range of strategy update.

[0094] Based on the framework of the above PPO algorithm, the present invention further proposes the IPO algorithm to solve the CMDP problem. The IPO algorithm inherits the clipping target of the PPO algorithm and expands the constraints through the logarithmic barrier function. For each problem that satisfies the constraints, if there is an indicator function satisfy:

[0095]

[0096] That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when any constraint is violated, the strategy needs to be adjusted first to meet the constraint. In this way, by expanding the target through the indicator function, the original CMDP problem can be simplified to an unconstrained optimization problem. The logarithmic barrier function is a differentiable approximation of the indicator function, which is defined as:

[0097]

[0098] Among them, k is a hyperparameter. At this time, the target is expanded by the logarithmic barrier function, and the final objective function of the IPO algorithm is:

[0099]

[0100] Step S2: Design a security layer based on the action mask in combination with the branch exchange mechanism of the distribution network, and apply the security layer to the output layer of the IPO strategy network so that the strategy output by the intelligent agent always meets the hard constraints of the DDNR task.

[0101] The principle of distribution network reconstruction based on branch exchange mechanism is: close a tie switch i to form a loop, and then open a segment switch j in the loop to restore the distribution network to a radial shape. This action can be regarded as a switch pair SP(i,j). Figure 2 Take an IEEE 14-node distribution network as an example. The system has 14 nodes and 16 branches. Node 1 is a transformer node, with 13 normally closed section switches and 3 normally open tie switches. Closing the three tie switches in sequence can form three loops, so feasible switch pairs will be generated in these three loops. For example, in the initial state, SP (14, 7) can be selected (close switch 14 and open switch 7) to complete the reconstruction of the distribution network topology and realize the transfer of energy between different feeders.

[0102] The present invention stores all feasible SP(i,j) at time t in the binary matrix Mt at time t. If the SP is feasible, Mt(i,j)=1, otherwise it is 0. Therefore, in each decision step, the exchange pair with Mt(i,j)=1 can be selected from the binary mask matrix M for reconstruction operation. Figure 2In the distribution network, the dimension of M is 16×16. There are not many feasible SPs at each moment, so M can be regarded as a sparse matrix. In order to match the design of the switch pair, the output of the IPO strategy network is set as a two-dimensional array h ij (s t ), whose dimension is the same as M. In order to output a feasible SP, a security layer based on an actionable action mask is added to the output layer of the policy network, which can be expressed as:

[0103]

[0104] In the formula, π ij (s t ) is a 2-dimensional array representing the probability distribution of SP(i,j). If Mt(i,j)=0, then π ij (s t )=0, thus ensuring the feasibility of the reinforcement learning agent's actions; represents the state value of the switch pair (i, j) at time t, k and l represent the number of all switches that can be closed and opened, and h kl (s t ) represents the hidden layer state value of all actions output by the action network at time t, Represents the state value of all switch pairs at time t.

[0105] The traditional branch exchange method uses a heuristic method to find the best SP. This solution only considers the optimal solution of the current decision step. However, DDNR is a typical sequential decision problem, and the current reconstruction solution will affect the subsequent benefits. The present invention uses the IPO algorithm to solve the best SP, which considers the cumulative report and cumulative constraints of the entire control horizon.

[0106] Step S3: Construct a spatiotemporal perception neural network model and apply the spatiotemporal perception neural network to the input layer of the IPO strategy network and the value network, so that the intelligent agent can better perceive the spatiotemporal variation characteristics of the distribution network flow during the DDNR process.

[0107] The spatiotemporal perception neural network model constructed by the present invention is composed of a GCN network and a GRU network, wherein GCN can effectively obtain and update the feature information of nodes from adjacent nodes by defining an invariant convolution kernel to perform convolution operations on graph structure data. The core idea is to apply graph convolution operations to aggregate the neighborhood information of nodes to obtain updated node representations. Specifically, by combining the adjacency matrix of the graph with the feature matrix and applying a nonlinear activation function, a new node representation is obtained.

[0108] GRU regulates the transmission and memory of information through a gating mechanism, thereby retaining important historical information while learning the current state. The structure of GRU includes a reset gate and an update gate. These two gating mechanisms can effectively control the impact of past information on the current state, thereby adapting to the characteristics of time series data. The reset gate determines how to combine new inputs with previous hidden states, while the update gate controls the update and retention process of hidden states.

[0109] Therefore, the update process of the spatiotemporal perception neural network can be expressed as the following equation:

[0110]

[0111] r t =σ(W r [f(X t ,A),h t-1 ]+b r ) (10)

[0112] z t =σ(W z [f(X t ,A),h t-1 ]+b z ) (11)

[0113] c t =tanh(W c [f(X t ,A),(r t *h t-1 )]+b c ) (12)

[0114] h t =z t *h t-1 +(1-z t )*c t (13)

[0115] In the above formula, X is the characteristic matrix of PDN, A is the adjacency matrix of PDN, represents the adjacency matrix plus self-loops, is the degree matrix of the adjacency matrix, σ is the nonlinear activation function, W is the learnable weight matrix, X′ is the new feature representation after GCN, r t is the reset gate, z t is the update gate, h t and h t-1 are the hidden states of the current moment and the previous moment respectively, W r , W z and W c are weight matrices, b r 、bz and b c is the bias term, c t is the candidate hidden state in the GRU network, X t Represents the characteristic vector of the distribution network at time t.

[0116] Step S4: Design a specific CMDP model for the DDNR task based on the improved IPO solution framework, including the specific forms of the state space, action space, reward function, constraint function, and state transfer function.

[0117] For the DDNR task of PDN, the present invention sets a specific CMDP model as follows:

[0118] State space: The state space of the present invention includes spatial topological information and electrical characteristics representing multiple time periods. Let the PDN state at the current time t be X t =(P t ,Q t ,V t ,I t ,P t loss ,A), where P t is the net active power injection amount of the node, Q t is the net reactive power injection amount of the node, V t is the node voltage amplitude, I t is the branch current amplitude, P t loss is the line active power loss, so s t =(X t ,X t-1 ,…,X t-w+1 ), using a moving window of Xt consisting of the past w steps to represent s t , in order to infer the future trends of various electrical quantities.

[0119] Action space: The present invention sets the action of DDNR as an SP. SP has been introduced in step S2. If the closed switch and the open switch are the same, it is considered that the PDN does not perform topology reconstruction at the current moment and does not incur switching costs.

[0120] Reward function: The reward function of the present invention is the network loss cost and switch action cost of PDN. The reward function at time t is Among them C p is the network loss cost, C s is the switching operation cost, is the active power loss on branch ij at time t;

[0121] Constraint function: The DDNR process should ensure that the node voltage and branch current do not violate the rules. The node voltage V iLimits: V min ≤V i ≤V max , Branch current I ij Limits to: I ij ≤I max ,ij∈ε,V min is the voltage lower limit, V max is the voltage upper limit, N is the number of distribution network node voltages, I max is the upper limit of branch current, and ε is the number of distribution network branches.

[0122] State transition probability distribution: State transition probability distribution Represents the DRL agent taking action a t When the state changes from s t Transfer to t+1 The probability of represents the action of the agent at time t, is a collection of actions, represents the state at time t and t+1, is the state set, Representing the power flow calculation process, DDNR will cause power flow changes, and the agent searches for the best strategy from these changes.

[0123] Step S5: Use the improved IPO algorithm constructed in steps S1, S2, and S3 to solve the CMDP model of step S4, design offline training rules to train the agent to learn the optimal DDNR strategy, then save the trained agent's strategy and apply the strategy to the online execution process of the distribution network.

[0124] The improved IPO algorithm proposed in the present invention for solving the DDRN problem can be divided into the exploration, training and execution stages. Figure 3 The overall framework of the proposed distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning is demonstrated.

[0125] In the exploration phase, the IPO algorithm interacts with the PDN environment based on the current strategy πθ to generate an experience trajectory, and the length of the experience trajectory represents the scope of the control horizon.

[0126] During the training phase, the parameters θ and μ of the actor network and critic network of the IPO agent are updated according to the current experience trajectory. The purpose is to maximize the cumulative discounted return of the control horizon and ensure that the current strategy satisfies the cumulative constraints. At the same time, the IPO agent can also capture the uncertainty of the PDN environment and the spatiotemporal dynamic characteristics of the DDNR problem from the experience trajectory.

[0127] In the execution phase, the trained policy network is saved, the spatiotemporal features of the environment are extracted through the spatiotemporal graph convolutional network, and the spatiotemporal features are input into the policy network with actionable action masks to generate a safe DDNR policy.

[0128] Reference Figure 4-6 , which is an embodiment of the present invention, provides a distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through experiments.

[0129] use Figure 2 and 4 The effectiveness of the proposed method was verified by the IEEE14-node test system and the IEEE33-node test system in the paper. In the IEEE14-node system, two PVs are located at nodes 2 and 13, and one WP is located at node 7. In the IEEE33-node system, three PVs are located at nodes 13, 25, and 31, and three WPs are located at nodes 7, 18, and 22. A certain amount of power load is aggregated on each node of the test system. The PV, WP, and power load data in this application are collected from real operating scenarios. In order to adapt to different test systems, the data is scaled to a certain proportion. One year of data was selected to train the DRL agent, and a typical day was selected in each of the two test systems to test the trained agent. The present invention uses one year of data to train the IPO agent for 1000 rounds, and randomly selects 10 days of data for interaction in each round, including 240 decision steps. The timestamp of each decision step is 1 hour, corresponding to the data resolution. The upper and lower limits of the node voltage are set to [0.95, 1.05]. The active power loss cost Cp is set to 40$ / MWh and the switch exchange cost is set to 0.5$.

[0130] In order to verify the effectiveness of the proposed safety reinforcement learning algorithm in solving DDNR tasks, the training process of the algorithm is analyzed. Figure 5 The average cumulative discounted reward and its sliding average of the algorithm of the present invention after 1000 rounds of training on two test systems are shown.

[0131] At the beginning of training, the randomly initialized parameters caused the agents to show low performance, and even the phenomenon of non-convergence of the power flow occurred on the IEEE14 system. This was due to the randomness of the policy network, which led to invalid actions taken by the DRL agent. As the training progresses, the actor and critic network parameters of the algorithm of the present invention are continuously updated, and the agent learns the correct strategy in this process. After 400 rounds of training, the agent can already make good actions and obtain a higher average cumulative discounted reward. Table 1 shows the average of the average network loss, average switch action cost, average voltage violation rate and average current violation rate of the last 200 rounds of the two test systems. It can be seen from the results that the proposed algorithm achieved a stable high return value in the later stage of training, and the strategy output by the algorithm of the present invention can reduce the operating cost of PDN and ensure the satisfaction of operating constraints.

[0132] Table 1

[0133]

[0134] In order to verify the effectiveness of the algorithm of the present invention and the advantages of DDNR performance, the trained agent of the present invention was tested on a typical day, and the DDNR schemes of the brute force search method (BFS) and the genetic algorithm were implemented to compare the performance with the proposed method. Figure 6 The typical daily operating costs of two test systems with three control schemes and without any control scheme are shown. Figure 6 It can be seen that compared with the scheme without any control, the total operating cost of the algorithm of the present invention on the two test systems decreased by 33.2% and 41.8%. Compared with the heuristic GA algorithm, the performance of the proposed method improved by 14.76% and 17%. Compared with the BFS method, the performance of the algorithm of the present invention improved by 3% and 6.7%. The BFS method has lower operating costs in some decision steps. This method only considers the immediate interests and does not take into account the interests of the entire control horizon. When the system is large, it is very time-consuming to calculate the flow of each possible action. Table 2 shows the action SP of the proposed algorithm in one day.

[0135] Table 2

[0136]

Claims

1. A distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning, characterized by: include, Step S1: The DDNR task of the active distribution network with high penetration of photovoltaic and wind power generation is regarded as a CMDP model, and a secure reinforcement learning solution framework based on the IPO algorithm is constructed. The logarithmic barrier function of IPO can enable the agent to meet the soft constraints of the DDNR task when learning the strategy; Step S2: Design a security layer based on the action mask in combination with the branch switching mechanism of the distribution network, and apply the security layer to the output layer of the IPO strategy network, so that the strategy output by the agent always meets the hard constraints of the DDNR task; Step S3: construct a spatiotemporal perception neural network model, and apply the spatiotemporal perception neural network to the input layer of the IPO strategy network and the value network, so that the intelligent agent can better perceive the spatiotemporal variation characteristics of the distribution network flow in the DDNR process; Step S4: Design a specific CMDP model for the DDNR task for the improved IPO solution framework, including the specific forms of the state space, action space, reward function, constraint function, and state transfer function; Step S5: Use the improved IPO algorithm constructed in steps S1, S2, and S3 to solve the CMDP model of step S4, design offline training rules to train the agent to learn the optimal DDNR strategy, then save the trained agent's strategy and apply the strategy to the online execution process of the distribution network.

2. The method for dynamic network reconstruction of distribution network based on constraint-enhanced security reinforcement learning according to claim 1, characterized in that: In step S1, the CMDP includes 6 elements, including the state space Action Space Reward Function Constraint Function State transition probability distribution Discount factor γ∈[0,,]; The DRL agent selects an action at based on the current state st, which is determined by a strategy π. The goal of the DRL agent is to find a strategy π that maximizes the discounted cumulative reward while satisfying the constraints. The expression is as follows: In the formula, Representative strategy π θ The cumulative discounted cost relative to the constraint, represents the feasible domain of strategy π, γ t is the discount factor, is the reward function at time t, d is the constraint limit, represents the random source expectation, represents the control horizon of the agent; the goal of CMDP is to select a policy πθ that satisfies the constraints to maximize the cumulative discounted return 3. The method for dynamic network reconstruction of distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 2, characterized in that: In step S1, the policy gradient algorithm uses the policy gradient theory to optimize the target J(π θ ), as follows: In the above formula, θ is the parameter of the policy network, π θ (a t |s t ) represents the current policy function according to s t Select action a t The probability of N is the length of a control horizon, and the state action value function Q(s t , a t ) is represented by a neural network with a parameter μ, and its calculation formula is as follows: In the above formula, r(s t′ , a t ,) is in state s t′ Next select action a t , the reward obtained, V(s t ) is the state value function; The policy gradient algorithm is a proximal policy optimization algorithm, and its objective function is expressed as: in: In the above formula, is the expectation, which represents the empirical average over a finite sample; In the strategy π θ The estimated value of the advantage function for the next t decision steps; Indicates the ratio of the new and old strategies; is the old strategy before the update; θ is the strategy parameter; clip(·) is the cutoff function, which controls the change between the old and new strategies within [1-ε, 1+ε], and ε is the cutoff constant used to set the range of strategy update.

4. The method for dynamic network reconstruction of distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 3, characterized in that: In the IPO algorithm of step S1, for each constraint satisfaction problem, there is an indicator function Satisfies the following expression: That is, when the constraints are met, the problem is solved as an unconstrained strategy optimization problem that only considers rewards; when any constraint is violated, the strategy needs to be adjusted to meet the constraint first; The logarithmic barrier function is defined as: Among them, k is a hyperparameter; At this time, the target is expanded by the logarithmic barrier function, and the final objective function of the IPO algorithm is:

5. The method for dynamic network reconstruction of distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 4, characterized in that: In step S2, the principle of the distribution network reconstruction based on the branch exchange mechanism is: close a tie switch i to form a loop, and then open a segment switch j in the loop to restore the distribution network to a radial shape, and regard this action as a switch pair SP (i, j); All feasible switch pairs SP(i, i) at time t are stored in the binary matrix Mt at time t. If the switch pair SP is feasible, Mt(i, j) = 1; otherwise, it is 0. At each decision step, the exchange pairs with Mt(i, j) = 1 are selected from the binary mask matrix M for reconstruction operation. The output of the IPO strategy network is set to a two-dimensional array h ij (s t ), whose dimension is the same as M. In order to output a feasible switch pair SP, a security layer based on the feasible action mask is added to the output layer of the policy network, which is expressed as: In the formula, π ij (s t ) is a 2-dimensional array representing the probability distribution of SP(i,j). If Mt(i,j)=0, then π ij (s t )=0; represents the state value of the switch pair (i, j) at time t, k and l represent the number of all switches that can be closed and opened, and h kl (s t ) represents the hidden layer state value of all actions output by the action network at time t, Represents the state value of all switch pairs at time t.

6. The method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 5, characterized in that: In step S3, the spatiotemporal perception neural network model includes a GCN network and a GRU network, wherein the GCN network performs a convolution operation on the graph structure data by defining an invariant convolution kernel; the GRU network adjusts the transmission and memory of information through a gating mechanism, and the structure of the GRU network includes a reset gate and an update gate; The updating process of the spatiotemporal-aware neural network is expressed as the following equation: r t =σ(W r [f(X t ,A),h t-1 ]+b r ) z t =σ(W z [f(X t ,A),h t-1 ]+b z ) c t =tanh(W c [f(X t ,A),(r t *h t-1 )]+b c ) h t =z t *h t-1 +(1-z t )*c t In the above formula, X is the characteristic matrix of PDN, A is the adjacency matrix of PDN, represents the adjacency matrix plus self-loops, is the degree matrix of the adjacency matrix, σ is the nonlinear activation function, W is the learnable weight matrix, X′ is the new feature representation after GCN, r t is the reset gate, z t is the update gate, h t and h t-1 are the hidden states of the current moment and the previous moment respectively, W r , W z and W c are weight matrices, b r , b z and b c is the bias term, c t is the candidate hidden state in the GRU network, X t Represents the characteristic vector of the distribution network at time t.

7. The method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 6, characterized in that: In step S4, the CMDP model includes: State space: The state space contains spatial topology information and electrical characteristics representing multiple time periods. Let the PDN state at the current time t be X t =(P t ,Q t ,V t ,I t ,P t loss ,A), where P t is the net active power injection amount of the node, Q t is the net reactive power injection amount of the node, V t is the node voltage amplitude, I t is the branch current amplitude, P t loss is the line active power loss, so s t =(X t ,X t-1 ,…,X t-w+1 ), using a moving window of Xt consisting of the past w steps to represent s t ; Action space: The action of DDNR is set as a switch pair SP. If the closed switch and the open switch are the same, it is considered that the PDN does not perform topology reconstruction at the current moment and does not incur switching costs. Reward function: The reward function is the network loss cost and switch action cost of PDN. The reward function at time t is Among them C p is the network loss cost, C s is the switching operation cost, is the active power loss on branch ij at time t; Constraint function: Node voltage V i The restrictions are: Branch current I ij Limits to: I ij ≤I max ,ij∈ε,V min is the voltage lower limit, V max is the voltage upper limit, N is the number of distribution network node voltages, I max is the upper limit of branch current, ε is the number of distribution network branches; State transition probability distribution: State transition probability distribution Represents the DRL agent taking action a t When the state changes from s t Transfer to t+1 The probability of represents the action of the agent at time t, is a collection of actions, represents the state at time t and t+1, is the state set, Represents the power flow calculation process.

8. The method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning as claimed in claim 7, characterized in that: The process is divided into exploration, training and execution phases; In the exploration phase, the IPO algorithm interacts with the PDN environment based on the current strategy πθ; In the training phase, the parameters θ and μ of the actor network and critic network of the IPO agent are updated according to the current experience trajectory. At the same time, the IPO agent captures the uncertainty of the PDN environment and the spatiotemporal dynamic characteristics of the DDNR problem from the experience trajectory. In the execution phase, the trained policy network is saved, the spatiotemporal features of the environment are extracted through the spatiotemporal graph convolutional network, and the spatiotemporal features are input into the policy network with actionable action masks to generate a safe DDNR policy.

9. A device, characterized in that: The device includes: one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors execute the distribution network dynamic network reconstruction method based on constraint-enhanced security reinforcement learning as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for dynamic network reconstruction of a distribution network based on constraint-enhanced security reinforcement learning as described in any one of claims 1 to 8 is implemented.

Images