A power distribution network reconstruction energy-saving loss reduction method and device based on security reinforcement learning

Abstract

The application discloses a power distribution network reconstruction energy-saving loss reduction method and device based on safety reinforcement learning. The method comprises the following steps: step S1: collecting historical operation data of the power distribution network as learning sample data of safety reinforcement learning; step S2: modeling the power distribution network reconstruction as a constrained Markov decision process, and building a reinforcement learning environment; step S3: offline training is carried out on the built reinforcement learning environment by using a constraint strategy optimization algorithm, and an optimal neural network model is obtained; and step S4: the power distribution network reconstruction is solved by using the optimal neural network model. The power distribution network reconstruction problem is formulated as a constrained Markov decision process, so that the switching cost and the network loss cost are minimized while the voltage safety is ensured.

Description

Technical Field

[0001] This invention relates to the field of distribution network technology, and in particular to a method and apparatus for energy saving and loss reduction in distribution network reconfiguration based on security reinforcement learning. Background Technology

[0002] The dynamic distribution network reconfiguration problem determines the optimal sequence of network reconfiguration within the operational area based on line resistance losses, load losses, or the managed capacity of distributed generation. Common operational constraints modeled in the distribution network reconfiguration problem include voltage amplitude limits and network radiation. Frequent switching of remotely controllable switches can lead to excessive equipment wear and increased costs.

[0003] Most existing solutions to the dynamic distribution network reconfiguration problem employ physical model-based control methods. The distribution network reconfiguration problem is often formulated as a mixed-integer or dynamic programming problem, and heuristic and metaheuristic algorithms are also widely used. However, existing physical model-based algorithms heavily rely on accurate knowledge of network topology and parameters. Regional power companies often struggle to maintain accurate network models, and parameter estimates for primary and secondary networks are inaccurate, thus model-based algorithms may be unreliable. Furthermore, model-based algorithms typically have high computational complexity, making computation very time-consuming and hindering real-time applications in distribution network reconfiguration.

[0004] Currently, deep reinforcement learning methods have been applied to some extent in distribution network reconfiguration. Firstly, deep reinforcement learning methods are model-free, therefore they do not require accurate distribution network parameters. Secondly, deep reinforcement learning methods based on offline strategies can save the trained optimal neural network model, enabling real-time online application of distribution network reconfiguration.

[0005] However, traditional deep reinforcement learning methods allow agents to freely explore any behavior during training, which can introduce serious safety issues into the distribution network reconfiguration process. Inappropriate behavior may lead to violations of power flow constraints and overvoltage and undervoltage in distribution feeders. Furthermore, traditional reinforcement learning methods mostly address voltage violations during training by adding penalty terms to the reward function, but designing the penalty coefficient is difficult in practice. An excessively large penalty coefficient results in a low reward value, preventing the agent from learning the optimal reconfiguration scheme; an excessively small penalty coefficient fails to adequately punish constraint violations, compromising the safety of distribution network operation. Summary of the Invention

[0006] This invention provides a method and apparatus for energy saving and loss reduction in distribution network reconfiguration based on secure reinforcement learning. The distribution network reconfiguration problem is modeled as a constrained Markov decision process, and a secure reinforcement learning method with constraint policy optimization is used for offline training and online testing. While ensuring that the voltage amplitude meets the constraints, it can significantly reduce distribution network losses, achieving a balance between safety and economy.

[0007] The technical solution of the present invention includes the following steps:

[0008] Step S1: Collect historical operating data of the distribution network as learning sample data for security reinforcement learning;

[0009] Step S2: Model the distribution network reconfiguration as a constrained Markov decision process and build a reinforcement learning environment;

[0010] Step S3: For the constructed reinforcement learning environment, use the constraint policy optimization algorithm for offline training to obtain the optimal neural network model;

[0011] Step S4: Solve for distribution network reconfiguration using the optimal neural network model.

[0012] In step S1, historical running data for n weeks is collected, where n > 2 and is an integer.

[0013] Historical operating data includes active and reactive power injected into each node of the distribution network.

[0014] In step S2, the constrained Markov decision process is as follows:

[0015] The Markov decision process is extended by an auxiliary cost function C, let J C (π) represents the expected discounted return of strategy π relative to the cost function:

[0016] Therefore, the feasible stationary strategies for a constrained Markov decision process are: In the formula, E represents the expectation, τ represents the trajectory τ = (s0, a0, s1, ...), τ ~ π indicates that the distribution on the trajectory τ depends on π, γ is the discount factor, d is the constraint, and a t Belongs to the action space, s t and s t+1 They belong to the state space respectively;

[0017] The objective function of the stationary strategy is:

[0018]

[0019] In the formula, Π represents the set of all fixed strategies. C It is the cost function C t+1A stationary strategy that limits future discounted returns, where J(π) represents the reward function R. t+1 Future discount returns.

[0020] Wherein, the reward function R t+1 for:

[0021]

[0022] In the formula, C l It equals the product of the fixed retail electricity price and the network loss, where the network loss is equal to the active power p at time t. t Reactive power q t and the radial structure of the power distribution network Related; C s A equals the product of the fixed cost of each switch and the number of switch actions. t For the action space, For the distribution network radial structure at time t, S t For state space;

[0023] Cost function C t+1 for:

[0024]

[0025] In the formula, N v It is the collection of all nodes that have voltage measuring devices; And v are the upper and lower limits of voltage, A t For the action space, S t This is the state space.

[0026] Among them, the state space

[0027] In the formula, p t =[p 1t ,p 2t ,…,p nt ], q t =[q 1t ,q 2t ,…,q nt ], p represents the radial structure of the distribution network at time t-1. it and q it Inject active and reactive power into node i at time t.

[0028] Step S3 includes the following steps:

[0029] Step S31: Solve the objective function transformation

[0030] Based on the corollary of constrained policy optimization theory, the objective function is transformed into the following surrogate optimization objective:

[0031]

[0032] In the formula, θ is the neural network parameter. If action a satisfies the policy The state distribution s under the given conditions, and These are related to future rewards. and future cost returns The relevant advantage function, d is the constraint, D KL (θ||θ i [s]) is distributed in π θ and The KL divergence, δ, is a very small value;

[0033] Step S32: Simplify the solution problem

[0034] The optimization objective in step S31 is approximated as the following optimization problem:

[0035]

[0036] In the formula, It's about the advantage function. The expected policy gradient in the future, where θ is the neural network parameter. If action a satisfies the policy The state distribution s under the given conditions; It represents future cost-return, and d represents the limitation; It's about the cost advantage function. Expected policy gradient in the future; It is distributed in π θ and The Laplace operator for the maximum KL-divergence, δ is a very small value;

[0037] Step S33: Use a constrained strategy optimization algorithm to solve the constructed constrained Markov decision process;

[0038] In each iteration, the Γ time step number is first updated, and the policy network parameters are run. Then collect the state-action group {(s t ,a t First, the gradient is estimated using importance sampling for the sequence t = 0, 1, ..., Γ-1; second, the policy optimization problem in step S32 is solved using an online search algorithm; finally, the optimal solution θ is used. i+1 To update the policy network parameters.

[0039] A power distribution network reconfiguration energy-saving and loss-reduction device based on security reinforcement learning, comprising:

[0040] The data acquisition module is used to collect historical operating data of the power distribution network as learning sample data for security reinforcement learning.

[0041] The modeling module is used to model the reconfiguration of the distribution network as a constrained Markov decision process and build a reinforcement learning environment.

[0042] The training module uses a constraint policy optimization algorithm to train offline in the constructed reinforcement learning environment to obtain the optimal neural network model.

[0043] The solution module solves for power distribution network reconfiguration using an optimal neural network model.

[0044] The present invention has the following beneficial effects:

[0045] 1. The algorithm proposed in this invention is model-free and does not require accurate distribution network topology parameters;

[0046] 2. This invention models the distribution network reconfiguration problem as a constrained Markov decision process, and formulates reward and cost functions respectively, thus avoiding the problem of difficulty in designing penalty coefficients in practice;

[0047] 3. The optimal neural network model trained using the constraint strategy optimization algorithm of this invention can be saved offline and applied online, enabling the solution of power distribution network reconfiguration in seconds;

[0048] 4. The constraint strategy optimization algorithm proposed in this invention is a safe reinforcement learning algorithm. During the training process, it can effectively reduce network loss while satisfying voltage constraints, thus achieving a balance between safety and economy. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention;

[0050] Figure 2 The reward curve for the training phase;

[0051] Figure 3 The constraint value curve for the training phase;

[0052] Figure 4 This shows the voltage distribution at each node on a certain day during the CPO testing phase. Detailed Implementation

[0053] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0054] like Figure 1 As shown, the present invention includes the following steps:

[0055] Step S1: Collect historical operating data of the distribution network as learning sample data for security reinforcement learning;

[0056] Step S2: Model the distribution network reconfiguration as a constrained Markov decision process and build a reinforcement learning environment;

[0057] Step S3: For the constructed reinforcement learning environment, use the constraint policy optimization algorithm for offline training to obtain the optimal neural network model;

[0058] Step S4: Solve for distribution network reconfiguration using the optimal neural network model. This involves inputting the active and reactive power data of each node in the current distribution network into the optimal neural network model to solve for the topological state of the distribution network and obtain the minimum operating cost for distribution network reconfiguration.

[0059] This invention formulates the distribution network reconfiguration problem as a constrained Markov decision process, ensuring voltage safety while minimizing switching costs and network loss costs. A constrained policy optimization (CPO)-based security reinforcement learning method is used to learn neural network policies. Unlike traditional deep reinforcement learning methods that allow agents to freely explore any behavior during training, the proposed method restricts exploration to security policies that satisfy AC power flow constraints during training. The proposed method is model-free, requiring no complete and accurate network parameters of the distribution network. Furthermore, it is an online algorithm that can achieve distribution network reconfiguration solutions in seconds.

[0060] In step S1, historical running data for n weeks is collected, where n > 2 and is an integer.

[0061] Historical operating data includes active and reactive power injected into each node of the distribution network.

[0062] Step S2 includes the following steps:

[0063] Step S21: Definition of Markov Decision Process

[0064] A Markov decision process is a tuple (S, A, P, r, γ, T) consisting of a set of states S, a set of actions A, and a set of state transition probabilities. The process consists of a reward function r(s,a), a discount coefficient γ∈[0,1], and a time range T. In a Markov decision process, the agent, at each discrete time step t, adjusts the state S of the environment. t ∈S select action A t ∈A, and then the agent receives a reward R. t+1 =r(S t A tThe environmental state will be determined according to the state transition probability P(S). t+1 |S t A t Transition to S t+1 When t = |T| and S |T| The process terminates when the state is terminated.

[0065] In reinforcement learning, the goal is to choose a policy π to maximize the performance metric J(π), which is often considered as an infinitely discounted total reward. τ represents the trajectory τ = (s0, a0, s1, ...), and τ ~ π means that the distribution on the trajectory τ depends on π.

[0066] Step S22: Definition of a Constrained Markov Decision Process

[0067] A constrained Markov decision process is a Markov decision process with added constraints, which limit the set of allowed policies. Specifically, the Markov decision process is extended with an auxiliary cost function C and a constraint d, where J... C (π) represents the expected discounted return of strategy π relative to the cost function:

[0068] Therefore, the feasible stationary strategy for a constrained Markov decision process is: In the formula, E represents the expectation, τ represents the trajectory τ = (s0, a0, s1, ...), τ ~ π indicates that the distribution on the trajectory τ depends on π, γ is the discount factor, d is the constraint, and a t Belongs to the action space, s t and s t+1 They belong to the state space respectively.

[0069] The objective function of the stationary strategy is:

[0070]

[0071] The objective of a running strategy π is to achieve the goal of running under the constraint Π, which does not exceed a cost threshold. C Under the condition of obtaining the maximum future reward J(π), where Π represents the set of all fixed strategies. C It is the cost function C t+1 For a stationary strategy where the future discounted return is limited to a certain range, J(π) is a function of the reward function R. t+1 Future discount returns.

[0072] Among them, action space A t

[0073] Action space is defined as: This represents the radial structure of the distribution network at time t, that is, This represents a directed graph generating forest associated with a non-splitter distribution network.

[0074] State space S t

[0075] The state space at time t is defined as follows:

[0076] In the formula, p t =[p 1t ,p 2t ,…,p nt ], q t =[q 1t ,q 2t ,…,q nt ], p represents the radial structure of the distribution network at time t-1. it and q it Inject active and reactive power into node i at time t.

[0077] Reward function R t+1

[0078] The goal of distribution network reconfiguration is to minimize network losses and switching costs, while the goal of reinforcement learning is to maximize future discounted returns. Therefore, the reward function is defined as the sum of the negative costs of line losses and the negative costs of switching actions:

[0079]

[0080] In the formula, C l It equals the product of the fixed retail electricity price and the network loss, where the network loss is equal to the active power p at time t. t Reactive power q t and the radial structure of the power distribution network Related; C s It equals the product of the fixed cost of each switch and the number of switch actions.

[0081] Cost function C t+1

[0082] The cost function C t+1 Defined as the sum of the absolute values ​​of voltage violations at all metering nodes:

[0083]

[0084] In the formula, N v It is the collection of all nodes that have voltage measuring devices; and v These are the upper and lower limits of the voltage.

[0085] In step S3, the constrained policy optimization algorithm is used for offline training. When the future reward converges to a stable value, the optimal neural network model is obtained.

[0086] Specifically, the following steps are included:

[0087] Step S31: Solve the objective function transformation

[0088] Based on the corollary of constrained policy optimization theory, the objective function is transformed into the following surrogate optimization objective:

[0089]

[0090] In the formula, θ is the neural network parameter. If action a satisfies the policy The state distribution s under the given conditions, and These are related to future rewards. and future cost returns The relevant advantage function, d is the constraint, D KL (θ||θ i [s]) is distributed in π θ and The KL divergence, δ, is a very small value.

[0091] Step S32: Simplify the solution problem

[0092] The optimization objective in step S31 is approximated by first performing a Taylor expansion on the objective function and constraint functions, resulting in a simplified optimization problem: the objective function is expanded into a linear function using a first-order Taylor expansion, one constraint condition is expanded into multiple linear constraints using a first-order Taylor expansion, and the other constraint condition is expanded into a quadratic function using a second-order Taylor expansion; the approximation is as follows:

[0093]

[0094] In the formula, It's about the advantage function. The expected policy gradient in the future, where θ is the neural network parameter. If action a satisfies the policy The state distribution s under the given conditions; It represents future cost-return, and d represents the limitation; It's about the cost advantage function. Expected policy gradient in the future; It is distributed in π θ and The Laplace operator for the maximum KL-divergence, δ is a very small value.

[0095] Step S33: Use a constrained strategy optimization algorithm to solve the constructed constrained Markov decision process. The pseudocode of the specific algorithm is shown in Algorithm 1.

[0096]

[0097]

[0098] In each iteration, the Γ time step number is first updated, and the policy network parameters are run. Then collect the state-action group {(s t ,a t Gradient estimation is performed using importance sampling on the interval |t=0,1,…,Γ-1}. as well as The specific formula is shown below:

[0099]

[0100]

[0101]

[0102]

[0103] In the formula, Γ is the total number of training time steps, and T is the training cycle. It is a Gaussian strategy. It is the policy gradient. and These are the policy network parameters. The advantage function is related to future rewards and costs, where γ is the discount factor and c t d is the cost function, and d is the constraint on the cost function.

[0104] Next, the strategy optimization problem in step S32 is solved using an online search algorithm; finally, the optimal solution θ is used. i+1 To update the policy network parameters.

[0105] Because the Deep Q-Network (DQN) algorithm allows for any free exploration behavior of the agent during training, this can lead to violations of power flow constraints and overvoltage and undervoltage behaviors in distribution feeders. The Constrained Policy Optimization (CPO) algorithm employed in this invention exhibits excellent performance in network loss control and voltage control.

[0106] This invention models the distribution network reconfiguration problem as a constrained Markov decision process, defining reward and cost functions separately without requiring manual intervention of penalty coefficients. The proposed security reinforcement learning method based on constrained policy optimization can efficiently train neural networks to generate optimal distribution network reconfiguration strategies that satisfy voltage constraints. Furthermore, the proposed method is model-free, requiring no accurate and complete distribution network parameters, and can effectively reduce distribution network line losses while ensuring distribution network security.

[0107] The following example uses a 16-node distribution network to illustrate this invention:

[0108] The distribution network has a base voltage of 100 MVA and a photovoltaic system is installed at node 11. Assuming each line segment has a remote control switch, the number of distribution network reconfiguration operations is calculated to be 190 using the matrix tree theorem. The cost C incurred by each line switchover is... s The cost C associated with network loss is $4. l The maximum and minimum node voltages are set to $0.13 / kWh. and v =0.95. The time series load data is taken from the Irish Energy Regulatory Commission's Smart Metering Project, and the solar power generation data are all from the Southern California site. The 26-week historical dataset is divided into a 25-week training dataset and a 1-week test dataset.

[0109] To verify the effectiveness of the proposed secure reinforcement learning method, namely the Constrained Policy Optimization (CPO) algorithm, this invention compares it with the traditional deep reinforcement learning algorithm, namely the Deep Q-Network (DQN) algorithm. For DQN to handle constraints, this invention adds a penalty term 1000*C to the reward function. t+1 This is to punish any behavior that violates voltage constraints.

[0110] 1. Economic Analysis

[0111] The reward curve during the training phase is as follows: Figure 2 As shown.

[0112] As the number of training steps increases, the reward values ​​of both the DQN and CPO algorithms steadily increase and eventually converge. The reward value is a negative sum of the action switching cost and the network loss cost. This indicates that both algorithms can learn energy-saving and loss-reduction strategies for distribution network reconfiguration, with CPO achieving a higher final reward value and lower economic cost. Compared to DQN, CPO can reduce daily operating costs by approximately 36.4%, thus demonstrating superior performance in network loss control.

[0113] 2. Security Analysis

[0114] The constraint value curve during the training phase is as follows: Figure 3As shown.

[0115] During training, it can be observed that the constraint values ​​of the DQN algorithm are violated more frequently, while the constraint values ​​of the CPO decrease to near zero after a few iterations and remain stable. Since the constraint values ​​are defined as voltage violations during training, the higher the constraint value, the greater the probability and severity of voltage violations. Therefore, the CPO is able to learn strategies that consider voltage safety during training.

[0116] The voltage distribution at each node on a certain day during the CPO testing phase is as follows: Figure 4 As shown.

[0117] A day was randomly selected from the test dataset and used as input to the offline-trained and saved model. The maximum and minimum voltage distribution of each node in the distribution network is as follows: Figure 4 As shown in the figure. It can be seen that, with the optimal neural network model trained by CPO, the voltages of the distribution network were maintained between 0.95 and 1.05 pu during the online testing phase.

[0118] Therefore, the CPO algorithm has good performance in voltage control.

[0119] 3. Real-time analysis

[0120] During the online testing phase, the calculation time for dynamic network reconstruction over 24 hours is only 0.37 seconds, thus meeting the requirements of real-time applications.

Claims

1. A method for energy saving and loss reduction in distribution network reconfiguration based on security reinforcement learning, characterized in that, Includes the following steps: Step S1: Collect historical operating data of the distribution network as learning sample data for security reinforcement learning; Step S2: Model the distribution network reconfiguration as a constrained Markov decision process and build a reinforcement learning environment; Step S3: For the constructed reinforcement learning environment, use the constraint policy optimization algorithm for offline training to obtain the optimal neural network model; Step S4: Solve for distribution network reconfiguration using the optimal neural network model; In step S2, the constrained Markov decision process is as follows: Through an auxiliary cost function Expanding the Markov decision-making process, making Representation strategy The expected discounted return function relative to the cost function: , Therefore, the feasible stationary strategies for a constrained Markov decision process are: In the formula, E represents the expectation. Representing the trajectory , Representing the trajectory The distribution on depends on , This is the discount factor. It is a limitation. Belongs to the action space, and They belong to the state space respectively; The objective function of the stationary strategy is: In the formula, Represents all fixed policy sets. It is the cost function A stable strategy that limits future discount returns. Representing the reward function The future discounted return function; Among them, the reward function for: In the formula, It equals the product of the fixed retail electricity price and the network loss, where the network loss is proportional to the time... active power value reactive power value and the radial structure of the power distribution network Related; It equals the product of the fixed cost of each switch and the number of switch actions. For the action space, For the distribution network in time The radial structure of the power distribution network For state space; Cost function for: In the formula, It is the collection of all nodes that have voltage measuring devices; and These are the upper and lower limits of voltage. For the action space, This is the state space.

2. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 1, characterized in that: In step S1, historical running data for n weeks is collected, where n > 2 and is an integer. The historical operating data includes the active power and reactive power values ​​injected into each node of the distribution network.

3. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 1, characterized in that: in, state space , In the formula, , , Indicates the distribution network in time The radiation structure, and For nodes In time Inject active and reactive power values ​​into the node.

4. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 1, characterized in that: Step S3 includes the following steps: Step S31: Solve for the objective function transformation; Step S32: Simplify the problem-solving process; Step S33: Use a constrained strategy optimization algorithm to solve the constructed constrained Markov decision process.

5. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 4, characterized in that: In step S31: Based on the corollary of constrained policy optimization theory, the objective function is transformed into the following surrogate optimization objective: In the formula, These are neural network parameters. In action Satisfaction Strategy The state below distributed, and These are related to future rewards. and future cost returns Related advantage functions, It is a limitation. It is distributed in and KL divergence, It is a threshold.

6. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 5, characterized in that: In step S32: The optimization objective of step S31 is approximated as the following optimization problem: In the formula, It concerns the dominance function. The expected policy gradient in the future, where, These are neural network parameters. In action Satisfaction Strategy The state below distributed; , It is the future cost return. It is a limitation; It concerns the cost advantage function. Expected policy gradient in the future; It is distributed in and The Laplace operator for the maximum KL-divergence, It is a threshold.

7. The energy-saving and loss-reduction method for distribution network reconfiguration based on security reinforcement learning according to claim 6, characterized in that: In step S33: In each iteration, first update Time steps, and running policy network parameters Then collect the state-action groups. First, the gradient is estimated using importance sampling; second, the policy optimization problem in step S32 is solved using an online search algorithm; finally, the optimal solution is used. To update the policy network parameters.

8. A power distribution network reconfiguration energy-saving and loss-reduction device based on security reinforcement learning, characterized in that, include: The data acquisition module is used to collect historical operating data of the power distribution network as learning sample data for security reinforcement learning. The modeling module is used to model the reconfiguration of the distribution network as a constrained Markov decision process and build a reinforcement learning environment. The training module uses a constraint policy optimization algorithm to train offline the reinforcement learning environment to obtain the optimal neural network model. The solution module solves for distribution network reconfiguration using an optimal neural network model. In the modeling module, the constrained Markov decision process is as follows: Through an auxiliary cost function Expanding the Markov decision-making process, making Representation strategy The expected discounted return function relative to the cost function: , Therefore, the feasible stationary strategies for a constrained Markov decision process are: In the formula, E represents the expectation. Representing the trajectory , Representing the trajectory The distribution on depends on , This is the discount factor. It is a limitation. Belongs to the action space, and They belong to the state space respectively; The objective function of the stationary strategy is: In the formula, Represents all fixed policy sets. It is the cost function A stable strategy that limits future discount returns. Representing the reward function The future discounted return function; Among them, the reward function for: In the formula, It equals the product of the fixed retail electricity price and the network loss, where the network loss is proportional to the time... active power value reactive power value and the radial structure of the power distribution network Related; It equals the product of the fixed cost of each switch and the number of switch actions. For the action space, For the distribution network in time The radial structure of the power distribution network For state space; Cost function for: In the formula, It is the collection of all nodes that have voltage measuring devices; and These are the upper and lower limits of voltage. For the action space, This is the state space.

Images