A Deep Reinforcement Learning-Based Fault Recovery Method for Distribution Networks

Abstract

This invention discloses a distribution network fault recovery method based on deep reinforcement learning, comprising the following steps: constructing an objective function and constraints for a distribution network system containing distributed generation sources, and establishing a distribution network fault recovery model; designing an action space based on loop-switch coding, and filtering infeasible solutions based on radial constraints; building a reinforcement learning framework for distribution network fault recovery, proposing a fault recovery strategy based on the PPO algorithm, training the strategy network, and applying the trained network model to distribution network fault recovery, enabling switch on / off selection based on system state, and achieving load transfer and power flow optimization. This invention effectively improves the fault recovery efficiency of the distribution network and reduces network losses, while ensuring the operational safety of the system.

Description

Technical Field

[0001] This invention belongs to the field of power distribution network fault recovery technology, specifically relating to a power distribution network fault recovery method based on deep reinforcement learning. Background Technology

[0002] In recent years, the demand for energy, especially electricity, has been increasing, and the shortage of primary energy and the environmental problems caused by its combustion have attracted widespread public attention. With the gradual increase in the proportion of distributed generation (DG) grid connection, the initial topology of the traditional distribution network has been altered, impacting the distribution network to a certain extent. The distribution network, located at the end of the power system, is a crucial link directly facing users and bears the important responsibility of supplying and distributing power to user loads. It not only needs to ensure stable operation under normal conditions but also should have strong response capabilities in the event of faults. Power supply security is crucial to the national economy and people's livelihood, and fault recovery is one of the important supports for ensuring power supply. Therefore, research on distribution network fault recovery is of great significance.

[0003] Distribution networks are characterized by "closed-loop design and open-loop operation," and network reconfiguration is a crucial means of fault recovery. During normal operation, sectionalizing switches are closed while tie switches are open to maintain a radial structure. When a fault occurs, the sectionalizing switches at both ends of the fault are immediately disconnected to isolate the fault. Then, the non-faulty loads downstream of the fault are transferred to other lines for short-term emergency power supply, minimizing power loss and preventing fault propagation, thus achieving distribution network fault recovery.

[0004] In research on distribution network fault recovery, various algorithms have been applied, primarily including traditional mathematical programming algorithms, heuristic algorithms, and deep reinforcement learning algorithms. Mathematical programming and heuristic algorithms, when dealing with large-scale distribution networks, require significant hardware computing power and are time-consuming, making it difficult to meet the real-time requirements of fault recovery. Deep reinforcement learning methods, on the other hand, can be pre-trained offline to obtain a well-learned network model, which can be directly invoked for rapid decision-making during recovery, resulting in a significant improvement in solution efficiency compared to traditional methods. With the expansion of distribution networks and the large-scale integration of distributed generation (DG), deep reinforcement learning methods are better suited to meet the demands of fault recovery. Therefore, researching distribution network fault recovery strategies based on deep reinforcement learning has important theoretical and practical significance. Summary of the Invention

[0005] The purpose of this invention is to provide a distribution network fault recovery method based on deep reinforcement learning, which can solve the problem that distribution networks with distributed generation sources cannot quickly and effectively restore loads when line faults occur, thereby improving the efficiency of distribution network fault recovery and ensuring the security of power flow.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] This invention proposes a distribution network fault recovery method based on deep reinforcement learning, which includes the following steps:

[0008] Step 1: Based on the distribution network system containing distributed generation, construct the objective function and constraints, and establish a distribution network fault recovery model;

[0009] Step 2: Design the action space based on loop-switch coding, and filter infeasible solutions based on radial constraints;

[0010] Step 3: Build a reinforcement learning framework for distribution network fault recovery, propose a fault recovery strategy based on the near-end policy optimization algorithm, train the agent, and apply the trained decision network to distribution network fault recovery. It can select the on / off state of the switch according to the system status to realize load transfer and power flow optimization.

[0011] In addition, the power distribution network fault recovery method based on deep reinforcement learning according to the above embodiments of the present invention may also have the following additional technical features:

[0012] In some embodiments of the present invention, in step 1, the parameters of the distribution network fault recovery model include node parameters, branch parameters, and distributed power source parameters. Applying reinforcement learning to perform distribution network fault recovery requires constructing a training set, a validation set, and a test set that includes load demand data, distributed renewable energy output data, and fault line data.

[0013] In some embodiments of the present invention, in step 1, the distribution network system model adopts the IEEE 33-node distribution network model, which provides detailed node and branch data, facilitating system analysis and optimization. Furthermore, to adapt to the current development needs and practical applications of distribution networks, two wind turbines and two photovoltaic distributed power sources are added to the standard model, providing more support for load restoration and power flow optimization.

[0014] Distributed generation (DG) makes full use of environmental resources with minimal pollution, effectively reducing the dependence of surrounding loads on the power grid. On one hand, during power system operation, DG provides a buffer, serving as a backup energy source to support the grid during peak demand periods, reducing system voltage fluctuations, and especially ensuring short-term emergency power supply to critical loads in extreme situations, thus improving system reliability. On the other hand, DG is distributed near loads, directly supplying power to nearby loads, eliminating the cost of erecting transmission lines and building substations, and significantly reducing network losses during power transmission.

[0015] In some embodiments of the present invention, in step 1, when a fault occurs in the distribution network, the losses should be minimized as much as possible, and the impact of the power outage on users should be reduced to a minimum. During the restoration process, parameters such as node voltage and branch power flow should be kept within allowable ranges to ensure the safe and stable restoration of power supply to the distribution network. Therefore, it is necessary to construct an objective function and constraints to establish a distribution network fault recovery model.

[0016] Step 1.1: Establish the objective function of the distribution network fault recovery model:

[0017]

[0018] In the formula, The weighting coefficients corresponding to the loss of load. The weighting coefficient corresponding to network loss; For the cost of load loss, The cost of network damage.

[0019]

[0020] In the formula, A set of nodes; For nodes The access status is a variable between 0 and 1, where 0 represents not connected and 1 represents connected; For nodes The active power load demand.

[0021]

[0022] In the formula, for The on / off state of a branch is a 0-1 variable, where 0 represents disconnection and 1 represents connection. for Network loss on branch lines.

[0023] Step 1.2: Establish constraints for the distribution network fault recovery model:

[0024] The constraints mainly include power balance constraints, line transmission power constraints, node voltage constraints, and network radial constraints, as shown below:

[0025] The power balance constraint:

[0026]

[0027] In the formula, , They are respectively with nodes Net inflow to the connected line node Active power and reactive power, and They are nodes The active and reactive power output of distributed photovoltaic systems, and They are nodes The active and reactive power outputs of the wind turbine generator set and For nodes The active and reactive load demand.

[0028] The line transmission power constraint:

[0029]

[0030] In the formula, and respectively flowing through Upper and lower limits of active power of the branch circuit; and respectively flowing through Upper and lower limits of reactive power of branch circuits; , respectively flowing through The active and reactive power of the branch circuit.

[0031] The node voltage constraint:

[0032]

[0033] In the formula, For nodes The voltage per unit value at that location, This represents the maximum deviation of the node voltage.

[0034] The network radial constraints:

[0035]

[0036] In the formula, and As an auxiliary variable, if for The parent node, then ,otherwise Conversely, if for The parent node, then ,otherwise ; For nodes The set of connected nodes This represents the set of root nodes. In a radial network, each node except the root node has one and only one parent node, while the root node has no parent node.

[0037] In some embodiments of the present invention, in step 2, to reduce the action space of distribution network reconfiguration and eliminate some infeasible solutions, the distribution network system is decomposed into m basic loops, each basic loop containing a tie switch and several segment switches. Based on basic loop theory, the basic loop numbers and switches are combined to construct an action space based on loop-switch coding. Each action includes the basic loop to be decided. and the switch that is disconnected within the loop , ,in This refers to the basic loop number. For the basic loop The set of switches included For switch serial numbers, The basic loop includes roads The number of switches. The loop-closing and loop-resolving decision-making steps for reconstruction are achieved through single-step decision-making, simplifying the reinforcement learning exploration process.

[0038] When making action selection, the basic loop of decision-making Only one switch can be selected. Disconnect. Backtracking by saving the selected action sequence can be mainly divided into two cases:

[0039] (1) If the basic loop is first tested Make a decision, that is, the saved action sequence does not contain a loop. First, close the connecting switch within the loop, then open the switch currently selected by the decision. ;

[0040] (2) If it is not the first time to access the basic loop Decision-making, i.e., the existence of loops in the saved sequence of actions. Then, it is necessary to first reclose the disconnect switch that was previously decided on in this basic loop, and then disconnect the currently selected switch. .

[0041] Action space is defined as:

[0042] This operation eliminates the need for two separate steps of selecting switches to merge and demerge the loops, simplifying the cumbersome decision-making process and also eliminating most switch combinations that could create islanding on non-common branches within the loop, thus improving decision-making efficiency.

[0043] In some embodiments of the present invention, in step 2, the power distribution network can be viewed as a graph composed of nodes and branches. As graph theory shows, a tree is a connected graph without loops; therefore, the radial structure of the power distribution network can be compared to a tree. The number of nodes is... For a distribution network with only one power source (main grid power supply) and a radial structure, the following two conditions must be met:

[0044] (1) The network is connected;

[0045] (2) There are on the network A closed branch.

[0046] When analyzing its radial conditions, first check whether the number of closed branches is sufficient, and secondly check its connectivity, i.e., whether there are any isolated islands.

[0047] In some embodiments of the present invention, step 3 requires transforming the distribution network optimization and reconfiguration problem into a typical Markov decision process, the state of which... ,action ,award The definition is as follows:

[0048] (1) State The state needs to consider all factors that may affect decision-making. For distribution network fault recovery, the operating state of the distribution network at the moment of the fault and the fault location information are key analytical data. Therefore, observing the state... This includes DG output data, load demand data, and line connectivity data for the distribution network. The line connectivity data implicitly contains information about the location of the faulty line. In addition, a loop status flag is set to distinguish between load transfer and optimization / reconfiguration decisions. The loop status of the faulty line is set to 1, which serves as the basis for shielding the loop.

[0049]

[0050] in, , These are the sets of active and reactive power demand for the node load at decision step t, respectively. , These are the sets of active and reactive power outputs of photovoltaic power in step t, respectively. , These are the sets of active and reactive power outputs from wind power, respectively. It is a set of line states. Compared with the optimized and reconstructed line states, it has an additional fault line flag bit. State 0 is open, 1 is closed, and 2 is fault. This is the loop masking flag for step t. The number of bits is equal to the number of basic loops. The initial state value is 0. When the line where the fault is located is selected to be disconnected, the state of the corresponding loop is set to 1, indicating that no further decisions will be made on this loop.

[0051] (2) Actions Actions are defined based on basic loops and include the decision-making loop and switch numbers.

[0052]

[0053] in, For loop-switch encoded actions, each action contains the basic loop for decision-making and the switch to be disconnected within that loop.

[0054] (3) Rewards The training cost mainly consists of three parts: load shedding cost, network loss cost, and safety constraint cost. The reward is obtained by weighted inversion of these three parts. The load shedding cost and network loss cost are the objective function described in step 1.1. The safety constraint cost is the penalty for exceeding the limits of system node voltage or branch power flow, i.e., when the power system node voltage or branch power flow exceeds the safe range, which can be expressed as:

[0055]

[0056] in, The cost of safety constraints for step t; The penalty coefficient for the branch line flow exceeding the limit. This is the penalty factor for exceeding the node voltage limit; This is a branch power flow limit violation indicator, a 0-1 variable representing... Does the branch road exceed the limit? This indicates that the branch line's power flow has exceeded the limit. This indicates that the branch line's power flow has not exceeded the limit; This is a node voltage over-limit flag, a 0-1 variable representing... Is the node out of bounds? This indicates that the node voltage has exceeded the limit. This indicates that the node voltage has not exceeded the limit.

[0057] Therefore, rewards It can be represented as:

[0058]

[0059] In the formula, The cost of unloaded load in step t, The network loss cost for t steps; Weights are imposed to account for safety constraints.

[0060] In some embodiments of the present invention, in step 3, the Proximal Policy Optimization (PPO) algorithm is a policy-based reinforcement learning algorithm. The PPO algorithm is based on the AC architecture and consists of three deep neural networks, namely the Actor_old, Actor, and Critic networks. The Actor network is used to select actions, inputting system state information and outputting the action selection probability in the current state, which needs to be trained and updated with gradient backpropagation. The Actor_old network is used to store the network parameters before updating the Actor network and is used in the optimization process to avoid excessive policy updates. The Critic network is used to calculate the state value, inputting system state information and outputting the value function of the current state, which needs to be trained and updated with gradient backpropagation.

[0061] In some embodiments of the present invention, in step 3, the training method specifically includes the following steps:

[0062] Step 3.1 Initialize the Actor_old, Actor, and Critic networks, where the Actor_old and Actor networks have the same structure;

[0063] Step 3.2 Set the total number of iterations as E, the current iteration number is denoted as epoch, initialize epoch = 1, and initialize the experience pool;

[0064] Step 3.3 Set the maximum number of steps in one iteration as T, the current action step number is t, and the policy update number is M;

[0065] Step 3.4 Initialize t = 1;

[0066] Step 3.5 Read the source-load data and fault location data and initialize the state;

[0067] Step 3.6 At the decision-making moment, first perform action screening based on the fault location and loop mask flag bit to obtain the set of executable actions, and then select an executable action;

[0068] Step 3.7 Execute action A , , ,

[0069] , , , ,

[0070] ,

[0071] , ,

[0065] , ,

[0063] , ,

[0064] ,

[0067] , t ,

[0068] ,

[0062] ,

[0066] , to perform power flow calculation, calculate the load shedding amount, network loss, voltage deviation, etc. after executing the action, and thus obtain the reward ;

[0069] Step 3.8 Store the past experience sequence in the experience pool;

[0070] Step 3.9 If the number of rounds t < T is satisfied, assign t + 1 to t and jump to step 3.6; otherwise, it means that the epoch-th iteration is completed;

[0071] Step 3.10 When learning from the data in the experience pool, first copy the parameters in the Actor network that obtained this batch of data to the Actor_old network;

[0072] Step 3.11 First, input the states of this batch of data into the Critic network to obtain the corresponding estimated value function. Then, use the rewards and the estimated value function to calculate the target value function and the advantage function for each time step;

[0073] Step 3.12 Use the advantage function and the target value function of this batch of data to calculate the loss function of PPO, perform M policy gradient updates on the parameters of the Actor network and the Critic network, and clear the experience pool;

[0074] Step 3.13 If the number of iterations epoch < E is satisfied, assign epoch + 1 to epoch and jump to Step 3.4. Otherwise, it means the iteration is completed and the network training ends.

[0075] In some embodiments of the present invention, in Step 3.6, the action screening process is specifically as follows: In the load transfer stage, set that only the switch of the faulty line can be decided to be disconnected. At this time, according to the t fault location information in the state S, add the actions that do not include the switch of the faulty line in the action space to the Mask sequence. The decision-making network will only select the switch of the faulty line to be disconnected. At the same time, according to the loop information of the selected action, close the tie switch of the basic loop to which the faulty line belongs, and set the corresponding loop flag to 1. In this way, a load transfer is completed. After the load transfer is completed, enter the power flow optimization stage. At this time, according to the information in the state, obtain the basic loops that need to be masked, add the actions that include this basic loop to the Mask sequence, and no longer change the loop where the fault is located. In addition, the power flow optimization stage also applies radial constraint detection to screen out infeasible solutions and perform action masking.

[0076] Compared with the prior art, the beneficial effects of the present invention are:

[0077] (1) The present invention proposes a two-stage fault recovery strategy of transfer-optimization. On the basis of load transfer, it further optimizes the power flow distribution of the system, avoids the power flow over-limit problem that may be caused by only load transfer, and ensures the safe and stable power supply of the system after fault recovery.

[0078] (2) The present invention better adapts to the fault recovery of the distribution network with DG. Compared with traditional optimization methods such as mathematical programming and heuristic algorithms, which are less efficient in dealing with the fault recovery problem of large-scale distribution networks, the fault recovery algorithm based on PPO effectively improves the decision-making and solution speed and meets the real-time requirements of the problem.

[0079] (3) The present invention has made certain improvements to the action set. Based on the loop-switch encoding method, it simplifies the cumbersome steps of the loop closing and closing operation decision during reconstruction. Furthermore, through the infeasible solution screening and action masking method, it further reduces the action space. While achieving the goal of power distribution network fault recovery, it effectively improves the training efficiency of the algorithm. Attached Figure Description

[0080] Figure 1 This is a fault recovery framework diagram in an embodiment of the present invention;

[0081] Figure 2 This is a basic loop partitioning diagram in an embodiment of the present invention;

[0082] Figure 3 This is a flowchart of the radial network detection method of the present invention;

[0083] Figure 4 This is the input-output diagram of the policy network in this embodiment of the invention;

[0084] Figure 5 This is a strategy network structure diagram in an embodiment of the present invention. Detailed Implementation

[0085] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0086] A distribution network fault recovery method based on deep reinforcement learning includes the following steps:

[0087] Step 1: Based on the distribution network system model containing distributed generation, construct the objective function and constraints, and establish an active distribution network fault recovery model;

[0088] The recovery process in a fault scenario differs from the optimization process in a normal operation scenario. During the decision-making process, the faulty line is always isolated and disconnected. Therefore, if the decision involves the basic loop containing the faulty line, the switch to be disconnected must be the faulty line itself; thus, unnecessary actions need to be eliminated. For example... Figure 1As shown, considering the complexity of the optimization problem and to prevent optimization difficulties due to decision-making confusion, the decision-making process is divided into two stages. First, the load transfer decision is completed with the load restoration amount as the primary objective, mainly involving the operation of tie switches. After the decision is made regarding the faulty switch, the loop to which the switch belongs is fixed, and the switch actions within this loop are masked, further optimizing the power flow distribution and network losses to maintain the stability of the system under fault conditions. The two-stage decision-making framework is implemented through cost weight changes and action masking mechanisms, requiring the construction of a unified distribution network fault recovery model.

[0089] Step 1.1: Establish the objective function of the distribution network fault recovery model:

[0090]

[0091] In the formula, The weighting coefficients corresponding to the loss of load. The weighting coefficient corresponding to network loss; For the cost of load loss, The cost of network damage.

[0092]

[0093] In the formula, A set of nodes; For nodes The access status is a variable between 0 and 1, where 0 represents not connected and 1 represents connected; For nodes The active power load demand.

[0094]

[0095] In the formula, for The on / off state of a branch is a 0-1 variable, where 0 represents disconnection and 1 represents connection. for Network loss on branch lines.

[0096] Step 1.2: Establish constraints for the distribution network fault recovery model:

[0097] The constraints mainly include power balance constraints, line transmission power constraints, node voltage constraints, and network radial constraints, as shown below:

[0098] The power balance constraint:

[0099]

[0100] In the formula, , They are respectively with nodes Net inflow to the connected line node Active power and reactive power, and They are nodes The active and reactive power output of distributed photovoltaic systems, and They are nodes The active and reactive power outputs of the wind turbine generator set and For nodes The active and reactive load demand.

[0101] The line transmission power constraint:

[0102]

[0103] In the formula, and respectively flowing through Upper and lower limits of active power of the branch circuit; and respectively flowing through Upper and lower limits of reactive power of branch circuits; , For flowing through The active and reactive power of the branch circuit.

[0104] The node voltage constraint:

[0105]

[0106] In the formula, For nodes The voltage per unit value at that location, This represents the maximum deviation of the node voltage.

[0107] The network radial constraints:

[0108]

[0109] In the formula, and As an auxiliary variable, if for The parent node, then ,otherwise Conversely, if for The parent node, then ,otherwise ; For nodes The set of connected nodes This represents the set of root nodes. In a radial network, each node except the root node has one and only one parent node, while the root node has no parent node.

[0110] Step 2: Design the action space using loop-switch encoding, and filter infeasible solutions based on radial constraints;

[0111] Step 2.1 Action Space Design Based on Loop-Switch Coding

[0112] To reduce the operation space for distribution network reconfiguration and filter out some infeasible solutions, the IEEE 33-node distribution network system is decomposed into five basic loops, each containing one tie switch and several sectionalizing switches. Figure 2 Taking the IEEE 33-node system as an example, the system is divided into five basic loops: L1, L2, L3, L4, and L5. The numbers are arranged according to the sequence of the tie switches, which are 33, 34, 35, 36, and 37 respectively. Loop L1 consists of branches 2, 3, 4, 5, 6, 7, 33, 20, 19, and 18; loop L2 consists of branches 9, 10, 11, 12, 13, 14, and 34; other loops, including their branch switches, are shown in the table below.

[0113] Table 1 Basic Loop and Intra-Loop Switches

[0114]

[0115] Based on the fundamental loop theory, the fundamental loop numbering and switch numbering are combined to construct an action space based on loop-switch coding; each action contains the fundamental loop to be decided. and the switch that is disconnected within the loop , ,in This refers to the basic loop number. For the basic loop The set of switches included For switch serial numbers, The basic loop includes roads The number of switches. The single-step decision-making process for reconstructing loop closure and reopening simplifies the reinforcement learning exploration process;

[0116] When making action selection, the basic loop of decision-making Only one switch can be selected. Disconnect. Backtracking by saving the selected action sequence can be mainly divided into two cases:

[0117] (1) If the basic loop is first tested Make a decision, that is, the saved action sequence does not contain a loop. First, close the connecting switch within the loop, then open the switch currently selected by the decision. ;

[0118] (2) If it is not the first time to access the basic loop Decision-making, i.e., the existence of loops in the saved sequence of actions. Then, it is necessary to first reclose the disconnect switch that was previously decided on in this basic loop, and then disconnect the currently selected switch.

[0119] Step 2.2 Infeasible solution selection based on radial constraints

[0120] In the distribution network optimization and reconfiguration model, a fundamental constraint is that the distribution network should always maintain an open-loop operation, i.e., maintain a radial network structure. Therefore, after making a decision, the agent needs to check whether the selected action violates this constraint. The distribution network can be viewed as a graph composed of nodes and branches. Graph theory shows that a tree is a connected graph without loops; therefore, the radial structure of the distribution network can be compared to a tree. A node has a... For a distribution network with only one power source (main grid power supply) and a radial structure, the following two conditions must be met:

[0121] (1) The network is connected;

[0122] (2) There are on the network A closed branch.

[0123] When analyzing its radial conditions, first check whether the number of closed branches is sufficient, and secondly check its connectivity, i.e., whether there are any isolated islands.

[0124] The specific testing steps are as follows, and the process is as follows: Figure 3 As shown:

[0125] (1) Read the initial network structure data

[0126] First, the node-branch data of the IEEE 33-node distribution network system is read to obtain the branch number and the node information at both ends of the branch, and a directed graph is constructed. Second, the entire line data is traversed, and lines in a closed state are added as directed edges to the connected graph according to the line state after the decision is completed, while lines in an open state are ignored.

[0127] (2) Detecting closed loops

[0128] Count the number of closed branches and see if it equals the number of nodes minus the number of power sources (considering only the root node's main network power supply). If the condition is met, further check the system connectivity; if not, it violates the radial constraint.

[0129] (3) Detect network connectivity

[0130] After satisfying condition (2), it is necessary to check whether the connectivity condition is met, that is, to check whether there are isolated nodes in the system. The correlation matrix composed of network data is used to search for whether there are any nodes that are not connected to the root node. If none are found, the distribution network satisfies the radial constraint.

[0131] Step 3: Construct a reinforcement learning model for fault recovery, train the PPO neural network model, and obtain the trained decision network;

[0132] First, the distribution network optimization and reconfiguration problem needs to be transformed into a typical Markov decision process, such as... Figure 4 As shown. Its state. ,action ,award The definition is as follows:

[0133] (1) State The state needs to consider all factors that may affect decision-making. For distribution network fault recovery, the operating state of the distribution network at the moment of the fault and the fault location information are key analytical data. Therefore, observing the state... This includes DG output data, load demand data, and line connectivity data for the distribution network. The line connectivity data implicitly contains information about the location of the faulty line. In addition, a loop status flag is set to distinguish between load transfer and optimization / reconfiguration decisions. The loop status of the faulty line is set to 1, which serves as the basis for shielding the loop.

[0134]

[0135] in, , These are the sets of active and reactive power demand for the node load at decision step t, respectively. , These are the sets of active and reactive power outputs of photovoltaic power in step t, respectively. , These are the sets of active and reactive power outputs from wind power, respectively. It is a set of line states. Compared with the optimized and reconstructed line states, it has an additional fault line flag bit. State 0 is open, 1 is closed, and 2 is fault. This is the loop masking flag for step t. The number of bits is equal to the number of basic loops. The initial state value is 0. When the line where the fault is located is selected to be disconnected, the state of the corresponding loop is set to 1, indicating that no further decisions will be made on this loop.

[0136] (2) Actions Actions are defined based on basic loops and include the decision-making loop and switch numbers.

[0137]

[0138] in, For loop-switch encoded actions, each action contains the basic loop for decision-making and the switch to be disconnected within that loop.

[0139] (3) Rewards The training cost mainly consists of three parts: load shedding cost, network loss cost, and safety constraint cost. The reward is obtained by weighted inversion of these three parts. The load shedding cost and network loss cost are the objective function described in step 1.1. The safety constraint cost is the penalty for exceeding the limits of system node voltage or branch power flow, i.e., when the power system node voltage or branch power flow exceeds the safe range, which can be expressed as:

[0140]

[0141] in, The cost of safety constraints for step t; The penalty coefficient for the branch line flow exceeding the limit. This is the penalty factor for exceeding the node voltage limit; This is a branch power flow limit violation indicator, a 0-1 variable representing... Does the branch road exceed the limit? This indicates that the branch line's power flow has exceeded the limit. This indicates that the branch line's power flow has not exceeded the limit; This is a node voltage over-limit flag, a 0-1 variable representing... Is the node out of bounds? This indicates that the node voltage has exceeded the limit. This indicates that the node voltage has not exceeded the limit.

[0142] Therefore, rewards It can be represented as:

[0143]

[0144] In the formula, The cost of unloaded load in step t, The network loss cost for t steps; Weighting the cost of safety constraints. Weighting for the cost of load loss. The weight corresponding to the network loss cost.

[0145] PPO is a deep reinforcement learning algorithm based on the Actor-Critic framework. For example... Figure 4 As shown, the main function of the Actor network is to obtain the probability distribution of the output action after the input state. First, the input state is used to extract features through a convolutional neural network to generate the probability distribution of the loop-switch action in the current state. Then, multinomial sampling is used to obtain the action values ​​of the segmented switch and the tie switch in the input state, thereby obtaining the switch state combination in the current system.

[0146] The main function of the Critic network is to estimate the state value. It receives the environmental state as input and outputs the expected return of that state, which is the state value function. Its output is used to calculate the Advantage function, an important parameter when updating the Actor network, which evaluates the advantage of performing an action in the current state relative to the average expected return. The Critic network is trained by minimizing the gap between the predicted state value function and the actual return, i.e., value function approximation.

[0147] The algorithm training process is as follows:

[0148] Step 3.1 Initialize the Actor_old, Actor, and Critic networks. Actor_old has the same network structure as the Actor network;

[0149] Step 3.2 Set the total number of iterations to E, denote the current iteration number as epoch, initialize epoch = 1, and initialize the experience pool;

[0150] Step 3.3 Set the maximum number of steps in one iteration to T, the current action step number to t, and the number of policy updates to M;

[0151] Step 3.4 Initialize t = 1;

[0152] Step 3.5 Read the source load data and fault location data, and initialize the state;

[0153] Step 3.6 At the decision moment, first perform action screening based on the fault location and loop mask flag bit to obtain the set of executable actions, and then select an executable action;

[0154] Step 3.7 Execute action A t , perform power flow calculation, calculate the load shedding, network loss, voltage deviation, etc. after executing the action, so as to obtain the reward ;

[0155] Step 3.8 Store the past experience sequence in the experience pool;

[0156] Step 3.9 If the number of rounds t < T is satisfied, assign t + 1 to t and jump to Step 3.6, otherwise, it means that the epoch - th iteration is completed;

[0157] Step 3.10 When learning from the data in the experience pool, first copy the parameters in the Actor network that obtained this batch of data to the Actor_old network;

[0158] Step 3.11 First, input the status of this batch of data into the Critic network to obtain the corresponding estimated value function. Then, use the reward and the estimated value function to calculate the target value function and the advantage function for each time step;

[0159] Step 3.12 Use the advantage function and the target value function of this batch of data to calculate the loss function of PPO, perform M policy gradient updates on the parameters of the Actor network and the Critic network, and clear the experience pool;

[0160] Step 3.13 If the number of iterations epoch < E is satisfied, assign epoch + 1 to epoch, and jump to Step 3.4. Otherwise, it means the iteration is completed and the network training ends.

[0161] The above content is only an example and explanation of the structure of the present invention. Those skilled in the art of this technology make various modifications or supplements to the described specific embodiments or use similar methods for substitution. As long as they do not deviate from the structure of the present invention or exceed the scope defined by this claim book, they should fall within the protection scope of the present invention.

Claims

1. A power distribution network fault restoration method based on deep reinforcement learning, characterized in that, Comprise the following steps: Step 1, based on the distribution network system containing distributed power supply, construct objective function and constraint condition, establish distribution network fault recovery model; Step 2, design the action space based on loop-switch coding, and screen out infeasible solutions based on radial constraints; The action space design based on loop-switch coding in step 2 is as follows: In order to reduce the action space of power distribution network reconstruction and filter out some infeasible solutions, the power distribution network system is decomposed into m basic loops, each of which contains a tie switch and a number of sectionalizing switches. Based on the basic loop theory, the basic loop number and switch are combined to construct the action space based on loop-switch coding. Each action contains a basic loop L p , p = 1, 2, …, m and the switches within the loop , where p is the basic loop number, , L p contains a set of switches, q is the switch number, and h p , L p contains the number of switches, and the loop closing and loop breaking decision step is realized by single-step decision, which simplifies the exploration process of reinforcement learning. When action selection is performed, the basic loop L p Only one switch can be selected Turn off, backtracking by saving the selected action sequence, divided into two cases: (1) If the first time the basic loop L p is decided, the action sequence saved does not contain the loop L p , then first close the contact in the loop, and then open the switch currently decided (2) If it is not the first time that the basic loop L p decision, i.e. the saved action sequence contains the loop L p then the disconnect switch previously decided on this basic loop L needs to be closed again before the currently selected switch is disconnected Step 3, build a reinforcement learning framework for distribution network fault recovery, propose a fault recovery strategy based on a proximal policy optimization algorithm, train the strategy network, and apply the trained network model to distribution network fault recovery, which can select switch on-off according to the system state, realize load transfer and power flow optimization.

2. The power distribution network fault restoration method based on deep reinforcement learning according to claim 1, characterized in that: The parameters of the distribution network fault recovery model in step 1 include node parameters, branch parameters and distributed power supply parameters. To apply reinforcement learning to distribution network fault recovery, a training set, a validation set and a test set containing load demand data, distributed new energy output data and fault line data need to be constructed.

3. The method of claim 1, wherein: In step 1, the IEEE33 node distribution network system model is used, which provides detailed node and branch data, facilitating system analysis and optimization. At the same time, to adapt to the development needs and actual application of current distribution network, two wind turbines and two photovoltaic distributed power supplies are added to the standard model, providing more support for load recovery and power flow optimization.

4. The power distribution system fault restoration method based on deep reinforcement learning of claim 1, wherein, In step 1: When a fault occurs in the distribution network, the loss should be minimized as much as possible, and the impact on users should be minimized. In the recovery process, the node voltage and branch power flow parameters should be within the allowed range to ensure the safe and stable recovery of the distribution network. Therefore, the objective function and constraint condition should be constructed to establish the distribution network fault recovery model. Step 1.1: Establish the objective function of the distribution network fault recovery model: min: F = ω blackout C blackout + ω loss C loss In the formula, ω blackout is the weight coefficient corresponding to the loss of load, ω loss is the weight coefficient corresponding to the network loss; C blackout is the loss of load cost, C loss is the network loss cost, where N is the set of nodes; u i is the access status of node i, which is a 0-1 variable, 0 for not accessing and 1 for accessing; is the active load demand of node i, In the formula, x ij is the on-off state of the i-j branch, and is a 0-1 variable, 0 being off and 1 being on; is the network loss of the i-j branch; Step 1.2: Establish the constraint condition of the distribution network fault recovery model: The constraint condition includes power balance constraint, line transmission power constraint, node voltage constraint and network radial constraint, as follows: The power balance constraint: where ΔP i line , are the active and reactive power injected into node i by the line connected to node i, P i pv and are the active and reactive power output of the distributed photovoltaic at node i, and are the active and reactive power output of the wind turbine at node i, P i load and are the active and reactive load demand at node i; The line transmission power constraint: wherein, and are the upper and lower limits of the active power flowing through the i-j branch, respectively; and are the upper and lower limits of the reactive power flowing through the i-j branch, respectively; ij , Q ij are the active and reactive power flowing through the i-j branch, respectively. The node voltage constraint: 1 - ζ ≤ U i ≤ 1 + ζ wherein U i is the voltage unit at node i, and ζ is the maximum deviation of the node voltage. The network radial constraint: where b i,j j,i are auxiliary variables, b i,j = 1 if j is the parent of i, otherwise b i,j = 0, and vice versa, b j,i = 1 if i is the parent of j, otherwise b j,i = 0; Ω i is the set of nodes connected to node i, and R g denotes the set of root nodes. In a radial network, each node has and only has one parent node except the root nodes which have no parent node.​ 5. The power distribution system fault restoration method based on deep reinforcement learning according to claim 1, wherein, In step 2, the infeasible solution screening based on radial constraints: one basic constraint condition of the distribution network optimization reconstruction model is that the distribution network should always maintain an open-loop operation state, i.e. maintain a radial network structure. Therefore, after the agent makes a decision, it needs to detect whether the selected action violates this constraint. The distribution network can be regarded as a graph composed of nodes and branches. According to graph theory, a tree is a connected graph without loops. Therefore, the radial structure of the distribution network can be compared with a tree. For a distribution network with n nodes and 1 power supply, if it is a radial structure, it needs to meet the following two conditions: (1) The network is connected; (2) There are n-1 closed branches in the network; When analyzing the radial condition, first detect whether the number of closed branches meets the condition, and then detect its connectivity, i.e. there is no island.

6. The power distribution system fault restoration method based on deep reinforcement learning of claim 1, wherein, In step 3: (1) State S t : The state needs to consider all factors that will affect the decision, for the power distribution network fault recovery problem, the power distribution network operating state at the time of fault and fault location information are the key analysis data, therefore, the observation state S t includes the DG output data of the power distribution network, load demand data and line on-off state data, among which the line on-off data implies the fault line location information, in addition, the loop state flag bit is set to distinguish the load transfer and optimization reconstruction two parts of the decision, the loop state belonging to the fault line is set to 1, as the basis for shielding the loop, where P t load , are the active and reactive power demand set of the t decision step node respectively; P t pv , are the active and reactive power output set of the t step photovoltaic respectively; P t wt , are the active and reactive power output set of the wind power respectively; is the line state set, compared with the line state of the optimization reconstruction, there is an additional fault line flag, state 0 is open, 1 is closed, 2 is fault; is the loop shielding mask flag, the number of bits is equal to the number of basic loops, the initial state value is 0, when the line on which the fault is selected is opened, the corresponding loop state is set to 1, indicating that the loop will not be decided in the future. (2) Action A t : The definition of an action is encoded based on a basic loop, containing the loop and switch number of the decision, wherein, are actions based on loop-switch encoding, each action comprising a basic loop of decisions and switches to be opened within this loop; (3) reward R t The training era cost includes three parts of load loss cost, network loss cost and security constraint cost, and the reward is obtained by weighting and taking the inverse of the three parts. The load loss cost and the network loss cost are the objective function, and the security constraint cost is the punishment of the out-of-limit of the system node voltage or branch flow, that is, the power system node voltage or branch flow exceeds the safe range, which can be expressed as: where, is the security constrained cost of t-step; ω a is the branch flow out-of-limit penalty coefficient, ω u is the node voltage out-of-limit penalty coefficient; μ ij,t is the branch flow out-of-limit flag, a 0-1 variable, representing whether the i-j branch is out-of-limit, μ ij,t = 1 indicates the branch flow is out-of-limit, μ ij,t = 0 indicates the branch flow is not out-of-limit; η i,t is the node voltage out-of-limit flag, a 0-1 variable, representing whether the i node is out-of-limit, η i,t = 1 indicates the node voltage is out-of-limit, η i,t = 0 indicates the node voltage is not out-of-limit, Thus, the reward R t may be expressed as: In the formula, ω blackout is a weight coefficient corresponding to load loss, ω loss is a weight coefficient corresponding to network loss, is a load loss cost of t step, is a network loss cost of t step; ω pf is a security constraint cost weight.

7. The method of claim 1, wherein, In step 3, the training method of the proximal policy optimization algorithm includes the following steps: Step 3.1 initializes the Actor_old, Actor and Critic networks, wherein the Actor_old has the same network structure as the Actor network; Step 3.2 sets the total number of iterations as E, and the current iteration number as epoch, initializes epoch = 1, and initializes the experience pool; Step 3.3 sets the maximum number of iterations as T, the current action step number as t, and the policy update number as M; Step 3.4 initializes t = 1; Step 3.5 reads the source data and fault location data, and initializes the state; Step 3.6 at the decision moment, first, according to the fault location and the loop mask flag, action screening is performed to obtain an executable action set, and then an executable action is selected; Step 3.7 perform action A t , perform power flow calculation, calculate loss of load, network loss and voltage deviation after action execution, and get reward R t ; Step 3.8 stores the past experience sequence into the experience pool; Step 3.9 if the number of rounds t < T is satisfied, t + 1 is assigned to t, and the step 3.6 is jumped to, otherwise, it is indicated that the epoch iteration is completed; Step 3.10 when learning the experience pool data, first, the parameters in the Actor network of the obtained batch data are copied to the Actor_old network; Step 3.11 first, the state of the batch data is input into the Critic network to obtain the corresponding estimated value function, then, the reward and the estimated value function are used to calculate the target value function and the advantage function of each time step; Step 3.12 using the advantage function and the target value function of the batch data, the loss function of the proximal policy optimization algorithm is calculated, and the parameters of the Actor network and the Critic network are updated for M times of policy gradient, and the experience pool is emptied; Step 3.13 if the iteration number epoch < E is satisfied, epoch + 1 is assigned to epoch, and the step 3.4 is jumped to, otherwise, it is indicated that the iteration is completed, and the network training is ended.

8. The power distribution network fault restoration method based on deep reinforcement learning according to claim 7, characterized in that, In the step 3.6, The action screening process is specifically as follows: firstly, in the load transfer stage, only the switch of the fault line can be determined to be opened, at this time, according to the fault location information in the state S t , the actions in the action space which do not contain the switch of the fault line are added to the Mask sequence, the network will only select the switch of the fault line to be opened, and according to the loop information of the selected action, the tie switch of the basic loop to which the fault line belongs is closed, and the flag of the corresponding loop is set to 1, thus completing a load transfer, after the load transfer is completed, the power flow optimization stage is entered, at this time, according to the information in the state S , the basic loop which needs to be shielded is obtained, the actions containing the basic loop are added to the Mask sequence, and the loop where the fault is located is no longer changed, in addition, the radial constraint detection is also applied in the power flow optimization stage to screen out the infeasible solution, and the action screening is performed, the process is referred to step 2.

Images