Power distribution network fault recovery method based on deep reinforcement learning

Abstract

The invention discloses a power distribution network fault recovery method based on deep reinforcement learning, and the method comprises the following steps: constructing an objective function and constraint conditions based on a power distribution network system containing a distributed power supply, and building a power distribution network fault recovery model; designing an action space based on loop-switch coding, and performing infeasible solution screening based on radial constraint; a reinforcement learning framework of power distribution network fault recovery is established, a fault recovery strategy based on a PPO algorithm is provided, a strategy network is trained, a trained network model is applied to power distribution network fault recovery, switch on-off selection can be performed according to a system state, and load transfer and power flow optimization can be realized. According to the invention, the fault recovery efficiency of the power distribution network is effectively improved, the network loss is reduced, and the operation safety of the system is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of distribution network fault recovery, and specifically relates to a distribution network fault recovery method based on deep reinforcement learning. Background Art

[0002] In recent years, people's demand for energy, especially electric energy, has been growing. The shortage of primary energy and the environmental problems caused by its combustion have attracted widespread public attention. With the gradual increase in the proportion of distributed generation (DG) connected to the grid, the initial topology of the traditional distribution network has changed, which has had a certain impact on the distribution network. The distribution network is at the end of the power system and is a key link directly facing users. It bears the important responsibility of supplying and distributing electricity to user loads. It is necessary not only to ensure stable operation under normal conditions, but also to have strong response capabilities in the event of a fault. Power supply is related to the national economy and people's livelihood, and fault recovery is one of the important supports for ensuring power supply. Therefore, the research on distribution network fault recovery is of great significance.

[0003] The distribution network has the characteristics of "closed-loop design and open-loop operation". Network reconstruction is an important means of fault recovery in the distribution network. When the distribution network is operating normally, the section switches are closed and the tie switches are disconnected to maintain the radial structure. When a fault occurs in the distribution network, the section switches at both ends of the fault are immediately disconnected to isolate the fault, and then the non-fault power-off load downstream of the fault is transferred to other lines for short-term emergency power supply, reducing power loss and preventing the spread of the fault, so as to achieve fault recovery in the distribution network.

[0004] Different types of algorithms have been applied in the research of distribution network fault recovery, mainly traditional mathematical programming algorithms, heuristic algorithms and deep reinforcement learning algorithms. When mathematical programming algorithms and heuristic algorithms are used for large-scale distribution network objects, they have high requirements on hardware computing power and take too long to implement, making it difficult to meet the real-time requirements of fault recovery. Deep reinforcement learning methods can perform offline training in advance to obtain a well-learned network model, and can directly call the model for rapid decision-making during recovery, which greatly improves the efficiency of solving problems compared to traditional methods. With the expansion of the scale of distribution networks and the large-scale grid connection of DGs, deep reinforcement learning methods can better adapt to fault recovery needs. Therefore, it is of great theoretical and practical significance to study distribution network fault recovery strategies based on deep reinforcement learning. Summary of the invention

[0005] The purpose of the present invention is to provide a distribution network fault recovery method based on deep reinforcement learning, which can solve the problem that a distribution network containing distributed power sources cannot quickly and effectively restore loads when a line fault occurs, thereby improving the efficiency of distribution network fault recovery and ensuring the safety of power flow.

[0006] To achieve the above object, the present invention adopts the following technical solution:

[0007] The present invention proposes a distribution network fault recovery method based on deep reinforcement learning, the method comprising the following steps:

[0008] Step 1: Based on the distribution network system containing distributed power sources, construct the objective function and constraint conditions, and establish a distribution network fault recovery model;

[0009] Step 2: Design an action space based on loop-switch encoding and screen infeasible solutions based on radial constraints;

[0010] Step 3: Build a reinforcement learning framework for distribution network fault recovery, propose a fault recovery strategy based on the proximal strategy optimization algorithm, train the intelligent agent, and apply the trained decision network to distribution network fault recovery. The switch on and off selection can be made according to the system status to achieve load transfer and power flow optimization.

[0011] In addition, the distribution network fault recovery method based on deep reinforcement learning according to the above embodiment of the present invention may also have the following additional technical features:

[0012] In some embodiments of the present invention, in step 1, the parameters of the distribution network fault recovery model include node parameters, branch parameters, and distributed power supply parameters. Applying reinforcement learning to perform distribution network fault recovery requires constructing a training set, a verification set, and a test set containing load demand data, distributed new energy output data, and fault line data.

[0013] In some embodiments of the present invention, in step 1, the distribution network system model adopts the IEEE33 node distribution network model, which provides detailed node and branch data to facilitate system analysis and optimization. At the same time, in order to adapt to the current development needs and practical applications of the distribution network, two wind power and two photovoltaic distributed power sources are added on the basis of the standard model to provide more support means for load recovery and power flow optimization.

[0014] Distributed generation can make full use of environmental resources with minimal pollution, effectively reducing the dependence of surrounding loads on power grid power supply. On the one hand, during the operation of the power system, the existence of DG gives the system a certain degree of buffering capacity, supporting the power grid as a backup energy source during peak power consumption, reducing system voltage fluctuations, and especially in extreme cases, ensuring short-term emergency power supply for critical loads and improving system power supply reliability. On the other hand, DG is distributed near the load and can directly supply power to nearby power loads, saving the cost of erecting transmission lines and building substations, and significantly reducing network losses in power transmission.

[0015] In some embodiments of the present invention, in step 1, when a fault occurs in the distribution network, the loss should be minimized as much as possible to minimize the impact of the power outage on users. During the recovery process, parameters such as node voltage and branch current are guaranteed to be within the allowable range to ensure that the distribution network can restore power safely and stably. Therefore, it is necessary to construct the objective function and constraint conditions to establish a distribution network fault recovery model.

[0016] Step 1.1: Establish the objective function of the distribution network fault recovery model:

[0017] min:F=ω blackout C blackout +ω loss C loss

[0018] In the formula, ω blackout is the weight coefficient corresponding to the load loss, ω loss is the weight coefficient corresponding to the network loss; C blackout is the load loss cost, C loss The cost of network loss.

[0019]

[0020] Where N is the node set; u i is the access status of node i, which is a 0-1 variable, 0 means no access, 1 means access; P i load is the active load demand of node i.

[0021]

[0022] In the formula, x ij is the on / off state of the ij branch, which is a 0-1 variable, 0 for disconnection and 1 for connection; is the network loss of the ij branch.

[0023] Step 1.2: Establish the constraints of the distribution network fault recovery model:

[0024] The constraints mainly include power balance constraints, line transmission power constraints, node voltage constraints, and network radial constraints, as shown below:

[0025] The power balance constraint:

[0026]

[0027] In the formula, ΔP i line , are the active power and reactive power of the line connected to node i flowing into node i, respectively, ipv and are the active and reactive output of distributed photovoltaic at node i, P i wt and are the active and reactive outputs of the wind turbine at node i, P i load and is the active and reactive load demand of node i.

[0028] The line transmission power constraint is:

[0029]

[0030] In the formula, and are the upper and lower limits of the active power flowing through the ij branch respectively; and are the upper and lower limits of reactive power flowing through the ij branch respectively; P ij , Q ij are the active power and reactive power flowing through the ij branch respectively.

[0031] The node voltage constraints are:

[0032] 1-ζ≤U i ≤1+ζ

[0033] Where U i is the voltage per unit value at node i, and ζ is the maximum deviation of the node voltage.

[0034] The network radial constraints:

[0035]

[0036] Where b i,j and b j,i is an auxiliary variable. If j is the parent node of i, then b i,j =1, otherwise b i,j = 0, otherwise, if i is the parent node of j, then b j,i =1, otherwise b j,i =0;Ω i is the set of nodes connected to node i, R g Represents the root node set. In a radial network, each node except the root node has only one parent node, while the root node has no parent node.

[0037] In some embodiments of the present invention, in step 2, in order to reduce the action space of distribution network reconstruction and screen out some infeasible solutions, the distribution network system is decomposed into m basic loops, each basic loop includes a tie switch and several segment switches, based on the basic loop theory, the basic loop number and the switch are combined together to construct an action space based on loop-switch coding; each action includes the basic loop L to be decided p ,p=1,2,...,m and the switches disconnected in the loop Where p is the basic loop number, The basic loop L p The switch set included, q is the switch number, h p The basic loop L p The number of switches included. The reconstructed loop closing and loop unclosing decision steps are realized through single-step decision making, which simplifies the process of reinforcement learning exploration;

[0038] When making action selection, the basic loop L decided p Only one switch can be selected Disconnect, backtrack by saving the selected action sequence, which can be divided into two situations:

[0039] (1) If the basic loop L is first p Make a decision, that is, the saved action sequence does not contain loop L p , then close the tie switch in the loop first, and then open the switch currently decided

[0040] (2) If it is not the first time to use the basic loop L p decision, that is, the saved action sequence contains the loop L p , you need to close the disconnected switch decided by the last time on this basic loop, and then disconnect the currently selected switch

[0041] The action space is defined as:

[0042]

[0043] This operation does not require the switch selection for closing and opening the ring in two steps, which simplifies the tedious decision-making steps and can also screen out most of the switch combination schemes that may cause islands on non-public branches within the ring, thereby improving decision-making efficiency.

[0044] In some embodiments of the present invention, in step 2, the distribution network can be regarded as a graph consisting of nodes and branches. According to graph theory, a tree is a connected graph without loops, so the radial structure of the distribution network can be compared with a tree. A distribution network with n nodes and 1 power source (main grid power source) needs to meet the following two conditions if it is a radial structure:

[0045] (1) The network is connected;

[0046] (2) There are n-1 closed branches in the network.

[0047] When analyzing its radial conditions, first check whether the number of closed branches is met, and then check its connectivity, that is, there are no isolated islands.

[0048] In some embodiments of the present invention, in step 3, the distribution network optimization reconstruction problem needs to be transformed into a typical Markov decision process, where the state S t 、Action A t , Reward R t The definition is as follows:

[0049] (1) State S t :The state needs to consider all factors that may affect the decision. For the distribution network fault recovery problem, the distribution network operation state and fault location information at the time of the fault are key analysis data. Therefore, the observation state S t It includes DG output data, load demand data and line on / off status data of the distribution network, among which the line on / off data implies the location information of the faulty line. In addition, a loop status flag is set to distinguish the load transfer and optimization reconstruction decisions, and the loop status to which the faulty line belongs is set to 1 as the basis for shielding the loop.

[0050]

[0051] Among them, P t load , are respectively the active and reactive demand sets of node loads at decision step t; P t pv , are the photovoltaic active and reactive power output sets of t steps respectively; P t wt , are wind power active and reactive power output sets respectively; is a line state set. Compared with the optimized and reconstructed line state, it has an additional fault line flag. State 0 is disconnected, 1 is closed, and 2 is faulty. It is the loop shielding mask flag of step t. The number of bits is equal to the number of basic loops. The initial state value is 0. When the line where the fault is located is disconnected, the corresponding loop state is set to 1, indicating that no decision will be made for this loop in the future.

[0052] (2) Action A t : The definition of the action is encoded based on the basic loop, including the decision loop and switch number.

[0053] A t ∈A loop_swi

[0054] Among them, A loop_swi The action space is based on loop-switch encoding, where each action contains the basic loop of decision making and the switch to be opened in this loop.

[0055] (3) Reward R t : The cost during training mainly includes three parts: load loss cost, network loss cost and safety constraint cost. The reward is obtained by weighted inversion of the three parts. Among them, the load loss cost and network loss cost are the objective function described in claim 4. The safety constraint cost is the penalty for the system node voltage or branch current exceeding the limit, that is, the power system node voltage or branch current exceeds the safety range, which can be expressed as:

[0056]

[0057] in, is the safety constraint cost of step t; ω a is the branch power flow over-limit penalty coefficient, ω u is the node voltage over-limit penalty coefficient; μ ij,t is the branch flow over-limit flag, which is a 0-1 variable representing whether the ij branch is over-limited, μ ij,t =1 means the branch current exceeds the limit, μ ij,t =0 means the branch current is within the limit; η i,t is the node voltage over-limit flag, which is a 0-1 variable representing whether the i node is over-limit, η i,t =1 indicates that the node voltage exceeds the limit, η i,t =0 means the node voltage is within the limit.

[0058] Therefore, the reward R t It can be expressed as:

[0059]

[0060] In the formula, is the load loss cost of step t, is the network loss cost of step t; pf is the security constraint cost weight.

[0061] In some embodiments of the present invention, in step 3, the Proximal Policy Optimization (PPO) algorithm is a policy-based reinforcement learning algorithm. The PPO algorithm is based on the AC architecture and consists of three deep neural networks in total, namely the Actor_old, Actor, and Critic networks. The Actor network is used to select actions, input system state information, and output the action selection probability in the current state, and it needs to be trained and updated with gradient backpropagation. The Actor_old network is used to store the network parameters before updating the Actor network and is used in the optimization process to avoid excessive policy update amplitude. The Critic network is used to calculate the state value, input system state information, and output the value function of the current state, and it needs to be trained and updated with gradient backpropagation.

[0062] In some embodiments of the present invention, in step 3, the training method specifically includes the following steps:

[0063] Step 3.1 Initialize the Actor_old, Actor, and Critic networks, where the Actor_old and Actor networks have the same structure;

[0064] Step 3.2 Set the total number of iterations to E, the current number of iterations is denoted as epoch, initialize epoch = 1, and initialize the experience pool;

[0065] Step 3.3 Set the maximum number of steps in one iteration to T, the current number of action steps is t, and the number of policy updates is M;

[0066] Step 3.4 Initialize t = 1;

[0067] Step 3.5 Read the source-load data and fault location data, and initialize the state;

[0068] Step 3.6 At the decision-making moment, first perform action screening according to the fault location and loop mask flag bit to obtain an executable action set, and then select an executable action;

[0069] Step 3.7 Execute action A t , perform power flow calculation, calculate the load shedding amount, network loss, voltage deviation, etc. after executing the action, so as to obtain the reward R t ;

[0070] Step 3.8 Store the past experience sequence in the experience pool;

[0071] Step 3.9 If the number of rounds t < T is satisfied, assign t + 1 to t, and jump to step 3.6, otherwise, it means that the epoch-th iteration is completed;

[0072] In step 3.10 when learning from the data in the experience pool, first copy the parameters in the Actor network that obtained this batch of data to the Actor_old network;

[0073] In step 3.11, first input the states of this batch of data into the Critic network to obtain the corresponding estimated value function. Then, use the rewards and the estimated value function to calculate the target value function and the advantage function for each time step;

[0074] In step 3.12, use the advantage function and the target value function of this batch of data to calculate the loss function of PPO, perform M times of policy gradient updates on the parameters of the Actor network and the Critic network, and empty the experience pool;

[0075] In step 3.13, if the number of iterations epoch < E is satisfied, assign epoch + 1 to epoch and jump to step 3.4; otherwise, it means the iteration is completed and the network training ends.

[0076] In some embodiments of the present invention, in step 3.6, the action screening process is specifically as follows: in the load transfer stage, set that only the switch of the faulty line can be decided to be disconnected. At this time, according to the t fault location information in the state S, add the actions in the action space that do not include the switch of the faulty line to the Mask sequence. The decision-making network will only select the switch of the faulty line to be disconnected. At the same time, according to the loop information of the selected action, close the tie switch of the basic loop to which the faulty line belongs, and set the corresponding loop flag to 1, so that a load transfer is completed. After the load transfer is completed, enter the power flow optimization stage. At this time, according to the information in the state, obtain the basic loops that need to be masked, add the actions including this basic loop to the Mask sequence, and no longer change the loop where the fault is located. In addition, the power flow optimization stage also applies the radial constraint detection to screen out infeasible solutions and perform action masking.

[0077] Compared with the prior art, the beneficial effects of the present invention are:

[0078] (1) The present invention proposes a two-stage fault recovery strategy of transfer - optimization. On the basis of load transfer, it further optimizes the power flow distribution of the system, avoids the power flow over-limit problem that may be caused by only load transfer, and ensures the safe and stable power supply of the system after fault recovery.

[0079] (2) The present invention better adapts to the fault recovery of distribution networks with DGs. Compared with traditional optimization methods such as mathematical programming and heuristic algorithms, which are less efficient in dealing with the fault recovery problems of large-scale distribution networks, the fault recovery algorithm based on PPO effectively improves the decision-making and solving speed and meets the real-time requirements of the problems.

[0080] (3) The present invention has made certain improvements to the action set. Based on the loop-switch encoding method, the tedious steps of making decisions on loop closing and loop opening operations during reconstruction are simplified. By screening infeasible solutions and using action masking methods, the action space is further reduced. While achieving the goal of distribution network fault recovery, the training efficiency of the algorithm is effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 is a fault recovery framework diagram in an embodiment of the present invention;

[0082] Figure 2 is a basic loop partition diagram in an embodiment of the present invention;

[0083] Figure 3 It is a radial network detection flow chart of the method of the present invention;

[0084] Figure 4 is a strategy network input and output diagram in an embodiment of the present invention;

[0085] Figure 5 It is a strategic network structure diagram in an embodiment of the present invention. DETAILED DESCRIPTION

[0086] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0087] A distribution network fault recovery method based on deep reinforcement learning includes the following steps:

[0088] Step 1: Based on the distribution network system model containing distributed power sources, construct the objective function and constraint conditions, and establish an active distribution network fault recovery model;

[0089] The recovery process in a fault scenario is different from the optimization process in a normal operation scenario. In the decision-making process, the faulty line is always isolated and disconnected. Therefore, if the decision is made within the basic ring where the faulty line is located, the disconnected switch can only be the faulty line, so it is necessary to shield unnecessary actions. Figure 1As shown in the figure, considering the complexity of problem optimization and preventing optimization difficulties caused by decision confusion, the decision-making process is divided into two stages. First, the load transfer decision is completed with the load recovery amount as the first goal, mainly operating the interconnection switch. After the decision is made to the faulty switch, the loop to which the switch belongs is fixed, and the switch action in this loop is shielded, and the power flow distribution and network loss are further optimized to maintain the stability of the system under fault operation. The two-stage decision framework is implemented through cost weight change and action shielding mechanism. First, a unified distribution network fault recovery model needs to be constructed.

[0090] Step 1.1: Establish the objective function of the distribution network fault recovery model:

[0091] min:F=ω blackout C blackout +ω loss C loss

[0092] In the formula, ω blackout is the weight coefficient corresponding to the load loss, ω loss is the weight coefficient corresponding to the network loss; C blackout is the load loss cost, C loss The cost of network loss.

[0093]

[0094] Where N is the node set; u i is the access status of node i, which is a 0-1 variable, 0 means no access, 1 means access; P i load is the active load demand of node i.

[0095]

[0096] In the formula, x ij is the on / off state of the ij branch, which is a 0-1 variable, 0 for disconnection and 1 for connection; is the network loss of the ij branch.

[0097] Step 1.2: Establish the constraints of the distribution network fault recovery model:

[0098] The constraints mainly include power balance constraints, line transmission power constraints, node voltage constraints, and network radial constraints, as shown below:

[0099] The power balance constraint:

[0100]

[0101] Where ΔP i line , are the active power and reactive power of the line connected to node i flowing into node i, respectively, i pv and are the active and reactive output of distributed photovoltaic at node i, P i wt and are the active and reactive outputs of the wind turbine at node i, respectively. i load and is the active and reactive load demand of node i.

[0102] The line transmission power constraint is:

[0103]

[0104] In the formula, and are the upper and lower limits of the active power flowing through the ij branch respectively; and are the upper and lower limits of reactive power flowing through the ij branch respectively; P ij , Q ij is the active power and reactive power flowing through the ij branch.

[0105] The node voltage constraints are:

[0106] 1-ζ≤U i ≤1+ζ

[0107] Where U i is the voltage per unit value at node i, and ζ is the maximum deviation of the node voltage.

[0108] The network radial constraints:

[0109]

[0110] Where b i,j and b j,i is an auxiliary variable. If j is the parent node of i, then b i,j =1, otherwise b i,j = 0, otherwise, if i is the parent node of j, then b j,i =1, otherwise b j,i =0;Ω i is the set of nodes connected to node i, R g Represents the root node set. In a radial network, each node except the root node has only one parent node, while the root node has no parent node.

[0111] Step 2: Design an action space encoded by loop-switch and screen infeasible solutions based on radial constraints;

[0112] Step 2.1 Action space design based on loop-switch encoding

[0113] In order to reduce the action space of distribution network reconstruction and filter out some infeasible solutions, the IEEE 33-node distribution network system is decomposed into five basic loops, each of which contains a tie switch and several section switches. Figure 2 Taking the IEEE33 node system shown in the figure as an example, the system is divided into 5 basic loops, namely L1, L2, L3, L4 and L5. The serial numbers are arranged in the order of the tie switches, including tie switches 33, 34, 35, 36 and 37. Loop L1 consists of branches 2, 3, 4, 5, 6, 7, 33, 20, 19 and 18; ring network L2 consists of branches 9, 10, 11, 12, 13, 14 and 34; other loops contain branch switches as shown in the table below.

[0114] Table 1 Basic loop and switches within the loop

[0115]

[0116] Based on the basic loop theory, the basic loop number and switch are combined to construct an action space based on loop-switch encoding; each action contains the basic loop L to be decided. p ,p=1,2,...,m and the switches disconnected in the loop Where p is the basic loop number, The basic loop L p The switch set included, q is the switch number, h p The basic loop L p The number of switches included. The reconstructed loop closing and loop unclosing decision steps are realized through single-step decision making, which simplifies the process of reinforcement learning exploration;

[0117] When making action selection, the basic loop L decided p Only one switch can be selected Disconnect, backtrack by saving the selected action sequence, which can be divided into two situations:

[0118] (1) If the basic loop L is first p Make a decision, that is, the saved action sequence does not contain loop L p , then close the tie switch in the loop first, and then open the switch currently decided

[0119] (2) If it is not the first time to use the basic loop L p decision, that is, the saved action sequence contains the loop L p , you need to close the disconnected switch decided by the last time on this basic loop, and then disconnect the currently selected switch

[0120] The action space is defined as:

[0121]

[0122] Step 2.2 Screening of infeasible solutions based on radial constraints

[0123] In the distribution network optimization and reconstruction model, a basic constraint is that the distribution network should always maintain an open-loop operation state, that is, maintain a radial network structure. Therefore, after the intelligent agent makes a decision, it is necessary to detect whether the selected action violates this constraint. The distribution network can be regarded as a graph composed of nodes and branches. From graph theory, we know that a tree is a connected graph without loops, so the radial structure of the distribution network can be compared with a tree. A distribution network with n nodes and 1 power source (main grid power supply) needs to meet the following two conditions if it is a radial structure:

[0124] (1) The network is connected;

[0125] (2) There are n-1 closed branches in the network.

[0126] When analyzing its radial conditions, first check whether the number of closed branches is met, and then check its connectivity, that is, there are no isolated islands.

[0127] The specific detection steps are as follows: Figure 3 As shown:

[0128] (1) Read the network initial structure data

[0129] First, the node-branch data of the IEEE33-node distribution network system is read, the branch sequence number and the node information at both ends of the branch are obtained, and a directed graph is constructed. Secondly, the entire line data is traversed, and according to the line status after the decision is completed, the lines with closed status are added to the connected graph as directed edges, and the lines with open status are ignored.

[0130] (2) Detection of closed loop

[0131] Count the number of closed branches to see if it is equal to the number of nodes minus the number of power supplies (consider only the power supply of the root node main network). If the conditions are met, further check the system connectivity. If not, the radial constraint is violated.

[0132] (3) Check network connectivity

[0133] After the condition in (2) is met, it is necessary to detect whether the connectivity condition is met, that is, to detect whether there are islands in the system. The association matrix composed of network data is used to search whether there are nodes that are not connected to the root node. If not, the distribution network satisfies the radial constraint.

[0134] Step 3: Build a reinforcement learning model for fault recovery, train the PPO neural network model, and obtain a trained decision network;

[0135] First, the distribution network optimization reconstruction problem needs to be transformed into a typical Markov decision process, such as Figure 4 Its status S t 、Action A t , Reward R t The definition is as follows:

[0136] (1) State S t :The state needs to consider all factors that may affect the decision. For the distribution network fault recovery problem, the distribution network operation state and fault location information at the time of the fault are key analysis data. Therefore, the observation state S t It includes DG output data, load demand data and line on / off status data of the distribution network, among which the line on / off data implies the location information of the faulty line. In addition, a loop status flag is set to distinguish the load transfer and optimization reconstruction decisions, and the loop status to which the faulty line belongs is set to 1 as the basis for shielding the loop.

[0137]

[0138] Among them, P t load , are respectively the active and reactive demand sets of node loads at decision step t; P t pv , are the photovoltaic active and reactive power output sets of t steps respectively; P t wt , are wind power active and reactive power output sets respectively; is a line state set. Compared with the optimized and reconstructed line state, it has an additional fault line flag. State 0 is disconnected, 1 is closed, and 2 is faulty. It is the loop shielding mask flag of step t. The number of bits is equal to the number of basic loops. The initial state value is 0. When the line where the fault is located is disconnected, the corresponding loop state is set to 1, indicating that no decision will be made for this loop in the future.

[0139] (2) Action A t : The definition of the action is encoded based on the basic loop, including the decision loop and switch number.

[0140] A t ∈A loop_swi

[0141] Among them, Aloop_swi The action space is based on loop-switch encoding, where each action contains the basic loop of decision making and the switch to be opened in this loop.

[0142] (3) Reward R t : The cost during training mainly includes three parts: load loss cost, network loss cost and safety constraint cost. The reward is obtained by weighted inversion of the three parts. Among them, the load loss cost and network loss cost are the objective function described in claim 4. The safety constraint cost is the penalty for the system node voltage or branch current exceeding the limit, that is, the power system node voltage or branch current exceeds the safety range, which can be expressed as:

[0143]

[0144] in, is the safety constraint cost of step t; ω a is the branch power flow over-limit penalty coefficient, ω u is the node voltage over-limit penalty coefficient; μ ij,t is the branch flow over-limit flag, which is a 0-1 variable representing whether the ij branch is over-limited, μ ij,t =1 means the branch current exceeds the limit, μ ij,t =0 means the branch current is within the limit; η i,t is the node voltage over-limit flag, which is a 0-1 variable representing whether the i node is over-limit, η i,t =1 indicates that the node voltage exceeds the limit, η i,t =0 means the node voltage is within the limit.

[0145] Therefore, the reward R t It can be expressed as:

[0146]

[0147] In the formula, is the load loss cost of step t, is the network loss cost of step t; pf is the safety constraint cost weight, ω blackout is the load loss cost weight, ω loss is the weight corresponding to the network loss cost.

[0148] PPO is a deep reinforcement learning algorithm based on the Actor-Critic framework. Figure 4 As shown in the figure, the main function of the Actor network is to obtain the probability distribution of the output action after the input state. First, the input state is feature extracted through a convolutional neural network to generate the probability distribution of the loop-switch action in the current state, and then the action values ​​of the segmented switch and the tie switch in the input state are obtained through polynomial sampling, and then the switch state combination in the current system is obtained.

[0149] The main function of the Critic network is to estimate the state value. It receives the environmental state as input and outputs the expected return of that state, which is the state value function. Its output is used to calculate the Advantage function, an important parameter during the update of the Actor network, which evaluates the advantage of executing an action in the current state relative to the average expected return. The Critic network is trained by minimizing the gap between the predicted state value function and the actual return, i.e., value function approximation.

[0150] The algorithm training process is as follows:

[0151] Step 3.1 Initialize the Actor_old, Actor, and Critic networks. Actor_old has the same network structure as the Actor network;

[0152] Step 3.2 Set the total number of iterations as E, denote the current iteration number as epoch, initialize epoch = 1, and initialize the experience pool;

[0153] Step 3.3 Set the maximum number of steps in one iteration as T, the current action step as t, and the policy update times as M;

[0154] Step 3.4 Initialize t = 1;

[0155] Step 3.5 Read the source-load data and fault location data to initialize the state;

[0156] Step 3.6 At the decision-making moment, first perform action screening based on the fault location and loop mask flag bit to obtain the set of executable actions, and then select an executable action;

[0157] Step 3.7 Execute action A t , perform power flow calculation, calculate the load shedding amount, network loss, voltage deviation, etc. after executing the action, so as to obtain the reward R t ;

[0158] Step 3.8 Store the past experience sequence in the experience pool;

[0159] Step 3.9 If the number of steps t < T is satisfied, assign t + 1 to t and jump to Step 3.6; otherwise, it means that the epoch - th iteration is completed;

[0160] Step 3.10 When learning from the data in the experience pool, first copy the parameters in the Actor network that obtained this batch of data to the Actor_old network;

[0161] Step 3.11 First, input the status of this batch of data into the Critic network to obtain the corresponding estimated value function. Then, use the reward and the estimated value function to calculate the target value function and the advantage function for each time step;

[0162] Step 3.12 Use the advantage function and the target value function of this batch of data to calculate the loss function of PPO, perform M times of policy gradient updates on the parameters of the Actor network and the Critic network, and empty the experience pool;

[0163] Step 3.13 If the number of iterations epoch < E is satisfied, assign epoch + 1 to epoch, and jump to Step 3.4. Otherwise, it means the iteration is completed and the network training ends.

[0164] The above content is only an example and explanation of the structure of the present invention. Those skilled in the art of this technology make various modifications or supplements to the described specific embodiments or use similar methods to replace them. As long as they do not deviate from the structure of the present invention or exceed the scope defined by this claim book, they should all belong to the protection scope of the present invention.

Claims

1. A distribution network fault recovery method based on deep reinforcement learning, characterized in that: The following steps are involved: Step 1: Based on the distribution network system containing distributed power sources, construct the objective function and constraint conditions, and establish a distribution network fault recovery model; Step 2: Design an action space based on loop-switch encoding and screen infeasible solutions based on radial constraints; Step 3: Build a reinforcement learning framework for distribution network fault recovery, propose a fault recovery strategy based on the proximal strategy optimization algorithm, train the strategy network, and apply the trained network model to distribution network fault recovery. Switch on and off selection can be made according to the system status to achieve load transfer and power flow optimization.

2. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: The parameters of the distribution network fault recovery model in step 1 include node parameters, branch parameters, and distributed power supply parameters. Applying reinforcement learning to perform distribution network fault recovery requires constructing a training set, a validation set, and a test set containing load demand data, distributed new energy output data, and fault line data.

3. The distribution network fault recovery method based on deep reinforcement learning according to claim 1 is characterized in that: In step 1, the IEEE33-node distribution network system model is used, which provides detailed node and branch data to facilitate system analysis and optimization. At the same time, in order to adapt to the development needs and practical applications of the current distribution network, two wind power units and two photovoltaic distributed power sources are added on the basis of the standard model to provide more support means for load recovery and power flow optimization.

4. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: In step 1: When a distribution network fails, the loss should be minimized as much as possible, the impact of power outage on users should be minimized, and parameters such as node voltage and branch current should be kept within the allowable range during the recovery process to ensure that the distribution network can restore power safely and stably. Therefore, it is necessary to construct the objective function and constraints to establish a distribution network fault recovery model. Step 1.1: Establish the objective function of the distribution network fault recovery model: min:F=ω blackout C blackout +oh loss C loss In the formula, ω blackout is the weight coefficient corresponding to the load loss, ω loss is the weight coefficient corresponding to the network loss; C blackout is the load loss cost, C loss For network loss cost, Where N is the node set; u i is the access status of node i, which is a 0-1 variable, 0 means no access, 1 means access; P i load is the active load demand of node i, In the formula, x ij is the on / off state of the ij branch, which is a 0-1 variable, 0 for disconnection and 1 for connection; is the network loss of the ij branch; Step 1.2: Establish the constraints of the distribution network fault recovery model: The constraints mainly include power balance constraints, line transmission power constraints, node voltage constraints, and network radial constraints, as shown below: The power balance constraint: In the formula, ΔP i line , are the active power and reactive power of the line connected to node i flowing into node i, respectively, i pv and are the active and reactive output of distributed photovoltaic at node i, P i wt and are the active and reactive outputs of the wind turbine at node i, respectively. i load and is the active and reactive load demand of node i; The line transmission power constraint is: In the formula, and are the upper and lower limits of the active power flowing through the ij branch respectively; and are the upper and lower limits of reactive power flowing through the ij branch respectively; P ij , Q ij are the active power and reactive power flowing through the ij branch respectively; The node voltage constraints are: 1-ζ≤U i ≤1+ζ Where U i is the voltage per unit value at node i, ζ is the maximum deviation of the node voltage; The network radial constraints: Where b i,j and b j,i is an auxiliary variable. If j is the parent node of i, then b i,j =1, otherwise b i,j = 0, otherwise, if i is the parent node of j, then b j,i =1, otherwise b j,i =0;Ω i is the set of nodes connected to node i, R g Represents the root node set. In a radial network, each node except the root node has only one parent node, while the root node has no parent node.

5. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: The action space design based on loop-switch encoding in step 2 is as follows: In order to reduce the action space of distribution network reconstruction and screen out some infeasible solutions, the distribution network system is decomposed into m basic loops, each of which contains a tie switch and several segment switches. Based on the basic loop theory, the basic loop number and the switch are combined to construct an action space based on loop-switch coding; each action contains the basic loop L to be decided p ,p=1,2,...,m and the switches disconnected in the loop Where p is the basic loop number, The basic loop L p The switch set included, q is the switch number, h p The basic loop L p The number of switches included. The reconstructed loop closing and loop unclosing decision steps are realized through single-step decision making, which simplifies the process of reinforcement learning exploration; When making action selection, the basic loop L decided p Only one switch can be selected Disconnect, backtrack by saving the selected action sequence, which can be divided into two situations: (1) If the basic loop L is first p Make a decision, that is, the saved action sequence does not contain loop L p , then close the tie switch in the loop first, and then open the switch currently decided (2) If it is not the first time to use the basic loop L p decision, that is, the saved action sequence contains the loop L p , you need to close the disconnected switch decided by the last time on this basic loop, and then disconnect the currently selected switch The action space is defined as:

6. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: In the step 2, the screening of infeasible solutions based on radial constraints: A basic constraint of the distribution network optimization reconstruction model is that the distribution network should always maintain an open-loop operation state, that is, maintain a radial network structure. Therefore, after the intelligent agent makes a decision, it is necessary to detect whether the selected action violates this constraint; The distribution network can be viewed as a graph consisting of nodes and branches. From graph theory, we know that a tree is a connected graph without loops. Therefore, the radial structure of the distribution network can be compared with a tree. A distribution network with n nodes and 1 power source needs to meet the following two conditions if it is a radial structure: (1) The network is connected; (2) There are n-1 closed branches in the network; When analyzing its radial conditions, first check whether the number of closed branches is met, and then check its connectivity, that is, there are no isolated islands.

7. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: In step 3: (1) State S t :The state needs to consider all factors that may affect the decision. For the distribution network fault recovery problem, the distribution network operation state and fault location information at the time of the fault are key analysis data. Therefore, the observation state S t It includes DG output data, load demand data and line on / off status data of the distribution network. The line on / off data implies the location information of the faulty line. In addition, the loop status flag is set to distinguish the load transfer and optimization reconstruction decisions. The loop status to which the faulty line belongs is set to 1 as the basis for shielding the loop. Among them, P t load , are respectively the active and reactive demand sets of node loads at decision step t; P t pv , are the photovoltaic active and reactive power output sets of t steps respectively; P t wt , are wind power active and reactive power output sets respectively; is a line state set. Compared with the optimized and reconstructed line state, it has an additional fault line flag. State 0 is disconnected, 1 is closed, and 2 is faulty. It is the loop shielding mask flag. The number of bits is equal to the number of basic loops. The initial state value is 0. When the faulty line is disconnected, the corresponding loop state is set to 1, indicating that no decision will be made for this loop in the future. (2) Action A t :The definition of the action is based on the encoding of the basic loop, including the decision loop and switch number, A t ∈A loop_swi Among them, A loop _ swi is an action space based on loop-switch coding, as specifically defined in claim 5; (3) Reward R t : The cost during training mainly includes three parts: load loss cost, network loss cost and safety constraint cost. The reward is obtained by weighted inversion of the three parts. Among them, the load loss cost and network loss cost are the objective function, and the safety constraint cost is the penalty for the system node voltage or branch flow exceeding the limit, that is, the power system node voltage or branch flow exceeds the safety range, which can be expressed as: in, is the safety constraint cost of step t; ω a is the branch power flow over-limit penalty coefficient, ω u is the node voltage over-limit penalty coefficient; μ ij,t is the branch flow over-limit flag, which is a 0-1 variable representing whether the ij branch is over-limited, μ ij,t =1 means the branch current exceeds the limit, μ ij,t =0 means the branch current is within the limit; η i,t is the node voltage over-limit flag, which is a 0-1 variable representing whether the i node is over-limit, η i,t =1 indicates that the node voltage exceeds the limit, η i,t =0 means the node voltage is within the limit. Therefore, the reward R t It can be expressed as: In the formula, is the load loss cost of step t, is the network loss cost of step t; pf is the security constraint cost weight.

8. A distribution network fault recovery method based on deep reinforcement learning according to claim 1, characterized in that: In step 3, the training method of the proximal strategy optimization algorithm specifically includes the following steps: Step 3.1 Initialize the Actor_old, Actor, and Critic networks, where the Actor_old has the same network structure as the Actor network; Step 3.2 Set the total number of iterations to E, denote the current iteration number as epoch, initialize epoch = 1, and initialize the experience pool; Step 3.3 Set the maximum number of steps in one iteration to T, the current action step number to t, and the policy update number to M; Step 3.4 Initialize t = 1; Step 3.5 Read the source-load data and fault location data, and initialize the state; Step 3.6 At the decision-making moment, first perform action screening according to the fault location and loop mask flag bit to obtain the set of executable actions, and then select an executable action; Step 3.7 Execute action A t , perform power flow calculation, calculate the load loss, network loss and voltage deviation after the action is executed, and then get the reward R t ; Step 3.8 Store the past experience sequence in the experience pool; Step 3.9 If the number of steps t < T, assign t + 1 to t and jump to Step 3.6; otherwise, it means that the epoch-th iteration is completed; Step 3.10 When learning the data in the experience pool, first copy the parameters in the Actor network that obtained this batch of data to the Actor_old network; Step 3.11 First input the state of this batch of data into the Critic network to obtain the corresponding estimated value function. Then, use the reward and the estimated value function to calculate the target value function and the advantage function at each time step; Step 3.12 Use the advantage function and the target value function of this batch of data to calculate the loss function of the proximal policy optimization algorithm, perform M times of policy gradient updates on the parameters of the Actor network and the Critic network, and empty the experience pool; Step 3.13 If the iteration number epoch < E, assign epoch + 1 to epoch and jump to Step 3.4; otherwise, it means that the iteration is completed and the network training ends.

9. A distribution network fault recovery method based on deep reinforcement learning according to claim 8, characterized in that: In the said Step 3.6: The specific action screening process is as follows: first, in the load transfer stage, the switch that can only decide the fault line is disconnected. At this time, according to the state S t The fault location information in the action space is added to the Mask sequence, and the action that does not contain the fault line switch in the action space is added to the Mask sequence. The decision network will only select the fault line switch to disconnect. At the same time, according to the loop information of the selected action, the tie switch of the basic loop to which the fault line belongs is closed, and the corresponding loop The flag is set to 1, which completes a load transfer. After the load transfer is completed, the flow optimization phase begins. The basic loop that needs to be shielded is obtained by using the information. The action containing this basic loop is added to the Mask sequence, and the loop where the fault is located is no longer changed. In addition, radial constraint detection is also applied in the power flow optimization stage to screen out infeasible solutions and perform action shielding. The process refers to step 2.

Images