Distributed power supply-containing power distribution network fault recovery method based on graph reinforcement learning
By modeling the distribution network topology as a graph structure and combining graph neural networks with reinforcement learning algorithms, the problems of high computational overhead and unstable training in traditional methods under dynamic topologies are solved, and efficient fault recovery decision-making and intelligent control are achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZAOZHUANG POWER SUPPLY COMPANY OF STATE GRID SHANDONG ELECTRIC POWER
- Filing Date
- 2026-01-27
- Publication Date
- 2026-04-24
AI Technical Summary
Traditional distribution network fault recovery methods have high computational overhead when facing large-scale networks or dynamic topologies, making it difficult to meet the real-time recovery requirements. Furthermore, deep reinforcement learning suffers from insufficient input representation, excessive action space, and unstable training in distribution network scenarios. The sparsity of reward signals also makes it difficult for agents to learn.
By employing graph reinforcement learning, the power distribution network topology is modeled as a graph structure, and embedded representation is performed using graph neural networks. Combined with reinforcement learning algorithms, a sparse reward allocation mechanism is designed to improve the generalization ability and learning efficiency of the agent.
It enables sensitive perception of changes in the distribution network topology, improves the intelligence level and execution efficiency of fault recovery decisions, and significantly enhances the resilience and self-healing capability of the distribution network.
Smart Images
Figure CN121920457A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of power system automation and fault recovery, specifically to a fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning, applicable to online fault recovery scenarios of distribution networks with a large number of distributed power sources and dynamically changing topologies. Background Technology
[0002] With the increasing proportion of new energy sources, distributed power sources such as photovoltaic, wind power, and energy storage systems are being widely connected to the distribution network, and the power system is gradually transforming into an active distribution network. These systems are characterized by complex structures, variable power flow, and highly dynamic operating states. When extreme weather, equipment aging, or operational disturbances cause line or node failures, the connectivity and reliability of the power grid are severely affected. How to quickly and reasonably restore critical loads and reconstruct the network topology after a fault has become a key issue in improving the resilience and self-healing capabilities of the distribution network. Traditional distribution network fault recovery mainly relies on heuristic rule-based or optimization algorithm-based methods, such as mixed-integer linear programming or metaheuristic search. While these methods can obtain relatively optimal solutions, they often require complex mathematical modeling and high computational overhead when dealing with large-scale networks or dynamic topologies, making it difficult to meet the real-time recovery requirements in disaster scenarios. Furthermore, the limited number of measurement points in the distribution network makes it difficult to fully perceive the system state, further limiting the practicality of traditional methods. In recent years, deep reinforcement learning has provided new ideas for intelligent control of power systems. However, traditional DRL still has limitations in distribution network scenarios: its input is usually a fixed-dimensional vector, which cannot fully represent the graph topology features of the distribution network; at the same time, the action space expands exponentially with the network size, leading to unstable training and difficulty in convergence. In addition, the reward signal for distribution network fault recovery tasks is highly sparsity, making it difficult for agents to learn effective policies from limited success samples.
[0003] To address the aforementioned issues, this invention proposes a graph reinforcement learning method that combines graph neural networks with reinforcement learning algorithms. By using graph structure encoding, it achieves sensitive perception of topological changes, thereby improving the agent's generalization ability and learning efficiency. However, existing research still lacks a systematic implementation framework applicable to distribution network restoration scenarios. Therefore, this invention proposes a distribution network fault restoration method and system based on graph reinforcement learning, achieving structured state representation, legal action selection, and sparse reward allocation, thus significantly improving the intelligence level and execution efficiency of restoration decision-making. Summary of the Invention
[0004] Objective: This invention proposes a method and system for distribution network fault recovery based on graph reinforcement learning, aiming to address the problems of insufficient state representation, excessive action space, slow training convergence, and weak generalization ability in traditional rule-based, optimization-based, or simple deep reinforcement learning methods for dynamic distribution network recovery. This invention models the distribution network topology as a graph structure and utilizes graph neural networks to embed network nodes and branches, combined with reinforcement learning algorithms to achieve optimal recovery decisions after a fault. To address the issues of sparse rewards and low training efficiency in distribution network recovery tasks, this invention designs a sparse reward allocation mechanism, distributing the sparse rewards generated by successful recovery at the end of each round back to each action according to weights, thereby increasing the learning signal density and accelerating convergence.
[0005] Technical solution: The specific steps of this invention are as follows: S1: Distribution System Topology Data Modeling. The distribution network is abstracted as a time-varying graph structure (nodes, branches, and tie switches), and multi-modal information such as electrical information, fault status, load level, and energy storage status are incorporated into the graph structure to form "situational awareness graph data" that can be updated in real time with faults, recovery actions, and the external environment. In the physical topology graph, all connected electrical nodes are marked and visited using a depth-first or breadth-first search method. The set of nodes visited in each search is merged into a fault recovery node until all nodes have been visited; a fault recovery topology graph is built with tie switches as edges.
[0006] S2: Multi-timescale, multi-agent fault recovery decision-making process modeling. A multi-agent decision-making process model for fault recovery is established, with the recovery amount of key loads within the distribution network as the objective function. The model follows six types of constraints: power flow constraints, node voltage constraints, line current constraints, distribution network topology constraints, and generator capacity constraints.
[0007] S3: Construct the agent-environment interaction interface, divided into power flow analysis and topology checking modules. A power flow model of the distribution system is established, where distributed generation (DG) is modeled as a constant voltage source, and its output will be part of the power flow results. Furthermore, a topology graph of the distribution network is created and saved for topology analysis. Then, upon receiving actions from the agent, a path between load nodes and DGs is searched using depth-first search, and radial topology constraints are checked. For control operations that do not violate topology constraints, the actions are executed and power flow calculations are performed. After all constraint checks and power flow calculations are completed, the current state of the system for the next stage is obtained, and the reward is calculated and fed back to the agent.
[0008] S4: Construct a graph reinforcement learning model. First, the actual distribution network containing distributed power sources is abstracted into graph data. The graph data includes two parts: the network topology G of the actual distribution network and its voltage, current and other feature data. Then, the abstracted graph data is input into the GRL model. In the GRL, a two-layer graph neural network is first built to process the graph data, complete the reception, information extraction, aggregation and transformation of the graph data, and then the processed feature information is passed to the downstream DQN task.
[0009] S5: Establish a priority experience playback and sparse reward allocation mechanism model. In the process of distribution network fault recovery decision-making, a memory will be generated after each complete action cycle. Before storing the memory in the memory pool, a judgment is made. If the timely reward of this memory meets the standard of the preferred memory, it will be stored in the memory bank. Otherwise, it will be skipped and the next action will be carried out directly.
[0010] S6: Model Training and Online Application: Combining graph reinforcement learning with the policy solving process, the model is trained to obtain the optimal resilience improvement policy, thereby enhancing the resilience of the distribution network. The agent model obtained during training is saved and used in the application phase to improve decision-making speed during application.
[0011] In step S1, the power distribution network can be abstracted as a set of vertices and edges, or a graph as a set of nodes and their relationships. Vertices, or nodes, generally point to the entity being studied, while edges point to the connections between entities, such as power lines between nodes in a power system. Mathematically, a graph is denoted as... Let V represent the set of nodes and E represent the set of edges. The adjacency matrix is commonly used to describe the structural information of a graph. Let the graph... It contains n nodes, and the node set is Then its adjacency matrix ,,in: Graph data consists of a graph G and feature data X representing the nodes and edges within the graph. Graph G represents the topology, while feature data X represents the characteristic information of nodes and edges. For example, in a power system, a graph abstracted from the actual system wiring relationships represents the connection relationships between nodes, i.e., the network topology. Node voltage and line current are feature data on nodes and edges, respectively. The graph and feature data together constitute the graph data.
[0012] A method for generating time-series diagrams for distribution network load recovery problems: at time... Abstracting the distribution network as a directed graph with attributes , where the set of nodes Includes busbars, load nodes, distributed generation (DG) units, and mobile energy storage units (MESS), edge collection It includes physical wiring and tie switches. A node feature vector is defined for each node. The components are arranged in a fixed order: node voltage magnitude and phase angle. Node active power injection Node reactive power injection Load importance weight Measurement confidence level Distance metric to the nearest DG If it is MESS, then it includes the battery charge rate. and mobility indicators .
[0013] For either side Define edge feature vectors The components include: line resistance, line reactance, current line current amplitude, upper limit of line heat capacity, and on / off status of the edges. The graph data is updated in real-time after action execution and fault occurrence, forming a time sequence graph. ,in Given the feature set of nodes and edges, perform a depth-first search or breadth-first search on connectivity at any given time step to identify sets of isolated subnets. The subgraph formed by the set of interconnecting switches is used as the "fault recovery topology" for subsequent action validity determination. The above graph modeling ensures that the graph neural network parameters can be shared when the number of nodes or edges changes with expansion or temporary failure, making the strategy transferable between topologies of different sizes.
[0014] In step S2, a multi-agent decision-making process model for fault recovery is established. The recovery problem is modeled as a partially observable stochastic game (POSG) with control at two time scales: a fast time domain and a slow time domain. Fast actions are switching operations; slow actions are MESS movement / scheduling. Define system state , For the first The model's objective function is to maximize the power restoration of the lost load, minimize the number of switching operations in the restoration strategy, and minimize network losses, i.e.: In the formula, where It is in the Step after node The increase in active power during recovery This represents the number of switching operations performed in this step. For network loss, For MESS mobile costs, If a safety constraint is violated, the value is 1. The optimization objective is to maximize the expected discounted return. System constraints include power flow constraints based on the branch-flow distributed power flow model, node voltage limits, line thermal capacity limits, single-source constraints for single microgrids, and MESS energy and mobility timing constraints.
[0015] The constraints mainly stem from the requirement to ensure the safe and stable operation of the power grid throughout the entire fault recovery process, including: 1) Power flow constraints in power distribution systems. The normal operation of a power distribution system requires satisfying power flow constraints, the specific expressions of which are as follows: 2) Node voltage constraints. When dynamically dividing a distribution network into microgrids, it is necessary to ensure that the node voltage remains within a reasonable range: In the formula: for Time Node The lower limit of voltage amplitude for Time Node The upper limit of voltage amplitude, The node at time t The voltage amplitude.
[0016] 3) Line current constraints. To prevent line overload, the line current in the distribution network system should be within its limits: In the formula: Let be the lower bound of the current in line n at time t. Let be the upper bound of the current in line n at time t. Let t be the current in line n at time t.
[0017] 4) Topology Constraints. During the post-disaster recovery phase, through dynamic microgrid grouping, each microgrid can utilize its own power generation resources to restore critical loads. For load restoration, a single-source, single-microgrid control strategy is adopted. That is, each microgrid is powered by only one distributed power source, and critical loads within the microgrid are powered through a single distributed power source and a single path. Each microgrid operates independently, and there are no pathways between any two microgrids. Considering that when the distribution network is decomposed into multiple isolated microgrids, it is necessary to ensure that the network always maintains a radial structure, the topology constraint for any originally connected distribution network can be expressed as: In the formula: To represent nodes and A binary value indicating whether the nodes are connected. and When connected, When node and When not connected, For a radial distribution network structure, the topological constraints can be simplified as follows: 5) Distributed power generation capacity constraints. As an emergency response resource, the output of distributed power sources is limited by their capacity throughout the recovery period, and their output should remain within the specified limits: In the formula: and They are respectively Power supply The lower limits of active power and reactive power output. and They are respectively Power supply The upper limit of active power and reactive power output.
[0018] In step S3, an agent-environment interaction interface (AEI) is constructed. The environment needs to receive actions from the agent and then provide feedback on the system state and reward. To achieve this, it is necessary to ensure that all parameters related to actions, states, and rewards are available, and that all violations of constraints can be detected in the environment. The construction process of the AEI is as follows: 1) Build a power distribution network system in Python and initialize the time to... The state is initialized to .
[0019] 2) Receive an action from the agent This set of actions is decomposed into energy storage distribution actions. With line action .
[0020] 3) Based on the action Perform topology analysis on the distribution network structure to determine whether any actions violate topology constraints. If a violation occurs, then... The state is terminated. = (If the condition is met, proceed to step 2); otherwise, proceed to step 4.
[0021] 4) Execute actions and run power flow analysis. If power flow constraints are violated, then... The state is terminated. = (If necessary, proceed to step 2); otherwise, proceed to step 5.
[0022] 5) In the power distribution system simulation environment, perform load restoration operations, calculate rewards, generate the final state, and calculate the final reward.
[0023] 6) and Feedback is given to the intelligent agent.
[0024] In step S4, a graph reinforcement learning model is constructed. First, the actual distribution network containing distributed generation is abstracted as graph data. This graph data includes two parts: the network topology G of the actual distribution network and its characteristic data such as voltage and current. Then, the abstracted graph data is input into the GRL model. The GRL first constructs a two-layer graph neural network to process the graph data, completing the reception, information extraction, aggregation, and transformation of the graph data. The processed feature information is then passed to the downstream task. In the downstream DQN framework, two layers of GCN are also embedded in both the eval net and the target net to further extract feature information. These two GCN layers are updated along with the DQN, and finally, the eval net outputs the action value.
[0025] Since the input data for each layer of the Graph Network Neural Network (GCN) is graph data, the graph G in the graph data can be constructed in real time according to the current actual network topology, provided that the connection relationships of each node in the current distribution network are obtainable. The input and output dimensional parameters of the graph neural network do not need to change. In other words, graph data of different dimensions can share GCN neural network parameters during propagation. For example, if the number of nodes in the current distribution network increases due to expansion, and the input of the neural network is node information such as node voltage, the input data dimension for the fully connected neural network will increase with the increase in the number of nodes. However, the neural network dimension in the model is fixed. In this case, the input dimension does not match the input layer dimension of the fully connected neural network, and the fully connected neural network in deep reinforcement learning will no longer be applicable. GCN, on the other hand, is not affected by changes in the number of nodes in graph G because the parameter matrix of each GCN layer is shared during the aggregation of node information. This is why the graph reinforcement learning distribution network fault recovery model proposed in this patent is endowed with the ability to adapt to changes in network topology.
[0026] Based on the above Graph Neural Network (GCN), a reinforcement learning MDP model is constructed: 1) Action Space. The action space consists of the agent's decision variables, including the energy storage allocation location and the execution state of the remote control switch. The action space can be represented as: In the formula: A binary variable representing the operation of a remote control switch. Indicates switch exist A disconnect operation is performed at all times, and a close operation is performed otherwise. Therefore, the action space... It is discrete, consisting of a finite number of binary quantities.
[0027] 2) State Space. The state space is divided into two categories. One category is load data, which is the load data restored after the decision is made; the other category is distributed generation data, which is the output limit and actual output data of distributed generation. According to the definition of state space S, its internal parameters contain continuous variables (load, power, voltage, current, etc.), because state space S is continuous.
[0028] 3) Reward Function. Constraints are categorized based on the severity of the penalty for violation. Strong constraints include power flow constraints, line current constraints, topology constraints, and generator capacity constraints in power distribution systems; weak constraints include node voltage constraints. The reward factor for violating the corresponding constraint and feeding it back to the agent is set as follows: In the formula, This represents the reward portion of the reward function for the i-th action in the current round. The power loss due to the load after the i-th action is completed. This refers to the network power loss after the i-th action is completed. The total load of the power distribution system is considered. Taking into account the switching operation time and error rate in the actual system, the number of switching actions should be as small as possible to achieve the same fault recovery effect. Therefore, the number of actions in the current round, i, is placed in the denominator. The additional reward portion represents the increase in load recovery rate compared to the previous action in this round, describing the new contribution of the current action to the fault recovery task. If the current action has an improved fault recovery rate compared to the previous action, it means that the current action has had a new positive impact on fault recovery; conversely, if... This indicates that the current action not only failed to restore more of the lost power load, but also further expanded the power outage area. Additionally, if the current action is the first action of this round, that is... When the system load recovery rate is 100% after the current action is completed, and all operational constraints are met, a larger sparse reward value is assigned to strengthen the guidance of the agent's learning direction.
[0029] In step S5, a model for priority experience replay and sparse reward allocation mechanism is established.
[0030] (1) Prioritize experience replay To improve sample utilization efficiency, this patent employs a preferred experience replay method suitable for offline reinforcement learning. Due to the existence of sparse rewards, most experiences stored in the experience pool do not receive rewards and therefore have no training guidance value. The preferred experience replay method selectively stores action memories with large TD-errors, retaining experiences with training value, thus saving experience pool capacity while improving the quality of memories in the pool. It is worth noting that experiences with high training value are not the same as experiences with high reward values, because during agent training, it can learn which actions are good from high-reward experiences and which actions are unsuitable from high-penalty experiences with negative rewards.
[0031] In the process of distribution network fault recovery decision-making, a memory is generated after each complete action cycle. Before storing the memory in the memory pool, a judgment is made. If the timely reward of this memory meets the criteria for preferred memory, it is stored in the memory bank; otherwise, it is skipped and the next action is carried out directly.
[0032] (2) Sparse reward distribution mechanism In distribution network fault recovery decision-making, a great reward is only awarded if the current action fully restores the lost load while satisfying various operational constraints. This also means the current action is the last action in the current round; therefore, the great reward in the reward function is sparse. To ensure the agent fully recognizes the value of the preceding preparatory actions, this patent designs a sparse reward allocation mechanism. If the current action earns a great reward, the sparse reward (great) is allocated proportionally to each action in the current round: In the formula, Let represent the total number of actions in the current round, and 'i' represent the i-th action in the current round. If the last action in this round receives a great reward, then actions closer to the last action (i.e., those further down the line) should be given more importance. Actions closer to the last action that receives a great reward will receive a higher proportion of sparse rewards, which aligns with practical needs.
[0033] In step S6, model training and online application take place. During the training phase, the evaluation network and target network are initialized, and an experience replay pool is established. Each training round begins with a random fault scenario. The agent selects an action based on its current state and a mask, and verifies the action results through power flow simulation. After execution, state transitions, rewards, and completion flags are recorded. At the end of each round, a sparse reward allocation mechanism is applied to adjust the rewards, and high-value experiences are written into the replay pool. Network updates employ mini-batch sampling, delayed updates of the target network, and a greedy exploration strategy to ensure learning stability and generalization ability. The specific training process is as follows: (1) Initialize the power distribution network: load, line, topology and other parameters are used to simulate the power system.
[0034] (2) Initialize the experience pool, Q-network, and target Q-network for each agent. Initialize the environment state. ,time .
[0035] (3) Use a greedy algorithm to select actions from the Q network .
[0036] (4) Interact with the power distribution system simulation environment through the AEI interface to perform corresponding topology analysis and power flow analysis, and generate new states. And calculate the reward Other relevant information, storage experience Go to the experience pool.
[0037] (5) Sample small batches of data from the experience pool.
[0038] Approximate target action value If the action Violation of strong constraints, ,otherwise .
[0039] (6) Minimize the loss function using the stochastic gradient descent algorithm and update the Q-network parameters. After every C updates, the target Q network is replaced with the Q network.
[0040] (7) If the maximum training cycle is reached, end the training and save the deep Q network; otherwise, go to (3).
[0041] The intelligent agent interacts with the environment through the AEI interface to obtain state-value information and approximates the value of the target action. The Q-network is updated and iterated using the stochastic gradient descent algorithm. A greedy algorithm is used to restrict the agent's free exploration and decision-making. Greedy algorithms represent the agent's choice of actions. Sometimes, The probability of choosing a random action. The algorithm selects the action with the highest probability. The greedy algorithm used in this patent is as follows: In the formula: The initial exploration rate, This is the rate decay factor, whose value is related to the state and the size of the action space. and These represent the current step number and the maximum training steps, respectively. The agent's free exploration rate during the training phase is adjusted through exponential decay. In the initial training phase, the agent is given high degrees of freedom to explore, fully exploring potential decision-making scenarios. Later in the training phase, the exploration rate is gradually reduced, driving the agent to choose the optimal decision and improving the convergence of the training process. Attached Figure Description
[0042] Figure 1 Flowchart of fault recovery process for distribution networks including distributed power sources Figure 2 Flowchart of reinforcement learning algorithm Figure 3 Sparse reward distribution mechanism diagram Detailed Implementation
[0043] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.
[0044] A fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning includes the following steps: S1: Distribution system topology data modeling. Based on the connection relationships between electrical nodes in the distribution network, an electrical topology diagram of the distribution network is established, and a fault recovery topology is constructed. Connection data of nodes and tie switches are read from the distribution network topology information. All connected electrical nodes are marked and visited in the physical topology diagram using a depth-first or breadth-first search. The set of nodes visited in each search is merged into a fault recovery node until all nodes have been visited. A fault recovery topology diagram is constructed with tie switches as edges.
[0045] S2: Establish a multi-agent decision-making process model for fault recovery, with the recovery amount of key loads within the distribution network as the objective function. The model follows six types of constraints: power flow constraints, node voltage constraints, line current constraints, distribution network topology constraints, and generator capacity constraints.
[0046] S3: Construct the agent-environment interaction interface, divided into power flow analysis and topology checking modules. A power flow model of the distribution system is established, where distributed generation (DG) is modeled as a constant voltage source, and its output will be part of the power flow results. Furthermore, a topology graph of the distribution network is created and saved for topology analysis. Then, upon receiving actions from the agent, a path between load nodes and DGs is searched using depth-first search, and radial topology constraints are checked. For control operations that do not violate topology constraints, the actions are executed and power flow calculations are performed. After all constraint checks and power flow calculations are completed, the current state of the system for the next stage is obtained, and the reward is calculated and fed back to the agent.
[0047] S4: Construct a graph reinforcement learning model. First, the actual distribution network containing distributed power sources is abstracted into graph data. The graph data includes two parts: the network topology G of the actual distribution network and its voltage, current and other feature data. Then, the abstracted graph data is input into the GRL model. In the GRL, a two-layer graph neural network is first built to process the graph data, complete the reception, information extraction, aggregation and transformation of the graph data, and then the processed feature information is passed to the downstream DQN task.
[0048] S5: Establish a priority experience playback and sparse reward allocation mechanism model. In the process of distribution network fault recovery decision-making, a memory will be generated after each complete action cycle. Before storing the memory in the memory pool, a judgment is made. If the timely reward of this memory meets the standard of the preferred memory, it will be stored in the memory bank. Otherwise, it will be skipped and the next action will be carried out directly.
[0049] S6: Model Training and Online Application: Combining graph reinforcement learning with the policy solving process, the model is trained to obtain the optimal resilience improvement policy, thereby enhancing the resilience of the distribution network. The agent model obtained during training is saved and used in the application phase to improve decision-making speed during application.
[0050] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning, characterized in that... Introducing graph neural networks into the distribution network fault recovery process enables unified modeling of network topology and electrical state. Through graph embedding representations of nodes and branches, the agent can automatically perceive topology changes, local voltage fluctuations, and load distribution differences, thereby rapidly generating optimal recovery decisions under different operating scenarios, significantly improving the algorithm's structural generalization ability and dynamic adaptability. Addressing the sparse reward problem in distribution network recovery tasks, a sparse reward allocation mechanism is proposed, distributing high-value terminal rewards to intermediate actions in rounds according to time weights, thereby increasing the reinforcement learning signal density and improving training convergence speed.
2. A fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning, comprising graph data modeling and a GRL model, characterized in that, Includes the following steps: S1: Power distribution system topology data modeling. Based on the connection relationship between electrical nodes in the power distribution network, establish the electrical topology of the power distribution network and construct the fault recovery topology. Read the connection data of nodes and tie switches from the power distribution network topology information. Mark and visit all connected electrical nodes in the physical topology map in a depth-first or breadth-first search manner. Merge the set of nodes visited in each search into a fault recovery node until all nodes have been visited. Establish a fault recovery topology using the interconnecting switch as an edge; S2: Establish a multi-agent decision-making process model for fault recovery, with the recovery amount of key loads within the distribution network as the objective function. The model follows six types of constraints: power flow constraints, node voltage constraints, line current constraints, distribution network topology constraints, and generator capacity constraints. S3: Construct the agent-environment interaction interface, divided into power flow analysis and topology checking modules. Establish a power flow model for the distribution system, where distributed generation (DG) is modeled as a constant voltage source, and its output will be part of the power flow results. Additionally, establish and save the topology graph of the distribution network for topology analysis. Then, upon receiving actions from the agent, search for paths between load nodes and DGs using depth-first search, checking radial topology constraints. For control operations that do not violate topology constraints, continue executing the actions and performing power flow calculations. Once all constraint checks and power flow calculations are complete, obtain the current state of the system for the next stage and calculate the reward to feed back to the agent. S4: Construct a graph reinforcement learning model. First, the actual distribution network containing distributed power sources is abstracted into graph data. The graph data includes two parts: the network topology G of the actual distribution network and its voltage, current and other feature data. Then, the abstracted graph data is input into the GRL model. In the GRL, a two-layer graph neural network is first built to process the graph data, complete the reception, information extraction, aggregation and transformation of the graph data, and then the processed feature information is passed to the downstream DQN task. S5: Establish a priority experience playback and sparse reward allocation mechanism model. In the process of distribution network fault recovery decision-making, a memory will be generated after each complete action cycle. Before storing the memory in the memory pool, a judgment is made. If the timely reward of this memory meets the standard of the preferred memory, it will be stored in the memory bank. Otherwise, it will be skipped and the next action will be carried out directly. S6: Model Training and Online Application: Combining graph reinforcement learning with the policy solving process, the model is trained to obtain the optimal resilience improvement policy, thereby enhancing the resilience of the distribution network. The agent model obtained during training is saved and used in the application phase to improve decision-making speed during application.
3. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S1, the power distribution system topology data is modeled. Specifically, this involves a time-series graph generation method for the power distribution network load recovery problem: the power distribution network is abstracted as a set of vertices and edges, denoted as... Let V represent the set of nodes and E represent the set of edges; the adjacency matrix is commonly used to describe the structural information of a graph. Let the graph... It contains n nodes, and the node set is Then its adjacency matrix ,in: Graph data consists of a graph G and feature data X representing the nodes and edges within that graph. Graph G represents the topology, while feature data X represents the characteristic information of nodes and edges. For example, in a power system, a graph abstracted from the actual system wiring relationships represents the connection relationships between nodes, i.e., the network topology. Node voltage and line current are feature data for nodes and edges, respectively. The graph and feature data together constitute the graph data. At time... Abstracting the distribution network as a directed graph with attributes , where the set of nodes Includes busbars, load nodes, distributed generation (DG) units, and mobile energy storage units (MESS), edge collection It includes physical wiring and tie switches. A node feature vector is defined for each node. The components are arranged in a fixed order: node voltage magnitude and phase angle. Node active power injection Node reactive power injection Load importance weight Measurement confidence level Distance metric to the nearest DG If it is MESS, then it includes the battery charge rate. and mobility indicators .
4. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S1, the power distribution system topology data is modeled. The topology data is updated in real time after actions are executed and faults occur, forming a time sequence diagram. : To either side Define edge feature vectors The components are: line resistance, line reactance, current line current amplitude, upper limit of line heat capacity, and on / off status of the edges. The graph data is updated in real-time after action execution and fault occurrence, forming a time sequence diagram. ,in Given the feature set of nodes and edges, perform a depth-first search or breadth-first search on connectivity at any given time step to identify sets of isolated subnets. The subgraph formed by the set of interconnecting switches is used as the "fault recovery topology" for subsequent action validity determination. This graph modeling ensures that graph neural network parameters can be shared when the number of nodes or edges changes with expansion or temporary failures, maintaining the transferability of strategies across topologies of different scales. A distribution network can be abstracted as a set of vertices and edges, or a graph as a set of nodes and their relationships. Vertices, or nodes, generally point to the entity under study, while edges point to the connections between entities, such as power lines between nodes in a power system. Mathematically, the graph is denoted as... Let V represent the set of nodes and E represent the set of edges; the adjacency matrix is commonly used to describe the structural information of a graph. Let the graph... It contains n nodes, and the node set is Then its adjacency matrix ,,in: Graph data consists of a graph G and feature data X of nodes and edges in the graph. Graph G represents the topology, and feature data X represents the feature information of nodes and edges. For example, in a power system, the graph abstracted from the actual system wiring relationship represents the connection relationship between nodes, i.e., the network topology relationship. Node voltage and line current are feature data on nodes and edges, respectively. The graph and feature data together constitute the graph data.
5. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S2, a multi-agent decision-making process model for fault recovery is established. The objective function of this model is to maximize the power restoration amount of the power-loss load, minimize the number of switching operations in the recovery strategy, and minimize network losses, i.e.: In the formula, m represents the number of loads in the system. The importance level coefficient for the j-th load depends on the load type (Level 1 load, Level 2 load, and Level 3 load). Let be the load shedding power of the j-th load node, which includes the power loss load caused by the initial line fault and the load actively shedding during the recovery process. s is the total number of all line switches in the system. If the j-th switch is opened or closed during the fault recovery strategy execution, then... ,otherwise , It is a system network loss; The constraints mainly stem from the requirement to ensure the safe and stable operation of the power grid throughout the entire fault recovery process, including: 1) Power flow constraints in power distribution systems: The normal operation of a power distribution system requires satisfying power flow constraints, the specific expressions of which are as follows: 2) Node voltage constraints: When dynamically dividing a distribution network into microgrids, it is necessary to ensure that the node voltage remains within a reasonable range. In the formula: for Time Node The lower limit of voltage amplitude for Time Node The upper limit of voltage amplitude, The node at time t The voltage amplitude; 3) Line current constraints: To prevent line overload, the line current in the distribution network system should be within its limits. In the formula: Let be the lower bound of the current in line n at time t. Let be the upper bound of the current in line n at time t. Let be the current in line n at time t; 4) Topology Constraints: During the post-disaster recovery phase, through dynamic microgrid grouping, each microgrid can utilize its own power generation resources to restore critical loads. For load restoration, a single-source, single-microgrid control strategy is adopted. That is, each microgrid is powered by only one distributed power source, and critical loads within the microgrid are powered through a single distributed power source and a single path. Each microgrid operates independently, and there are no pathways between any two microgrids. Considering that when the distribution network is decomposed into multiple isolated microgrids, it is necessary to ensure that the network always maintains a radial structure, the topology constraint for any originally connected distribution network can be expressed as: In the formula: To represent nodes and A binary value indicating whether the nodes are connected. and When connected, When node and When not connected, For a radial distribution network structure, the topological constraints can be simplified as follows: 5) Distributed power generation capacity constraints: As an emergency response resource, the output of distributed power sources is limited by their capacity throughout the recovery period, and their output should be within the specified limits. In the formula: and They are respectively Power supply The lower limits of active power and reactive power output. and They are respectively Power supply The upper limit of active power and reactive power output.
6. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S3, an agent-environment interaction interface is constructed. The environment needs to receive actions from the agent and then provide feedback on the system state and reward. To achieve this, it is necessary to ensure that all parameters related to actions, states, and rewards are available, and that all violations of constraints can be detected in the environment. The construction process of the AEI is as follows: 1) Build a power distribution network system in Python and initialize the time to... The state is initialized to ; 2) Receive an action from the agent This set of actions is decomposed into energy storage distribution actions. With line action ; 3) Based on the action Perform topology analysis on the distribution network structure to determine whether any actions violate topology constraints. If a violation occurs, then... The state is terminated. = If the condition is met, proceed to step 2; otherwise, proceed to step 4. 4) Execute actions and run power flow analysis; if power flow constraints are violated, then... The state is terminated. = (If necessary, proceed to step 2); otherwise, proceed to step 5. 5) In the power distribution system simulation environment, perform load restoration operations, calculate rewards, generate the final state, and calculate the final reward; 6) and Feedback is given to the intelligent agent.
7. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S4, a graph reinforcement learning model is constructed. First, the actual distribution network containing distributed power sources is abstracted into graph data. The graph data includes two parts: the network topology G of the actual distribution network and its characteristic data such as voltage and current. Then, the abstracted graph data is input into the GRL model. The GRL first builds a two-layer graph neural network to process the graph data, completing the reception, information extraction, aggregation and transformation of the graph data. Then, the processed feature information is passed to the downstream task. In the downstream DQN framework, two layers of GCN are also embedded in the eval net and the target net to further extract feature information. These two layers of GCN will be updated along with the DQN. Finally, the eval net outputs the action value. Since the input data for each layer of the Graph Neural Network (GCN) is graph data, the graph G in the graph data can be constructed in real time according to the current actual network topology, provided that the connection relationships of each node in the current distribution network are obtainable. The input and output dimensional parameters of the graph neural network do not need to change. This means that graph data of different dimensions can share GCN neural network parameters during propagation. For example, if the number of nodes in the current distribution network increases due to expansion, and the input of the neural network is node information such as node voltage, the input data dimension for the fully connected neural network will increase with the increase in the number of nodes. However, the neural network dimension in the model is fixed. In this case, the input dimension does not match the input layer dimension of the fully connected neural network, and the fully connected neural network in deep reinforcement learning will no longer be applicable. The GCN, on the other hand, is not affected by changes in the number of nodes in graph G because the parameter matrix of each GCN layer is shared during the aggregation of node information. This is why the graph reinforcement learning distribution network fault recovery model proposed in this patent is endowed with the ability to adapt to changes in network topology. Based on Graph Neural Networks (GCN), a reinforcement learning-based MDP model is constructed: 1) Action space: The action space consists of the agent's decision variables, including the energy storage allocation location and the execution state of the remote control switch. The action space can be represented as: In the formula: A binary variable representing the operation of a remote control switch. Indicates switch exist A disconnect operation is performed at all times, and a close operation is performed otherwise; therefore, the action space... It is discrete, consisting of a finite number of binary quantities; 2) State space: The state space is divided into two categories: one is load data, which is the load data restored after the execution of decisions; the other is distributed power source data, which is the output limit and actual output data of distributed power sources. According to the definition of state space S, its internal parameters contain continuous variables (load, power, voltage, current, etc.), because state space S is continuous.
8. The fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S5, the reinforcement learning reward function is designed. Constraints are categorized based on the severity of the penalty for violation. Strong constraints include power flow constraints, line current constraints, topology constraints, and generator capacity constraints; weak constraints include node voltage constraints. The reward factor for violating the corresponding constraint and feeding it back to the agent is set as follows: In the formula, This represents the reward portion of the reward function for the i-th action in the current round. The power loss due to the load after the i-th action is completed. This refers to the network power loss after the i-th action is completed. The total load of the power distribution system is considered. Taking into account the switching operation time and error rate in the actual system, the number of switching actions should be minimized to achieve the same fault recovery effect. Therefore, the number of actions in the current round, i, is placed in the denominator. The additional reward portion represents the increase in load recovery rate compared to the previous action in this round, describing the new contribution of the current action to the fault recovery task. If the current action has an improved fault recovery rate compared to the previous action, it means that the current action has had a new positive impact on fault recovery; conversely, if... This indicates that the current action not only failed to restore more of the lost power load, but also further expanded the power outage area. Additionally, if the current action is the first action of this round, that is... When the additional reward value is 0, great is the sparse reward value. When the system load recovery rate is 100% after the current action is completed and various operating constraints are met, a larger sparse reward value is assigned to strengthen the guidance of the agent's learning direction.
9. A fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S5, a model for priority experience replay and sparse reward allocation mechanism is established: (1) Prioritize experience replay To improve sample utilization efficiency, this patent employs a preferred experience replay method suitable for offline reinforcement learning. Due to sparse rewards, most experiences stored in the experience pool do not receive rewards and therefore have no training guidance value. The preferred experience replay method selectively stores action memories with large TD-errors, retaining only those with training value, thus saving experience pool capacity while improving memory quality. It is important to note that experiences with high training value are not equivalent to experiences with high reward values. During agent training, high-reward experiences can teach which actions are good, while high-penalty experiences with negative rewards can teach which actions are unsuitable. In the process of distribution network fault recovery decision-making, a memory is generated after each complete action cycle. Before storing the memory in the memory pool, a judgment is made. If the timely reward of this memory meets the criteria for preferred memory, it is stored in the memory bank; otherwise, it is skipped and the next action is carried out directly. (2) Sparse reward distribution mechanism In distribution network fault recovery decision-making, a great reward is only awarded if the current action fully restores the lost load while satisfying various operational constraints. This also means the current action is the last action in the current round; therefore, the great reward in the reward function is sparse. To ensure the agent fully recognizes the value of the preceding preparatory actions, this patent designs a sparse reward allocation mechanism. If the current action earns a great reward, the sparse reward (great) is allocated proportionally to each action in the current round: In the formula, Let i represent the total number of actions in the current round, and i represent the i-th action in the current round. If the last action in this round receives a great reward, then actions closer to the last action (i.e., those further back) should be given more importance. Actions closer to the last action that receives a great reward will receive a higher proportion of sparse rewards, which is consistent with actual needs.
10. A fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning according to claim 1, characterized in that: In step S6, model training and online application take place. During the training phase, the evaluation network and target network are initialized, and an experience replay pool is established. Each training round starts with a random fault scenario. The agent selects an action based on the current state and mask, and the action result is verified through power flow simulation. After execution, state transitions, rewards, and completion flags are recorded. At the end of each round, a sparse reward allocation mechanism is applied to adjust the reward, and high-value experiences are written into the replay pool. Network updates employ mini-batch sampling, delayed updates of the target network, and a greedy exploration strategy to ensure learning stability and generalization ability. The specific training process is as follows: (1) Initialize the power distribution network: load, lines, topology and other parameters are used to simulate the power system; (2) Initialize the experience pool, Q-network, and target Q-network for each agent. Initialize the environment state. ,time ; (3) Use a greedy algorithm to select actions from the Q network ; (4) Interact with the power distribution system simulation environment through the AEI interface to perform corresponding topology analysis and power flow analysis, and generate new states. And calculate the reward Other relevant information, storage experience Go to the experience pool; (5) Sample small batches of data from the experience pool; Approximate target action value If the action Violation of strong constraints, ,otherwise ; (6) Minimize the loss function using the stochastic gradient descent algorithm and update the Q-network parameters. After every C updates, the target Q network is replaced with the Q network; (7) If the maximum training cycle is reached, end the training and save the deep Q network; otherwise, go to (3). The intelligent agent interacts with the environment through the AEI interface to obtain state-value information and approximates the value of the target action. The Q-network is updated and iterated using the stochastic gradient descent algorithm. A greedy algorithm is used to restrict the agent's free exploration and decision-making. Greedy algorithms represent the agent's choice of actions. Sometimes, The probability of choosing a random action. The patent uses a greedy algorithm to select the action with the highest probability, as follows: In the formula: This is the initial exploration rate. The rate decay factor is used to explore the state and the size of the action space. and These represent the current step number and the maximum training step number, respectively. The free exploration rate of the agent during the training phase is adjusted through exponential decay. In the initial stage of training, the agent is given a high degree of freedom to explore in order to fully explore potential decision-making situations. In the later stage of training, the exploration rate is gradually reduced to drive the agent to choose the optimal decision and improve the convergence of the training process. The beneficial effects of this invention are as follows: Compared with existing technologies, the fault recovery method for distribution networks with distributed power sources based on graph reinforcement learning proposed in this invention has the following advantages and positive effects: This invention introduces graph neural networks into the distribution network fault recovery process, achieving unified modeling of network topology and electrical state. Through graph embedding representation of nodes and branches, the agent can automatically perceive topology changes, local voltage fluctuations, and load distribution differences, thereby quickly generating optimal recovery decisions under different operating scenarios, significantly improving the algorithm's structural generalization ability and dynamic adaptability. Addressing the sparse reward problem in distribution network recovery tasks, this invention proposes a sparse reward allocation mechanism, distributing high terminal rewards to intermediate actions in rounds according to time weights, greatly increasing the reinforcement learning signal density and accelerating training convergence. Simultaneously, combining priority experience replay and target network delayed update strategies makes the agent more stable during training, avoiding Q-value oscillations and overfitting problems. This invention utilizes a power flow simulation environment to automatically filter illegal operations before each action selection and verifies voltage and current constraints through power flow calculations before execution, ensuring that all recovery actions meet physical safety conditions. This mechanism effectively prevents problems such as ring network formation, voltage exceeding limits, and line overload, guaranteeing the engineering feasibility of the recovery scheme and the safe and reliable operation of the system. In summary, this invention can significantly improve the risk resistance and recovery level of power distribution networks under extreme disaster conditions while ensuring technical feasibility, and has high practical value and application prospects.