Power grid resilience improvement method and system based on multi-agent deep reinforcement learning

Abstract

The application provides a power grid resilience improvement method and system based on multi-agent deep reinforcement learning, and relates to the technical field of power grid dispatching; including obtaining multi-source data of a power grid resilience improvement system; constructing a topology graph based on the multi-source data to obtain a multi-source topology graph; extracting graph features based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power supply graph node features and mobile energy storage vehicle distribution graph node features; processing the multi-source topology graph and the multi-source graph node features through a multi-agent deep reinforcement learning model to obtain a power grid control strategy; the multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network; the multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network and a power generation control agent actor network. The application realizes the collaborative complementation of multi-source information and enhances the global optimization capability of the recovery strategy.

Description

Technical Field

[0001] This invention relates to the field of power grid dispatching technology, specifically to a method and system for improving power grid resilience based on multi-agent deep reinforcement learning. Background Technology

[0002] A power grid resilience enhancement system refers to an intelligent management system that enables the power grid to proactively defend against, respond quickly to, and efficiently restore power supply after disturbances such as extreme weather, natural disasters, or major faults, in order to ensure continuous power supply to critical loads, minimize power outage losses, and ultimately restore the power grid to its normal function.

[0003] Existing methods for improving grid resilience are insufficient in integrating and utilizing information from multiple systems. Decision-making is often limited to local sensing data from a single system. For example, grid nodes can only obtain data such as their own voltage, current, and power status; mobile energy storage vehicles can only obtain data such as their own location and remaining power; and meteorological monitoring points can only obtain data such as wind speed and rainfall at their location.

[0004] However, this traditional approach focuses on optimization of a single aspect, neglecting collaborative optimization of multiple resources in a real-world context and lacking global scheduling of various resources. This creates "information silos" and limits the global optimization capability of power grid resilience recovery strategies. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a power grid resilience enhancement system based on multi-agent deep reinforcement learning, which solves the problem of existing methods lacking multi-source information synergy and complementarity.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: Firstly, this application provides a method for improving power grid resilience based on multi-agent deep reinforcement learning, including: Acquire multi-source data for the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data; A topology map is constructed based on the multi-source data to obtain a multi-source topology map; the multi-source topology map includes a power grid topology map, a distributed power source map, and a mobile energy storage vehicle distribution map. Based on the multi-source topology graph, graph features are extracted to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features. The trained multi-agent deep reinforcement learning model processes the multi-source topology graph and the node features of the multi-source graph to obtain the power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

[0007] Preferred options also include: Build a multi-agent deep reinforcement learning model and determine the model's state space, action space, state transition function, and reward function; A virtual interactive environment is constructed, and its simulation model and constraints are defined. The simulation model includes a power system physical model, a component dynamic probabilistic fault model under disasters, and a traffic network and mobile resource model. The constraints include operational process constraints and recovery process constraints. The operational process constraints include voltage safety constraints, line capacity constraints, generator output constraints, node power balance, and topology connection constraints. The recovery process constraints include limits on the number of switching operations, mobile energy storage scheduling constraints, traffic accessibility constraints, critical load priority constraints, and resource quantity constraints. The multi-agent deep reinforcement learning model is progressively trained using a virtual interactive environment based on a course-based learning method, resulting in a fully trained multi-agent deep reinforcement learning model.

[0008] Preferably, the course-based learning method progressively trains the multi-agent deep reinforcement learning model through a virtual interactive environment, including: Store the experience tuples from each training round of interaction into a shared experience replay pool. Randomly sample small batches of experimental data from the experience replay pool; Calculate the temporal difference objective based on the aforementioned small-batch experimental data; Based on the aforementioned time-series differential objective, the commentator network is updated by minimizing the mean square error loss, and the updated commentator network outputs the global state-action value of the power grid. The actor network is updated by maximizing the expected cumulative return based on the policy gradient method and the global state-action value of the power grid.

[0009] Preferably, the process of using a trained multi-agent deep reinforcement learning model to process the multi-source topology graph and multi-source graph node features to obtain a power grid control strategy includes: The intelligent agent actor network reconstructs the power grid topology map to obtain the switching action decisions for each route; the ... The mobile energy storage scheduling intelligent agent network processes the graph node features and edge connection data of the mobile energy storage vehicle to obtain the probability of the mobile energy storage vehicle connecting to each grid node, the charging and discharging power at the time of connection, and the connection status at the time of connection with the node; the mobile energy storage scheduling intelligent agent network consists of a third GAT layer and a fourth GAT layer. The power generation control agent network processes the graph node features and edge data of the distributed power source to obtain continuous control commands for the active and reactive power output of the distributed power source; the power generation control agent network consists of a fifth GAT layer and a sixth GAT layer.

[0010] Preferably, the commentator network consists of a graph attention encoder and a multilayer perceptron; the graph attention encoder consists of a seventh GAT layer and an eighth GAT layer; the multilayer perceptron consists of a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0011] Preferably, the reward function is: in: In the formula, It is an adjustable positive weighting coefficient used to balance multiple objectives such as recovery amount, operating cost, safety, and synergy. , ; This indicates a reward for the amount of recovery. This indicates a penalty for operating costs; This indicates a security penalty; Indicates a collaborative reward; Indicates the load is t The power increment that is restored at any time, i.e. t Time and t The power supply difference at time 1; For the set of critical loads; For the set of important loads; For a single load set; For the set of all switches; for t The state at any given moment: 0 indicates disconnection, 1 indicates connection; express Time's up t The number of switching actions at any given time; Represents a node i The actual voltage amplitude at the current moment, Represents a node i The upper limit of voltage safety. Represents a node i The lower limit of voltage safety; This is an indicator function that takes the value 1 when a successful collaborative event occurs, such as a mobile energy storage vehicle supplying power to an island formed by reconfiguration, and 0 otherwise; Represents the set of lines in a power distribution network; Indicates the line The actual current amplitude; Indicates the line The maximum current value.

[0012] Preferably, the course-based learning method progressively trains the multi-agent deep reinforcement learning model through a virtual interactive environment, including: Monitor the average reward of the multi-agent deep reinforcement learning model over multiple consecutive training cycles in the current training phase; Once the average reward reaches a stable level, the system will automatically switch from the current training phase to the next training phase.

[0013] Secondly, this application provides a power grid resilience enhancement system based on multi-agent deep reinforcement learning, comprising: The data acquisition module acquires multi-source data from the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data. The topology graph construction module constructs a topology graph based on the multi-source data to obtain a multi-source topology graph; the multi-source topology graph includes a power grid topology graph, a distributed power source graph, and a mobile energy storage vehicle distribution graph. The feature extraction module performs graph feature extraction based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features; The strategy generation module processes the multi-source topology graph and the node features of the multi-source graph using a trained multi-agent deep reinforcement learning model to obtain a power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

[0014] Thirdly, this application provides a computer-readable storage medium storing a computer program for improving power grid resilience based on multi-agent deep reinforcement learning, wherein the computer program causes a computer to execute the power grid resilience improvement method based on multi-agent deep reinforcement learning as described above.

[0015] Fourthly, this application provides an electronic device, comprising: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing grid resilience enhancement methods based on multi-agent deep reinforcement learning as described above.

[0016] (III) Beneficial Effects This invention provides a method and system for improving power grid resilience based on multi-agent deep reinforcement learning. Compared with existing technologies, it has the following advantages: 1. This invention acquires multi-source data such as power grid physical and topology data, component vulnerability parameters, geospatial and traffic data, and meteorological and disaster scenario data, and constructs a multi-source topology map based on this multi-source data, which includes a power grid topology map, a distributed power source map, and a mobile energy storage vehicle distribution map. This breaks through the limitations of traditional single-system local perception, realizes the synergistic complementarity of multi-source information such as power grid, traffic, and meteorology, and enhances the global optimization capability of power grid resilience recovery strategies.

[0017] 2. This invention obtains the features of power grid nodes, distributed power source nodes, and mobile energy storage vehicle distribution nodes through graph feature extraction. It also uses graph attention networks to extract features of the power grid topology, which can accurately capture the electrical coupling relationships and spatial dependencies between nodes. This provides multi-system coupling information for the decision-making of each agent, effectively making up for the shortcomings of traditional methods that rely on a single decision basis. It significantly improves the accuracy and collaborative efficiency of agent decision-making and provides reliable decision support for improving power grid resilience. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic flowchart illustrating the power grid resilience enhancement method provided in this application embodiment; Figure 2A flowchart illustrating a method for improving power grid resilience according to another embodiment of this application; Figure 3 This is a schematic diagram of the power grid resilience enhancement system provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] This application provides a method and system for improving power grid resilience based on multi-agent deep reinforcement learning. It solves the problem that existing methods lack the synergistic complementarity of multi-source information such as power grid, transportation, and meteorology, realizes the unified consideration of multi-source information, and improves the generation and global optimization capabilities of power grid recovery strategies when disasters occur.

[0022] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows: In the face of uncertain extreme weather conditions, the failure scenarios of the power grid are equally difficult to determine. Improving load recovery efficiency to a greater extent in such uncertain real-world environments is key to enhancing grid resilience. Existing research largely relies on historical data to simulate catastrophic failure scenarios and employs purely data-driven decision-making models, such as deep Q-networks and near-end policy optimization machine learning algorithms. However, historical data cannot cover all possible future scenarios, resulting in weak generalization ability. Furthermore, the internal mechanisms of purely data-driven decision-making models are opaque, leading to a lack of interpretability in the decision-making process. Moreover, the performance ceiling of these methods based on historical data simulation and purely data-driven modeling is determined by data quality; the quality of the dataset directly affects the fitting effect of the trained model, ultimately making it difficult to effectively support the improvement of power grid resilience.

[0023] Power system recovery involves achieving multiple objectives, including critical load restoration, operating costs, system security, and collaborative efficiency. These objectives are subject to trade-offs, and relying solely on single-system studies such as grid topology reconstruction and mobile energy storage vehicle dispatching is insufficient for achieving coordinated optimization of these multiple objectives. Therefore, improving grid resilience exhibits characteristics such as multi-temporal and spatial coupling, weak global information perception, and conflicting objectives. Traditional methods struggle to simultaneously meet the demands in terms of dynamic control, generalization, and coordination. Against this backdrop, constructing a smart grid system capable of rapidly restoring power supply and withstanding natural disasters has become an essential requirement for building resilient urban power grids.

[0024] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0025] like Figure 1 As shown, this embodiment provides a method for improving power grid resilience based on multi-agent deep reinforcement learning, specifically including the following steps: Step S110: Obtain multi-source data from the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data. Step S120: Construct a topology map based on multi-source data to obtain a multi-source topology map; the multi-source topology map includes a power grid topology map, a distributed power source map, and a mobile energy storage vehicle distribution map; Step S130: Extract graph features based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features; Step S140: The trained multi-agent deep reinforcement learning model processes the multi-source topology graph and the node features of the multi-source graph to obtain the power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

[0026] In one embodiment, step S110 involves acquiring multi-source data from the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and traffic data, and meteorological and disaster scenario data. Specifically, the power grid physical and topology data are obtained from the databases of the power industry's dedicated production management system (PMS), geographic information system (GIS), and supervisory control and data acquisition (SCADA) system, which contain complete power grid connection diagrams, including the location, connection relationship, and related electrical parameters such as resistance, reactance, capacity, and voltage of all busbars, transmission / distribution lines, transformers, and disconnectors.

[0027] Simultaneously, load data under historical typhoon disaster scenarios was acquired, and the loads were categorized into three types: critical loads, important loads, and general loads. Critical loads are those where power outages would threaten lives or disrupt social order, such as hospitals, emergency command centers, fire stations, military facilities, and public security facilities. Important loads are those where power outages, while not threatening lives, would cause significant economic losses, such as large shopping malls, high-speed rail stations, airports, data centers, and large industrial production bases. General loads are those where power outages would not affect people's normal lives in the short term, such as residential areas, small businesses, and small production bases.

[0028] The component vulnerability parameters are key parameters of the "vulnerability curves" for components such as lines and towers, obtained from equipment manufacturer technical manuals, power industry design standards, and equipment damage analysis reports after historical typhoon disasters. Specifically, these are the wind speed thresholds in the Logistic function below. and shape parameters .

[0029] Geospatial and traffic data consists of road network vector data covering the area where the power grid data is located, obtained from public maps and traffic management departments, including information such as road class and speed limits.

[0030] The meteorological and disaster scenario data are obtained from meteorological departments, including wind speed data and typhoon movement trajectory data under historical typhoon disasters in the region.

[0031] In one embodiment, after obtaining multi-source data, the multi-source data is preprocessed; wherein, the preprocessing includes data cleaning, alignment and normalization.

[0032] Specifically, for missing electrical parameters such as resistance, reactance, capacity, and voltage due to incomplete records or outdated equipment, the average values ​​of known equipment of the same model and batch are used to fill the gaps. For missing spatiotemporal sequence data due to sparse monitoring stations or transmission failures, data from neighboring stations are used to infer the missing values. For values ​​that cannot be filled, they are marked as special values ​​and assumed to have a failure probability of 1 in the simulation. A sensitivity analysis is then performed to assess the impact of these parameters on the overall recovery strategy to confirm the rationality of the assumptions. Data that is clearly inconsistent with common sense, such as negative values ​​for wind speed, resistance, or reactance, are directly discarded.

[0033] After data cleaning, all processed data are unified to the same spatial coordinate system and time base to complete data alignment.

[0034] After data alignment, parameters with significantly different dimensions and numerical ranges are normalized to ensure stable convergence in subsequent training. Specifically, for parameters that are always positive, such as voltage amplitude, impedance, and wind speed, the formula is used. Mapped to Interval; for parameters with positive and negative values such as active power and reactive power, the formula is mapped to the interval.

[0035] In one embodiment, step S120 constructs a multi-source topology graph based on multi-source data to obtain a multi-source topology graph; the multi-source topology graph includes a power grid topology graph, a distributed power source graph, and a mobile energy storage vehicle distribution graph; In this embodiment, after preprocessing the data to obtain the target data set, it is also necessary to construct a multi-source graph topology. Specifically, the power grid is abstracted as a power grid topology graph . Among them, the node is the busbar, and the features include data such as the normalized voltage reference, load value, and load type; the edge is the corresponding power transmission or distribution line connecting the nodes, and the edge attributes include data such as the normalized impedance, capacity, and initial state. The distributed power source area is used as the graph , the node is the small power source in the distributed power source, and the edge is the distribution line and switch between the nodes. The mobile energy storage vehicle is used as the node of the graph , and the edge is the distribution line and switch between the nodes.

[0036] Construct the traffic routes between each node into a traffic network graph for subsequent calculation of the moving time.

[0037] Step S130 extracts graph features based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features; Specifically, taking the initial feature vector and edge connection data of each node in the power grid topology graph as input, using a multi-head graph attention network layer, each layer calculates the attention coefficient between the node and its neighbor for weighted aggregation of neighbor information. The specific calculation process of feature extraction is as follows: Among them, the attention coefficient is calculated by the following formula: In the formula is the initial feature vector of the node , including the normalized voltage reference, load value, load type, etc., is the node The output feature vector, i.e. the updated features; For nodes The set of neighboring nodes; The weight matrix is ​​trainable. It is a trainable attention vector; This represents a vector concatenation operation; It is a non-linear activation function; It is a modified linear unit with leakage, used for attention score calculation.

[0038] Step S140: The trained multi-agent deep reinforcement learning model processes the multi-source topology graph and the node features of the multi-source graph to obtain the power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

[0039] In one embodiment, the training process of a multi-agent deep reinforcement learning model includes the following steps: Step S210: Build a multi-agent deep reinforcement learning model and determine the model's state space, action space, state transition function, and reward function. Step S220: Build a virtual interactive environment and define the simulation model and constraints of the virtual interactive environment. The simulation model includes a power system physical model, a component dynamic probabilistic fault model under disaster, and a traffic network and mobile resource model. The constraints include operational process constraints and recovery process constraints. The operational process constraints include voltage safety constraints, line capacity constraints, generator output constraints, node power balance, and topology connection constraints. The recovery process constraints include limits on the number of switching operations, mobile energy storage scheduling constraints, traffic accessibility constraints, critical load priority constraints, and resource quantity constraints. Step S230: Based on the course learning method, the multi-agent deep reinforcement learning model is progressively trained through a virtual interactive environment to obtain the trained multi-agent deep reinforcement learning model.

[0040] Specifically, we first define three types of intelligent agents: the power grid network reconfiguration intelligent agent, the mobile energy storage dispatch intelligent agent, and the power generation control intelligent agent.

[0041] Understandably, during the power system recovery process, a sub-region of the power grid that is disconnected from the main grid due to a fault but can still maintain local power supply presents an "island" state. At this time, the network reconstruction agent controls the tie switches and sectionalizing switches in the power grid to change the network topology, thereby reorganizing the non-faulty areas into one or more stable islands that can be connected to the main grid or operate independently. The number of islands depends on the number and location of the faults.

[0042] The role of the mobile energy storage dispatching intelligent agent is to direct mobile energy storage vehicles to move to the fault area, connect lines, and charge and discharge, providing temporary power to the isolated area to restore power supply to the fault area.

[0043] The role of the power generation control agent is to regulate the output of distributed power sources within the island in order to maintain the real-time power balance and stability of the island.

[0044] In one embodiment, a multi-agent actor network is constructed.

[0045] Specifically, the variables involved in the construction of the actor network include the number of power grid bus nodes (Node_num), node state feature dimension (State_dim), total number of lines (Edge_num), number of mobile energy storage vehicles (N_vehicle), mobile energy storage vehicle state feature dimension (State_dim_vehicle), number of distributed power sources (N_generator), and distributed power source state features (State_dim_dg).

[0046] Based on the different types of intelligent agents, three types of agent networks are constructed: power grid network reconstruction agent network, mobile energy storage dispatch agent network, and power generation control agent network.

[0047] The construction of the intelligent agent network for power grid network reconstruction is described. The intelligent agent network for power grid network reconstruction adopts a two-layer graph attention network (GAT) architecture, specifically designed to process power grid topology information and output decisions on the opening and closing actions of each line switch. The intelligent agent network for power grid network reconstruction consists of a first GAT layer, a second GAT layer, and a first fully connected layer.

[0048] The first GAT layer has a node input feature dimension of State_dim and an output feature dimension of GAT_hidden_1. This layer calculates the association weights between a node and its neighboring nodes through an attention mechanism, aggregates first-order neighbor information, and outputs a preliminary node embedding representation with a data structure of [Node_num, GAT_hidden_1]. The output result is then processed... The activation function processes the data, which is then input into the second GAT layer.

[0049] The second GAT layer has a node input feature dimension of GAT_hidden_1. This layer further aggregates higher-order neighbor information to generate a 2-dimensional high-level feature representation for each node, indicating the action tendency and risk level associated with that node. If the node is a power source or critical load node, the action tendency associated with it is greater; if the node is a network hub, the risk associated with it is higher. The output data structure of this layer is [Node_num, 2].

[0050] The endpoint feature concatenation method is used to concatenate the feature vectors of the two nodes connected by each edge to form the feature of that edge, thereby obtaining the edge feature matrix [Edge_num, 4].

[0051] The edge matrix is ​​input into the first fully connected layer, which linearly maps the 4D edge features to the 2D action space. This mapping is then fed into the Softmax function, which outputs a probability matrix of [Edge_num, 2], representing the switching action probability distribution for each line. The i-th row represents the action probability of line i. ,satisfy .

[0052] Construction of a Mobile Energy Storage Dispatch Agent Network. The mobile energy storage dispatch agent network adopts a two-layer GAT architecture combined with a multi-output head architecture to handle the dynamic dispatching problem of mobile energy storage vehicles. This part of the input... The graph node characteristics and edge connection data are used to output the probability of selecting a mobile energy storage vehicle to connect to each grid node, the charging and discharging power at the time of connection, and the connection status when connected to a node. The mobile energy storage scheduling intelligent agent network consists of a third GAT layer and a fourth GAT layer.

[0053] The third GAT layer has a node input feature dimension of State_dim_vehicle and an output feature dimension of GAT_hidden_2. This layer aggregates the correlation information between vehicles and between vehicle-grid nodes, and the output data structure is [N_vehicle, GAT_hidden_2]. The output results are processed... The activation function processes the data, which is then input into the fourth GAT layer.

[0054] The fourth GAT layer has a node input feature dimension of GAT_hidden_2 and an output feature dimension of Node_num+1+2, corresponding to the total dimension of the three output heads: target node selection output head, charge / discharge power output head, and connection status output head. Specifically, the target node selection output head has a dimension of Node_num, the charge / discharge power output head has a dimension of 1, and the connection status output head has a dimension of 2. This layer further extracts high-level scheduling features to prepare for multi-objective decision-making.

[0055] The multi-output head layer further processes the output data from the three output heads. First, the data output from the target node selection output head is transformed into a probability distribution of [p_1, p_2, … p_Node_num] using Softmax to represent the probability of selecting each grid node as the destination. Then, the data output from the charge / discharge power output head is constrained to the range [-1, 1] using the Tanh activation function to represent normalized charge / discharge commands, where negative commands represent charging and positive commands represent discharging. Finally, the data output from the connection status output head is transformed into a probability distribution of [p_connected, p_disconnected] using the Softmax function to represent the probability of connection or disconnection.

[0056] Construction of the Power Generation Control Agent Network. The power generation control agent network adopts a two-layer GAT architecture, with the input being... The graph node features and edge data are used to output continuous control commands for the active and reactive power output of the distributed power source. The power generation control intelligent agent network consists of the fifth and sixth GAT layers.

[0057] The fifth GAT layer aggregates the electrical coupling relationships between generators and system state information. The node input feature structure for this layer is [N_generator, State_dim_dg], and the output data structure is [N_generator, GAT_hidden_3]. The output results are processed... The activation function processes the data, which is then input into the sixth GAT layer.

[0058] The sixth GAT layer has a node input feature dimension of GAT_hidden_3 and an output feature dimension of 2, corresponding to the active and reactive power control dimensions. This layer generates the output setpoints for each generator. The output data structure is [N_generator,2]. Finally, the Tanh activation function constrains the output to the range [-1,1], resulting in an output data structure of [N_generator_1,2], where the first column represents the active power setpoint and the second column represents the reactive power setpoint. The actual output is calculated through inverse normalization, as shown in the following formula: In one embodiment, the aforementioned multi-agent deep reinforcement learning model, after training, processes the multi-source topology graph and the features of the multi-source graph nodes to obtain a power grid control strategy, including: The intelligent agent actor network reconstructs the power grid topology map to obtain the switching action decisions for each route; the ... The mobile energy storage scheduling intelligent agent network processes the graph node features and edge connection data of the mobile energy storage vehicle to obtain the probability of the mobile energy storage vehicle connecting to each grid node, the charging and discharging power at the time of connection, and the connection status at the time of connection with the node; the mobile energy storage scheduling intelligent agent network consists of a third GAT layer and a fourth GAT layer. The power generation control agent network processes the graph node features and edge data of the distributed power source to obtain continuous control commands for the active and reactive power output of the distributed power source; the power generation control agent network consists of a fifth GAT layer and a sixth GAT layer.

[0059] In one embodiment, a critic network is constructed. The critic network consists of a graph attention encoder and a multilayer perceptron; the graph attention encoder consists of a seventh GAT layer and an eighth GAT layer; the multilayer perceptron consists of a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0060] The role of the critic network is to evaluate the value of the global state of the power grid topology and the joint actions obtained by the power grid reconfiguration agent, the mobile energy storage dispatch agent, and the generation control agent, in order to guide the updates of the individual actor networks. See also Figure 2 Fault data is input into the actor network and outputs actions. After the fault data forms the edge connection matrix and node state matrix of the power grid topology graph, it is integrated into the critic network. Subsequently, the critic network forms a global state-action value to guide the updates of each actor network. In this embodiment, a centralized training critic network is used.

[0061] The input to the commentator network consists of three parts: the first part is the global node state matrix of the power grid topology graph, with the data structure [Node_num, State_dim]; the second part is the edge connection matrix of the power grid topology graph, with the data structure [2, Edge_num]; and the third part is the joint action vector of all agents, with the data structure [1, Action_dim_total]. Where Action_dim_total = Edge_num × 2 + N_vehicle × (Node_num + 1 + 2) + N_generator × 2.

[0062] The critic network consists of a graph attention encoder and a multilayer perceptron.

[0063] This embodiment describes the construction of a graph attention encoder. First, the input data for the global state and joint actions of the power grid topology graph is formatted. Based on the global node state and joint action information of the power grid topology graph, the input features of each node can be obtained. ,in This represents the original state characteristics of node i; This represents the local action encoding related to node i, with the dimension of Action_dim_per_node. The local action encoding consists of three parts: node connection action, mobile energy storage vehicle connection action, and distributed power generation action, corresponding to the actions obtained by the grid network reconfiguration agent, the mobile energy storage scheduling agent, and the generation control agent, respectively. The node connection action represents the probability of opening or closing the switch action of the line connecting node i to other nodes, with a feature dimension of Link_edge_num×2. The mobile energy storage vehicle connection action represents the connection status between the mobile energy storage vehicle and the node, including node selection, charging / discharging power, and the probability of connection and disconnection, with a feature dimension of 4. The distributed power generation action represents the power settings of the distributed power generation connected to node i, including the set values ​​of the active power P and reactive power Q of the power generation, with a feature dimension of 2.

[0064] When selecting node connection action, mobile energy storage vehicle connection action, and distributed power source action simultaneously, Action_dim_per_node = Link_edge_num × 2 + 4 + 2.

[0065] Therefore, the data input structure of the graph attention encoder is [Node_num, State_dim+Action_dim_per_node].

[0066] The graph attention encoder consists of a seventh GAT layer and an eighth GAT layer.

[0067] The seventh GAT layer's node input feature dimension is State_dim + Action_dim_per_node, and the output feature dimension is GAT_hidden_4. This layer accepts features concatenated from global node state features and action codes of each agent. The input data structure is [Node_num, State_dim + Action_dim_per_node], and the output data structure is [Node_num, GAT_hidden_4]. The output result is processed... The activation function processes the data, which is then input into the eighth GAT layer.

[0068] The node input feature dimension of the eighth GAT layer is GAT_hidden_4, and the output feature dimension is GAT_hidden_5. The input data structure is [Node_num, GAT_hidden_4], and the output data structure is [Node_num, GAT_hidden_5]. The output result is also processed... The activation function is used for processing.

[0069] The eighth GAT layer further aggregates global topological information, and its output node features are processed by a global attention pooling layer to obtain a graph-level representation vector [1, GAT_hidden_5].

[0070] This embodiment constructs a multilayer perceptron. First, the graph-level representation vector output by the graph attention encoder is concatenated with the joint action vector of all agents to obtain a concatenated feature with the structure [1, GAT_hidden_5+Action_dim_total]. Then, the concatenated feature is input into the multilayer perceptron to obtain the global state-action value of the power grid. The multilayer perceptron consists of a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

[0071] The second fully connected layer has a feature input dimension of GAT_hidden_5 + Action_dim_total and an output feature dimension of FC_hidden1. The input data structure is [1, GAT_hidden_5 + Action_dim_total], where 1 represents the number of graphs, i.e., a global vector. The output data structure is [1, FC_hidden1]. Activation function processing.

[0072] The third fully connected layer has a feature input dimension of FC_hidden1 and an output feature dimension of FC_hidden2. The input data structure is [1, FC_hidden1], and the output data structure is [1, FC_hidden2]. Activation function processing.

[0073] The fourth fully connected layer has a feature input dimension of FC_hidden2 and an output feature dimension of 1. The input data structure is [1, FC_hidden2], and the output is a scalar value with a data structure of [1, 1]. This yields the global state-action value of the power grid. .

[0074] In one embodiment, an interactive environment is constructed.

[0075] In this embodiment, a simulation model of the interactive environment and corresponding model constraints are defined, including the definition of a power system physical model, a component dynamic probabilistic fault model under disaster, and a traffic network and mobile resource model. In addition, the state space, action space, state transition function, and reward function of reinforcement learning are set.

[0076] In this embodiment, a physical model of the power system is defined. The operating state of the power grid is described using AC power flow equations: in .

[0077] In the formula , They are nodes The injected active power and reactive power; For nodes The voltage amplitude; For nodes The voltage amplitude; For nodes , The voltage phase angle difference between them; , Let be the real and imaginary parts of the network admittance matrix; It is the set of all nodes in the system.

[0078] After the power grid reconfiguration agent, mobile energy storage dispatch agent, and generation control agent output their respective actions through the corresponding actor network, the AC power flow equations are solved using the power system physical model to determine whether the action will lead to voltage overruns, line overloads, or power imbalances. The grid state S at the next time step is then updated based on the combined actions of the agents. t+1 This includes variables such as the voltage at each node, phase angle, active and reactive power of the line, and load recovery at the next moment.

[0079] In this embodiment, a dynamic probabilistic failure model for components under disaster conditions is defined. Traditional power grid disaster simulations typically employ deterministic fault lists or simple hard-threshold probabilistic models. These methods cannot accurately characterize the progressive failure process of components under extreme stress conditions and can lead to an uneven simulation environment, which is detrimental to the learning of data-driven algorithms. Therefore, considering the above conditions, this paper adopts the vulnerability curve of the Logistic function: In the formula Indicates at time step Time element The probability of failure; For simulation time steps; Indicates power grid components (such as lines and towers); For a moment Acting on components Wind speed above, in units of ; For components The design wind speed threshold; The curve shape parameter represents the component's sensitivity to wind speed.

[0080] The dynamic probabilistic fault model of components under disaster conditions calculates the instantaneous fault probability of each component by taking into account the typhoon path and wind speed, thereby providing the input for the fault of the entire system.

[0081] In this embodiment, a traffic network and a mobile resource model are defined. The road network is modeled using graph theory to describe the dynamic position and speed changes of the mobile energy storage vehicle. in A diagram representing a transportation network; For the set of vertices; Let it be the set of edges; , These represent the position coordinates of the mobile energy storage vehicle at times t+1 and t, respectively. Indicates vehicle exist The velocity vector at any given moment is constrained by road speed limits and vehicle performance, while its direction is determined by path planning. This is the simulation time step; for The global road traffic coefficient at any given time is obtained from the specific environmental conditions.

[0082] The traffic network and mobile resource model provides physical reachability constraints and travel time calculations for the scheduling of mobile intelligent vehicles, ensuring the feasibility of energy storage scheduling schemes in actual road networks.

[0083] The system must meet the following constraints to operate: (1) Setting constraints during operation. Constraints during operation mainly include voltage safety constraints, line capacity constraints, generator output constraints, node power balance and topology connection constraints.

[0084] Voltage safety constraints: Line capacity constraints: Generator output constraints: Node power balancing: Topology connectivity constraints: The power grid must maintain a radial operation with no loops.

[0085] in Represents a node The voltage value on; The set of all nodes; For the line Current on; This is the set of all routes; For generator The power generation capacity; This is the set of all generators; It provides active and reactive power for power generation and energy storage; The active and reactive power required by the load.

[0086] (2) Setting of recovery process constraints. Recovery process constraints mainly include limits on the number of switching operations, mobile energy storage scheduling constraints, traffic accessibility constraints, critical load priority constraints, and resource quantity constraints.

[0087] Switch operation limit: Mobile energy storage dispatch constraints: Accessibility constraints: Mobile resources need to be within the road network Internal movement.

[0088] in Indicates the line The switch state is 0 for open and 1 for closed; Indicates a change in switch state; Indicates the maximum number of operations allowed; This indicates the state of charge of mobile energy storage, i.e., the percentage of remaining battery capacity. This indicates the minimum percentage of remaining battery charge allowed to prevent over-discharge; This indicates the maximum percentage of remaining battery power allowed to prevent overcharging.

[0089] (3) Priority constraint for critical loads: The principle of priority restoration of load importance shall be met, that is, the power supply of critical loads shall be guaranteed first, then the power supply of important loads shall be guaranteed, and finally the power supply of general loads shall be guaranteed.

[0090] Resource quantity constraint: The total number of mobile energy storage vehicles is less than or equal to the total number of available vehicles.

[0091] In this embodiment, the state space, action space, state transition function, and reward function of the multi-agent deep reinforcement learning model are determined.

[0092] This invention defines the state space at time t as... .in Let t represent the electrical state of all grid nodes at time t, i.e., voltage, power, phase angle, etc. Let represent the power grid topology at time t. The information includes the typhoon's location and wind speed at time t, as well as the location and status of the mobile energy storage vehicle.

[0093] Action space is defined as . These represent the actions of the power grid reconfiguration agent, the mobile energy storage dispatch agent, and the power generation control agent at time t.

[0094] This embodiment defines a state transition function. The environment is determined based on the joint actions of the agents. Based on the current disaster intensity, the power grid state at the next moment is calculated using AC power flow equations, component vulnerability models, and traffic movement models. .

[0095] This embodiment designs a comprehensive multi-objective reward function to guide the agent in collaborative optimization.

[0096] in: In the formula, It is an adjustable positive weighting coefficient used to balance multiple objectives such as recovery amount, operating cost, safety, and synergy. , ; This indicates a reward for the amount of recovery. This indicates a penalty for operating costs; This indicates a security penalty; Indicates a collaborative reward; Indicates the load is t The power increment that is restored at any time, i.e. t Time and t The power supply difference at time 1; For the set of critical loads; For the set of important loads; For a single load set; For the set of all switches; for t The state at any given moment: 0 indicates disconnection, 1 indicates connection; express Time's up t The number of switching actions at any given time; Represents a node i The actual voltage amplitude at the current moment, Represents a node i The upper limit of voltage safety. Represents a node i The lower limit of voltage safety; This is an indicator function that takes the value 1 when a successful collaborative event occurs, such as a mobile energy storage vehicle supplying power to an island formed by reconfiguration, and 0 otherwise; Represents the set of lines in a power distribution network; Indicates the line The actual current amplitude; Indicates the line The maximum current value.

[0097] In one embodiment, critic network updates and actor network updates are performed.

[0098] Temporal difference learning is employed. This is achieved by minimizing the mean square error loss. To update network parameters, where This is the discount factor.

[0099] Discount factor It is used to measure the current value of future rewards.

[0100] Given that power grid restoration tasks require medium- to long-term planning, a pre-set... The search range is [0.9, 0.99], for example, values ​​of [0.90, 0.95, 0.97, 0.99]. Under the same course learning settings, network structure, and random seed, for different... Values ​​are used to train multiple groups of agent policies.

[0101] In a fixed set of representative simulation experiments, the final performance of each strategy is evaluated. Evaluation metrics include, but are not limited to, the total recovery of critical loads throughout the recovery cycle, the time required to reach 80% recovery rate, and cumulative operating costs. Finally, plotting... The graph shows the relationship between the values ​​and the aforementioned performance metrics. Observing the graph, we can determine the optimal overall performance of the test set. The value is used as a fixed parameter for subsequent training and evaluation.

[0102] A policy gradient method is employed. Each actor network maximizes its expected cumulative reward. To update parameters In gradient calculation, the global value provided by the critic network is used. As a benchmark.

[0103] In one embodiment, the aforementioned course-based learning method progressively trains the multi-agent deep reinforcement learning model through a virtual interactive environment, including: Step S310: Store the experience tuples of each training round of interaction into the shared experience replay pool. Step S320: Randomly sample small batches of experimental data from the experience replay pool; Step S330: Calculate the time-series difference objective based on the small batch of experimental data; Step S340: Based on the time-series differential objective, update the commentator network by minimizing the mean square error loss, and output the global state-action value of the power grid through the updated commentator network; Step S350: Update the actor network by maximizing the expected cumulative reward based on the policy gradient method and the global state-action value of the power grid.

[0104] After completing the policy network design and interactive environment construction, the model training and optimization phase begins. This phase aims to enable the multi-agent system to learn efficient and collaborative power system recovery strategies through continuous interaction between the agents and the simulation environment, utilizing curriculum learning strategies and reinforcement learning algorithms.

[0105] This embodiment employs a progressive training approach based on course learning, specifically divided into three stages, as detailed below: Phase 1 Training: In the IEEE 9-node test system, a fixed fault is set up on a pre-selected single line or node, and only the power grid reconfiguration agent is activated. The goal of this phase is to enable the agent to learn to locate the fault, isolate the faulty area, and restore power to the non-faulty area.

[0106] The second phase of training extends the environment to an IEEE 33-node distribution network, introducing random numbers to randomly generate 3-5 faulty nodes within the network. An energy storage dispatching agent and a generation control agent are then added. The goal of this phase is to enable the agents to learn the coordination of different types of resources. For example, the grid network reconfiguration agent isolates fault points through network reconfiguration, leading to islanding. The mobile energy storage dispatching agent then moves to supply power, while the generation control agent adjusts output to maintain island stability.

[0107] The third phase of training involves implementing a complete dynamic probabilistic fault model and three intelligent agents in the IEEE 39-node transmission network and 3×IEEE 33 distribution network. Fault input is based on typhoon wind speed. The goal of this phase is to train all agents to develop a forward-looking and robust globally optimal recovery strategy under highly uncertain disaster environments, balancing recovery speed, operating costs, and system safety.

[0108] In this embodiment, progressive training includes the following steps: Step S410: Monitor the average reward of the multi-agent deep reinforcement learning model over multiple consecutive training cycles in the current training phase; Step S420: Once the average reward reaches a stable plateau, the system automatically switches from the current training phase to the next training phase.

[0109] Understandably, a preset switching threshold is set during the course learning phase. When the agent's average reward for multiple consecutive training cycles in the current phase reaches the preset threshold, it will automatically switch to the next course learning phase.

[0110] In this embodiment, experience is collected and stored during the training process.

[0111] Specifically, at each training time step t, each agent, based on the current policy of its actor network, performs training based on local observations. Select Action The actions of all agents constitute a joint action. Input the simulation environment. The environment calculates the next state based on the physical model. and instant team rewards The experience tuple from this interaction Stored in a shared experience replay pool.

[0112] In this embodiment, network parameters are updated during the training process.

[0113] Specifically, a small batch of experimental data is randomly sampled periodically from the experience replay pool.

[0114] Computing the temporal difference objective ,in As a discount factor, Estimate the value of the critic network's output in the next state by minimizing the mean squared error loss. Update the reviewer network parameters using the Adam optimizer.

[0115] We use the policy gradient method to estimate the advantage provided by the critic network. We use weights to maximize the expected cumulative return. Specifically, this is achieved by minimizing the actor's loss. Update, among which For the dominant function, For the actor network strategy, the Adam optimizer is also used to update the actor network.

[0116] In one embodiment, hyperparameter tuning and model evaluation are performed during model training.

[0117] Hyperparameter tuning includes determining key hyperparameters through Bayesian optimization and exploring discount factors based on the training experience pool. To balance immediate and long-term rewards; for actors and critics networks in arrive Set independent learning rates for different order sizes.

[0118] Model convergence evaluation includes plotting the average team cumulative reward curve and the critical load recovery rate curve during the training process as monitoring indicators. When the curves no longer rise significantly and tend to stabilize over a large number of consecutive training cycles, the model is considered converged. Then, the converged model is tested on the simulation environment set to failure mode and typhoon disaster trajectory to evaluate its generalization performance. Finally, the set of actor network parameters with the best performance is saved.

[0119] like Figure 3 As shown, this embodiment of the invention also provides a power grid resilience enhancement system based on multi-agent deep reinforcement learning, comprising: The data acquisition module acquires multi-source data from the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data. The topology graph construction module constructs a topology graph based on the multi-source data to obtain a multi-source topology graph; the multi-source topology graph includes a power grid topology graph, a distributed power source graph, and a mobile energy storage vehicle distribution graph. The feature extraction module performs graph feature extraction based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features; The strategy generation module processes the multi-source topology graph and the node features of the multi-source graph using a trained multi-agent deep reinforcement learning model to obtain a power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

[0120] It is understood that the power grid resilience enhancement system based on multi-agent deep reinforcement learning provided in this embodiment of the invention corresponds to the power grid resilience enhancement method based on multi-agent deep reinforcement learning described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the power grid resilience enhancement method based on multi-agent deep reinforcement learning, and will not be repeated here.

[0121] This invention also provides a computer-readable storage medium storing a computer program for a power grid resilience enhancement method based on multi-agent deep reinforcement learning, wherein the computer program causes a computer to execute the power grid resilience enhancement method based on multi-agent deep reinforcement learning as described above.

[0122] This application also provides an electronic device, including: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the grid resilience enhancement method based on multi-agent deep reinforcement learning as described above.

[0123] In summary, compared with existing technologies, it has the following beneficial effects: 1. The embodiments of the present invention construct a multi-dimensional coupled simulation environment that includes a power system physical model, a component dynamic probabilistic fault model under disasters, and a traffic network and mobile resource model. This provides a training environment for intelligent agents that closely resembles the real world, enhancing the generalization ability and adaptability of the model.

[0124] 2. The embodiments of the present invention formalize the problem of improving grid resilience into a distributed partially observable Markov decision process, and define the grid network reconfiguration agent, the mobile energy storage scheduling agent, the generation control agent and their coordination mechanism, supporting the unified optimization of local observation and global objectives.

[0125] 3. In this embodiment of the invention, graph attention networks are used to extract features from the power grid topology, capture the electrical coupling relationships and spatial dependencies between nodes, provide multi-system coupling information for the agent's decision-making, and improve the accuracy and efficiency of decision-making and collaboration.

[0126] 4. This invention employs a "centralized training, distributed execution" architecture. During training, a centralized commentator network is used to evaluate the global state-action value. The objective function value is obtained by weighting multiple objectives, including load power recovery, switching operation costs, grid operation safety, and the coordination among agents, thereby guiding and optimizing the collaborative effect of all agent strategies. During execution, each agent relies entirely on its trained actor network, independently and quickly outputting control actions based solely on its local observation information. For example, the grid network reconstruction agent observes node fault information, the mobile energy storage scheduling agent observes the location and power information of mobile energy storage vehicles, and the generation control agent observes the output information of distributed power sources. Each actor network generates control commands independently and in parallel, achieving the goals of collaborative optimization and real-time decision-making. Simultaneously, a graph attention network is used to extract node features of the grid topology, enabling each agent to perceive and understand the network's connection structure during decision-making, further improving the accuracy of multi-agent collaborative decision-making. 5. The embodiments of the present invention gradually transition from simple scenarios to complex multi-fault environments through course learning, and combine the experience playback mechanism to gradually improve the robustness and adaptability of the intelligent agent in uncertain environments.

[0127] 6. The embodiments of the present invention design a multi-objective reward function that integrates load power recovery amount, operating cost, system security and agent coordination. Through weight adjustment, a dynamic balance is achieved among the multiple objectives, guiding the agent and realizing the optimal power system recovery strategy.

[0128] 7. The embodiments of the present invention adopt a course learning and reinforcement learning training paradigm. Training starts from a fixed single fault and single agent scenario, gradually increasing the number of fault occurrences and introducing a component failure probability model under typhoon disaster to ensure the randomness of fault occurrence and increase the number of agents. The agents learn autonomously through experience generated by interacting with the simulation environment. The experience is stored in a shared replay pool for training, thereby reducing the model's dependence on data, enhancing the model's generalization ability, and overcoming the dependence between data and model.

[0129] 8. By employing reinforcement learning and an interactive autonomous learning strategy, the reliance on large-scale historical fault data is reduced, enhancing the method's practicality in data-scarce scenarios. Through a multi-objective reward function and collaborative mechanism, the recovery process balances critical load power restoration, operating costs, system security, and resource coordination, maximizing overall benefits.

[0130] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for improving power grid resilience based on multi-agent deep reinforcement learning, characterized in that, include: Acquire multi-source data for the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data; A topology graph is constructed based on the multi-source data to obtain a multi-source topology graph. The multi-source topology map includes a power grid topology map, a distributed power source map, and a mobile energy storage vehicle distribution map; Based on the multi-source topology graph, graph features are extracted to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features. The trained multi-agent deep reinforcement learning model processes the multi-source topology graph and the node features of the multi-source graph to obtain the power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

2. The method for improving power grid resilience according to claim 1, characterized in that, Also includes: Build a multi-agent deep reinforcement learning model and determine the model's state space, action space, state transition function, and reward function; A virtual interactive environment is constructed, and its simulation model and constraints are defined. The simulation model includes a power system physical model, a component dynamic probabilistic fault model under disasters, and a traffic network and mobile resource model. The constraints include operational process constraints and recovery process constraints. The operational process constraints include voltage safety constraints, line capacity constraints, generator output constraints, node power balance, and topology connection constraints. The recovery process constraints include limits on the number of switching operations, mobile energy storage scheduling constraints, traffic accessibility constraints, critical load priority constraints, and resource quantity constraints. The multi-agent deep reinforcement learning model is progressively trained using a virtual interactive environment based on a course-based learning method, resulting in a fully trained multi-agent deep reinforcement learning model.

3. The method for improving power grid resilience according to claim 2, characterized in that, The curriculum-based learning method progressively trains the multi-agent deep reinforcement learning model through a virtual interactive environment, including: Store the experience tuples from each training round of interaction into a shared experience replay pool. Randomly sample small batches of experimental data from the experience replay pool; Calculate the temporal difference objective based on the aforementioned small-batch experimental data; Based on the aforementioned time-series differential objective, the commentator network is updated by minimizing the mean square error loss, and the updated commentator network outputs the global state-action value of the power grid. The actor network is updated by maximizing the expected cumulative return based on the policy gradient method and the global state-action value of the power grid.

4. The method for improving power grid resilience according to claim 1, characterized in that, The multi-agent deep reinforcement learning model, after training, processes the multi-source topology graph and the features of the multi-source graph nodes to obtain a power grid control strategy, including: The power grid network is reconstructed using an intelligent agent network to process the power grid topology and obtain switching action decisions for each route. The intelligent agent network for power grid network reconstruction includes a first GAT layer, a second GAT layer, and a first fully connected layer. The mobile energy storage scheduling intelligent agent network processes the graph node features and edge connection data of the mobile energy storage vehicle to obtain the probability of the mobile energy storage vehicle connecting to each grid node, the charging and discharging power at the time of connection, and the connection status at the time of connection; the mobile energy storage scheduling intelligent agent network includes a third GAT layer and a fourth GAT layer. The power generation control agent network processes the graph node features and edge data of the distributed power source to obtain continuous control commands for the active and reactive power output of the distributed power source; the power generation control agent network includes a fifth GAT layer and a sixth GAT layer.

5. The method for improving power grid resilience according to claim 4, characterized in that, The critic network consists of a graph attention encoder and a multilayer perceptron; the graph attention encoder includes a seventh GAT layer and an eighth GAT layer; the multilayer perceptron includes a second fully connected layer, a third fully connected layer, and a fourth fully connected layer.

6. The method for improving power grid resilience according to claim 2, characterized in that, The reward function is: in: In the formula, It is an adjustable positive weighting coefficient used to balance multiple objectives such as recovery amount, operating cost, safety, and synergy. , ; This indicates a reward for the amount of recovery. This indicates a penalty for operating costs; This indicates a security penalty; Indicates a collaborative reward; Indicates the load is t The power increment recovered at each moment, i.e. t Time and t The power supply difference at time 1; For the set of critical loads; For the set of important loads; For a single load set; For the set of all switches; for t The state at any given moment: 0 indicates disconnection, 1 indicates connection; express Time's up t The number of switching actions at any given time; Represents a node i The actual voltage amplitude at the current moment, Represents a node i The upper limit of voltage safety, Represents a node i The lower limit of voltage safety; This is an indicator function that takes the value 1 when a successful collaborative event occurs, such as a mobile energy storage vehicle supplying power to an island formed by reconfiguration, and 0 otherwise; Represents the set of lines in a power distribution network; Indicates the line The actual current; Indicates the line The maximum current value.

7. The method for improving power grid resilience according to claim 2, characterized in that, The curriculum-based learning method progressively trains the multi-agent deep reinforcement learning model through a virtual interactive environment, including: Monitor the average reward of the multi-agent deep reinforcement learning model over multiple consecutive training cycles in the current training phase; Once the average reward reaches a stable level, the system will automatically switch from the current training phase to the next training phase.

8. A power grid resilience enhancement system based on multi-agent deep reinforcement learning, characterized in that, include: The data acquisition module acquires multi-source data from the power grid resilience enhancement system; the multi-source data includes power grid physical and topology data, component vulnerability parameters, geospatial and transportation data, and meteorological and disaster scenario data. The topology graph construction module constructs a topology graph based on the multi-source data to obtain a multi-source topology graph. The multi-source topology map includes a power grid topology map, a distributed power source map, and a mobile energy storage vehicle distribution map; The feature extraction module performs graph feature extraction based on the multi-source topology graph to obtain multi-source graph node features; the multi-source graph node features include power grid graph node features, distributed power source graph node features, and mobile energy storage vehicle distribution graph node features; The strategy generation module processes the multi-source topology graph and the node features of the multi-source graph using a trained multi-agent deep reinforcement learning model to obtain a power grid control strategy. The multi-agent deep reinforcement learning model includes a multi-agent actor network and a critic network. The multi-agent actor network includes a power grid network reconstruction agent actor network, a mobile energy storage scheduling agent actor network, and a power generation control agent actor network.

9. A computer-readable storage medium, characterized in that, It stores a computer program for improving power grid resilience based on multi-agent deep reinforcement learning, wherein the computer program causes a computer to execute the power grid resilience improvement method based on multi-agent deep reinforcement learning as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including methods for performing the grid resilience enhancement method based on multi-agent deep reinforcement learning as described in any one of claims 1 to 7.

Images