Power grid reconfiguration method, system, device and medium

Abstract

The application discloses a power grid reconstruction method, system, device and medium, comprising: acquiring state information when a target power grid is in a fault state; inputting the state information into a pre-trained policy network to obtain original decision scores of each candidate recovery action; wherein the candidate recovery action is used for recovering the target power grid to a normal operating state, and the policy network is pre-trained by using a recovery trajectory through imitation learning in an offline stage; determining effectiveness information of each candidate recovery action according to the state information and a preset power grid operation constraint; performing correction processing on the original decision scores based on the effectiveness information to obtain corrected decision scores; determining and executing a target recovery action from each candidate recovery action according to the corrected decision scores to reconstruct the target power grid. Through pre-training by imitation learning and soft mask dynamic constraint, the application improves the rapidity and robustness of grid reconstruction in a random complex fault scenario.

Description

Technical Field

[0001] This application belongs to the field of power system operation and control technology, and in particular relates to a power grid reconfiguration method, system, equipment and medium. Background Technology

[0002] After a power outage occurs, the power system needs to gradually restore power from a state of complete darkness or partial power outage; this process is called grid reconfiguration. Disturbances such as lightning strikes, wildfires, equipment failures, and cyberattacks can all cause large-scale power outages, requiring reconfiguration to restore the system to normal operation as quickly as possible.

[0003] Among related technologies, network reconfiguration methods mainly include three categories: mixed-integer programming, heuristic algorithms, and deep reinforcement learning. Mixed-integer programming methods have high computational complexity and long solution time as the system scale increases; heuristic algorithms are prone to getting trapped in local optima and lack stability; although deep reinforcement learning methods have the potential for real-time decision-making, in random and complex fault scenarios, the agent starts from scratch to explore and experiment, and the state space and action space expand exponentially, making it difficult to learn a reconfiguration strategy that satisfies the constraints of topological connectivity, power flow safety, and device availability within a reasonable time. Summary of the Invention

[0004] In view of the above, this application provides a power grid reconfiguration method, system, device, medium, and product to address some of the deficiencies mentioned in the background art.

[0005] In a first aspect, embodiments of this application provide a power grid reconfiguration method, including: Obtain the status information of the target power grid when it is in a fault state; The state information is input into a pre-trained policy network to obtain the original decision scores for each candidate recovery action; wherein, the candidate recovery action is used to restore the target power grid to normal operation, and the policy network is pre-trained using the recovery trajectory through imitation learning in the offline stage; Based on the status information and preset power grid operation constraints, the validity information of each candidate recovery action is determined; Based on the validity information, the original decision score is corrected to obtain a corrected decision score; Based on the modified decision score, a target recovery action is determined from each of the candidate recovery actions and executed to reconstruct the target power grid.

[0006] In a second aspect, embodiments of this application provide a power grid reconfiguration system, comprising: The information acquisition module is configured to acquire status information of the target power grid when it is in a fault state. The decision scoring module is configured to: input the state information into a pre-trained policy network to obtain the original decision scores for each candidate recovery action; wherein the candidate recovery action is used to restore the target power grid to normal operation. The validity determination module is configured to: determine the validity information of each candidate recovery action based on the status information and preset power grid operation constraints; The decision correction module is configured to: correct the original decision score based on the validity information to obtain a corrected decision score; The power grid reconfiguration module is configured to: determine and execute a target recovery action from each of the candidate recovery actions based on the corrected decision score, so as to reconfigure the target power grid.

[0007] In a third aspect, this application provides a computer device including one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method described in the first aspect.

[0008] In a fourth aspect, this application provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the one or more processors to perform the method described in the first aspect.

[0009] The power grid reconfiguration method, system, device, and medium provided in this disclosure pre-train the policy network using imitation learning, enabling it to make initial feasible decisions before online reinforcement learning begins. This alleviates the cold start problem caused by the random initialization agent frequently selecting invalid actions in a high-dimensional action space. By applying a soft mask mechanism to invalid actions with dynamic penalties related to policy confidence, it balances constraint guidance and policy exploration, improving training convergence speed and stability. By regenerating action validity labels and soft mask corrections based on real-time states at each decision step, the recovery strategy can be dynamically adjusted according to subsequent random faults or equipment state changes, improving the reconfiguration robustness and generalization ability under complex fault scenarios. Attached Figure Description

[0010] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 A flowchart illustrating a power grid reconfiguration method provided in an embodiment of this application is shown; Figure 2This illustration shows a schematic diagram of the framework structure of a power grid reconfiguration process provided in an embodiment of this application; Figure 3 This paper shows a schematic diagram of the structure of a power grid reconfiguration system provided in an embodiment of this application; Figure 4 A schematic diagram of a more specific device hardware structure provided in an embodiment of this application is shown. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0013] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0014] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with specific embodiments and the accompanying drawings.

[0015] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0016] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0017] For clarity, before describing the specific technical solutions of the embodiments of this application, the technical terms involved in the embodiments of this application will be explained first.

[0018] A policy network is a neural network model used to output the probability of selecting each candidate recovery action based on the current state of the power grid. In this embodiment, the policy network is pre-trained using recovery trajectories through imitation learning in the offline stage, and its parameters are updated through interaction with the power grid simulation environment via reinforcement learning in the online stage. This network is used to generate the original decision scores for each candidate recovery action at each decision step.

[0019] Value Network: A neural network model used to estimate the state value of the current power grid. In this embodiment, the value network and the policy network together form an actor-critic architecture. The state value output by the value network is used to calculate the advantage function, and the advantage function is used to update the parameters of the policy network.

[0020] Agent: Refers to a decision-making entity composed of a policy network, a value network, and an action sampling module. In the embodiments of this application, the agent is used to receive the current state information of the power grid, generate an action probability distribution through the policy network, sample and determine the target recovery action after soft masking correction, and execute it, thereby realizing interaction with the power grid simulation environment.

[0021] Recovery Trajectory: Refers to the optimal or suboptimal sequence of recovery actions generated by the optimization solver under typical fault scenarios, satisfying power grid operation constraints. In the embodiments of this application, the recovery trajectory includes the power grid state at multiple consecutive time steps and the recovery action corresponding to each power grid state. The recovery actions include bus recovery actions and load recovery actions, which are used to pre-train the policy network through behavior cloning.

[0022] Validity Information: This refers to the marking information used to identify whether each candidate recovery action meets the execution conditions under the current power grid state. In the embodiments of this application, the validity information is generated based on the current topological connectivity of the power grid, the availability of equipment, and the power grid operation constraints. Candidate recovery actions that meet the execution conditions are marked as valid recovery actions, and candidate recovery actions that do not meet the execution conditions are marked as invalid recovery actions.

[0023] Soft masking refers to a method of dynamically penalizing the scores of invalid actions in the original decision scores output by the policy network. In this embodiment, the soft masking applies a penalty value positively correlated with the maximum original decision score in the current sub-action space to the original decision scores of invalid actions, reducing the selection probability of invalid actions to less than the selection probability of valid actions but greater than zero, thus distinguishing it from the hard masking method that directly sets the selection probability of invalid actions to zero.

[0024] Dynamic Penalty: refers to the penalty value applied to the original decision score of an invalid recovery action, the penalty intensity of which adaptively changes with the current confidence level of the policy network. In the embodiments of this application, the dynamic penalty is determined based on the maximum value among the original decision scores of all candidate recovery actions in the current sub-action space and a preset penalty coefficient. The larger the maximum value, the larger the penalty value; the smaller the maximum value, the smaller the penalty value.

[0025] Reward feedback refers to the scalar reward value returned by the power grid simulation environment after performing a recovery action, based on the action's execution result. In this embodiment, the reward feedback is determined based on at least one of the following: the amount of restored load, the recovery status of critical loads, constraint violation penalties, switching operation costs, and recovery time costs. It is used to evaluate the merits of actions and update the policy network parameters during reinforcement learning.

[0026] As described in the background section, the relevant technologies have many problems in the context of power grid reconfiguration.

[0027] First, the network reconfiguration method based on mixed integer programming in related technologies needs to incorporate discrete decision variables such as equipment recovery status and recovery timing, along with power flow and safety constraints, into the optimization model for iterative solution. As the number of candidate black-start power sources, the number of faulty components, and the system scale increase, the number of scenarios that need to be solved increases dramatically, and the overall computation time increases significantly, making it difficult to meet the requirements for rapid decision-making during power outage recovery.

[0028] Secondly, heuristic-based network reconstruction methods in related technologies search for optimal recovery paths in the feasible recovery scheme space based on preset evaluation indicators, search rules, or population evolution mechanisms. These methods typically assume that the states of critical components are relatively fixed during the recovery process, and the generated recovery schemes mainly correspond to the fault conditions at the time of solution. However, under conditions such as extreme weather, network attacks, or cascading disturbances, critical components such as transmission lines and substations may experience subsequent random faults during the recovery process, causing the originally planned recovery path to no longer meet topological connectivity, power flow safety, or equipment availability constraints. Existing methods need to re-identify the fault state and perform optimization solutions or heuristic searches again, resulting in slow response times and potential delays in the recovery process.

[0029] Furthermore, while deep reinforcement learning-based network reconstruction methods possess real-time decision-making potential, the agent needs to adapt simultaneously to different black-start unit configurations, different combinations of line faults, and potential subsequent faults during the recovery process. This leads to an exponential expansion of the state space and action space with the number of scenarios. Traditional deep reinforcement learning relies on trial-and-error exploration from scratch. Without effective constraint guidance, the agent is prone to choosing ineffective actions that violate network topology constraints, device availability constraints, or power flow safety constraints. This results in sparse rewards, low training sample utilization, and slow learning convergence, making it difficult to learn a network reconstruction strategy that combines speed, safety, and robustness within a reasonable training time.

[0030] Furthermore, deep reinforcement learning methods in related technologies often employ either maskless or hard masking mechanisms to handle action constraints. Maskless methods have low exploration efficiency, as the agent directly samples from the entire action space, making it difficult to distinguish between valid and invalid actions in the current state, and easily leading to repeated exploration among a large number of actions that violate constraints. While hard masking methods can set the probability of invalid actions to zero, they completely block the possibility of the agent obtaining feedback information from invalid actions and constraint boundaries, resulting in insufficient learning ability of the policy towards constraint boundaries.

[0031] Therefore, how to overcome the computational efficiency bottleneck of mixed integer programming and heuristic algorithms and the insufficient adaptability of static schemes in random complex fault scenarios, while solving the difficulties of cold start, invalid exploration and constraint learning faced by deep reinforcement learning methods, and achieving fast, safe and adaptive network reconstruction decisions has become an urgent technical problem to be solved in this field.

[0032] Figure 1 A flowchart illustrating a power grid reconfiguration method provided in an embodiment of this application is shown; Figure 2 This diagram illustrates a framework structure of a power grid reconfiguration process provided in an embodiment of this application. Figure 1 and Figure 2 As shown, the power grid reconfiguration method in this disclosure further includes the following steps: S102: Obtain the status information of the target power grid when it is in a fault state.

[0033] When a power outage occurs in the power grid, it is necessary to first obtain the current operating status of the power grid as input for subsequent decision-making. Status information can include the topology of the power grid (such as bus sets, branch sets, and switch status) and operating parameters (such as voltage of each node, power flow of each branch, generator output, load power, and switching status).

[0034] In some alternative embodiments, this status information can be acquired in real time by Supervisory Control and Data Acquisition (SCADA) and Phasor Measurement Unit (PMU) deployed in the power grid.

[0035] Because the power grid state changes rapidly after a power outage, the latest state information must be retrieved before each decision-making step to ensure that the information used for the decision is consistent with the actual state of the power grid and to avoid decision-making errors due to state lag. The state information also includes line fault status, which indicates whether each transmission line has experienced a fault at the current moment. This information will be used to subsequently determine the effectiveness of each candidate restoration action.

[0036] In some optional embodiments, before acquiring the status information, this application may also initiate a reconfiguration process in response to detecting a power outage event in the target power grid. The power outage event may be triggered by detecting a system frequency below a preset threshold, a continuous drop in bus voltage, or a circuit breaker tripping signal.

[0037] S104: Input the state information into a pre-trained policy network to obtain the original decision scores for each candidate recovery action.

[0038] After obtaining the current power grid status information, the status information needs to be input into a pre-trained policy network, which will then provide a preference score for each candidate recovery action.

[0039] The policy network is a core component of the agent in deep reinforcement learning. In this embodiment, the policy network is a neural network model, which may include an input layer, multiple fully connected layers, and an output layer. The input layer receives the encoded state information vector, and after feature extraction and nonlinear transformation through multiple fully connected layers, the output layer outputs the original decision score corresponding to each candidate recovery action. The original decision score is the original logit value, which characterizes the policy network's preference for performing the action in the current state; a higher value indicates that the policy network is more inclined to choose that action.

[0040] In this embodiment, the candidate restoration action can include two types of cooperating action components. The first type is a bus restoration action, used to select the next bus to be restored in the target power grid from multiple candidate buses; this action is a multi-classification decision. The second type is a load restoration action, used to determine whether each load in the target power grid should be engaged; this action includes multiple binary classification decisions, with one binary classification decision corresponding to each load. This hybrid action space modeling method can simultaneously express the operations of both restoration path selection and load engagement decision-making, adapting to the actual operational characteristics of grid reconfiguration tasks.

[0041] Since the output layer of the policy network needs to process both multi-classification and binary classification actions simultaneously, in this embodiment, the output layer of the policy network outputs raw logit values ​​for multiple categories for bus recovery actions, with each category corresponding to a candidate bus; for load recovery actions, it outputs a raw logit value for each load.

[0042] It should be noted that the policy network is pre-trained offline using imitation learning and recovery trajectories; the training process will be detailed later. Offline pre-training enables the policy network to possess preliminary feasible decision-making capabilities before entering online decision-making, thus solving the cold start problem caused by traditional deep reinforcement learning methods that involve trial and error from scratch.

[0043] S106: Determine the validity information of each candidate recovery action based on the status information and the preset power grid operation constraints.

[0044] The raw decision scores output by the policy network only reflect the network's preference for actions, but do not take into account the current physical constraints of the power grid. For example, the policy network may prefer to restore a certain bus, but the line connecting that bus is currently in a fault state, making the action practically impossible. Without this constraint, the agent may repeatedly choose ineffective actions, leading to inefficient decision-making.

[0045] To address this issue, this application determines the validity of each candidate recovery action based on current power grid status information and preset power grid operation constraints. The preset power grid operation constraints may include at least one of the following: topology connectivity constraints, branch capacity constraints, node voltage constraints, load recovery constraints, and action execution constraints.

[0046] Specifically, the first step is to determine the current topological connectivity of the target power grid and the availability status of each device within the target power grid based on the status information. For example, based on the open / closed status of switches and the fault status of lines, it can be determined which busbars can form a path, which devices are currently available, and which devices are in a fault-prone and unavailable state.

[0047] Then, based on the topology connectivity, equipment availability, and grid operation constraints, it is determined whether each candidate restoration action meets the preset execution conditions. For example, for a bus restoration action, it is necessary to determine whether the candidate bus can establish a connection path with the energized area through a currently closable switch; if the candidate bus can connect with the energized area, the bus restoration action is valid; otherwise, it is invalid. For a load restoration action, it is necessary to determine whether the bus containing the candidate load is already energized and whether restoring the load will cause branch overload or voltage exceedance; if the bus containing the load is already energized and the operation constraints are met after restoration, the load restoration action is valid; otherwise, it is invalid.

[0048] Finally, candidate recovery actions that meet the execution conditions are marked as valid recovery actions, and candidate recovery actions that do not meet the execution conditions are marked as invalid recovery actions. The validity information includes the marking results of each candidate recovery action as valid or invalid.

[0049] S108: Based on the validity information, the original decision score is corrected to obtain a corrected decision score.

[0050] In the above steps, the policy network outputs the raw decision scores for each candidate action; the validity determination module gives a label indicating whether each candidate action is "valid" or "invalid". However, there are two technical difficulties in effectively combining the two to improve decision-making efficiency.

[0051] On the one hand, if no invalid actions are processed (i.e., no masking method), the agent directly samples in the complete action space. Due to the strict constraints of power grid operation, a large number of actions are invalid in any state. The agent will frequently select invalid actions, resulting in sparse rewards and low utilization of training samples.

[0052] On the other hand, if the probability of selecting invalid actions is set to zero (i.e., hard masking), although it can avoid selecting invalid actions, it completely blocks the possibility of the agent obtaining feedback information from invalid actions and constraint boundaries, resulting in insufficient learning ability of the policy on constraint boundaries, and also losing the opportunity to explore better solutions that may exist near the constraint boundaries.

[0053] To overcome the aforementioned difficulties, this application proposes a soft masking mechanism. Under the soft masking mechanism, for candidate actions marked as invalid recovery actions, their corresponding original decision scores are subject to dynamic penalties, resulting in revised decision scores. Conversely, for candidate actions marked as valid recovery actions, their original decision scores remain unchanged.

[0054] Specifically, for any sub-action space, let the original decision score of the candidate recovery action within that sub-action space be raw_logit, and the original decision score corresponding to the invalid recovery action be corrected as follows: soft_mask_logit = raw_logit-λ×max_logit Where soft_mask_logit represents the decision score after soft mask correction, max_logit represents the maximum value among the original decision scores of all candidate recovery actions in the sub-action space, and λ is a preset penalty coefficient used to control the penalty intensity.

[0055] The technical implication of this modified formula is that for invalid recovery actions, their decision score is subtracted by a penalty value (λ×max_logit) that is positively correlated with the maximum decision score in the current sub-action space. Since the penalty term includes max_logit, this penalty value can adaptively adjust as the current confidence of the policy network changes.

[0056] In the early stages of training, the policy network parameters have not yet converged, the original decision scores of each action are relatively flat, the absolute value of max_logit is relatively small, so the penalty value λ×max_logit is also small, and invalid actions still retain a relatively high selection probability, which is conducive to retaining the ability to explore the area near invalid actions, enabling the agent to collect feedback information near the constraint boundary.

[0057] As training progresses, the policy network gradually converges, the original decision scores of effective actions gradually increase, and max_logit also increases accordingly. The penalty value λ×max_logit increases accordingly, and the probability of selecting ineffective actions is further reduced, which helps to improve the utilization rate of available actions and makes the decision more focused on effective recovery paths.

[0058] Another key design feature of this correction method is that after the corrected decision score for an invalid action is corrected by `soft_mask_logit = raw_logit - λ × max_logit`, the selection probability of that invalid action, although significantly reduced when calculated using the softmax or sigmoid function, remains greater than zero. This is because the output range of both the softmax and sigmoid functions is (0, 1), and the output probability is always greater than zero, regardless of how small the input value is (as long as it is not negative infinity). This characteristic ensures that invalid actions retain the possibility of being selected during training, allowing the agent to continuously obtain feedback information from the constraint boundaries and strengthen its cognitive learning ability regarding these boundaries.

[0059] This is fundamentally different from the hard masking method (which directly sets the logit of invalid actions to negative infinity, making the softmax output probability strictly zero). The hard masking method completely blocks the agent from exploring the constraint boundary, while the soft masking method in this application retains a limited opportunity to explore the constraint boundary while suppressing invalid actions.

[0060] In this embodiment, the aforementioned soft mask correction is applied to different sub-action spaces. Since candidate recovery actions include two different action components: bus recovery actions (multi-classification) and load recovery actions (multiple binary classifications), their action spaces have different scales and data distribution characteristics. Unified processing would lead to mutual interference between the probability scales of different action types. Therefore, for the bus recovery action subspace and the load recovery action subspace, their respective maximum values ​​(max_logit) are calculated, their respective penalty values ​​are determined, and the aforementioned soft_mask_logit = raw_logit - λ × max_logit formula is applied for soft mask correction. The corrected selection probability is calculated independently for each, thereby ensuring that the probability distribution within each sub-action space is reasonable and does not interfere with each other.

[0061] S110: Determine and execute the target recovery action from each of the candidate recovery actions based on the modified decision score to reconstruct the target power grid.

[0062] After obtaining the revised decision score, the target recovery actions to be actually implemented need to be determined accordingly.

[0063] First, based on the original decision scores of effective recovery actions and the revised decision scores of ineffective recovery actions, the selection probability corresponding to each candidate recovery action is calculated. Specifically, for the bus recovery action subspace, since this subspace is a multi-class decision space, the Softmax function is applied to the revised or original scores of each candidate bus to obtain the probability that each candidate bus will be selected for execution: P(a_i) = exp(score_i) / Σ_jexp(score_j) Where P(a_i) represents the selection probability of the i-th candidate bus recovery action, and score_i is the decision score of the i-th candidate action (the original score for valid actions and the revised score for invalid actions). The Softmax function converts the scores of multiple categories into a probability distribution, and the sum of the selection probabilities of all candidate actions is 1.

[0064] For the load recovery action subspace, since this subspace includes multiple independent binary classification decisions, the Sigmoid function is applied to each load to obtain the probability of each load being selected for deployment: P(l_k) = 1 / (1+exp(-score_k)) Where P(l_k) represents the probability of the k-th load being restored, and score_k is the decision score for that load restoration action (the original score for valid actions and the revised score for invalid actions). The Sigmoid function maps the score of a single binary action to the interval (0, 1), representing the probability that the action is executed.

[0065] Then, action sampling is performed based on the calculated selection probabilities. Specifically, for bus restoration actions, the next target bus to be restored is determined by sampling according to the multi-class probability distribution; for load restoration actions, each load is determined by sampling according to the binary probability corresponding to each load to determine whether each load should be restored.

[0066] After determining the target recovery action, execute the action in the power grid simulation environment. The execution process includes: updating the operable switch state based on the target bus selection action to establish a power supply path between the target bus and the energized area; updating the load connection state based on the load recovery action; and performing power flow calculations or constraint checks to determine whether the power grid operation constraints are met after the action is executed. If the action results in an infeasible state, the environment records the constraint violation type and applies a penalty in the reward calculation; if the action is feasible, the system state is updated and the process proceeds to the next time step.

[0067] The power grid simulation environment is used to simulate the operating characteristics of a real power grid, receive recovery actions output by the agent, perform state updates, and return new state information and reward feedback for the agent's online learning. Since the cost of repeated trial and error in a real power grid is too high for online reinforcement learning, and may even lead to secondary accidents, using a simulation environment to replace the real power grid for training is a necessary technical choice.

[0068] Through the above steps, this application completes a reconfiguration action of a decision step. In fact, power grid reconfiguration is a multi-step sequential decision-making process. After executing a recovery action, the power grid state changes, and the agent needs to return to S102 to obtain the updated state information, execute S104 to S110 again, output and execute the next action, and so on until the target power grid is restored to normal operation.

[0069] In the above process, steps S102 to S110 have fully described the process of making online decisions using a trained policy network. However, it should also be noted how the policy network is pre-trained. Since the training process of the policy network involves two stages—imitation learning pre-training and online reinforcement learning update—these will be described separately below.

[0070] First, regarding the pre-training of the policy network. In traditional deep reinforcement learning methods, the parameters of the policy network are typically initialized randomly. The agent, initially unaware of the power grid's topology and operational constraints, can only explore through random trial and error. However, power grid reconfiguration is a complex decision-making problem with a high-dimensional state space, a mixed action space, and strict operational constraints. A randomly initialized agent is highly likely to choose invalid actions in the early stages of training, such as failing to connect the target bus, mistakenly connecting loads lacking power supply capabilities, or violating power flow safety constraints. This results in sparse rewards, low sample utilization, and slow convergence.

[0071] To overcome the aforementioned cold start problem, this application introduces an imitation learning pre-training stage before online reinforcement learning. The specific steps are as follows.

[0072] First, based on the optimization objective and grid operation constraints, multiple recovery trajectories are generated when the target grid is in a preset fault state. A recovery trajectory refers to a complete sequence of recovery actions from a power outage state to a state of normal operation. The optimization objective is at least one of maximizing the restored load, minimizing the recovery time, or reducing grid losses. In this embodiment, a mixed-integer programming solver is used to solve the grid reconfiguration optimization model, generating an expert recovery strategy that satisfies the grid operation constraints. The recovery trajectory includes multiple grid states of the target grid at different times and the recovery actions corresponding to each grid state.

[0073] Although mixed-integer programming solvers are computationally expensive, they guarantee the generation of optimal or near-optimal solutions that satisfy constraints under a given fault scenario. Therefore, they are suitable for generating high-quality expert demonstration data offline. This expert demonstration data contains information on the optimal actions that the power grid should take in various states during the recovery process, forming the foundation for policy network learning.

[0074] Then, a trajectory dataset is constructed based on multiple power grid states and the corresponding recovery actions for each power grid state. Each trajectory in the trajectory dataset is a complete "state-action" sequence, recording the power grid state and its corresponding optimal recovery action at each decision step in a predetermined fault scenario, from the initial power outage state to the completion of recovery.

[0075] Finally, the initial policy network is trained based on the trajectory dataset to obtain a pre-trained policy network. The specific training method is behavior cloning.

[0076] It should be noted that behavioral cloning is a type of imitation learning algorithm. In this application, the policy network learns the expert's recovery decision-making logic by directly imitating the "state-action" pairings in the expert's trajectory. Specifically, for any recovery trajectory in the trajectory dataset, the power grid state in that recovery trajectory is input into the initial policy network, and the network outputs the predicted bus recovery action and the predicted load recovery action.

[0077] Since the policy network needs to process two different action components simultaneously: bus recovery actions (multi-class) and load recovery actions (multiple binary classifications), the training loss also includes two parts: A first loss, the multi-class cross-entropy loss, is determined based on the difference between the predicted bus recovery action and the corresponding bus recovery action in the recovery trajectory; the second loss, the sum of the binary cross-entropy losses for each load, is determined based on the difference between the predicted load recovery action and the corresponding load recovery action in the recovery trajectory. The total loss function is: L_BC = L_bu + α × L_load Where L_BC represents the total loss of behavior cloning, L_bus represents the multi-class cross-entropy loss corresponding to the bus restoration action, L_load represents the sum of the binary cross-entropy losses corresponding to the load restoration action, and α is a preset weighting coefficient used to control the proportion of load restoration loss in the total loss. The specific value of α can be determined according to the emphasis requirements of bus restoration accuracy and load restoration accuracy—when the business focuses more on the accuracy of the bus restoration path, α can be decreased to reduce the weight of load restoration loss; when the business focuses more on the accuracy of load deployment, α can be increased to increase the weight of load restoration loss. By minimizing the above total loss to adjust the parameters of the policy network, the pre-trained policy network is obtained.

[0078] Through the aforementioned pre-training process, the policy network has initially acquired feasible decision-making capabilities before entering online reinforcement learning. It can output recovery actions that conform to the power grid operation constraints based on the current power grid state, thereby effectively alleviating the cold start problem caused by random initialization.

[0079] Secondly, regarding online reinforcement learning updates Although the pre-trained policy network already possesses a certain decision-making ability, the expert trajectories used in the pre-training stage only cover a limited number of preset fault scenarios, while the types and combinations of faults faced by the actual power grid are more complex and diverse. Therefore, reinforcement learning is still needed in the online stage to continuously optimize the policy network in actual interactions, thereby improving its generalization ability and robustness.

[0080] During the online reinforcement learning phase, after each decision step determines and executes the target recovery action, the power grid simulation environment returns reward feedback based on the action execution result. The agent updates the policy network parameters based on the reward feedback, thereby gradually optimizing the network.

[0081] Specifically, the agent interacts with the power grid simulation environment. At the start of each training round, the power grid simulation environment randomly generates or reads a fault scenario as the current training scenario. At each decision step, the agent acquires the current state information, generates actions through the policy network, and executes the actions after validity judgment and soft mask correction. When the round's end conditions are met (such as all recoverable loads being restored, reaching the preset maximum number of recovery steps, no actions available, or the system entering a terminated state), the round ends, and the next training round begins.

[0082] Reward feedback is the core training signal for reinforcement learning. In this embodiment, reward feedback is determined based on at least one of the following: the amount of load restored, the status of important load restoration, constraint violation penalties, switching operation costs, and restoration time costs. Specifically, when an action successfully restores the load, the system provides a positive reward; the greater the amount of load restored and the more important loads restored, the higher the reward value. When an action leads to constraint violations (such as overload or voltage exceeding limits), the system provides a negative reward (penalty). When a switching operation is performed or restoration time is consumed, the system provides a smaller negative reward to reflect the operation cost and restoration time cost. Through this multi-objective comprehensive reward design, the agent gradually learns a strategy that balances restoration efficiency and safety during training.

[0083] After obtaining the reward feedback, it needs to be used to update the parameters of the policy network. This embodiment employs the A2C (Advantage Actor-Critic) algorithm for online reinforcement learning. The A2C algorithm simultaneously maintains a policy network (Actor) and a value network (Critic). The policy network outputs the probability of action selection, while the value network estimates the state value of the current power grid state; both work together to achieve policy optimization.

[0084] Specifically, the state information before the target recovery action, the target recovery action, the reward feedback, and the latest state information after execution are input into the value network to calculate the temporal difference error (TD error). The temporal difference error is calculated as follows: TD_error = r + γ × V(s') - V(s) Where r represents the reward feedback obtained after performing the target recovery action, γ is the discount factor (a preset constant with a value between 0 and 1, used to adjust the weight of future rewards), V(s) represents the value network's estimate of the state value of state s before the action is performed, and V(s') represents the value network's estimate of the state value of state s' after the action is performed.

[0085] Temporal difference error reflects the difference between the estimated value of the current state and the target value, and is a core signal for judging the quality of an action. When TD_error > 0, it means that the actual reward plus the value of subsequent states is higher than the original estimate, indicating that the action performed is better than expected, and the probability of this action being selected should be increased; when TD_error < 0, it means that the action performed is worse than expected, and the probability of this action being selected should be decreased.

[0086] Then, the advantage function is calculated based on the temporal difference error. In the A2C algorithm, the advantage function A(s,a) measures the superiority of performing a specific action a relative to the average level (i.e., the current state value V(s)) in the current state s. The advantage function directly uses the temporal difference error as its estimate: A(s,a) =TD_error=r+γ×V(s')-V(s) A positive dominance function indicates that the action is better than average and its probability of being selected should be increased; a negative dominance function indicates that the action is worse than average and its probability of being selected should be decreased.

[0087] The parameters of the policy network are updated based on the advantage function. The loss function of the policy network is: Loss_policy=-logπ(a|s)×A(s,a) Here, π(a|s) represents the probability that the policy network selects action a in the current state s, logπ(a|s) is the logarithmic value of this probability, and A(s,a) is the advantage function. The meaning of this loss function is: for actions with a positive advantage function, increase their logarithmic probability (i.e., increase the probability that the action is selected); for actions with a negative advantage function, decrease their logarithmic probability (i.e., decrease the probability that the action is selected). By optimizing this loss function through gradient descent, the parameters of the policy network are updated.

[0088] Meanwhile, the loss function of the value network is: Loss_value=(TD_error)² That is, by minimizing the square of the temporal difference error, the value network's estimation of state value becomes increasingly accurate. The parameters of the value network are updated synchronously.

[0089] Through the aforementioned online reinforcement learning update process, the policy network is continuously optimized in actual interactions, gradually improving its adaptability to different fault scenarios and the quality of recovery decisions.

[0090] Finally, regarding continuous decision-making mechanisms based on real-time state feedback In this embodiment, the decision-making process described in steps S102 to S110 is executed in each decision step. After each recovery action is completed, the agent returns to S102 to reacquire the current state information of the power grid and regenerates the action validity flag and soft mask correction result based on the latest power grid state information. This continuous decision-making mechanism based on real-time state feedback enables the recovery strategy to dynamically adjust subsequent actions as the power grid state changes in real time.

[0091] Its technical advantage lies in the fact that when a subsequent random fault occurs during the recovery process (such as a restored line tripping again, a black start unit suddenly failing and shutting down), the state information obtained by the agent in the next decision loop already includes the above changes. The validity determination module will re-evaluate the validity of each candidate action based on the new fault state, and the soft mask correction will also be recalculated based on the new state, thereby automatically avoiding infeasible recovery paths and replanning subsequent solutions, without having to re-execute the complete optimization solution or search process as in traditional methods.

[0092] Through the synergistic effect of all the above steps, this application can improve the feasible strategy search capability of the agent in the early stage of training, reduce the proportion of invalid action exploration, improve the training convergence speed and stability, and enhance the speed, security and robustness of network reconstruction under complex random fault scenarios, while satisfying the network connectivity, topology constraints and load recovery constraints.

[0093] In some optional embodiments, the status information includes at least one of the following: a set of energized buses, a set of de-energized buses, a set of restored loads, a set of unrestored loads, line fault status, and electrical quantities of each node or branch. This status information collectively constitutes a complete description of the current operating state of the power grid, providing sufficient input information for subsequent validity assessment and decision-making.

[0094] In some optional embodiments, the power grid operation constraints include at least one of topology connectivity constraints, branch capacity constraints, node voltage constraints, load restoration constraints, and action execution constraints. Specifically, topology connectivity constraints require restored areas to remain connected to avoid islanding; branch capacity constraints require that the power flow of each branch does not exceed its thermal stability limit; node voltage constraints require that the voltage of each node be maintained within allowable ranges; load restoration constraints require that load restoration meet priority and tiered requirements; and action execution constraints require that the restoration actions themselves (such as switching operation sequence, generator starting conditions, etc.) meet equipment operating procedures.

[0095] In some optional embodiments, the action validity determination module determines whether each candidate action is valid based on the current power grid topology and physical constraints. Specifically, for target bus selection actions, if a bus cannot form a feasible connection with the energized area through the currently operable switch, it is marked as an invalid action; for load restoration actions, if the area where a load is located does not have a power supply path or violates capacity, voltage, or other constraints after restoration, it is marked as an invalid action.

[0096] In some optional embodiments, the reward feedback is determined comprehensively based on the amount of recovered load, the status of critical load recovery, constraint violation penalties, switching operation costs, and recovery time costs. This multi-objective comprehensive reward design enables the agent to gradually learn strategies that balance recovery efficiency and safety during training.

[0097] It should be noted that the policy network and value network described in the above embodiments are both neural network models, and their network structure and parameters are continuously updated during training. The method proposed in this application solves the cold start problem of deep reinforcement learning through imitation learning pre-training, balances invalid action constraints and policy exploration through a soft mask dynamic penalty mechanism, and enhances the adaptability to subsequent random faults through real-time state feedback decision-making, thus forming a complete power grid reconfiguration solution.

[0098] Compared with related technologies, this application has the following advantages: First, this application pre-trains the policy network through imitation learning, generates recovery trajectories in typical fault scenarios using an optimization solver, and enables the policy network to learn expert recovery sequences and action selection patterns through behavior cloning. This effectively solves the cold start problem caused by deep reinforcement learning agents exploring from scratch in related technologies. By transferring the prior knowledge of the optimization solver to the policy network, the agent possesses preliminary feasible decision-making capabilities before online reinforcement learning begins. This avoids the problems of reward sparsity and low training sample utilization caused by randomly initialized agents frequently selecting invalid actions in a high-dimensional mixed action space, significantly improving training convergence speed and initial policy feasibility.

[0099] Second, this application employs a soft-masking mechanism to impose a dynamic penalty related to policy confidence on invalid recovery actions, effectively addressing the problems of low exploration efficiency in maskless methods and the obstruction of constraint boundary feedback in related technologies. The soft mask generates action validity information based on the current power grid topology connectivity, equipment availability, and operational constraints, penalizing the original decision scores of invalid actions, reducing the selection probability of invalid actions to less than that of valid actions but greater than zero. This mechanism reduces the exploration of invalid actions, concentrating exploration near the feasible recovery region, while retaining limited opportunities for constraint boundary exploration, enabling the agent to obtain feedback information from the constraint boundaries. It balances constraint guidance and policy exploration, improving training sample utilization and policy generalization ability.

[0100] Third, this application effectively solves the exploration-exploitation imbalance problem caused by hard masks completely blocking invalid actions in the early and late stages of training by adopting a mechanism where the intensity of the dynamic penalty adapts to the current confidence of the policy network. In the early stages of training, the distribution of action scores is relatively flat and the penalty is relatively weak, which helps preserve the exploration space. In the later stages of training, the advantage of effective actions becomes more obvious, and the penalty increases accordingly, which helps suppress invalid actions and improve the utilization rate of available actions. This mechanism can dynamically balance exploration and exploitation without relying on fixed training rounds, improving training stability and the quality of the final policy.

[0101] Fourth, this application effectively solves the problem in related technologies where static recovery schemes generated by mixed-integer programming and heuristic algorithms cannot be dynamically adjusted when encountering subsequent random faults during the recovery process. This is achieved through a step-by-step decision-making mechanism that regenerates action validity information and soft-mask corrections based on real-time state at each decision step. The agent re-acquires the current power grid state at each time step and generates action validity markers and soft-mask action probabilities accordingly. This allows subsequent recovery actions to be adjusted as the real-time state changes, enhancing the strategy's generalization ability and robustness against different black-start power supply configurations, different line fault combinations, and subsequent faults during the recovery process.

[0102] Fifth, this application specifically designs the hybrid action space for grid reconfiguration, dividing recovery actions into multi-classification actions for selecting the next target bus to be recovered and multiple binary classification actions for determining whether each load has been recovered. This combined action structure is maintained in pre-training loss and online reinforcement learning updates. This action modeling method can simultaneously express recovery path selection and load deployment decisions, adapting to the actual operational characteristics of grid reconfiguration tasks, enabling the policy network to simultaneously learn the recovery path selection rules and load deployment rules of experts.

[0103] In summary, this application provides a power grid reconfiguration method based on imitation learning pre-training and soft mask-guided deep reinforcement learning. Through the collaborative mechanism of imitation learning pre-training, soft mask dynamic constraints, and real-time state feedback decision-making, it effectively improves the speed, security, and robustness of grid reconfiguration under random and complex fault scenarios, and has significant engineering application value.

[0104] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a power grid reconfiguration method.

[0105] Figure 3 A schematic diagram of a power grid reconfiguration method provided in an embodiment of this application is shown.

[0106] refer to Figure 3 As shown, the power grid reconfiguration method includes: The information acquisition module 202 is configured to acquire the status information of the target power grid when it is in a fault state.

[0107] The decision scoring module 204 is configured to: input the state information into a pre-trained policy network to obtain the original decision scores for each candidate recovery action; wherein the candidate recovery action is used to restore the target power grid to normal operation.

[0108] In some optional embodiments, the decision scoring module 204 further performs the following steps: Based on preset optimization objectives and power grid operation constraints, multiple recovery trajectories are generated when the target power grid is in a preset fault state; wherein, the recovery trajectory includes multiple power grid states of the target power grid at different times and the recovery action corresponding to each power grid state; A trajectory dataset is constructed based on multiple power grid states and the recovery action corresponding to each power grid state. The recovery actions include bus recovery actions to indicate the next bus to be recovered in the target power grid, and load recovery actions to indicate whether each load in the target power grid has been recovered. For any recovery trajectory in the trajectory dataset, the power grid state in the recovery trajectory is input into the initial strategy network, and the bus recovery action prediction result and the load recovery action prediction result are output. The first loss is determined based on the difference between the predicted bus recovery action and the corresponding bus recovery action in the recovery trajectory; The second loss is determined based on the difference between the predicted load recovery action and the corresponding load recovery action in the recovery trajectory; The parameters of the initial policy network are adjusted based on the first loss, the second loss, and the preset weight coefficients to obtain the pre-trained policy network.

[0109] The validity determination module 206 is configured to determine the validity information of each candidate recovery action based on the status information and preset power grid operation constraints.

[0110] In some optional embodiments, the validity determination module 206 further performs the following steps: Based on the status information, determine the current topological connectivity of the target power grid and the availability status of each device in the target power grid; Based on the topological connectivity, the availability status, and the power grid operation constraints, determine whether each candidate recovery action meets the preset execution conditions. Candidate recovery actions that meet the execution conditions are marked as valid recovery actions, and candidate recovery actions that do not meet the execution conditions are marked as invalid recovery actions, including: A dynamic penalty is applied to the original decision score corresponding to the invalid recovery action to obtain a revised decision score; Based on the original decision score corresponding to the effective recovery action and the modified decision score corresponding to the invalid recovery action, calculate the selection probability of each candidate recovery action. The probability of being selected for the invalid recovery action is not zero, but is less than the probability of being selected for the valid recovery action.

[0111] The decision correction module 208 is configured to: correct the original decision score based on the validity information to obtain a corrected decision score.

[0112] In some optional embodiments, the decision correction module 208 further performs the following steps: Determine the maximum value among the original decision scores for each of the candidate recovery actions; The penalty value for the invalid recovery action is determined based on the maximum value and a preset penalty coefficient; wherein the penalty value is positively correlated with the maximum value; The penalty value is superimposed on the original decision score corresponding to the invalid recovery action to obtain the corrected decision score of the invalid recovery action.

[0113] The power grid reconfiguration module 210 is configured to: determine and execute a target recovery action from each of the candidate recovery actions based on the modified decision score, so as to reconfigure the target power grid.

[0114] In some optional embodiments, the power grid reconfiguration module 210 further performs the following steps: Obtain the latest status information of the target power grid after the target recovery action is executed, and the reward feedback corresponding to the target recovery action; The state information before the target recovery action, the target recovery action, the reward feedback, and the latest state information are input into the pre-generated value network to calculate the temporal difference error. The dominance function is calculated based on the time-series difference error. The parameters of the policy network are updated according to the dominance function to obtain the updated policy network; The parameters of the value network are updated based on the time-series difference error to obtain the updated value network; The target power grid is reconstructed based on the updated policy network and the updated value network.

[0115] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0116] The apparatus of the above embodiments can be used to implement the corresponding power grid reconfiguration method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0117] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the power grid reconfiguration method described in any of the above embodiments.

[0118] Figure 4 This embodiment illustrates a more specific hardware structure of a computer device, which may include a processor 810, a memory 820, an input / output interface 830, a communication interface 840, and a bus 850. The processor 810, memory 820, input / output interface 830, and communication interface 840 are interconnected internally via the bus 850.

[0119] The processor 810 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the power grid reconfiguration method provided in any of the foregoing embodiments of this specification.

[0120] The memory 820 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 820 can store operating devices and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 820 and is called and executed by the processor 810.

[0121] The input / output interface 830 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0122] The communication interface 840 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0123] Bus 850 includes a pathway for transmitting information between various components of the device, such as processor 810, memory 820, input / output interface 830, and communication interface 840.

[0124] It should be noted that although the above-described device only shows the processor 810, memory 820, input / output interface 830, communication interface 840, and bus 850, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0125] The electronic devices described above are used to implement the corresponding power grid reconfiguration methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0126] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the power grid reconfiguration method as described in any of the above embodiments.

[0127] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0128] The aforementioned non-transitory computer-readable storage media can be any available medium or data storage device that a computer can access, including but not limited to magnetic storage (e.g., floppy disks, hard disks, magnetic tapes, magneto-optical disks (MOs), etc.), optical storage (e.g., CDs, DVDs, BDs, HVDs, etc.), and semiconductor storage (e.g., ROMs, EPROMs, EEPROMs, non-volatile memory (NAND flash), solid-state drives (SSDs)).

[0129] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the power grid reconfiguration method as described in any of the embodiments in the exemplary method section above, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0130] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0131] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0132] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0133] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A power grid reconfiguration method, characterized in that, include: Obtain the status information of the target power grid when it is in a fault state; The state information is input into a pre-trained policy network to obtain the original decision scores for each candidate recovery action; wherein, the candidate recovery action is used to restore the target power grid to normal operation, and the policy network is pre-trained using the recovery trajectory through imitation learning in the offline stage; Based on the status information and preset power grid operation constraints, the validity information of each candidate recovery action is determined; Based on the validity information, the original decision score is corrected to obtain a corrected decision score; Based on the modified decision score, a target recovery action is determined from each of the candidate recovery actions and executed to reconstruct the target power grid.

2. The power grid reconfiguration method according to claim 1, characterized in that, Before inputting the state information into the pre-trained policy network, the method further includes: Based on preset optimization objectives and power grid operation constraints, multiple recovery trajectories are generated when the target power grid is in a preset fault state; wherein, the recovery trajectory includes multiple power grid states of the target power grid at different times and the recovery action corresponding to each power grid state; A trajectory dataset is constructed based on multiple power grid states and the recovery action corresponding to each power grid state. The pre-trained policy network is obtained by training the initial policy network using behavior cloning based on the trajectory dataset.

3. The power grid reconfiguration method according to claim 2, characterized in that, The recovery actions include bus recovery actions for instructing the next bus to be recovered in the target power grid, and load recovery actions for instructing whether each load in the target power grid has been recovered; The step of training the initial policy network based on the trajectory dataset using behavior cloning to obtain the pre-trained policy network includes: For any recovery trajectory in the trajectory dataset, the power grid state in the recovery trajectory is input into the initial strategy network, and the bus recovery action prediction result and the load recovery action prediction result are output. The first loss is determined based on the difference between the predicted bus recovery action and the corresponding bus recovery action in the recovery trajectory; The second loss is determined based on the difference between the predicted load recovery action and the corresponding load recovery action in the recovery trajectory; The parameters of the initial policy network are adjusted based on the first loss, the second loss, and the preset weight coefficients to obtain the pre-trained policy network.

4. The power grid reconfiguration method according to claim 1, characterized in that, The determination of the validity information of each candidate recovery action based on the status information and preset power grid operation constraints includes: Based on the status information, determine the current topological connectivity of the target power grid and the availability status of each device in the target power grid; Based on the topological connectivity, the availability status, and the power grid operation constraints, determine whether each candidate recovery action meets the preset execution conditions. Candidate recovery actions that meet the execution conditions are marked as valid recovery actions, and candidate recovery actions that do not meet the execution conditions are marked as invalid recovery actions.

5. The power grid reconfiguration method according to claim 4, characterized in that, The step of correcting the original decision score based on the validity information to obtain a corrected decision score includes: A dynamic penalty is applied to the original decision score corresponding to the invalid recovery action to obtain a revised decision score; Based on the original decision score corresponding to the effective recovery action and the modified decision score corresponding to the invalid recovery action, calculate the selection probability of each candidate recovery action. The probability of being selected for the invalid recovery action is not zero, but is less than the probability of being selected for the valid recovery action.

6. The power grid reconfiguration method according to claim 5, characterized in that, The step of applying a dynamic penalty to the original decision score corresponding to the invalid recovery action to obtain a revised decision score includes: Determine the maximum value among the original decision scores for each of the candidate recovery actions; The penalty value for the invalid recovery action is determined based on the maximum value and a preset penalty coefficient; wherein the penalty value is positively correlated with the maximum value; The penalty value is superimposed on the original decision score corresponding to the invalid recovery action to obtain the corrected decision score of the invalid recovery action.

7. The power grid reconfiguration method according to claim 1, characterized in that, After determining and executing the target recovery action, the method further includes: Obtain the latest status information of the target power grid after the target recovery action is executed, and the reward feedback corresponding to the target recovery action; The state information before the target recovery action, the target recovery action, the reward feedback, and the latest state information are input into the pre-generated value network to calculate the temporal difference error. The dominance function is calculated based on the time-series difference error. The parameters of the policy network are updated according to the dominance function to obtain the updated policy network; The parameters of the value network are updated based on the time-series difference error to obtain the updated value network; The target power grid is reconstructed based on the updated policy network and the updated value network.

8. A power grid reconfiguration system, characterized in that, include: The information acquisition module is configured to acquire status information of the target power grid when it is in a fault state. The decision scoring module is configured to: input the state information into a pre-trained policy network to obtain the original decision scores for each candidate recovery action; wherein, the candidate recovery action is used to restore the target power grid to normal operation, and the policy network is pre-trained using the recovery trajectory through imitation learning in the offline stage; The validity determination module is configured to: determine the validity information of each candidate recovery action based on the status information and preset power grid operation constraints; The decision correction module is configured to: correct the original decision score based on the validity information to obtain a corrected decision score; The power grid reconfiguration module is configured to: determine and execute a target recovery action from each of the candidate recovery actions based on the modified decision score, so as to reconfigure the target power grid.

9. A computer device, characterized in that, It includes one or more processors, memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the one or more programs include instructions for performing the method of any one of claims 1 to 7.

10. A non-volatile computer-readable storage medium containing a computer program, characterized in that, When the computer program is executed by one or more processors, the one or more processors perform the method according to any one of claims 1 to 7.

Images