A method and system for generating power grid fault plans based on deep reinforcement learning

Abstract

The present invention discloses a method and system for generating a power grid fault plan based on deep reinforcement learning, relating to the technical field of power system main grid operation and dispatching, and is used to address the technical problem that traditional power grid fault plan generation methods result in insufficient adaptability of power grids in complex operating environments. The method includes constructing a power system fault plan model when a target power grid fault recovery plan needs to be generated for a faulty power grid; converting the power system fault plan model to generate a Markov decision process model; using a power network simulation tool to simulate a pre-operation of the faulty power grid to generate a baseline state of the faulty power grid; and using a dual deep reinforcement learning Q network to solve the Markov decision process model based on the baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid.

Description

Technical Field

[0001] The present invention relates to the technical field of power system main grid operation and dispatching, and in particular to a method and system for generating a power grid fault plan based on deep reinforcement learning. Background Art

[0002] With the continuous expansion of the power grid and the widespread application of power electronic equipment, the complexity of system operation has increased significantly, and various potential failure risks have also increased accordingly. Therefore, in modern power systems, the generation of fault plans is an important part of ensuring the safe and stable operation of the power grid.

[0003] From traditional issues like equipment overloads and line short circuits to newer issues like voltage fluctuations and harmonic interference brought on by the integration of renewable energy, these issues can all trigger cascading failures and even widespread power outages. Therefore, a scientific and sound fault response plan must not only consider basic operations like fault isolation and load transfer, but also balance multiple objectives such as cost-effectiveness, reliability, and rapid response.

[0004] Traditional methods for generating grid fault emergency plans often rely on expert experience or rule-based optimization algorithms. These methods simplify complex grid models through convex relaxation or linearization techniques, and then use optimization solvers to calculate emergency plans. However, with the large-scale integration of renewable energy and the widespread use of flexible resources (such as energy storage and demand response), grid operations are becoming highly uncertain and dynamic. These changes have resulted in traditional methods, which rely on precise grid parameter and device models, becoming less adaptable to complex operating environments. Summary of the Invention

[0005] The present invention provides a method and system for generating a power grid fault plan based on deep reinforcement learning, which is used to solve the technical problem that traditional power grid fault plan generation methods cause the power grid to show insufficient adaptability in complex operating environments.

[0006] A first aspect of the present invention provides a method for generating a power grid fault plan based on deep reinforcement learning, comprising:

[0007] Respond to generation requests and build a power system fault plan model;

[0008] Converting the power system fault plan model to generate a Markov decision process model;

[0009] Using a power network simulation tool to simulate a pre-operation of the faulty power grid to generate a baseline state of the faulty power grid;

[0010] A dual deep reinforcement learning Q network is used to solve the Markov decision process model according to the baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid.

[0011] Optionally, the adopting of a dual deep reinforcement learning Q network to solve the Markov decision process model according to a reference state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid includes:

[0012] Determining whether the baseline state meets the preset risk condition;

[0013] If satisfied, the target policy network in the dual deep reinforcement learning Q network is used to generate multiple actions according to the baseline state;

[0014] Evaluate each of the actions using the target evaluation network in the dual deep reinforcement learning Q network, and output an evaluation value corresponding to each of the actions;

[0015] Sort the evaluation values ​​corresponding to the actions in descending order, and take the action corresponding to the evaluation value at the first preset digit as the intermediate action;

[0016] Calculating a reward value corresponding to each intermediate action based on the plurality of intermediate actions using a reward function in the Markov decision process model;

[0017] Select the intermediate action corresponding to the maximum reward value as the target action for the current time step;

[0018] Based on the target action of the current time step, the current operating state of the fault power grid is generated, and the current time step is counted in real time;

[0019] Determining whether the current operating state satisfies the preset risk condition and whether the current time step reaches the preset time step;

[0020] If the current operating state does not meet the preset risk condition or the current time step reaches the preset time step, a target power grid fault recovery plan for the faulty power grid is generated according to the target action of the current time step and the target actions of multiple historical time steps.

[0021] Optionally, it also includes:

[0022] If the current running state satisfies the preset risk condition and the current time step does not reach the preset time step, taking the target action of the current time step as the target action of the historical time step;

[0023] Taking the current operating state of the faulty power grid as a new reference state, and jumping to the step of using the target policy network in the dual deep reinforcement learning Q network to generate multiple actions according to the reference state until the current operating state fails to meet the preset risk condition or the current time step reaches the preset time step;

[0024] A target power grid fault recovery plan for the faulty power grid is generated according to a target action for the current time step and target actions for multiple historical time steps determined when the current operating state does not satisfy a preset risk condition or the current time step reaches the preset time step.

[0025] Optionally, the training process of the target evaluation network in the dual deep reinforcement learning Q network is specifically as follows:

[0026] Based on the power network simulation tool, generating a training data set;

[0027] Using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in the multiple state-action pairs in the training data set, and generating a training evaluation value corresponding to the action in each state-action pair;

[0028] Sort the training evaluation values ​​corresponding to the actions in each state-action pair in descending order, assign a preset first weight to the actions in the state-action pair corresponding to the training evaluation values ​​of the first preset second number of digits, and assign a preset second weight to the actions in the state-action pair corresponding to the remaining training evaluation values;

[0029] Substituting the training evaluation value corresponding to the action in each state-action pair, the preset first weight, and the preset second weight into a preset loss function and performing derivative, and outputting a model gradient;

[0030] Using the model gradient to update the model parameters of the initial evaluation network, determine the intermediate evaluation network, and count the number of model updates in real time;

[0031] Determining whether the model update count reaches a preset update threshold;

[0032] If achieved, the intermediate evaluation network is used as the trained target evaluation network.

[0033] Optionally, it also includes:

[0034] If the number of model updates does not reach the preset update threshold, the intermediate evaluation network is used as a new initial evaluation network, and the step of using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in the multiple state-action pairs in the training data set to generate training evaluation values ​​corresponding to the actions in each state-action pair is executed, until the number of model updates reaches the preset update threshold;

[0035] The intermediate evaluation network determined when the number of model updates reaches the preset update threshold is used as the trained target evaluation network.

[0036] Optionally, the objective function corresponding to the power system fault plan model is specifically:

[0037] ;

[0038] ;

[0039] ;

[0040] in, - are the weights of various evaluation indicators (including node load loss, generator output adjustment, and the status of the backup line at time t), represents the node load loss; Indicates generator output adjustment; is a 0-1 variable, indicating the status of the backup line at time t, where =1 means the status of the backup line at time t is in operation, =0 means the status of the backup line at time t is disconnected, is the status of the backup line at time t-1; M is the number of backup equipment (including lines and generators) in the system; 、 are the collections of load nodes and generator nodes respectively; and are the active load of load node i before and after the fault plan is activated; and They are the output of the generator node before and after the fault plan is activated.

[0041] A second aspect of the present invention provides a power grid fault plan generation system based on deep reinforcement learning, comprising:

[0042] The response module is used to respond to the generation request and build a power system fault plan model;

[0043] A conversion module, used to convert the power system fault plan model to generate a Markov decision process model;

[0044] A simulation module, configured to simulate a pre-operation of a faulty power grid using a power network simulation tool to generate a reference state of the faulty power grid;

[0045] A solution module is used to solve the Markov decision process model according to the reference state of the faulty power grid using a dual deep reinforcement learning Q network to generate a target power grid fault recovery plan for the faulty power grid.

[0046] A third aspect of the present invention provides a computer device comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for generating a power grid fault plan based on deep reinforcement learning as described in any one of the above items.

[0047] A fourth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed, implements the steps of the method for generating a power grid fault plan based on deep reinforcement learning as described in any one of the above items.

[0048] A fifth aspect of the present invention provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to perform the steps of the power grid fault plan generation method based on deep reinforcement learning as described in any one of the above items.

[0049] It can be seen from the above technical solutions that the present invention has the following advantages:

[0050] The above-mentioned scheme of the present invention provides a method for generating a power grid fault plan based on deep reinforcement learning. When it is necessary to generate a target power grid fault recovery plan for a faulty power grid, a power system fault plan model is constructed; the power system fault plan model is converted to generate a Markov decision process model; a power network simulation tool is used to simulate the pre-operation of the faulty power grid to generate a reference state of the faulty power grid; a dual deep reinforcement learning Q network is used to solve the Markov decision process model according to the reference state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid; based on the above-mentioned scheme, the present invention utilizes a deep reinforcement learning optimization strategy to realize the rapid generation of power grid fault recovery plans, thereby improving the adaptability of the power grid to complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0052] Figure 1 A flowchart of a method for generating a power grid fault plan based on deep reinforcement learning provided in Example 1 of the present invention;

[0053] Figure 2 A flowchart illustrating the implementation of the early warning mechanism provided in the first embodiment of the present invention;

[0054] Figure 3 A flowchart of the steps of the training process of the target evaluation network in the dual deep reinforcement learning Q network provided in the second embodiment of the present invention;

[0055] Figure 4 A schematic diagram of the process of training the target evaluation network in the dual deep reinforcement learning Q network provided in the second embodiment of the present invention;

[0056] Figure 5 A schematic diagram of a flow chart of a method for generating a power grid fault plan based on deep reinforcement learning provided in the second embodiment of the present invention;

[0057] Figure 6 This is a structural block diagram of a power grid fault plan generation system based on deep reinforcement learning provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0058] The embodiments of the present invention provide a method and system for generating a power grid fault plan based on deep reinforcement learning, which are used to solve the technical problem that traditional power grid fault plan generation methods cause the power grid to show insufficient adaptability in complex operating environments.

[0059] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0060] See also Figure 1 , Figure 1 A flowchart of the steps of a method for generating a power grid fault plan based on deep reinforcement learning is provided in Example 1 of the present invention.

[0061] The present invention provides a method for generating a power grid fault plan based on deep reinforcement learning, comprising:

[0062] Step 101: In response to a generation request, a power system fault contingency plan model is constructed.

[0063] It should be noted that in response to the generation request, that is, when the target grid fault recovery plan of the fault grid needs to be generated, the power system fault plan model is constructed. Among them, the objective function corresponding to the power system fault plan model comprehensively considers the start-up action of the backup line and generator after the system fault is restored and the load loss , generator output adjustment , as shown below:

[0064] ;

[0065] ;

[0066] ;

[0067] in, - are the weights of various evaluation indicators (including node load loss, generator output adjustment, and the status of the backup line at time t), represents the node load loss; Indicates generator output adjustment; is a 0-1 variable, indicating the status of the backup line or generator m at time t, where =1 means the status of the standby line or generator m at time t is in operation, =0 means the status of the standby line or generator m at time t is disconnected; is the status of the backup line at time t-1; M is the number of backup equipment (including lines and generators) in the system; 、 are the collections of load nodes and generator nodes respectively; and are the active load of load node i before and after the fault plan is activated; and are the output of generator node i before and after the fault plan action (i.e., the target action of the current time step).

[0068] Furthermore, for the constraints corresponding to the power system fault plan model, considering the power flow constraint and the switch operation number constraint, the following are shown:

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] ;

[0078] in, is a 0-1 variable used to indicate the circuit breaker status ( =1 means that line ij is closed at time t, =0 means line ij is disconnected at time t); is the square of the voltage at node i at time t; 、 are the active and reactive powers flowing through line ij at time t respectively; 、 are the active power and reactive power injected into the system at node j at time t respectively; 、 are the sets of parent nodes and child nodes of node j in the system respectively; M1, M2, M3, and M4 are all sufficiently large positive numbers; and are the operating conditions of the backup line or generator m at time t before and after the fault, both are 0-1 variables; is the maximum number of actions; is the square of the voltage at node j at time t; is the element in the i-th row and j-th column of the real part of the grid node admittance matrix; is the element in the i-th row and j-th column of the imaginary part of the grid node admittance matrix; is the phase angle difference between node i and node j at time t; 、 are the active and reactive powers flowing through line jk at time t respectively; 、 are the active power and reactive power injected into node i respectively; 、 are the active load and reactive load of load node i after the fault plan is activated; 、 They are the output size of generator node i after the fault contingency plan is activated and the reactive power of generator node i after the fault contingency plan is activated.

[0079] Step 102: Convert the power system fault plan model to generate a Markov decision process model.

[0080] It's important to note that the evolution of power systems during fault recovery adheres to the Markov property, meaning that the next change in network topology and the activation of backup power sources are solely dependent on the current state and are not constrained by historical paths. Reinforcement learning methods, leveraging this property, aim to improve overall profitability by generating and optimizing continuous action sequences. Given that fault recovery requires minimizing total cycle loss within a limited number of actions, Deep Reinforcement Learning (DRL), with its ability to adapt to complex dynamic environments, is an ideal tool for generating fault response plans.

[0081] Furthermore, the present invention converts the power system fault plan model into a generative Markov decision process model, whose corresponding state space defines all possible states of the environment in reinforcement learning, provides the agent with the environmental information required for decision-making, and is the input part of the neural network. Set to:

[0082] ;

[0083] in, and are the active power and reactive power transmitted by line ij, is the voltage at node i, and are the start and stop states of the line and generator respectively, Numbers the system fault condition.

[0084] Furthermore, for the action space corresponding to the Markov decision process model, the present invention combines the start and stop of the backup line and the generator and , and the generator output and , together constitute the set G as the action space.

[0085] Furthermore, for the reward function corresponding to the Markov decision process model, the reward function is used to guide the agent to learn how to optimize its behavior in a specific task to maximize long-term benefits. The instant reward of the present invention comprehensively considers the load loss , generator output adjustment Since the reinforcement learning algorithm converges to the direction of maximum reward value, the reward function is set to the opposite form:

[0086] ;

[0087] in, is the reward value at the current time step t (t moment); is a function related to the number of actions, expressed as follows:

[0088] ;

[0089] in, is the action status of the standby line or generator m at time step t. It is a 0-1 variable, indicating the operational status of the standby line or generator m at time t. It is 1 when the line or generator is operational, and 0 otherwise.

[0090] Furthermore, in addition to the reward information related to the grid status, the influence of the constraints must still be considered. If the constraints are violated, the agent will be penalized:

[0091] ;

[0092] in, and are the reward and punishment coefficients respectively; Represents the penalty for violating the constraint at the current time step t, which is a negative value; is the penalty at the current time step t.

[0093] Step 103: Use a power network simulation tool to simulate the pre-operation of the faulty power grid to generate a reference state of the faulty power grid.

[0094] It should be noted that power network simulation tools (such as Pandapower) can be used to simulate the consequences of not taking any action on the grid, generating a baseline state under the current state (i.e., the baseline state of the faulty grid). This allows assessment of potential grid risks if no intervention is taken, providing a basis for subsequent decision-making.

[0095] Step 104: Use a dual deep reinforcement learning Q network to solve the Markov decision process model according to the baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid.

[0096] It should be noted that for the agent of the Dueling Deep Q-Network (DuelingDQN) algorithm, its Q-value function can be expressed as:

[0097] ;

[0098] in, is the Q-value function in the reinforcement learning algorithm; is the state value function; is the advantage function; For averaging operation; For status; For action; are the parameters of the Q network; are the parameters of the state value function network; is the parameter of the advantage function network; Dueling DQN splits the Q function into a state value function and the state-action advantage function , achieving independent learning of state value and action advantage, thereby improving the effectiveness of the training process. In the fault plan generation model designed by the present invention, Considering only the state of the power system, It reflects the actions of the intelligent agent.

[0099] Furthermore, the present invention introduces a dual deep Q-network (DDQN) structure, which effectively alleviates the problem of Q-value over-estimation by periodically fixing the target Q-value (evaluation value), thereby stabilizing the update process of the intelligent agent.

[0100] Specifically, step 104 may include the following sub-steps S41-:

[0101] Step S41: Determine whether the baseline state meets the preset risk condition;

[0102] Step S42: If satisfied, use the target policy network in the dual deep reinforcement learning Q network to generate multiple actions based on the baseline state;

[0103] Step S43: Evaluate each action using the target evaluation network in the dual deep reinforcement learning Q network, and output the evaluation value corresponding to each action;

[0104] Step S44: sort the evaluation values ​​corresponding to the actions in descending order, and take the action corresponding to the evaluation value at the first preset digit as the intermediate action;

[0105] Step S45: Calculate the reward value corresponding to each intermediate action based on the multiple intermediate actions using the reward function in the Markov decision process model;

[0106] Step S46: Select the intermediate action corresponding to the maximum reward value as the target action for the current time step;

[0107] Step S47: Based on the target action of the current time step, generate the current operating state of the faulty power grid and count the current time step in real time;

[0108] Step S48: determine whether the current operating state meets the preset risk condition and whether the current time step reaches the preset time step;

[0109] Step S49: If the current operating state does not meet the preset risk condition or the current time step reaches the preset time step, a target grid fault recovery plan for the faulty grid is generated based on the target action of the current time step and the target actions of multiple historical time steps.

[0110] It's important to note that this invention incorporates an Early Warning (EW) mechanism to monitor grid status and trigger prudent decisions at critical moments, thereby improving system robustness and fault tolerance. This mechanism addresses the power system's high sensitivity to topology changes and fault recovery, ensuring that DRL agents can select safe and effective actions in complex operational scenarios.

[0111] See also Figure 2 At each time step, the early warning mechanism first simulates the result of not taking any action on the power grid through a power network simulation tool (such as Pandapower) as the baseline state. If at this time, any transmission line or transformer is heavily / overloaded, or there is a load loss, indicating that the system is in a potential risk or fault state, the DRL agent will select the N with the highest Q value (evaluation value) from the action space. g The agent performs actions and further simulates the impact of these actions on the grid state through simulation tools. The simulation evaluation indicators include load loss and generator output adjustment in the objective function. The action with the highest reward in the simulation results is selected for execution, and the experience is stored in the experience replay buffer for subsequent agent updates. If there is no heavy / overload or load loss or the time step reaches the threshold, the agent chooses not to take any action and this round ends.

[0112] Specifically, the present invention obtains grid status data, i.e., the baseline status of the faulty grid, including line load, node voltage, etc., through a power system simulation tool (such as Pandapower). Determine whether the baseline status meets the preset risk conditions, that is, check whether any of the following conditions exist in the baseline status: ① The transmission line or transformer is heavily loaded or overloaded (for example, the load exceeds the capacity threshold); ② There is a loss of load (the load cannot supply power normally). Determine whether the system is in a potential risk or fault state as a basis for triggering subsequent actions. If the above situation exists, it indicates that the system is at risk, that is, the baseline status meets the preset risk conditions, and proceeds to the next step. If there is no heavy / overload and loss of load, that is, the baseline status does not meet the preset risk conditions, it indicates that the system (grid) is in a safe state, and the DRL agent chooses not to take any action to avoid unnecessary intervention and maintain system stability.

[0113] Furthermore, the target policy network in the dual deep reinforcement learning Q network is used to generate multiple actions according to the baseline state, and each action is evaluated by the target evaluation network in the dual deep reinforcement learning Q network, and the evaluation value (Q value) corresponding to each action is output, and the preset first number N with the highest Q value is selected. g Actions (such as N g =5), as an intermediate action.

[0114] Furthermore, the reward value corresponding to each intermediate action is calculated, and the intermediate action with the highest reward value is selected and executed in the actual power grid to obtain the current operating state of the faulty power grid. The experience (state, action, reward, next state) is stored in the experience replay buffer.

[0115] Furthermore, the system determines whether the current operating state meets the preset risk conditions and whether the current time step has reached the preset time step. Specifically, the system checks whether any of the following conditions exist in the current operating state: ① The transmission line or transformer is overloaded or overloaded (for example, the load exceeds the capacity threshold); ② There is a load loss (the load cannot supply power normally). If any of these conditions exist, the system is at risk, meaning that the current operating state meets the preset risk conditions. If none of these conditions exist, the system is not at risk, meaning that the current operating state does not meet the preset risk conditions. When the current operating state does not meet the preset risk conditions or the current time step reaches the preset time step, a target grid fault recovery plan is generated for the faulty grid based on the target actions for the current time step and all the target actions for the historical time steps. The target actions in the target grid fault recovery plan are then executed sequentially according to the time steps in the actual faulty grid.

[0116] Optionally, it also includes:

[0117] If the current running state meets the preset risk condition and the current time step does not reach the preset time step, the target action of the current time step is used as the target action of the historical time step;

[0118] The current operating state of the faulty power grid is used as the new baseline state, and the target policy network in the dual deep reinforcement learning Q network is used to jump to the execution step to generate multiple actions according to the baseline state until the current operating state does not meet the preset risk condition or the current time step reaches the preset time step;

[0119] A target grid fault recovery plan for the faulty grid is generated according to a target action of the current time step and target actions of multiple historical time steps determined when the current operating state does not meet a preset risk condition or the current time step reaches a preset time step.

[0120] It should be noted that if the current operating state meets the preset risk condition and the current time step does not reach the preset time step, the target action of the current time step is used as the target action of the historical time step, the current operating state of the faulty power grid is used as the new reference state, and the process jumps to step S42 until the current operating state does not meet the preset risk condition or the current time step reaches the preset time step. According to the target action of the current time step determined when the current operating state does not meet the preset risk condition or the current time step reaches the preset time step and the target actions of multiple historical time steps, a target power grid fault recovery plan for the faulty power grid is generated.

[0121] In this embodiment, the present invention ensures the safe and efficient decision-making of the DRL agent in the power system through real-time status monitoring and multi-action simulation based on the early warning mechanism.

[0122] In an embodiment of the present invention, a method for generating a power grid fault plan based on deep reinforcement learning is provided. When a target power grid fault recovery plan needs to be generated for a faulty power grid, a power system fault plan model is constructed; the power system fault plan model is converted to generate a Markov decision process model; a power network simulation tool is used to simulate a pre-operation of the faulty power grid to generate a baseline state of the faulty power grid; a dual deep reinforcement learning Q network is used to solve the Markov decision process model based on the baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid; based on the above scheme, the present invention utilizes a deep reinforcement learning optimization strategy to achieve rapid generation of power grid fault recovery plans, thereby improving the power grid's adaptability to complex environments. At the same time, based on the real-time state of the power grid (including fault location, load conditions, and equipment parameters), an interactive framework between the DRL agent and the power grid simulation environment is constructed, and a reward mechanism is used to drive the continuous optimization of the plan generation strategy.

[0123] For better explanation, refer to Figure 3 , shows a flowchart of the steps of the training process of the target evaluation network in the dual deep reinforcement learning Q network provided by the second embodiment of the present invention. The process may include the following steps:

[0124] Step 301: Generate a training data set based on a power network simulation tool.

[0125] It should be noted that the training process of the target evaluation network is as follows Figure 4 As shown, the method of the present invention integrates the following three key technologies: 1) Experience replay buffer: By configuring the experience replay buffer, the intelligent agent can achieve off-policy training. The buffer stores historical experience data and weakens the strong correlation between continuous training data through random sampling, thereby improving the stability and efficiency of the learning process. 2) Importance sampling: Importance sampling technology is used to improve the learning efficiency of the algorithm and the quality of the final strategy. This technology measures the importance of the data by calculating its absolute temporal difference (TD) error, and preferentially samples data with higher importance from the memory buffer during training. 3) Double Deep Q-Network (DDQN), that is, the dual deep reinforcement learning Q network structure: The present invention introduces a dual deep Q-network (DDQN) structure, which effectively alleviates the problem of Q-value over-estimation by periodically fixing the target Q value, thereby stabilizing the update process of the intelligent agent.

[0126] Furthermore, imitation learning is inherently a supervised learning method and is often used to pre-train DRL agents by providing a good initial policy (in the form of neural network weights). A large dataset was generated using a power grid simulator, which was then further processed and used to train the Dueling DQN agent. This process enables the DRL agent to acquire a good distribution over different input states.

[0127] Specifically, the present invention first generates the training dataset using a power network simulation tool (such as Pandapower or other similar tools). Based on the physical model and operating rules of the power system, the simulation tool outputs data containing multiple scenarios. Each scenario includes a state vector (such as node voltage and line load) and an action option (such as commissioning a backup line or starting a backup generator). After preprocessing, this data is converted into a training dataset. The training dataset includes multiple standardized state-action pairs, which are used for subsequent neural network training.

[0128] In this embodiment, the present invention generates a data set containing at least 10,000 state-action pairs from a simulation tool, and divides the data into a training set, a validation set, and a test set according to a certain ratio.

[0129] Step 302: Use the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in multiple state-action pairs in the training data set to generate training evaluation values ​​corresponding to the actions in each state-action pair.

[0130] It should be noted that for each state s, all actions a are calculated i The Q value (training evaluation value) forms a quantity, and arranged in descending order, where The agent (initial evaluation network) takes the |A|th action in state s Predicted Q value.

[0131] Step 303: sort the training evaluation values ​​corresponding to the actions in each state-action pair in descending order, and assign a preset first weight to the actions in the state-action pairs corresponding to the training evaluation values ​​of the first preset second digit, and assign a preset second weight to the actions in the state-action pairs corresponding to the remaining training evaluation values.

[0132] It should be noted that, before selecting, a second number N of high Q value actions are preset, and a preset first weight is given (e.g. 0.8), the remaining actions are given the preset second weight (e.g. 0.2).

[0133] In this embodiment, by properly setting and ,This invention ensures that the agent is more sensitive to high-reward actions during training, thereby accelerating ,policy optimization and improving decision quality.

[0134] Step 304: Substitute the training evaluation value, preset first weight, and preset second weight corresponding to the action in each state-action pair into the preset loss function and derive the derivative to output the model gradient.

[0135] It should be noted that the Adam optimizer (learning rate is set to 0.0001) is used to minimize the preset loss function by gradient descent, and training is performed for 100 epochs or until the validation set loss converges.

[0136] Furthermore, during training, this paper proposes a novel weighted mean-squared error (MSE) loss function (preset loss function) to optimize the agent's Q-value prediction ability. This loss function takes into account the importance differences of action rewards. By assigning higher weights to high-reward actions, the agent can quickly focus on high-quality strategies early in training. Specifically, the loss function is defined as follows:

[0137] ;

[0138] in, is the loss value; For agents in state Next action for i Predicted Q value; is the target Q value, usually provided by expert data or simulator; is the size of the action space; N is the number of high Q-value actions, which are selected based on the Q-value in descending order; and is the weight, satisfying , and .

[0139] Step 305: Use the model gradient to update the model parameters of the initial evaluation network, determine the intermediate evaluation network, and count the number of model updates in real time.

[0140] Step 306: Determine whether the number of model updates reaches a preset update threshold.

[0141] Step 307: If the target is reached, the intermediate evaluation network is used as the trained target evaluation network.

[0142] It should be noted that after pre-training, the deep reinforcement learning agent has a good initial strategy, which enables it to converge to the optimal solution more quickly in the subsequent reinforcement learning phase. Imitation learning provides a good initial strategy, and then DRL is used to train the agent to acquire long-term planning capabilities and a globally focused strategy. For this problem, traditional epsilon-greedy exploration methods are inefficient. First, the action space is very large and the Markov chain is very long. Second, the agent can easily get stuck in a local optimum. Therefore, this paper proposes a guided exploration method. At each time step of the agent, Ng actions with the highest Q values ​​are selected and their results are simulated and evaluated in real time. Then, the action with the highest reward is selected and implemented, and this experience is stored in an experience replay pool. Guided exploration helps the agent further extract good actions. With the help of the action simulation function, the training process is more stable, and better experience is stored and used to update the agent. Therefore, guided exploration significantly improves training efficiency.

[0143] Optionally, it also includes:

[0144] If the number of model updates does not reach the preset update threshold, the intermediate evaluation network is used as the new initial evaluation network, and the execution jumps to the step of using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in multiple state-action pairs in the training dataset, and generating training evaluation values ​​corresponding to the actions in each state-action pair, until the number of model updates reaches the preset update threshold;

[0145] The intermediate evaluation network determined when the number of model updates reaches the preset update threshold is used as the trained target evaluation network.

[0146] It should be noted that if the number of model updates does not reach the preset update threshold, the intermediate evaluation network will be used as the new initial evaluation network, and the process will jump to step 302 until the number of model updates reaches the preset update threshold; the intermediate evaluation network determined when the number of model updates reaches the preset update threshold will be used as the trained target evaluation network.

[0147] As a comparison of technical effects, we can refer to existing technologies. In modern power systems, the generation of fault plans is an important part of ensuring the safe and stable operation of the power grid. Traditional methods mainly rely on expert experience or rule-based optimization algorithms to simplify complex power grid models through convex relaxation or linearization techniques, and then use optimization solvers to calculate plans. However, with the large-scale access of renewable energy and the widespread application of flexible resources (such as energy storage and demand response), the operating status of the power grid has become highly uncertain and dynamic. These changes have caused traditional methods to rely on precise power grid parameters and equipment models, showing limitations such as insufficient adaptability and slow response speed in complex operating environments, making it difficult to meet the needs of efficient and rapid decision-making.

[0148] In recent years, deep reinforcement learning, as a data-driven optimization method, has shown potential in power system fault recovery. Through real-time interaction with the power grid environment, DRL can learn strategies that adapt to dynamic changes, especially when dealing with high-dimensional state spaces and complex decision-making scenarios. However, DRL faces the challenge of low training efficiency in practical applications. Specifically, DRL training usually starts with a random strategy, and the agent needs to explore effective strategies through a large amount of trial and error. Under conditions of a large action space and a complex power grid environment, the training time is significantly extended, and the quality of the initial strategy is low, which limits its application in scenarios with high real-time requirements.

[0149] To address the above issues, the present invention proposes a method for generating power grid fault plans based on deep reinforcement learning. This method uses historical expert plan data to initialize the strategy network through imitation learning, providing a high-quality starting point for DRL and significantly reducing random exploration time. At the same time, the plan is optimized through real-time interaction between DRL and the power grid environment, adapting to dynamically changing operating conditions, thereby overcoming the limitations of traditional methods and improving the efficiency and reliability of fault plan generation. Figure 5 Grid status data, including line loads and node voltages, is obtained from power system simulation tools (such as PandaPower). Training samples are generated from historical expert plan data through imitation learning to provide an initial policy for the intelligent agent. At each time step, Ng actions (e.g., Ng = 5) with the highest Q-values ​​are selected for simulation evaluation to optimize action selection. Importance sampling prioritizes high-value experiences from the experience replay buffer to accelerate training convergence. The next state and reward are obtained through interaction with the grid environment, and the agent's policy network is updated. The early warning mechanism is an adaptive technology used in this invention to generate power system fault plans, designed to enhance the agent's decision-making robustness and fault tolerance. Its core function is to trigger cautious decisions when potential risks arise by monitoring grid status in real time. This invention uses imitation learning to generate a good initial policy for the D3QN intelligent agent, significantly reducing exploration and training time. Furthermore, a guided exploration approach is used to train the agent, rather than traditional Epsilon greedy exploration. Furthermore, a prioritized sampling mechanism is used to accelerate training convergence. An early warning mechanism is also designed to improve system robustness. In response to the shortcomings of traditional power system fault plan generation methods when dealing with large-scale, dynamic power grids, the present invention strives to achieve rapid generation of fault plans by combining imitation learning accelerated training and deep reinforcement learning optimization strategies, improve their adaptability to complex environments, and generate plans that meet technical indicators and operational requirements.

[0150] In summary, this invention uses imitation learning to provide an optimized initial strategy for the Dueling DQN agent. Combined with deep reinforcement learning (DRL) and guided exploration, it significantly improves the response speed of fault plan generation. This invention uses imitation learning to pre-train the agent and, through an early warning mechanism, triggers multi-action simulations in potential risk states to ensure decision-making safety. Imitation learning provides the initial strategy, a weighted MSE loss function optimizes the pre-training process, guided exploration reduces invalid attempts through high-Q-value action simulations, and a prioritized sampling mechanism accelerates convergence. Finally, a weighted MSE loss function is designed to assign higher weights to high-reward actions, and a prioritized sampling mechanism optimizes empirical selection through importance sampling. Compared to traditional DRL training with uniform sampling, the present invention significantly improves the quality of fault plans. This effect is directly due to the technical features' ability to prioritize learning and empirical optimization of high-quality actions, ensuring the accuracy and reliability of the plans.

[0151] In an embodiment of the present invention, the present invention combines imitation learning and guided exploration, and uses historical expert plan data to train the initial Q network, providing a high-starting point strategy for DRL, significantly reducing ineffective exploration during training, and providing a good initial strategy for the Dueling DQN agent, reducing subsequent random exploration time. Training data is collected by interacting with the power grid environment. The strategy of Dueling DQN is optimized. A batch of experiences is sampled from the experience replay buffer (using a priority sampling mechanism to prioritize experiences with high importance). The mean square error between the predicted Q value and the target Q value is minimized through gradient descent. The weights of the main Q network are copied to the target Q network to maintain training stability. The optimal plan is generated through multiple iterations. The above steps are repeated until the agent strategy converges or the predetermined number of training steps is reached. The final output Dueling DQN agent can quickly generate a fault plan that meets real-time and reliability requirements.

[0152] See also Figure 6 , Figure 6 This is a structural block diagram of a power grid fault plan generation system based on deep reinforcement learning provided in Example 3 of the present invention.

[0153] The present invention provides a power grid fault plan generation system based on deep reinforcement learning, comprising:

[0154] The response module 601 is used to respond to the generation request and build a power system fault plan model;

[0155] The conversion module 602 is used to convert the power system fault plan model to generate a Markov decision process model;

[0156] The simulation module 603 is used to simulate the pre-operation of the faulty power grid using a power network simulation tool to generate a reference state of the faulty power grid;

[0157] The solution module 604 is configured to solve the Markov decision process model according to the reference state of the faulty power grid using a dual deep reinforcement learning Q network to generate a target power grid fault recovery plan for the faulty power grid.

[0158] Furthermore, the solution module 604 is specifically configured to:

[0159] Determine whether the baseline state meets the preset risk conditions;

[0160] If satisfied, the target policy network in the dual deep reinforcement learning Q network is used to generate multiple actions based on the baseline state;

[0161] Each action is evaluated separately through the target evaluation network in the dual deep reinforcement learning Q network, and the evaluation value corresponding to each action is output;

[0162] Sort the evaluation values ​​corresponding to the actions in descending order, and take the action corresponding to the evaluation value in the first preset digit as the intermediate action;

[0163] The reward function in the Markov decision process model is used to calculate the reward value corresponding to each intermediate action based on multiple intermediate actions;

[0164] Select the intermediate action corresponding to the maximum reward value as the target action for the current time step;

[0165] Based on the target action of the current time step, the current operating status of the faulty power grid is generated and the current time step is counted in real time;

[0166] Determine whether the current operating state meets the preset risk conditions and whether the current time step reaches the preset time step;

[0167] If the current operating state does not meet the preset risk condition or the current time step reaches the preset time step, a target grid fault recovery plan for the faulty power grid is generated based on the target action of the current time step and the target actions of multiple historical time steps.

[0168] In an optional system embodiment, the system further includes:

[0169] The first module is used to use the target action of the current time step as the target action of the historical time step if the current operating state meets the preset risk condition and the current time step does not reach the preset time step;

[0170] The second module is used to use the current operating state of the faulty power grid as the new baseline state and jump to the target policy network in the dual deep reinforcement learning Q network to generate multiple action steps based on the baseline state until the current operating state fails to meet the preset risk condition or the current time step reaches the preset time step;

[0171] The third module is used to generate a target power grid fault recovery plan for the faulty power grid based on the target action of the current time step and the target actions of multiple historical time steps determined when the current operating state does not meet the preset risk condition or the current time step reaches the preset time step.

[0172] In an optional system embodiment, the system further includes:

[0173] The fourth module is used to generate a training data set based on a power network simulation tool;

[0174] The fifth module is used to evaluate the actions in multiple state-action pairs in the training dataset using the initial evaluation network in the dual deep reinforcement learning Q network, and generate training evaluation values ​​corresponding to the actions in each state-action pair;

[0175] a sixth module, configured to sort the training evaluation values ​​corresponding to the actions in each state-action pair in descending order, and assign a preset first weight to the actions in the state-action pairs corresponding to the training evaluation values ​​of the first preset second number of digits, and assign a preset second weight to the actions in the state-action pairs corresponding to the remaining training evaluation values;

[0176] The seventh module is used to substitute the training evaluation value corresponding to the action in each state-action pair, the preset first weight, and the preset second weight into the preset loss function and derive it, and output the model gradient;

[0177] The eighth module is used to update the model parameters of the initial evaluation network using the model gradient, determine the intermediate evaluation network, and count the number of model updates in real time;

[0178] The ninth module is used to determine whether the number of model updates reaches a preset update threshold;

[0179] The tenth module is used to use the intermediate evaluation network as the trained target evaluation network if it is achieved.

[0180] In an optional system embodiment, the system further includes:

[0181] The eleventh module is configured to use the intermediate evaluation network as a new initial evaluation network if the number of model updates does not reach a preset update threshold, and jump to executing the step of using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in multiple state-action pairs in the training data set, and generate training evaluation values ​​corresponding to the actions in each state-action pair, until the number of model updates reaches the preset update threshold;

[0182] The twelfth module is used to use the intermediate evaluation network determined when the number of model updates reaches a preset update threshold as the trained target evaluation network.

[0183] Furthermore, the objective function corresponding to the power system fault plan model is specifically:

[0184] ;

[0185] ;

[0186] ;

[0187] in, - are the weights of various evaluation indicators (including node load loss, generator output adjustment, and the status of the backup line at time t), represents the node load loss; Indicates generator output adjustment; is a 0-1 variable, indicating the status of the backup line at time t, where =1 means the status of the backup line at time t is in operation, =0 means the status of the backup line at time t is disconnected, is the status of the backup line at time t-1; M is the number of backup equipment (including lines and generators) in the system; 、 are the collections of load nodes and generator nodes respectively; and are the active load of load node i before and after the fault plan is activated; and They are the output of the generator node before and after the fault plan is activated.

[0188] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0189] An embodiment of the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory; when the computer program is executed by the processor, the processor executes the steps of the method for generating a power grid fault plan based on deep reinforcement learning as described in any of the above embodiments.

[0190] An embodiment of the present invention also provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the method for generating a power grid fault plan based on deep reinforcement learning as in any of the above embodiments are implemented.

[0191] An embodiment of the present invention also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the method for generating a power grid fault plan based on deep reinforcement learning as in any of the above embodiments.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0193] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0194] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating a power grid fault plan based on deep reinforcement learning, characterized in that: include: Respond to generation requests and build a power system fault plan model; Converting the power system fault plan model to generate a Markov decision process model; Using a power network simulation tool to simulate a pre-operation of the faulty power grid to generate a baseline state of the faulty power grid; A dual deep reinforcement learning Q network is used to solve the Markov decision process model according to a baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid; The dual deep reinforcement learning Q network is used to solve the Markov decision process model according to the baseline state of the faulty power grid to generate a target power grid fault recovery plan for the faulty power grid, including: Determining whether the baseline state meets a preset risk condition; If satisfied, the target policy network in the dual deep reinforcement learning Q network is used to generate multiple actions according to the baseline state; Evaluate each of the actions using the target evaluation network in the dual deep reinforcement learning Q network, and output an evaluation value corresponding to each of the actions; Sort the evaluation values ​​corresponding to the actions in descending order, and take the action corresponding to the evaluation value at the first preset digit as the intermediate action; Calculating a reward value corresponding to each intermediate action based on the plurality of intermediate actions using a reward function in the Markov decision process model; Select the intermediate action corresponding to the maximum reward value as the target action for the current time step; Based on the target action of the current time step, the current operating state of the fault power grid is generated, and the current time step is counted in real time; Determining whether the current operating state satisfies the preset risk condition and whether the current time step reaches the preset time step; If the current operating state does not satisfy the preset risk condition or the current time step reaches the preset time step, generating a target power grid fault recovery plan for the faulty power grid according to the target action of the current time step and the target actions of multiple historical time steps; The objective function corresponding to the power system fault plan model is specifically: ； ； ； in, - are the weights of each evaluation index, represents the node load loss; Indicates generator output adjustment; is a 0-1 variable, indicating the status of the backup line at time t, where =1 means the status of the backup line at time t is in operation, =0 means the status of the backup line at time t is disconnected, is the status of the backup line at time t-1; M is the number of backup devices in the system; 、 are the collections of load nodes and generator nodes respectively; and are the active load of load node i before and after the fault plan is activated; and They are the output of the generator node before and after the fault plan is activated.

2. The method for generating a power grid fault plan based on deep reinforcement learning according to claim 1, characterized in that: Also includes: If the current running state satisfies the preset risk condition and the current time step does not reach the preset time step, taking the target action of the current time step as the target action of the historical time step; Taking the current operating state of the faulty power grid as a new reference state, and jumping to the step of using the target policy network in the dual deep reinforcement learning Q network to generate multiple actions according to the reference state until the current operating state fails to meet the preset risk condition or the current time step reaches the preset time step; A target power grid fault recovery plan for the faulty power grid is generated according to a target action for the current time step and target actions for multiple historical time steps determined when the current operating state does not satisfy a preset risk condition or the current time step reaches the preset time step.

3. The method for generating a power grid fault plan based on deep reinforcement learning according to claim 1, characterized in that: The training process of the target evaluation network in the dual deep reinforcement learning Q network is as follows: Based on the power network simulation tool, generating a training data set; Using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in the multiple state-action pairs in the training data set, and generating a training evaluation value corresponding to the action in each state-action pair; Sort the training evaluation values ​​corresponding to the actions in each state-action pair in descending order, assign a preset first weight to the actions in the state-action pair corresponding to the training evaluation values ​​of the first preset second number of digits, and assign a preset second weight to the actions in the state-action pair corresponding to the remaining training evaluation values; Substituting the training evaluation value corresponding to the action in each state-action pair, the preset first weight, and the preset second weight into a preset loss function and performing derivative, and outputting a model gradient; Using the model gradient to update the model parameters of the initial evaluation network, determine the intermediate evaluation network, and count the number of model updates in real time; Determining whether the model update count reaches a preset update threshold; If achieved, the intermediate evaluation network is used as the trained target evaluation network.

4. The method for generating a power grid fault plan based on deep reinforcement learning according to claim 3, characterized in that: Also includes: If the number of model updates does not reach the preset update threshold, the intermediate evaluation network is used as a new initial evaluation network, and the step of using the initial evaluation network in the dual deep reinforcement learning Q network to evaluate the actions in the multiple state-action pairs in the training data set to generate training evaluation values ​​corresponding to the actions in each state-action pair is executed, until the number of model updates reaches the preset update threshold; The intermediate evaluation network determined when the number of model updates reaches the preset update threshold is used as the trained target evaluation network.

5. A power grid fault plan generation system based on deep reinforcement learning, applied to the power grid fault plan generation method based on deep reinforcement learning according to claim 1, characterized in that: include: The response module is used to respond to the generation request and build a power system fault plan model; A conversion module, used to convert the power system fault plan model to generate a Markov decision process model; A simulation module, configured to simulate a pre-operation of a faulty power grid using a power network simulation tool to generate a reference state of the faulty power grid; A solution module is used to solve the Markov decision process model according to the reference state of the faulty power grid using a dual deep reinforcement learning Q network to generate a target power grid fault recovery plan for the faulty power grid.

6. A computer device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor performs the steps of the power grid fault plan generation method based on deep reinforcement learning as described in any one of claims 1 to 4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the method for generating a power grid fault plan based on deep reinforcement learning as described in any one of claims 1 to 4 is implemented.

8. A computer program product, characterized in that The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions, wherein when the program instructions are executed by a computer, the computer is caused to execute the method for generating a power grid fault plan based on deep reinforcement learning as described in any one of claims 1 to 4.

Images