Power grid security enhanced intelligent scheduling method and device based on twinning adversarial imitation learning, and storage medium

Abstract

The invention discloses a power grid safety enhanced intelligent scheduling method and device based on twinborn adversarial imitation learning and a storage medium, and belongs to the technical field of power scheduling. The method comprises the following steps: acquiring current real-time state information of a power grid system; inputting the real-time state information into a pre-trained scheduling strategy agent to generate a corresponding real-time scheduling action; and judging whether the real-time scheduling action in the current state is an expert-level action or not based on a state action data pair pool formed by the scheduling strategy agent in a training stage, and outputting a final scheduling strategy to a real environment according to a judgment result. Through real-time data analysis, intelligent decision and expert verification, the method can automatically adapt to the complexity and dynamic nature of the power grid, not only realizes efficient, safe and stable operation of the power grid, but also provides powerful support for intelligent management of the power grid.

Description

Technical Field

[0001] The present invention relates to a method, device and storage medium for intelligent dispatching of power grid security enhancement based on twin adversarial imitation learning, and belongs to the technical field of power dispatching. Background Art

[0002] The power grid system is a key infrastructure in modern society. The safety and reliability of its operation directly affect the country's economic development and people's daily life. However, with the expansion of the scale and complexity of the power grid, the traditional power grid dispatching method has gradually become incapable of dealing with complex environments and uncertain factors. Most traditional dispatching strategies are based on preset rules or simple optimization algorithms, so they often lack flexibility in dealing with variable risks in complex dynamic environments. Especially in the case of sudden failures and abnormal weather, it is difficult to respond quickly and adjust the dispatching strategy, which greatly increases the possibility of serious consequences such as large-scale power outages.

[0003] In this context, intelligent dispatching methods based on machine learning have emerged, especially the application of reinforcement learning technology in power grid dispatching, which provides a new way to improve the safety and efficiency of the power grid. Reinforcement learning gradually optimizes strategies to achieve the desired goals through interaction with the environment, and has achieved remarkable results in the fields of automatic control, robotics, and resource management. However, standard reinforcement learning methods face some challenges in the application of power grid dispatching, such as safety during exploration, rapid response to emergencies, and the ability to learn effective strategies from sparse rewards. For this reason, safe reinforcement learning has gradually become the focus of attention.

[0004] Safety reinforcement learning introduces safety constraints in the learning process. By considering the safety indicators of the system when optimizing the strategy, it ensures that the strategy will not cause the system to enter a dangerous state under any circumstances. In view of this, twin adversarial imitation learning combines the concept of safety reinforcement learning and introduces twin network structure and adversarial learning mechanism, so that the system can not only imitate the optimal scheduling behavior of experts, but also identify and respond to potential threats, thereby enhancing the robustness and adaptability of the system. However, although the application of twin adversarial imitation learning in some complex systems has shown its potential, there is still a lack of a more systematic solution in the field of intelligent scheduling for power grid security enhancement. Summary of the invention

[0005] The purpose of the present invention is to provide a method, device and storage medium for intelligent scheduling based on twin adversarial imitation learning to enhance power grid security, aiming to achieve global optimal intelligent scheduling while ensuring the safety of the power grid system by independently evaluating and optimizing scheduling actions in the training stage and the decision-making stage.

[0006] In order to achieve the above objectives / solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0007] The present invention provides a method for intelligent dispatching of power grid security enhancement, which comprises: Obtain the current real-time status information of the power grid system; Input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; Based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, it is determined whether the real-time scheduling action in the current state is an expert-level action, and then the final scheduling strategy, that is, the active output value of the unit is output to the real environment according to the judgment result.

[0008] The power grid security enhanced intelligent dispatching method based on twin adversarial imitation learning proposed in the present invention can not only effectively utilize historical data and simulation environment for continuous optimization, but also ensure the safety and efficiency of dispatching decisions in a complex and changeable power grid operating environment, thereby providing strong support for the safe and stable operation of smart grids and has strong practicality.

[0009] Optionally, the current real-time status information of the power grid system is expressed as: ; In the formula, represents the state vector set, Indicates the voltage value of each node in the power grid, Indicates the current value of each line in the power grid, Indicates the power status of power generation and load, including active power and reactive power. Indicates time The load demand of each node or area in the power grid, the state vector will be input into the dispatch strategy agent and the optimal action generator as the basis for dispatch strategy generation and evaluation.

[0010] Optionally, the training process of the scheduling strategy agent includes: The acquired historical status information of the power grid system is input into the optimal action generator and the dispatch strategy agent to generate the optimal dispatch action and the common dispatch action respectively; Inputting the optimal scheduling action and the common scheduling action into the first discriminator to perform comparative evaluation of the strategies, generating corresponding reward values ​​according to the comparative evaluation results, and feeding back the reward values ​​to the scheduling strategy agent; Repeat the above steps until the common scheduling action generated by the scheduling strategy agent in the current state is continuously identified as the optimal scheduling action and the training converges. In other words, the action output by the scheduling strategy agent is input into the first discriminator, and if the first discriminator continuously identifies it as the optimal scheduling action, the training converges.

[0011] Optionally, during the training process, the state-action data pair pool stores historical state information and the optimal scheduling action corresponding to the historical state information in the form of key-value pairs for the scheduling strategy agent to call in the decision-making stage.

[0012] Optionally, the optimal action generator is based on the optimal power flow model and, according to the acquired historical status information of the power grid system, solves the corresponding optimization problem with the optimization goals of minimizing power generation costs and energy losses and maximizing the active output of new energy, and generates the optimal scheduling action that meets the power grid operation constraints at the current moment.

[0013] Optionally, the objective function of the optimal power flow model is expressed as: ; In the formula, Indicates the number of thermal power units, Indicates the number of new energy units, represents the cost function of the thermal power unit, represents the wind curtailment rate function of new energy units, Indicates the number The active power output of thermal power units, Indicates the number of the thermal power unit. Indicates the number The active power output of the new energy units, Indicates the number of the new energy unit.

[0014] Optionally, the cost function of the thermal power unit is expressed as: ; In the formula, represents the cost function of the thermal power unit, , and Represents thermal power unit cost factor.

[0015] Optionally, the wind abandonment rate function of the new energy unit is expressed as: ; In the formula, represents the wind curtailment rate function of new energy units, Indicates the number The maximum active output of the new energy unit.

[0016] Optionally, during the training process, the scheduling strategy agent optimizes the scheduling strategy based on the SAC algorithm, and the corresponding loss function is expressed as: ; In the formula, represents the loss function, represents the mathematical expectation of the policy network Q function, express The current state of the power grid, express Scheduling actions at the moment, represents the policy function, express The action value function at the moment, Represents the entropy adjustment parameter, which is used to adjust the exploration and stability of the strategy. express In Status The entropy under Indicates in status Next, strategy Probability distribution over actions.

[0017] Optionally, the strategy of the scheduling strategy agent is updated by gradient descent, and the corresponding update rule is expressed as: ; In the formula, represents the network parameters of the target scheduling strategy agent, represents the learning rate, Represents the gradient of the loss function.

[0018] Optionally, the action value function is used to evaluate the long-term reward of an action in the current state, which can be specifically expressed as: ; In the formula, express The action value function at the moment, It indicates that the expected value of the next state is calculated based on the current state and action. express The current state of the power grid, represents the state transition probability distribution, Represents the current reward value, represents the discount factor, express The action value function at the moment, express Scheduling actions at the moment, express The maximum action value function at the moment is the maximum expected benefit that the system can bring by taking the optimal action in the next step.

[0019] Optionally, the calculation formula of the reward value is as follows: ; In the formula, Represents the scoring result of the scheduling strategy agent.

[0020] Optionally, the scoring function used by the first discriminator in the process of strategy comparison and evaluation is expressed as: ; In the formula, Indicates the rating results. represents the Sigmoid activation function, and Represent different weight matrices, represents the rectified linear unit activation function, express The corresponding action under the state, and They represent different biases respectively.

[0021] Optionally, the loss function corresponding to the first discriminator is expressed as: ; In the formula, represents the loss function, It indicates the expected value calculation of the optimal action and the corresponding state combination. Indicates the expected value calculation for common actions and corresponding state combinations. represents the optimal scheduling action, Indicates a normal scheduling action. Represents the scoring result of the optimal action generator.

[0022] Optionally, judging whether the real-time scheduling action in the current state is an expert-level action based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, and then outputting the final scheduling strategy to the real environment according to the judgment result, includes: Acquire historical state information in the state-action data pair pool that meets a preset state distance with the real-time state information, and a historical optimal scheduling action corresponding to the historical state information; By evaluating the similarity value between the real-time scheduling action and the historical optimal scheduling action, determining whether the real-time scheduling action in the current state is an expert-level action; If so, the real-time scheduling action is output to the real environment as the final scheduling strategy; if not, the real-time status information is input to the optimal action generator, and the optimal scheduling action generated by the optimal action generator is output to the real environment as the final scheduling strategy.

[0023] Optionally, a second identifier is used to determine whether the real-time scheduling action in the current state is an expert-level action. If the similarity value is higher than a preset threshold, the real-time scheduling action in the current state is considered to be an expert-level action.

[0024] Optionally, the similarity value is calculated as follows: ; In the formula, Represents the similarity value, Indicates the historical optimal scheduling action corresponding to the historical state information close to the real-time state information, Represents a real-time scheduling action.

[0025] Optionally, the loss function corresponding to the second discriminator is expressed as: ; In the formula, represents the loss function of the second discriminator, Indicates the security risk of the current action. represents the output score of the second discriminator, that is, in state Next, action The higher the value, the safer the action.

[0026] Optionally, the second discriminator inherits the evaluation criteria and weight parameters learned by the first discriminator in the training phase through a parameter transfer mechanism, for rapid convergence in the decision phase.

[0027] In a second aspect, the present invention further provides a power grid security enhancement intelligent dispatching device, which includes: The data acquisition module is configured to acquire the current real-time status information of the power grid system; The scheduling strategy preliminary determination module is configured to input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; The scheduling strategy finalization module is configured to determine whether the real-time scheduling action in the current state is an expert-level action based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, and then output the final scheduling strategy to the real environment according to the judgment result.

[0028] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the intelligent dispatching method for enhancing power grid security as described in the first aspect.

[0029] Compared with the prior art, the present invention has the following beneficial effects: The present invention uses a twin adversarial imitation learning framework to enable the system to generate a global optimal scheduling strategy that meets the safety requirements of the power grid, avoiding the local optimal problem of traditional scheduling methods; secondly, the present invention uses two independent discriminators to evaluate the scheduling actions in the training stage and the decision-making stage respectively, ensuring that the actions of the system in each stage meet safety standards and ensure the safety of power grid operation; in addition, the scheduling strategy agent of the present invention is based on the reinforcement learning ability of the SAC algorithm, and can generate efficient scheduling strategies in real time in a complex power grid environment, thereby improving the real-time response capability of the system; in addition, the present invention also designs a state-action data pool to ensure that the system can effectively utilize historical scheduling experience for dynamic optimization, ensuring that the scheduling strategy can execute the optimal decision under similar conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a flow chart of the decision-making application phase of the dispatching strategy agent in the intelligent dispatching method for strengthening power grid security based on twin adversarial imitation learning of the present invention; Figure 2 It is a flow chart of the dispatching strategy agent model training phase in the power grid security enhancement intelligent dispatching method based on twin adversarial imitation learning of the present invention; Figure 3 It is a schematic diagram showing the parameter transfer process of the discriminator network in the training stage and the deduction stage in the power grid security enhancement intelligent dispatching method based on twin adversarial imitation learning of the present invention; Figure 4 Shown is a schematic diagram of the overall process of the power grid security enhancement intelligent scheduling method based on twin adversarial imitation learning of the present invention. DETAILED DESCRIPTION

[0031] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0032] The term "and / or" is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the associated objects are in an "or" relationship.

[0033] Example 1 This embodiment introduces a power grid security enhancement intelligent dispatching method, which includes: Obtain the current real-time status information of the power grid system; Input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; Based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, it is determined whether the real-time scheduling action in the current state is an expert-level action, and then the final scheduling strategy, that is, the active output value of the unit is output to the real environment according to the judgment result.

[0034] In the actual application process of this embodiment, the current state data of the real power grid system is first obtained, and the relevant data is input into the trained dispatching strategy agent based on the SAC algorithm. The dispatching strategy agent gives a dispatching strategy according to the input power system state, and inputs the dispatching strategy into the second discriminator. Furthermore, the second discriminator will identify whether it is an expert action based on the output action of the dispatching strategy agent, the system state, and the state action data pool formed in the previous training stage. If it is identified as an expert action, the action output by the dispatching agent is directly output to the real environment; if it is identified as a non-expert action, the current power grid state is input into the optimal action generator, and the optimal action generator gives the optimal action to the real environment.

[0035] Example 2 In this embodiment, the core of the whole system is an architecture based on twin adversarial imitation learning. This architecture achieves dual optimization of scheduling strategy generation and safety assessment through the collaborative work of scheduling strategy agent, optimal action generator, two independent discriminators and state action data pair pool. Figures 1 to 4 Based on Example 1, this example introduces in detail the specific implementation process of the power grid security enhancement intelligent dispatching strategy.

[0036] 1. Training Phase refer to Figure 1 ,During the training phase, the scheduling strategy is continuously optimized through adversarial learning between the scheduling strategy agent and the optimal action generator, making it close to the global optimum.

[0037] The scheduling strategy agent performs reinforcement learning based on the SAC algorithm (Soft Actor-Critic), and gradually learns to generate a scheduling strategy close to the global optimal. At the same time, the system also provides the optimal scheduling solution under the current state as a reference through the optimal action generator, and uses the first discriminator (i.e. Figure 4 The discriminator 1) in the scheduling strategy agent evaluates the actions generated by the scheduling strategy agent to help the scheduling strategy agent optimize the strategy. The specific implementation steps are as follows: The current state data of the power grid is obtained through the sensor network or SCADA system, and the state data includes at least: voltage, current, power and load. The above state data can be represented as a vector set: ; In the formula, represents the state vector set, Indicates the voltage value of each node in the power grid, Indicates the current value of each line in the power grid, Reflects the power status of power generation and load, including active power and reactive power. Indicates time The load demand of each node or area in the power grid. In actual application, the above-mentioned state space expression is preferred in this embodiment.

[0038] In addition, the state space It can also be expressed by the following formula, and its corresponding sets include: the active output set of all units, the upper limit set of all unit adjustment values ​​at the current moment, the lower limit set of all unit adjustment values, the set of all unit load active output changes, and the set of total predicted loads at the current moment.

[0039] ; In the formula, represents the set of active outputs of all units at the previous moment, Indicates the upper limit set of active output adjustment values ​​of all units at the current moment, Represents the lower limit set of active output adjustment values ​​of all units at the current moment, It represents the set of load active output changes between the next moment and the current moment. Represents the set of active outputs of all unit loads at the current moment.

[0040] Furthermore, the state vector will be input into the scheduling strategy agent and the optimal action generator as the basis for scheduling strategy generation and evaluation.

[0041] Based on the state vector of the above power grid system, the optimal action generator generates the optimal dispatching action based on the optimal power flow model (OPF), and the dispatching strategy agent generates the common dispatching strategy based on the SAC algorithm.

[0042] SAC is a deep reinforcement learning algorithm based on policy gradient. Its main feature is to encourage exploration by maximizing the entropy of the strategy and prevent the strategy from falling into the local optimum. In the application of power grid dispatch, the SAC algorithm can help the dispatch strategy agent make better dispatch decisions in an uncertain dynamic environment.

[0043] In order to balance the stability and exploration of the strategy, the SAC algorithm introduces an entropy term to encourage the strategy to explore more in the early stages. The larger the entropy, the higher the strategy uncertainty, and the more the agent explores the environment, which helps to find the global optimal solution.

[0044] The goal of the SAC algorithm is to maximize the following loss function: ; In the formula, represents the loss function, represents the mathematical expectation of the policy network Q function, Indicates the power grid status. Indicates a scheduling action. represents the policy function, express The action value function at the moment, Represents the entropy adjustment parameter, which is used to adjust the exploration and stability of the strategy. express In Status Entropy under It is an indicator of uncertainty, used to indicate the Next Strategy The degree of confusion about the probability distribution of various possible actions, Indicates that the status Next, strategy Probability distribution over actions.

[0045] The strategy of the scheduling strategy agent is updated by gradient descent, and the corresponding update rule is expressed as: ; In the formula, represents the network parameters of the target scheduling strategy agent, represents the learning rate, Represents the gradient of the loss function; Among them, the policy function :The dispatch strategy agent is based on the current power grid status , generate scheduling actions through policy functions The policy function is optimized through continuous iteration and eventually learns the optimal scheduling policy.

[0046] Action-Value Function : Value function It is used to evaluate the long-term return of an action in the current state. The corresponding calculation formula is: ; In the formula, express The action value function at the moment, It indicates that the expected value of the next state is calculated based on the current state and action. express The current state of the power grid, represents the state transition probability distribution, represents the current reward, Represents the discount factor, usually selected as 0.99, which is used to balance the impact of current returns and future returns. express The action value function at the moment, express Scheduling actions at the moment, express The maximum action value function at the moment is the maximum expected benefit that the system can bring by taking the optimal action in the next step.

[0047] Through this training method, the dispatching strategy agent gradually learns how to generate dispatching strategies close to the global optimality to effectively respond to various disturbances in the dynamic environment of the power grid.

[0048] The optimal action generator is based on the optimal power flow model (OPF) of the power grid. By combining historical operation data and the real-time status of the system, it generates the optimal dispatching action at the current moment as a reference for the dispatching strategy agent to generate actions in the decision-making stage. It is worth noting that in the actual application process, an expert imitation module is also involved, which is used to collect and store expert dispatching actions and provide them to the dispatching strategy agent for imitation learning, that is, the optimal dispatching action generated by the optimal action generator.

[0049] The optimal power flow model is a typical power system optimization problem, whose goal is to minimize the power generation cost and energy loss and maximize the active output of renewable energy while ensuring the security of the power grid.

[0050] Among them, the optimization objective function of the optimal power flow model is expressed as: ; In the formula, Indicates the number of thermal power units, Indicates the number of new energy units, represents the cost function of the thermal power unit, represents the wind curtailment rate function of new energy units, Indicates the number The active power output of thermal power units, Indicates the number of the thermal power unit. Indicates the number The active power output of the new energy units, Indicates the number of the new energy unit.

[0051] Optionally, the cost function of the thermal power unit is expressed as: ; In the formula, represents the cost function of the thermal power unit, , and Represents thermal power unit cost factor.

[0052] Optionally, the wind abandonment rate function of the new energy unit is expressed as: ; In the formula, represents the wind curtailment rate function of new energy units, Indicates the number The maximum active output of the new energy unit.

[0053] Furthermore, the optimal action generator and the scheduling strategy agent input the generated optimal scheduling action and the ordinary scheduling action into the first discriminator respectively. The first discriminator compares the ordinary scheduling action and the optimal scheduling action, evaluates the difference between the two, and feeds back a reward signal to the scheduling strategy agent to help it continuously improve and complete the optimization of the scheduling strategy.

[0054] The first discriminator consists of several convolutional layers and fully connected layers, and the input is the current power grid state and two dispatch actions (optimal scheduling action) and (Ordinary scheduling actions generated by ordinary strategies) The network corresponding to the first discriminator extracts the feature representations of the state and action through the feature extraction layer, and outputs the scores of the two actions through the fully connected layer.

[0055] In this embodiment, the first discriminator adopts the structure of deep convolutional neural network and multi-layer fully connected network to improve the action discrimination ability in the power grid dispatching process. Specifically, the first discriminator consists of 5 convolutional layers and 5 fully connected layers, which are used to input the power grid state. , the optimal dispatching action, and the ordinary dispatching action generated by the common strategy are feature extracted and fused, and finally the score of the dispatching action is output. First, the input layer receives the power grid status and dispatching action, and the input data is normalized to ensure the stability of the model learning in the high-dimensional feature space. In the feature extraction part, 5 convolutional layers process the input data in turn, and the number of convolution kernels increases from 64 to 1024. The convolution kernels that increase layer by layer are used to capture the complex spatial patterns and high-order features in the power grid operation status. Each convolutional layer is followed by a ReLU activation function and a maximum pooling layer to enhance the nonlinear expression ability of the network and gradually reduce the feature dimension, thereby effectively reducing the computational complexity.

[0056] After the convolutional layer extracts high-dimensional features, the state features and action features are flattened and concatenated to form a unified joint feature representation, which is used to capture the complex relationship between the power grid state and the dispatch action. Next, the joint features are processed step by step through 5 fully connected layers, each of which compresses the feature dimension from 8192 to 512 in turn, thereby gradually extracting more discriminative feature information. The ReLU activation function is used between the fully connected layers to enhance the nonlinear fitting ability of the network, and the Dropout mechanism (ratio of 0.5) is added to prevent overfitting and improve the generalization ability of the model under complex power grid operating conditions.

[0057] Finally, the features processed by the fully connected layer are output through the Sigmoid activation function to obtain a score value, which is used to judge whether the input scheduling action is close to the optimal action. The score range is between 0 and 1, representing the probability estimate of the discriminator that the action is the optimal scheduling. The combination of deep convolution and multi-layer full connection enables the first discriminator to have powerful feature extraction and discrimination capabilities, and can effectively learn the complex mapping relationship between the power grid state and the action, thereby continuously optimizing the scheduling strategy during the training process, and ultimately improving the safety, stability and economy of the power grid.

[0058] The goal of the first discriminator is to distinguish the optimal action from the common strategy and to generate scores to feed back to the scheduling strategy agent. The scoring function used in the decision evaluation process of discriminator 1 can be expressed as: ; In the formula, Indicates the rating results. represents the Sigmoid activation function (such as sigmoid), and Represent different weight matrices, represents the rectified linear unit activation function, express The corresponding action under the state, and They represent different biases respectively.

[0059] The loss function of the first discriminator is used to measure the difference between the optimal action and the ordinary action. The loss function is defined as: ; In the formula, represents the loss function, It indicates the expected value calculation of the optimal action and the corresponding state combination. Indicates the expected value calculation for common actions and corresponding state combinations. represents the optimal scheduling action, Indicates a normal scheduling action. represents the scoring result of the optimal action generator, Represents the scoring result of the scheduling strategy agent.

[0060] By minimizing the above loss function, the first discriminator can gradually improve its discrimination ability to prompt the scheduling strategy agent to generate a better scheduling strategy.

[0061] Furthermore, the first discriminator will score the result Converted into a corresponding reward value, which is used to feed back to the scheduling strategy agent to guide it to generate better actions in the future. The calculation formula is: ; Furthermore, the first discriminator outputs the optimal dispatching strategy to the real power grid and stores the state-action pairs into the state-action data pair pool for use in the subsequent decision-making stage.

[0062] It is worth noting that during the training process, the optimal action generated each time is consistent with the corresponding power grid state. It will be stored in the state-action data pair pool. The data pair pool records the historical data during system training. The main functions are as follows: Storage state action pair: data pair pool The combination of state and optimal action is stored in the form of. As the system is trained, the data pool gradually accumulates a large number of historical state and scheduling action pairs. Utilization of historical experience: In the decision-making stage, the scheduling strategy agent can query data for historical scheduling actions under similar conditions in the pool and optimize the current scheduling strategy based on historical experience. This process helps the system make better decisions in complex environments and avoid repeated mistakes. Prioritized experience playback: The data pool also supports a prioritized experience playback mechanism, which assigns higher priority to critical states based on the importance of historical states and actions, helping the system pay more attention to states and dispatch strategies that have a greater impact on grid operation. Among them, critical states refer to states with large TD errors. Large TD errors indicate that the current strategy's estimate of the state has a large deviation from the actual result. By giving priority to replaying these states, the system can effectively correct strategy estimates and improve the accuracy of dispatch strategies.

[0063] Repeat the above steps until the strategy generated by the scheduling strategy agent is close to the optimal scheduling action and the training converges.

[0064] Decision-making stage refer to Figure 2 The task of the decision-making stage is to use the trained dispatching strategy agent to generate dispatching actions in the actual operation of the power grid, and to use the second discriminator (i.e. Figure 4 The discriminator in 2) performs safety assessment on these actions to ensure the safe operation of the system in a complex power grid environment. The specific implementation steps are as follows: First, the real-time status data of the power grid is obtained and input into the trained dispatch strategy agent.

[0065] In the decision-making stage, the real-time status data of the power grid is collected through the sensor network or SCADA system. Input into the trained scheduling strategy agent, the agent will Generates the current scheduling action .

[0066] Since the agent has been optimized by the SAC algorithm during the training phase, the dispatch actions it generates have high global optimality and safety. However, due to the dynamic complexity of the power grid, the generated actions need to be further evaluated for safety by the second discriminator.

[0067] It is worth noting that the two discriminators in this embodiment are designed independently and are respectively applied in the training stage and the decision-making stage to effectively ensure the rationality and safety of the scheduling actions during generation and execution.

[0068] The second discriminator is responsible for the safety assessment of the actions generated by the scheduling strategy agent in the decision-making stage. By combining the current state with the historical data in the state-action data pool, the second discriminator determines whether the current action meets the expert-level scheduling standards. The specific implementation is as follows: Parameter passing: Reference Figure 3The figure describes the parameter transfer process of the discriminator network in the training and deduction stages. The parameters of the middle part of the first discriminator neural network trained offline are copied to the second discriminator in the online deduction. The second discriminator inherits the evaluation criteria and weight parameters learned by the first discriminator in the training stage through the parameter transfer mechanism. In this way, the second discriminator has a higher evaluation ability in the initial stage, which can accelerate the convergence of the decision stage.

[0069] Evaluation process: At every decision moment , the second discriminator inputs the current state and dispatch actions , and retrieve historical states similar to the current state from the state-action data pool and the optimal scheduling action By comparing the current action with the historical optimal action, the second discriminator determines the rationality of the current action.

[0070] It is worth noting that the above-mentioned "similar" judgment criteria are mainly based on the feature similarity between the real-time status information and the historical status information. In order to determine the similarity between the two, the following methods and standards can be used: Euclidean distance, specific feature thresholds, and setting the allowable deviation range for key features (such as load level, voltage level, etc.). Calculate the Euclidean distance between the historical information and the real-time information. In other words, if the key features of the real-time status are within the allowable range, they can be considered similar.

[0071] Similarity metric: The similarity is calculated as: ; If the similarity is higher than the preset threshold , then the second discriminator believes that the current action It is an expert-level action and the safety meets the requirements.

[0072] Decision output: According to the evaluation results, the output of the second discriminator is divided into two cases: if the current action meets the expert standard, the action is directly output and executed; and if the current action does not meet the expert standard, the system will state Passed to the optimal action generator, which generates an alternative optimal action and executes it.

[0073] Among them, the loss function of the second discriminator adds a safety constraint term to ensure that the generated dispatch action will not cause risks to the power grid. The corresponding loss function calculation formula is: ; in represents the loss function of the second discriminator, Indicates the security risk of the current action, and evaluates it in combination with historical data. represents the output score of the second discriminator, that is, in state Next, action The higher the value, the safer the action.

[0074] In addition, in order to adapt to the ever-changing dynamic environment during the operation of the power grid, the second discriminator continuously adjusts and optimizes the evaluation criteria through online learning and fine-tuning mechanisms in the decision-making stage. The specific process is as follows: Online learning: By incrementally learning from new data in grid operation, the second discriminator is able to update its evaluation model based on the latest operating conditions.

[0075] Fine-tuning strategy: In order to maintain the stability of the evaluation criteria, the learning rate of online learning is usually set low to ensure that changes during the fine-tuning process do not have too large an impact on the existing criteria.

[0076] The state-action data pair pool is one of the key modules of the entire system. It records a large amount of historical data during the training phase and provides support for the generation and evaluation of scheduling strategies by querying these data during the decision-making phase.

[0077] Data pair storage structure: The state-action data pair pool stores states and optimal scheduling actions in the form of key-value pairs. Can correspond to one or more optimal actions The data pool in this case can be expressed as: ; Data query: In the decision-making stage, the actions generated by the scheduling strategy agent can query the state-action data for similar states in the pool and refer to the historical optimal scheduling actions to optimize the execution effect of the current scheduling strategy.

[0078] Priority experience replay mechanism: The data pool introduces a priority experience replay mechanism, which assigns higher priority to historically important states and scheduling actions, ensuring that the system can effectively use key data for strategy optimization.

[0079] This embodiment successfully achieves the optimization and security of power grid dispatching strategy through innovative design of twin adversarial imitation learning framework, reinforcement learning of SAC algorithm, independent discriminator design and introduction of state-action data pair pool. Through specific implementation strategies, the system can ensure the global optimality, real-time and security of dispatching strategy, and has broad application prospects in complex power grid environment.

[0080] Finally, it is worth noting that the dispatch strategy proposed in this embodiment not only shows significant progress in multiple indicators compared with the existing traditional rule-based dispatch method and the existing intelligent dispatch method, but also has significant improvements in dispatch response time, system stability, emergency response capability and overall dispatch efficiency. For example, in terms of dispatch response time, traditional dispatch strategies often take 10-15 minutes to analyze and generate dispatch actions, while this embodiment can give an accurate dispatch plan within 1-2 minutes by combining reinforcement learning and expert imitation learning; in terms of system stability, traditional dispatch is prone to frequent manual adjustments and it is difficult to quickly adapt to the drastic fluctuations of load, while the intelligent dispatch system of this embodiment achieves higher stability and security by learning the load state of the power grid in real time; in addition, this embodiment also performs well in emergency response capability, and can quickly identify and output the optimal response plan when a system failure occurs, significantly shortening the response time of emergency dispatch. Through the collaborative work of the dispatch strategy agent and the discriminator, the intelligent dispatch strategy provided by this embodiment realizes the dynamic optimization and security of power grid dispatch, showing significant differences and progress from traditional dispatch methods, and further improving the safety and economy of the power grid system in actual application scenarios.

[0081] Example 3 Based on the same inventive concept as embodiment 1 or 2, this embodiment introduces a power grid security enhancement intelligent dispatching device, which includes: The data acquisition module is configured to acquire the current real-time status information of the power grid system; The scheduling strategy preliminary determination module is configured to input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; The scheduling strategy finalization module is configured to determine whether the real-time scheduling action in the current state is an expert-level action based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, and then output the final scheduling strategy to the real environment according to the judgment result.

[0082] Combination Figures 1 to 4 The power grid security enhancement intelligent dispatching device can execute the power grid security enhancement intelligent dispatching method introduced in the above-mentioned embodiment 1 or 2, and the specific functions of each functional module are described as follows.

[0083] In the actual application process of this embodiment, each functional module closely cooperates in the workflow to ensure that the dispatching action generated by the dispatching strategy agent can dynamically adapt to the load fluctuation and fault handling requirements of the power grid, effectively improving the intelligence level and safety of power grid dispatching. First, the data acquisition module is responsible for real-time acquisition of the current state information of the power grid system, including key parameters such as voltage, current, load level, frequency, etc., to provide basic data for dispatching decisions. Then, the dispatching strategy preliminary determination module inputs the collected real-time state information into the pre-trained dispatching strategy agent to generate corresponding real-time dispatching actions. Based on the improved SAC algorithm and historical data learning, the agent can quickly adapt to the current power grid operation state and output the preliminary dispatching strategy. Subsequently, the dispatching strategy final determination module judges the real-time dispatching action generated under the current state based on the state action data pool formed by the dispatching strategy agent in the training phase, and evaluates whether it is an "expert-level" action. This module performs similarity analysis on the expert-level data in the pool by comparing the current action with the state action data. If it is judged that the current action meets the expert standard, it is directly output as the final dispatching strategy to the real environment; otherwise, the power grid state information is re-input into the optimal action generator to generate an improved dispatching strategy and apply it to the power grid.

[0084] Example 4 This embodiment introduces a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the power grid security enhancement intelligent scheduling method described in Embodiment 1 or 2 are implemented.

[0085] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0086] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for intelligent dispatching of power grid security enhancement, characterized in that: include: Obtain the current real-time status information of the power grid system; Input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; Based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, it is determined whether the real-time scheduling action in the current state is an expert-level action, and then the final scheduling strategy is output to the real environment according to the judgment result.

2. The method for intelligent dispatching of power grid security enhancement according to claim 1, characterized in that: The current real-time status information of the power grid system is expressed as: ； In the formula, represents the state vector set, Indicates the voltage value of each node in the power grid. Indicates the current value of each line in the power grid, Indicates the power status of power generation and load, including active power and reactive power. Indicates time The load demand at each node or area in the power grid.

3. The intelligent dispatching method for strengthening power grid security according to claim 2 is characterized in that: The training process of the scheduling strategy agent includes: The acquired historical status information of the power grid system is input into the optimal action generator and the dispatch strategy agent to generate the optimal dispatch action and the common dispatch action respectively; Inputting the optimal scheduling action and the common scheduling action into the first discriminator to perform comparative evaluation of the strategies, generating corresponding reward values ​​according to the comparative evaluation results, and feeding back the reward values ​​to the scheduling strategy agent; Repeat the above steps until the common scheduling action generated by the scheduling strategy agent in the current state is continuously identified as the optimal scheduling action and the training converges; During the training process, the state-action data pair pool stores historical state information and the optimal scheduling action corresponding to the historical state information in the form of key-value pairs, which are called by the scheduling strategy agent in the decision-making stage.

4. The method for intelligent dispatching for strengthening power grid security according to claim 3 is characterized in that: The optimal action generator is based on the optimal power flow model and the historical state information of the power grid system, with the optimization goals of minimizing the power generation cost and energy loss and maximizing the active output of new energy, to solve the corresponding optimization problem and generate the optimal dispatching action that meets the power grid operation constraints at the current moment; The objective function of the optimal power flow model is expressed as: ； In the formula, Indicates the number of thermal power units, Indicates the number of new energy units, represents the cost function of the thermal power unit, represents the wind curtailment rate function of new energy units, Indicates the number The active power output of thermal power units, Indicates the number of the thermal power unit. Indicates the number The active power output of the new energy units, Indicates the number of the new energy unit; The cost function of the thermal power unit is expressed as: ； In the formula, represents the cost function of the thermal power unit, , and Represents thermal power unit Cost coefficient of The wind abandonment rate function of the new energy unit is expressed as: ； In the formula, represents the wind curtailment rate function of new energy units, Indicates the number The maximum active output of the new energy unit.

5. The method for intelligent dispatching for strengthening power grid security according to claim 4 is characterized in that: During the training process, the scheduling strategy agent optimizes the scheduling strategy based on the SAC algorithm, and the corresponding loss function is expressed as: ； In the formula, represents the loss function, represents the mathematical expectation of the policy network Q function, express The current state of the power grid, express Scheduling actions at the moment, represents the policy function, express The action value function at the moment, represents the tuning parameter of entropy, express In Status The entropy under Indicates in status Next, strategy Probability distribution of actions; The strategy of the scheduling strategy agent is updated by gradient descent, and the corresponding update rule is expressed as: ； In the formula, represents the network parameters of the target scheduling strategy agent, represents the learning rate, surface shows the gradient of the loss function; Wherein, the action value function is expressed as: ； In the formula, express The action value function at the moment, It indicates that the expected value of the next state is calculated based on the current state and action. express The current state of the power grid, represents the state transition probability distribution, Represents the current reward value, represents the discount factor, express The action value function at the moment, express Scheduling actions at the moment, express The maximum action value function at the moment; The calculation formula of the reward value is as follows: ； In the formula, Represents the scoring result of the scheduling strategy agent.

6. The intelligent dispatching method for strengthening power grid security according to claim 5 is characterized in that: The scoring function used by the first discriminator in the process of strategy comparison and evaluation is expressed as: ； In the formula, Indicates the rating results. represents the Sigmoid activation function, and Represent different weight matrices, represents the rectified linear unit activation function, express The corresponding action under the state, and They represent different biases respectively; The loss function corresponding to the first discriminator is expressed as: ； In the formula, represents the loss function, It indicates the expected value calculation of the optimal action and the corresponding state combination. Indicates the expected value calculation for common actions and corresponding state combinations. represents the optimal scheduling action, Indicates a normal scheduling action. Represents the scoring result of the optimal action generator.

7. The method for intelligent dispatching for strengthening power grid security according to claim 6, characterized in that: The state-action data pair pool formed by the scheduling strategy agent in the training phase determines whether the real-time scheduling action in the current state is an expert-level action, and then outputs the final scheduling strategy to the real environment according to the judgment result, including: Acquire historical state information in the state-action data pair pool that meets a preset state distance with the real-time state information, and a historical optimal scheduling action corresponding to the historical state information; By evaluating the similarity value between the real-time scheduling action and the historical optimal scheduling action, determining whether the real-time scheduling action in the current state is an expert-level action; If so, the real-time scheduling action is output to the real environment as the final scheduling strategy; if not, the real-time status information is input to the optimal action generator, and the optimal scheduling action generated by the optimal action generator is output to the real environment as the final scheduling strategy.

8. The method for intelligent dispatching for strengthening power grid security according to claim 7, characterized in that: The second discriminator is used to determine whether the real-time scheduling action in the current state is an expert-level action. If the similarity value is higher than a preset threshold, the real-time scheduling action in the current state is considered to be an expert-level action. The calculation formula of the similarity value is as follows: ； In the formula, Represents the similarity value, Indicates the historical optimal scheduling action corresponding to the historical state information close to the real-time state information, Indicates real-time scheduling actions; The loss function corresponding to the second discriminator is expressed as: ； In the formula, represents the loss function of the second discriminator, Indicates the security risk of the current action. surface Show in status Next, action Safety assessment of The second discriminator inherits the evaluation criteria and weight parameters learned by the first discriminator in the training phase through a parameter transfer mechanism, which is used for rapid convergence in the decision phase.

9. A power grid security enhancement intelligent dispatching device, characterized in that: include: The data acquisition module is configured to acquire the current real-time status information of the power grid system; The scheduling strategy preliminary determination module is configured to input the real-time status information into a pre-trained scheduling strategy agent to generate corresponding real-time scheduling actions; The scheduling strategy finalization module is configured to determine whether the real-time scheduling action in the current state is an expert-level action based on the state-action data pair pool formed by the scheduling strategy agent in the training phase, and then output the final scheduling strategy to the real environment according to the judgment result.

10. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the power grid security enhancement intelligent dispatching method as described in any one of claims 1 to 8 are implemented.

Images