A reinforcement learning method and system for real-time control security of distribution networks based on quadratic programming

Abstract

A reinforcement learning method for real-time control security of distribution networks based on quadratic programming includes the following steps: Offline training of the DRL agent: Constructing a DRL agent using the Actor-Critic algorithm architecture, modeling the distribution network optimization control problem as a Markov decision process, defining the state space, action space, and multi-objective reward function, and training the DRL agent for distribution network optimization control through multiple rounds of exploration and learning in a simulation environment; Online execution and immediate safety correction. Through rapid checking supported by a physical model and quadratic programming correction, provable mathematical safety guarantees are provided for each online control action, fundamentally solving the core problem that standard DRL algorithms cannot inherently guarantee that actions meet the physical safety constraints of the power grid. This method, while ensuring the decision-making speed and policy autonomy of the DRL agent, generates reliable and safe actions in real time, achieving computationally efficient and automatically intervened real-time safety control of the distribution network.

Description

Technical Field

[0001] This invention relates to the field of photovoltaic power generation technology, and in particular to a reinforcement learning method and system for real-time control security of distribution networks based on quadratic programming. Background Technology

[0002] As renewable energy penetration in power distribution networks increases, the power grid's operating mode is shifting from a traditional passive mode to an active mode. While this shift brings flexibility and economic benefits, it also introduces serious challenges caused by bidirectional power flow and high uncertainty, especially the safety issue of voltage exceeding limits, which seriously threatens the stable operation of the power grid.

[0003] Deep Reinforcement Learning (DRL), as an intelligent control technique that does not require a precise system model, has shown great potential in the field of distribution network optimization control. DRL agents, through trial and error with the power grid environment, can learn complex, nonlinear control strategies end-to-end to optimize system operation (e.g., minimizing network losses or maximizing local absorption). Typical DRL algorithms, such as the Actor-Critic method, consist of an "actor" network that outputs actions and a "critic" network that evaluates the value of those actions.

[0004] However, in pursuing long-term reward optimization, the standard DRL algorithm cannot inherently guarantee that every action of its output strictly meets the physical safety constraints of the power grid. This has become the biggest obstacle for DRL technology to move from simulation to practical application. Summary of the Invention

[0005] To address the aforementioned shortcomings, the present invention aims to propose a reinforcement learning method and system for real-time control security of power distribution networks based on quadratic programming.

[0006] To achieve this objective, the present invention adopts the following technical solution: a reinforcement learning method for real-time control security of distribution networks based on quadratic programming, comprising the following steps: Offline training of DRL agents: Construct a DRL agent with an Actor-Critic algorithm architecture, model the distribution network optimization control problem as a Markov decision process, define the state space, action space and multi-objective reward function, and train the DRL agent for distribution network optimization control through multiple rounds of exploration and learning in a simulation environment; Online execution and instant security correction: Step A: Generate raw actions: The DRL agent outputs raw control actions based on the real-time operating status of the power distribution network; Step B: Rapid Check Based on Safety Margin: The rapid check module assesses the safety risks of the original control actions with low computational cost, specifically including: The power flow sensitivity matrix is ​​used to linearly predict the voltage of the distribution network system after the execution of the original control action. The power flow sensitivity matrix is ​​the degree of response of the node voltage to the power injection change of the PV inverter, which is obtained by inverting the grid Jacobian matrix near the current operating point. The predicted node voltage is compared with a preset internal safety margin boundary, which is a boundary obtained by setting a safety margin based on the physical safety boundary of the distribution network. If all predicted node voltages are within the internal safety margin boundary, the original control action is deemed safe and is directly adopted and executed; if any predicted node voltage exceeds the internal safety margin boundary, the original control action is deemed to have a potential safety risk and action correction is initiated. Action execution: The adopted original control action or the modified control action is sent to the PV inverter for execution.

[0007] Preferably, the action correction in step B includes: constructing and solving a quadratic programming (QP) problem to obtain the corrected control action under the premise of satisfying the physical hard constraints of the distribution network; The objective function of the quadratic programming (QP) problem is to minimize the deviation between the corrected action and the original action. The objective function is constructed using the L2 norm, and the constraints include the upper and lower limits of the reactive power output of the PV inverter and the physical safety constraints of the distribution network node voltage.

[0008] The preferred objective function expression for the quadratic programming (QP) problem is: ; in For the corrected action, This is the original action; The constraints of the objective function of the quadratic programming (QP) problem are: ; ; in Let be the lower limit and upper limit of the reactive power that the k-th PV can generate at time t, respectively. These are the lower and upper limits of node voltage safety, respectively. Let be the voltage amplitude at time t of the i-th node. The degree of response of the node voltage to changes in PV inverter power injection is defined by the action after correction at time t and the action after correction at time t-1.

[0009] Preferably, the expression for the multi-objective reward function is as follows: ;in This is a penalty item for voltage exceeding the limit. Network damage penalties These are the weighting coefficients; ; ; Where N is the total number of nodes. For the branch at time t Active power loss.

[0010] The preferred formula for linear voltage prediction is expressed as follows: .

[0011] Preferably, the lower limit of the internal safety margin boundary is the node voltage safety lower limit plus the safety margin, and the upper limit of the internal safety margin boundary is the node voltage safety upper limit minus the safety margin.

[0012] A reinforcement learning system for real-time control security of distribution networks based on quadratic programming, wherein the reinforcement learning method for real-time control security of distribution networks based on quadratic programming includes: The DRL agent training module is used for offline training of DRL agents using the Actor-Critic algorithm architecture. It completes the definition of the state space, action space and multi-objective reward function of the Markov decision process, and obtains the DRL agent for distribution network optimization control through multi-round exploration and learning in the simulation environment. The instant safety correction module, which communicates with the DRL agent training module, includes a rapid inspection unit and an action correction unit. The rapid inspection unit is used to quickly detect safety risks in the original control actions output by the DRL agent based on the real-time status of the distribution network, realize voltage linear prediction through the power flow sensitivity matrix, and determine whether to initiate action correction after comparing with the internal safety margin boundary. The action correction unit is used to construct and solve a quadratic programming QP problem when the rapid inspection unit determines that the original control action has potential safety risks, and make minimum corrections to the original control action to obtain a corrected action that satisfies the physical hard constraints. The action execution module communicates with the real-time safety correction module and is used to send the final control action output by the real-time safety correction module to the PV inverter in the distribution network for execution.

[0013] One of the above technical solutions has the following advantages or beneficial effects: Through rapid checking and secondary planning correction supported by physical models, a provable mathematical safety guarantee is provided for each online control action, fundamentally solving the core problem that the standard DRL algorithm cannot inherently guarantee that the action meets the physical safety constraints of the power grid. While ensuring the decision-making speed and policy autonomy of the DRL agent, this method generates reliable and safe actions in real time, realizing computationally efficient and automatically intervened real-time safety control of the distribution network. Attached Figure Description

[0014] Figure 1 One example of the method of the present invention is a flowchart in one embodiment; Figure 2 The present invention is illustrated in one embodiment of the system. Detailed Implementation

[0015] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0016] In the description of embodiments of the present invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0017] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0018] like Figures 1-2 As shown, a reinforcement learning method for real-time control security of distribution networks based on quadratic programming includes the following steps: Offline Training of the DRL Agent: A DRL agent using the Actor-Critic algorithm architecture is constructed, modeling the distribution network optimization control problem as a Markov decision process. A state space, action space, and multi-objective reward function are defined. Through multiple rounds of exploration and learning in a simulation environment, the DRL agent for distribution network optimization control is trained. The state space includes the bus voltage amplitude, load active power, load reactive power, photovoltaic active power output, and photovoltaic reactive power output of each node in the distribution network. The action space represents the reactive power output of the photovoltaic inverter. The multi-objective reward function includes voltage limit violation penalties and network loss penalties, used to guide the DRL agent in learning a long-term control strategy that balances safety and economy.

[0019] Online execution and instant security correction: Step A: Generate raw actions: The DRL agent outputs raw control actions based on the real-time operating status of the power distribution network; Step B: Rapid Check Based on Safety Margin: The rapid check module assesses the safety risks of the original control actions with low computational cost, specifically including: The power flow sensitivity matrix is ​​used to linearly predict the voltage of the distribution network system after the execution of the original control action. The power flow sensitivity matrix is ​​the degree of response of the node voltage to the power injection change of the PV inverter, which is obtained by inverting the grid Jacobian matrix near the current operating point. The predicted node voltage is compared with a preset internal safety margin boundary, which is a boundary obtained by setting a safety margin based on the physical safety boundary of the distribution network. If all predicted node voltages are within the internal safety margin boundary, the original control action is deemed safe and is directly adopted and executed; if any predicted node voltage exceeds the internal safety margin boundary, the original control action is deemed to have a potential safety risk and action correction is initiated. Action execution: The adopted original control action or the modified control action is sent to the PV inverter for execution.

[0020] This invention proposes a quadratic programming-based reinforcement learning method for real-time control security in distribution networks. It constructs a DRL agent with an Actor-Critic algorithm architecture and completes offline training. Simultaneously, it designs an instantaneous safety correction module, including a fast check module based on safety margin and an action correction module based on quadratic programming, to achieve online execution and instantaneous safety correction. Addressing the issues of existing technologies where DRL algorithms cannot guarantee actions meet grid physical safety constraints and where existing DNN-MILP architectures suffer from high computational complexity and unnecessary continuous intervention, this invention places the high-complexity DRL agent learning process in the offline training phase. During online execution, the DRL agent first outputs the original control action. Then, the fast check module uses the power flow sensitivity matrix to linearly predict the system voltage after the action. Combined with an internal safety margin boundary that is more stringent than the physical boundary, it quickly judges the safety risk of the action. Only when the predicted value exceeds the margin boundary is the quadratic programming-based action correction module activated, performing minimal correction to the original action while satisfying the hard physical constraints of the grid. In other operating conditions, the original actions of the DRL agent are directly adopted. This implementation method avoids the high computational overhead of solving large-scale MILP problems at every decision time step in existing technologies. The complex online optimization process is only initiated when there is a potential safety risk, which greatly reduces the amount of online computation and meets the performance requirements of real-time control of the distribution network. At the same time, it abandons the complete replacement of Actor network decision-making in existing technologies and realizes automatic intervention only when potential risks are detected. It gives full play to the advantages of the DRL agent as a fast decision-maker and retains the control strategy learned from long-term reward optimization. In addition, through fast checking and quadratic programming correction supported by physical models, provable mathematical safety guarantees are provided for each online control action. This fundamentally solves the core problem that the standard DRL algorithm cannot inherently guarantee that the action meets the physical safety constraints of the power grid. While ensuring the decision-making speed and policy autonomy of the DRL agent, this method generates reliable and safe actions in real time, realizing computationally efficient and automatically intervened real-time safety control of the distribution network.

[0021] Preferably, the action correction in step B includes: constructing and solving a quadratic programming (QP) problem to obtain the corrected control action under the premise of satisfying the physical hard constraints of the distribution network; The objective function of the quadratic programming (QP) problem is to minimize the deviation between the corrected action and the original action. The objective function is constructed using the L2 norm, and the constraints include the upper and lower limits of the reactive power output of the PV inverter and the physical safety constraints of the distribution network node voltage.

[0022] The preferred objective function expression for the quadratic programming (QP) problem is: ; in For the corrected action, This is the original action; The constraints of the objective function of the quadratic programming (QP) problem are: ; ; in Let be the lower limit and upper limit of the reactive power that the k-th PV can generate at time t, respectively. These are the lower and upper limits of node voltage safety, respectively. Let be the voltage amplitude at time t of the i-th node. The degree of response of the node voltage to changes in PV inverter power injection is defined by the action after correction at time t and the action after correction at time t-1.

[0023] To address the issues of existing technologies, such as the inability of standard DRL algorithms to guarantee that actions meet power grid physical safety constraints, and the high computational complexity and poor real-time performance of existing DNN-MILP architectures, this paper proposes a solution by constructing a quadratic programming QP problem in the action correction stage. This solution uses the L2 norm as the objective function and incorporates the upper and lower limits of reactive power output from PV inverters and the voltage physical safety constraints of distribution network nodes. When a potential safety risk is detected in the original action output by the DRL agent, this solution constructs the QP objective function with the goal of minimizing the deviation between the corrected action and the original action. Simultaneously, it integrates the equipment operation constraints of PV inverters and the voltage physical safety constraints of the distribution network core as hard constraints into the QP problem solution process. Compared to existing technologies that transform deep neural networks into large-scale MILP problems and solve them at every decision step, quadratic programming, as a convex optimization problem, is far less difficult to solve than NP-hard MILP problems. It offers faster solution speeds, lower computational overhead, and enables real-time control of the distribution network. The algorithm completes action correction within the specified time requirement, balancing computational efficiency and real-time performance. Furthermore, by setting hard constraints, it ensures that the corrected action strictly meets the physical safety constraints of the power grid and the operational constraints of the equipment. This fundamentally solves the core problem that the standard DRL algorithm cannot inherently guarantee action safety. Moreover, this QP problem is only solved when there is a potential risk in the original action, rather than being continuously executed at every decision time step, avoiding unnecessary complex calculations. In addition, the objective function constructed using the L2 norm achieves minimal correction to the original action, maximizing the preservation of the control strategy learned by the DRL agent from long-term reward optimization, and fully leveraging the advantages of the DRL agent as a fast decision-maker. Its beneficial effects are reflected in providing strict mathematical safety guarantees for real-time control actions in the distribution network, eliminating safety risks such as voltage exceeding limits, significantly reducing the computational complexity of online correction, meeting the millisecond-level real-time control performance requirements of the distribution network, and protecting the integrity and economy of the DRL agent strategy, avoiding unnecessary performance losses.

[0024] Preferably, the expression for the multi-objective reward function is as follows: ;in This is a penalty item for voltage exceeding the limit. Network damage penalties These are the weighting coefficients; ; ; Where N is the total number of nodes. For the branch at time t Active power loss.

[0025] The preferred formula for linear voltage prediction is expressed as follows: .

[0026] Preferably, the lower limit of the internal safety margin boundary is the node voltage safety lower limit plus the safety margin, and the upper limit of the internal safety margin boundary is the node voltage safety upper limit minus the safety margin.

[0027] A reinforcement learning system for real-time control security of distribution networks based on quadratic programming, wherein the reinforcement learning method for real-time control security of distribution networks based on quadratic programming includes: The DRL agent training module is used for offline training of DRL agents using the Actor-Critic algorithm architecture. It completes the definition of the state space, action space and multi-objective reward function of the Markov decision process, and obtains the DRL agent for distribution network optimization control through multi-round exploration and learning in the simulation environment. The instant safety correction module, which communicates with the DRL agent training module, includes a rapid inspection unit and an action correction unit. The rapid inspection unit is used to quickly detect safety risks in the original control actions output by the DRL agent based on the real-time status of the distribution network, realize voltage linear prediction through the power flow sensitivity matrix, and determine whether to initiate action correction after comparing with the internal safety margin boundary. The action correction unit is used to construct and solve a quadratic programming QP problem when the rapid inspection unit determines that the original control action has potential safety risks, and make minimum corrections to the original control action to obtain a corrected action that satisfies the physical hard constraints. The action execution module communicates with the real-time safety correction module and is used to send the final control action output by the real-time safety correction module to the PV inverter in the distribution network for execution.

[0028] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0029] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A reinforcement learning method for real-time control security of distribution networks based on quadratic programming, characterized in that, Includes the following steps: Offline training of DRL agents: Construct a DRL agent with an Actor-Critic algorithm architecture, model the distribution network optimization control problem as a Markov decision process, define the state space, action space and multi-objective reward function, and train the DRL agent for distribution network optimization control through multiple rounds of exploration and learning in a simulation environment; Online execution and instant security correction: Step A: Generate raw actions: The DRL agent outputs raw control actions based on the real-time operating status of the power distribution network; Step B: Rapid Check Based on Safety Margin: The rapid check module assesses the safety risks of the original control actions with low computational cost, specifically including: The power flow sensitivity matrix is ​​used to linearly predict the voltage of the distribution network system after the execution of the original control action. The power flow sensitivity matrix is ​​the degree of response of the node voltage to the power injection change of the PV inverter, which is obtained by inverting the grid Jacobian matrix near the current operating point. The predicted node voltage is compared with a preset internal safety margin boundary, which is a boundary obtained by setting a safety margin based on the physical safety boundary of the distribution network. If all predicted node voltages are within the internal safety margin boundary, the original control action is deemed safe and is directly adopted and executed; if any predicted node voltage exceeds the internal safety margin boundary, the original control action is deemed to have a potential safety risk and action correction is initiated. Action execution: The adopted original control action or the modified control action is sent to the PV inverter for execution.

2. The reinforcement learning method for real-time control security of distribution networks based on quadratic programming according to claim 1, characterized in that, The action correction in step B includes: constructing and solving a quadratic programming QP problem to obtain the corrected control action under the premise of satisfying the physical hard constraints of the distribution network; The objective function of the quadratic programming (QP) problem is to minimize the deviation between the corrected action and the original action. The objective function is constructed using the L2 norm, and the constraints include the upper and lower limits of the reactive power output of the PV inverter and the physical safety constraints of the distribution network node voltage.

3. The reinforcement learning method for real-time control security of distribution networks based on quadratic programming according to claim 2, characterized in that, The objective function expression for the quadratic programming (QP) problem is: ； in For the corrected action, This is the original action; The constraints of the objective function of the quadratic programming (QP) problem are: ； ； in Let be the lower limit and upper limit of the reactive power that the k-th PV can generate at time t, respectively. These are the lower and upper limits of node voltage safety, respectively. Let be the voltage amplitude at time t of the i-th node. The degree of response of the node voltage to changes in PV inverter power injection is defined by the action after correction at time t and the action after correction at time t-1.

4. The reinforcement learning method for real-time control security of distribution networks based on quadratic programming according to claim 1, characterized in that, The expression for the multi-objective reward function is as follows: ;in This is a penalty item for voltage exceeding the limit. Network damage penalties These are the weighting coefficients; ； ； Where N is the total number of nodes. For the branch at time t Active power loss.

5. The reinforcement learning method for real-time control security of distribution networks based on quadratic programming according to claim 1, characterized in that, The formula for linear voltage prediction is expressed as follows: 。 6. The reinforcement learning method for real-time control security of distribution networks based on quadratic programming according to claim 1, characterized in that, The lower limit of the internal safety margin boundary is the node voltage safety lower limit plus the safety margin, and the upper limit of the internal safety margin boundary is the node voltage safety upper limit minus the safety margin.

7. A reinforcement learning system for real-time control security of distribution networks based on quadratic programming, using the reinforcement learning method for real-time control security of distribution networks based on quadratic programming as described in any one of claims 1 to 6, characterized in that, include: The DRL agent training module is used for offline training of DRL agents using the Actor-Critic algorithm architecture. It completes the definition of the state space, action space and multi-objective reward function of the Markov decision process, and obtains the DRL agent for distribution network optimization control through multi-round exploration and learning in the simulation environment. The instant safety correction module, which communicates with the DRL agent training module, includes a rapid inspection unit and an action correction unit. The rapid inspection unit is used to quickly detect safety risks in the original control actions output by the DRL agent based on the real-time status of the distribution network, realize voltage linear prediction through the power flow sensitivity matrix, and determine whether to initiate action correction after comparing with the internal safety margin boundary. The action correction unit is used to construct and solve a quadratic programming QP problem when the rapid inspection unit determines that the original control action has potential safety risks, and make minimum corrections to the original control action to obtain a corrected action that satisfies the physical hard constraints. The action execution module communicates with the real-time safety correction module and is used to send the final control action output by the real-time safety correction module to the PV inverter in the distribution network for execution.

Images