Active Distribution Network Cooperative Optimization Method and System Based on Multi-Agent Reinforcement Learning

Abstract

This invention provides a multi-agent reinforcement learning-based active distribution network collaborative optimization method and system, belonging to the field of active distribution network operation optimization and intelligent control. It constructs a globally linearized voltage sensitivity model; treats each distributed resource as an agent, inputs the local observation vector of each agent into the policy network to obtain the original action, and calculates the actual node voltage; verifies whether the actual node voltage meets the voltage safety constraints; if so, the original action is used as the execution instruction; otherwise, proceeds to step 4; step 4: online correction, jointly solving the active power injection adjustment, reactive power injection adjustment, and node voltage relaxation variables to obtain candidate safe actions; step 5: power flow verification, updating the linearized error compensation term; if the actual node voltage does not meet the voltage safety constraints, substitute the updated linearized error compensation term into step 4 to solve the action correction amount, iteratively executing steps 4 and 5; improving the prediction accuracy of the safety projection layer and the reliability of action correction.

Description

Technical Field

[0001] This invention relates to the field of active distribution network operation optimization and intelligent control technology, and in particular to an active distribution network collaborative optimization method and system based on multi-agent reinforcement learning. Background Technology

[0002] With the large-scale grid connection of high proportions of renewable energy, the operating characteristics of active distribution networks are becoming increasingly complex, facing multiple challenges such as voltage exceeding limits, power fluctuations, and reduced economic efficiency. Distributed resource collaborative optimization is a key means to improve the resilience of distribution network operation, promote the consumption of renewable energy, and achieve multi-objective collaborative optimization.

[0003] Traditional power system optimization methods are mainly based on mathematical programming models, such as mixed-integer linear programming, stochastic programming, or robust optimization. While these methods can obtain exact solutions in deterministic or finite uncertainty scenarios, their performance heavily depends on accurate system model parameters and prior probability distribution assumptions about the uncertainties. Faced with high-dimensional optimization problems with strong uncertainties on both the source and load sides, traditional methods often suffer from heavy computational burdens and struggle to handle complex nonlinear constraints and trade-offs between multiple objectives.

[0004] In recent years, data-driven methods, represented by deep reinforcement learning, have shown great potential in the field of power system optimization and scheduling due to their powerful sequential decision-making and high-dimensional space processing capabilities. Existing research has applied reinforcement learning to real-time scheduling and control of power systems, but these studies still have significant shortcomings in ensuring the real-time satisfaction of hard safety constraints of the power grid (such as node voltage limits). Current mainstream solutions typically only add a penalty term to the reward function, which is a "soft constraint" approach and cannot verify the physical feasibility of actions before each decision is executed. When reinforcement learning agents explore high-dimensional action spaces, they are prone to generating invalid actions that lead to voltage exceeding limits or even power flow non-convergence. This not only reduces training efficiency but also introduces safety risks in practical deployments.

[0005] Specifically, existing technologies have the following shortcomings: First, existing reinforcement learning-based optimization methods lack a verifiable safe execution layer. Most of them transform safety constraints into penalty terms in the reward signal, which is a post-hoc punishment rather than a pre-hoc guarantee. This means that the feasibility cannot be verified through a physical model before the action is executed, posing a safety hazard. More importantly, such methods cannot assess the potential physical safety risks (i.e., counterfactual risks) inherent in the agent's original decision actions, causing the agent to learn through trial and error in the long term, resulting in low training efficiency and difficulty in developing autonomous safety awareness.

[0006] Second, when simultaneously optimizing multiple control variables (active and reactive power), the action space exhibits high-dimensional, continuous, and hybrid characteristics, and is coupled with the strongly nonlinear power flow equations of the power grid. Existing methods lack an efficient action projection mechanism that can automatically and in real time correct the agent's economic optimization intentions into feasible instructions that satisfy all power grid security constraints. In particular, there is a lack of a safe execution mechanism that can perform multi-round iterative correction and deviation compensation based on the physical model, making it difficult to cope with the linearization model errors caused by complex nonlinear power flows.

[0007] Third, power system optimization is essentially a multi-objective optimization problem, requiring consideration of multiple dimensions such as voltage stability, network losses, operational economics, and renewable energy consumption. Existing methods typically use linear weighting to transform multiple objectives into a single objective, making it difficult to obtain a balanced Pareto optimal solution set. Furthermore, the weighting settings rely on experience. How can we decouple security constraints from economic objectives to alleviate the difficulty in balancing security penalties and economic goals in traditional single reward functions?

[0008] Fourth, in distributed resource collaboration scenarios, how to design a multi-agent framework to coordinate local decisions with the global optimization goal of the power distribution network, avoid suboptimal solutions, and at the same time ensure the scalability and computational efficiency of the algorithm is a challenge faced in practical engineering applications.

[0009] Among related technologies, the paper "Active Distribution Network Voltage Regulation Strategy Based on Multi-Agent Deep Reinforcement Learning" (Huang Yu et al., Institute of Carbon Neutrality Advanced Technology, Nanjing University of Posts and Telecommunications, *Automation of Electric Power Systems*, 2025, 49(18): 65-73) studies an active distribution network voltage collaborative regulation method based on multi-agent deep reinforcement learning, providing a data-driven approach for distributed collaborative decision-making of multiple controllable resources. However, this type of method still needs to further address the issues of physical feasibility verification before the execution of reinforcement learning actions and explicit guarantees of node voltage hard constraints. The paper "A Novel Two-Stage Robust Configuration Optimization Model for Flexible Resources Considering Uncertainty" (Zhu Haijun et al., School of Economics and Management, North China Electric Power University, *Electric Power Construction*, 2024, 45(7): 1-11) uses a two-stage robust optimization method to handle the uncertainty in flexible resource configuration, which can improve the adaptability of configuration schemes to uncertain operating scenarios. However, this type of method relies on explicit models and optimization solutions, and may face a large computational burden when facing high-dimensional and rapidly changing online control scenarios. Therefore, there is an urgent need for an active distribution network control method that can integrate data-driven decision-making with physical model constraints, while taking into account collaborative optimization, action executability, and operational safety. Summary of the Invention

[0010] The technical problem to be solved by this invention is: how to solve the problems that existing active distribution network optimization methods are highly dependent on accurate models, have difficulty in handling high-dimensional continuous action spaces and multi-objective collaborative optimization problems, and that existing reinforcement learning methods cannot verifiably guarantee grid security constraints such as node voltages before decision execution.

[0011] This invention solves the above-mentioned technical problems through the following technical solution: a multi-agent reinforcement learning-based active distribution network cooperative optimization method, comprising:

[0012] Step 1: Based on the active distribution network system of distributed resource collaboration, establish the operation model of each distributed resource and the system constraints, and construct a global linearized voltage sensitivity model; Step 2: Treat each distributed resource as an independent intelligent agent and divide the active distribution network node into multiple non-overlapping regions. Obtain the local observation vector of each intelligent agent in its respective region, input the local observation vector of each intelligent agent into the corresponding policy network, and output the original action. Step 3: Calculate the actual node voltage based on the original action and verify whether the actual node voltage meets the voltage safety constraints. If it does, use the original action as the final execution instruction; otherwise, proceed to step 4. Step 4: The original action is corrected online through the global linearized safety projection layer. A convex optimization problem is constructed with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint to participate in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved together to obtain candidate safety actions. Step 5: Perform real power flow verification based on candidate safety actions, and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, output the candidate safety action as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, substitute the updated linearization error compensation term back into the convex optimization problem in step 4, resolve the action correction amount, and iteratively execute steps 4 and 5 until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

[0013] This invention first constructs a physical model for multi-objective scheduling of an active distribution network and a globally linearized voltage sensitivity model based on network topology. Then, it formalizes the real-time scheduling problem of the active distribution network into a multi-agent sequential decision-making problem. Finally, through a closed-loop control process of "multi-agent decision-making—real power flow verification—safety projection correction—real power flow review—error compensation iteration," it achieves collaborative optimization scheduling of distributed resources. The upper-layer multi-agent reinforcement learning policy network is responsible for generating original actions that reflect economic and collaborative optimization intentions. The middle-layer safety projection layer corrects potentially unsafe original actions into candidate safe actions based on the linearized voltage sensitivity model. The bottom layer reviews the candidate safe actions through real power flow calculations and updates the linearized error compensation term based on the deviation between the real node voltage and the linearized predicted node voltage, thereby improving the prediction accuracy and action correction reliability of the safety projection layer.

[0014] This invention innovatively proposes a hybrid driving paradigm of "data-driven decision-making—physical model verification—safety projection correction." This method not only utilizes multi-agent reinforcement learning to learn economic optimization strategies from high-dimensional, uncertain operational data, but also ensures that control commands meet the hard safety constraints of the power grid through real power flow verification and a safety projection layer before action execution. This overcomes the limitations of traditional reinforcement learning methods that only approximate safety constraints through rewards and penalties, achieving a shift from "soft constraint learning" to "hard constraint guarantee."

[0015] Preferably, the distributed resources include dispatchable thermal power units, photovoltaic power generation units, wind power generation units, distributed energy storage systems, and flexible loads. The operation models for each distributed resource include: dispatchable thermal power units simultaneously satisfying active power output constraints and power factor constraints; for photovoltaic power generation units or wind power generation units, under a given active power output, the reactive power output of the photovoltaic power generation unit or wind power generation unit is controlled by adjusting the inverter power coefficient, and the operation model reflects the reactive power regulation capability and operating range; for distributed energy storage systems, a dynamic equation for energy storage based on the state of charge is established, and charging and discharging efficiency and self-discharge rate are considered; for flexible loads, a model for flexible loads participating in demand response is established, characterizing the range of proportions that flexible loads can be reduced or transferred. System constraints include: node power balance constraints, upper and lower limits of active power output of adjustable thermal power units, upper and lower limits of power factor of adjustable thermal power units, upper and lower limits of reactive power coefficient of inverters connected to photovoltaic or wind power generation units, upper and lower limits of output of distributed energy storage systems, upper and lower limits of state of charge of distributed energy storage systems, upper and lower limits of flexible load reduction or transfer coefficients, node voltage constraints, branch current constraints, and substation capacity constraints.

[0016] Preferably, the process of constructing a globally linearized voltage sensitivity model includes: Calculate the reference impedance based on the reference voltage and reference power of the active distribution network system; The topology of the active distribution network system is analyzed using the shortest path algorithm in graph theory. The resistance sensitivity matrix and reactance sensitivity matrix of the node pairs are calculated and then converted into per-unit values. Establish a linear approximation relationship between the node injection power increment and the linearized predicted node voltage.

[0017] Preferably, the linear approximation relationship between the node injection power increment and the linearized predicted node voltage is as follows:

[0018]

[0019] in, For a moment Linearized prediction of node voltage, To balance the node voltage, For a moment The active target adjustment amount, For a moment The reactive power target adjustment amount, , Representing time respectively Active load vector and reactive load vector at nodes , These represent the active power injection vector and reactive power injection vector that have been actually applied to each controllable resource, respectively. , These are the active resource-node mapping matrix and the reactive resource-node mapping matrix, respectively. For linearization error compensation term, It is a column vector of all 1s. Represents the resistance sensitivity matrix per-unit value, Represents the reactance sensitivity matrix The per-unit value.

[0020] The preferred method for calculating convex optimization problems is as follows:

[0021]

[0022]

[0023] in, for The amount of active power injection adjustment at any given moment. for The amount of reactive power injection adjustment at any given moment. This is the lower limit relaxation variable for the node voltage. This is a relaxation variable for the upper limit of the node voltage. , , These represent the weighting coefficients for active power adjustment, reactive power adjustment, and voltage slack, respectively. This is a vector of all 1s, used to sum over slack variables; , These represent the lower and upper bound vectors of the node voltage, respectively. and They represent The active and reactive target adjustment quantities obtained from the mapping of the agent's original actions at any given time. For a moment Linearized prediction of node voltage.

[0024] This invention introduces a real-time security layer with iterative closed-loop correction capability. This security layer utilizes a topology-adaptive sensitivity model to transmit the agent's original actions through a... P - Q The joint convex optimization problem is projected in real time to the feasible region that satisfies all power grid security constraints.

[0025] Preferably, the process of updating the linearization error compensation term based on the deviation between the actual node voltage corresponding to the candidate safety action and the linearized predicted node voltage corresponding to the candidate safety action includes: Calculate the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. :

[0026] in, Indicates the first The actual node voltage in the next iteration. Indicates the first Linearized prediction of node voltage in the next iteration; According to the The deviation between the actual node voltage and the linearized predicted node voltage in the next iteration Update the linearization error compensation term:

[0027] in, This represents the updated linearization error compensation term. Indicates the first The linearization error compensation term used in the next iteration This is the deviation smoothing factor. .

[0028] Based on the updated linearization error compensation term, the linearized predicted node voltage is recalculated, and the safety projection optimization problem is resolved to improve the approximation accuracy of the linearized model to the actual power flow results.

[0029] Preferably, during the iterative execution of steps 4 and 5, one or more nodes with the greatest voltage exceedance are identified as a set of dangerous nodes based on the actual power flow calculation results. and dangerous nodes The upper and lower voltage limits are set with additional voltage safety margins. : like Then the lower voltage limit in the next safe projection of that node. Adjusted to ; like Then the upper limit of voltage in the next safe projection of that node. Adjusted to .

[0030] in, Indicates the first The actual node voltage in the next iteration. For voltage safety margin, and Indicates dangerous nodes The upper and lower voltage limits.

[0031] This invention constructs a real-time security projection layer with iterative closed-loop correction capability. It utilizes a voltage sensitivity model built offline based on network topology to project the original actions output by the agent into a feasible domain that satisfies all power grid security constraints in real time. Combined with real power flow verification and error compensation mechanisms, it improves the prediction accuracy and security correction capability of the linearized model under complex operating conditions.

[0032] Preferably, the training process of the policy network includes: From the experience replay buffer Randomly sample a batch of data ,in, Indicates the current state. Indicates a safety action. As a reward, For the new global state transitioned to by the environment, each agent will... Local observation vector at the next time step Input agent corresponding to the Actor target network , Represents the Actor target network The parameters are used to obtain the deterministic actions of the agents, and the deterministic actions of all agents constitute the target action; The target action and the new global state transferred by the environment are input into Critic Target Network 1 and Critic Target Network 2, respectively. The minimum value between the outputs of Critic Target Network 1 and Critic Target Network 2 is then used. Calculate the target value; Input the joint state-action pairs of all agents at the current moment into Critic Network 1 and Critic Network 2 respectively, and Critic Network 1 outputs the current value. The Critic network outputs the current value. The parameters of Critic Network 1 and Critic Network 2 are updated by minimizing the time difference error. Each intelligent agent Local observation vector at the current moment Input Actor Network , For Actor Network The parameters are used to obtain new actions. Based on new actions and the removal of intelligent agents Historical actions of other intelligent agents Constructing joint actions The current global state and joint actions Input Critic Network 1 and Critic Network 2, Critic Network 1 outputs the current value. The Critic network outputs the current value. Get the current value Current value minimum value Maximize the target value using gradient ascent method Update the parameters of the Actor network to obtain the policy network.

[0033] Preferred target value for:

[0034] in, for Global rewards gained at any time As a discount factor, The minimum value between the outputs of Critic target network one and Critic target network two. ,in, This indicates the new global state to which the environment has transitioned. This represents the deterministic action output by the Actor target network under the local observation vector at the next time step. , Indicates that the intelligent agent The local observation vector is input into the agent at the next time step. The corresponding Actor target network yields the deterministic actions of the agent. , This represents the output of Critic target network one. For the parameters of Critic target network one, This represents the output of the Critic target network 2. These are the parameters of Critic target network 2.

[0035] Preferred, Global rewards gained at each moment for:

[0036] in, For a moment Daily comprehensive operating cost For a moment The active network loss, This is to account for the potential risk of voltage exceeding limits after the original action is executed. , , These represent the cost weighting coefficient, network loss weighting coefficient, and counterfactual risk penalty coefficient, respectively; among them, the potential voltage over-limit risk after the original action is executed. for:

[0037] in, It is a set of nodes in a distributed resource-coordinated active distribution network system. To ensure the safety of the projection layer, a global linearized voltage sensitivity model is used to predict the voltage of the original action at the node. Voltage estimate, and These represent the lower and upper limits of the node voltage, respectively.

[0038] This invention pioneers a counterfactual risk guidance mechanism that incorporates the risk of the original action into the reward function, achieving an evolution from "dependence on security layer correction" to "autonomous agent safety," fundamentally solving the engineering risk of unsafe actions generated during reinforcement learning exploration.

[0039] This invention also provides an active distribution network cooperative optimization system based on multi-agent reinforcement learning, comprising: The model building module is used to establish the operation model of each distributed resource and the system constraints for an active distribution network system based on distributed resource collaboration, and to build a global linearized voltage sensitivity model. The agent decision-making module is used to treat each distributed resource as an independent agent, divide the active distribution network node into multiple non-overlapping regions, obtain the local observation vector of each agent in its own region, input the local observation vector of each agent into the corresponding policy network, and output the original action. The original action verification module is used to perform real power flow calculations based on the original actions to obtain the real node voltages, and to verify whether the real node voltages meet the voltage safety constraints. If they do, the original actions are directly used as the final execution instructions; otherwise, the module proceeds to the safety projection module. The safety projection module is used to correct the original actions online through a global linearized safety projection layer. It constructs a convex optimization problem with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved together to obtain candidate safety actions. The deviation compensation and iteration module is used to perform real power flow verification based on candidate safety actions and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, the candidate safety action is output as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, the updated linearization error compensation term is substituted back into the convex optimization problem in the safety projection module to solve the action correction amount again, and the safety projection module and the deviation compensation and iteration module are iteratively executed until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

[0040] The advantages provided by this invention are: 1. This invention achieves accuracy and efficiency in safety correction through refined and differentiated physical constraint modeling and incremental optimization mechanisms. The safety layer not only uniformly handles node voltage constraints but also models power factor constraints, inverter capacity circle constraints (using a second-order cone form), and active-reactive coupling relationships for different resource characteristics of DG, PV / WT, ESS, and FL. In particular, for energy storage systems, the safety layer jointly models their active power SOC dynamic constraints and apparent power capacity constraints, fully exploring their potential in voltage regulation. Incremental modeling avoids redundant state calculations and introduces a "dangerous node" identification and additional margin application mechanism to adaptively strengthen voltage control of weak links. Combined with an iterative closed loop of "projection-power flow verification-deviation compensation," linearization model errors are effectively compensated, ensuring the final reliability of the safety correction results.

[0041] 2. This invention significantly improves the convergence stability and practicality of the algorithm by constructing a training environment that integrates high-fidelity simulation (Pandapower) and a data-driven model, and by employing a stable multi-agent training algorithm. The introduction of a safety layer decouples complex physical constraint verification from the learning process, greatly reducing the invalid space of agent exploration and thus accelerating training convergence. The proposed architecture decouples decision-making from safety, making the trained policy more interpretable and providing confidence in engineering deployment. This offers an efficient, robust, and safe solution for the online intelligent optimization of large-scale active distribution networks. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of an active distribution network system with distributed resource collaboration in the active distribution network collaborative optimization method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention. Figure 2 This is a schematic diagram of the active distribution network cooperative optimization method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention. Figure 3 This is a flowchart of the active distribution network collaborative optimization method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention; Figure 4(a) is a graph showing the relationship between the reward value of the round reward and the number of training rounds in the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 4(b) shows the relationship between the loss value of the commentator loss and the number of training rounds in the simulation analysis 1 of the multi-agent reinforcement learning active distribution network cooperative optimization method provided in Embodiment 1 of the present invention. Figure 4(c) is a graph showing the relationship between the loss value of the actor loss and the number of training rounds in the simulation analysis 1 of the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 5(a) shows the relationship between the security layer utilization rate and the number of training rounds in simulation analysis 1 for the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 5(b) is a graph showing the relationship between the action difference value and the number of training rounds in simulation analysis 1 for the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 6(a) is a graph showing the relationship between the operating cost and the number of training rounds of the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention in simulation analysis 1. Figure 6(b) shows the relationship between system loss and training rounds in simulation analysis 1 for the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 6(c) is a graph showing the relationship between the number of voltage exceedances and the number of training rounds in Simulation Analysis 1 for the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 7(a) shows the relationship between the average node voltage and time of the active distribution network cooperative optimization method based on multi-agent reinforcement learning provided in Embodiment 1 of the present invention in Simulation Analysis 2. Figure 7(b) shows the relationship between active power loss and time in simulation analysis 2 of the active distribution network cooperative optimization method of multi-agent reinforcement learning provided in Embodiment 1 of the present invention. Figure 8(a) shows the relationship between electricity price fluctuation and system operating cost and time in simulation analysis 2 of the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 8(b) shows the relationship curves of energy storage agent power, SOC and time in simulation analysis 2 for the multi-agent reinforcement learning active distribution network collaborative optimization method provided in Embodiment 1 of the present invention. Figure 8(c) is a graph showing the relationship between the flexible load reduction coefficient and time in simulation analysis 2 of the active distribution network collaborative optimization method of multi-agent reinforcement learning provided in Embodiment 1 of the present invention. Figure 9(a) is a schematic diagram comparing the active power output of new energy sources and the active power output of traditional units in simulation analysis 2 of the active power optimization method of multi-agent reinforcement learning for distribution networks provided in Embodiment 1 of the present invention. Figure 9(b) is a schematic diagram showing the comparison between the main grid interactive power and the reactive power output of traditional generating units in simulation analysis 2 of the active distribution network collaborative optimization method of multi-agent reinforcement learning provided in Embodiment 1 of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0044] Example 1 See Figure 3 This embodiment provides a multi-agent reinforcement learning-based active distribution network collaborative optimization method, including the following steps: Step 1: Establish an active distribution network multi-objective scheduling model that considers the coordination of multiple types of distributed resources. Figure 1Based on the standard active distribution network topology IEEE 33-node system shown, a simulation test environment is established that includes a high proportion of distributed renewable energy (PV, wind power) and diverse loads. In this embodiment, the load demand of the nodes, renewable energy output, main grid electricity purchase price, output of distributed resources (dispatchable units, PV, wind turbines, energy storage, flexible loads) at the previous moment, and the state of charge (SOC) of energy storage are collected as state variables. The constructed active distribution network multi-objective scheduling model is based on multi-source collaborative complementarity. It not only considers the independent operation constraints of each distributed resource (dispatchable units, PV, wind turbines, energy storage, flexible loads), but also emphasizes the collaborative capabilities of each distributed resource in terms of power regulation and voltage support. At the same time, system power balance constraints, equipment output constraints, node voltage safety constraints, branch current constraints, and dynamic energy constraints of energy storage are set.

[0045] by Figure 1 Taking the active distribution network topology of the IEEE 33-node standard system as an example, the node set is defined. Set with branch roads , , , , , These represent the node sets to which the dispatchable thermal power unit (DG), photovoltaic power generation unit (PV), wind power generation unit (WT), distributed energy storage system (ESS), and slashable flexible load (FL) belong, respectively, with node numbers... . Figure 1 In this active distribution network topology IEEE 33-node system, dispatchable thermal power units DG1 and DG2 are connected at nodes 11 and 29 respectively; photovoltaic power generation units PV1, PV2, and PV3 are connected at nodes 19, 24, and 32 respectively; wind power generation units WT1, WT2, and WT3 are connected at nodes 5, 15, and 30 respectively; distributed energy storage system ESS1 is connected at node 18; and reducible flexible load FL1 is connected at node 10. Figure 1 Node 1) is connected to the main grid, and each device accesses the network through power electronic devices such as inverters, exhibiting bidirectional power flow characteristics. The system model and constraints are as follows: Step 1.1: Establish the physical model and constraints that the active distribution network topology IEEE 33-node system must meet for operation. The operation models for each distributed resource include: dispatchable thermal power units simultaneously satisfying active power output constraints and power factor constraints; for photovoltaic (PV) or wind power units, under a given active power output, the reactive power output of PV or wind power units is controlled by adjusting the inverter power coefficient, and the operation model reflects reactive power regulation capability and operating range; for distributed energy storage systems, establish energy storage dynamic equations based on state of charge (SOC) to describe its charging and discharging process, considering charging and discharging efficiency and self-discharge rate; for flexible loads, establish a model for flexible load participation in demand response, characterizing the range of proportions of flexible loads that can be reduced or transferred.

[0046] A set of physical and safety constraints that the system must satisfy for operation is defined. These constraints include: node power balance constraints (i.e., the sum of active and reactive power injected into each node must equal the sum of outflow power and losses), equipment operation constraints, system safety constraints, and substation capacity constraints. Equipment operation constraints include upper and lower limits on the output of adjustable thermal power units, renewable energy (photovoltaic power generation units, wind power generation units) inverters, and distributed energy storage systems, as well as power change rate limits. In this embodiment, equipment operation constraints include: upper and lower limits on the active power output of adjustable thermal power units, upper and lower limits on the power factor of adjustable thermal power units, upper and lower limits on the reactive power coefficient of inverters connected to photovoltaic or wind power generation units, upper and lower limits on the output of distributed energy storage systems, upper and lower limits on the state of charge of distributed energy storage systems, and upper and lower limits on the flexible load reduction or transfer coefficient. System safety constraints include node voltage constraints and branch current constraints. Node voltage constraints refer to the requirement that all node voltages must be maintained within the allowable safe range (e.g., ...). to Branch current constraints mean that the current in all branches must not exceed their current-carrying limits. Substation capacity constraints are used to limit the power purchased from the main grid.

[0047] In this embodiment, at any time For each node in the network The injected and outflowing power must be balanced for any node. The following active and reactive power balance equations must be satisfied:

[0048]

[0049] in, , Representing time respectively Mainnet Injection Node Active power and reactive power , Representing time respectively Thermal power unit injection node Active power and reactive power , Representing time respectively Photovoltaic unit injection node Active power and reactive power , Representing time respectively Fan unit injection node Active power and reactive power Indicates time Energy storage system injection node The active power (positive for discharging, negative for charging). , Representing nodes respectively At any moment Active load and reactive load , Representing nodes respectively ,node At any moment voltage amplitude, For nodes With nodes Branch road Complex admittance, For electrical conductivity, For susceptance, The imaginary unit, For nodes With nodes The voltage phase angle difference between them This represents the set of branches.

[0050] At the same time, the operation of all types of equipment must not exceed their physical limits, and the thermal power units can be dispatched ( It is necessary to simultaneously satisfy the active power output limit and power factor constraint, the specific constraints of which are as follows: Constraints on effort output:

[0051] Power factor constraint:

[0052] in, , Representing time respectively Thermal power unit injection node Active power and reactive power , Representing nodes respectively The maximum and minimum allowable power factor of the thermal power unit. For nodes The rated apparent capacity of the thermal power unit.

[0053] Photovoltaic power generation unit ( ) and wind power generation unit ( Given a positive power output, its reactive power output can be continuously adjusted within an allowable range by the inverter's power coefficient, with the specific constraints as follows:

[0054]

[0055] in, , Representing time respectively Photovoltaic unit injection node Active power and reactive power Active power output is determined by the environment (considered as input), while reactive power output is adjusted through the power coefficient. For nodes The apparent capacity of the inverter connected to the photovoltaic unit. For nodes The reactive power coefficient of the inverter connected to the photovoltaic unit. , These are the minimum and maximum values ​​of the reactive power coefficient of the inverter connected to the photovoltaic unit, respectively.

[0056]

[0057]

[0058] in, , Representing time respectively Fan unit injection node Active power and reactive power Active power output is determined by the environment (considered as input), while reactive power output is adjusted through the power coefficient. For nodes The apparent capacity of the inverter connected to the wind turbine unit. For nodes The reactive power coefficient of the inverter connected to the wind turbine unit. , These are the minimum and maximum values ​​of the reactive power coefficient of the inverter connected to the wind turbine unit, respectively.

[0059] Specifically, the operation of photovoltaic power generation units and wind power generation units must satisfy the apparent power capacity circle constraint. This constraint serves as a direct constraint in the safety projection layer optimization problem, ensuring that the inverter capacity is not exceeded, i.e.:

[0060]

[0061] For distributed energy storage systems ( The energy storage dynamic equations, power, and SOC constraints are as follows: Energy storage dynamic equation: ,like (Charge) ,like (Discharge) in, For nodes Energy storage system at all times energy, For nodes Energy storage system at all times energy, Indicates time Energy storage system injection node active power, Self-discharge rate (unit: , (in hours) For charging efficiency, For discharge efficiency, use during charging. Use during discharge .

[0062] Power and SOC constraints: ,

[0063] ,

[0064] in, Indicates time Energy storage system injection node active power, , These represent the maximum charging power and discharging power, dynamically calculated based on the real-time state of charge. For nodes Energy storage system at all times energy, Indicates the maximum usable energy stored. For nodes Energy storage system at all times The state of charge, , These represent the minimum and maximum states of charge (SOC) of the energy storage system, respectively. The maximum charging power is dynamically calculated based on the real-time SOC. Discharge power The calculation formula is:

[0065]

[0066] in, This refers to the discharge efficiency.

[0067] Further considering the inverter capacity constraints of the energy storage system, the energy storage system is allowed to provide reactive power support in voltage regulation, with the specific constraints as follows: ,

[0068] in, For nodes The apparent capacity of the inverter connected to the energy storage system. Indicates time Energy storage system injection node reactive power, The range is determined by apparent capacity. With active power A joint decision.

[0069] Flexible load The demand response is then mediated through an adjustable reduction factor, which also has upper and lower limits, with the specific constraints as follows: Demand response model and reduction factor constraints: ,

[0070] ,

[0071] in, Represents a node At any moment The original flexible load power, that is, the user's original power consumption before participating in demand response. Represents a node At any moment The actual flexible load power, that is, the power consumption after demand response (reduction or transfer), This represents the load reduction / transfer factor (negative values ​​indicate reduction, positive values ​​indicate transfer). , These represent the minimum and maximum values ​​of the load reduction / transfer coefficient, respectively.

[0072] In addition, to ensure the safe and stable operation of the system, the voltage amplitude of all nodes must be maintained within the preset safety range, the current of each branch must not exceed its thermal stability limit, and the power purchased from the main grid is also constrained by the substation capacity, as follows: Node voltage constraints: ,

[0073] in, Represents a node At any moment voltage amplitude, , These represent the lower and upper limits of the voltage, respectively. This represents the set of nodes in an active distribution network system.

[0074] Branch current constraints: ,

[0075] in, Indicates time node With nodes Branch road The current, This indicates the maximum value of the branch current. This represents the set of branches in an active distribution network system.

[0076] Substation capacity constraints:

[0077] in, , Representing time respectively Active power and reactive power injected into the main grid This indicates the maximum capacity of the substation.

[0078] Step 1.2: To achieve rapid and secure verification and online correction of reinforcement learning actions, this invention synchronously constructs a globally linearized voltage sensitivity model based on the active distribution network topology. The specific process includes: Step 1.2.1: Set the reference voltage of the network (active distribution network system) to... The reference power is Then the reference impedance for:

[0079] Among them, the reference voltage The unit is kV, and the reference power is... The unit is MVA.

[0080] Step 1.2.2: Analyze the topology of the active distribution network system using the shortest path algorithm in graph theory, and calculate the common path impedance between node pairs. For any node... and nodes compute nodes Path to Balanced Node and Node The set of common branches leading to the equilibrium node is obtained, and the resistance and reactance of each common branch are summed to obtain the resistance sensitivity matrix. and reactance sensitivity matrix Then, based on the reference impedance... The resistance sensitivity matrix and reactance sensitivity matrix Convert them to per-unit value form respectively:

[0081]

[0082] in, Represents the resistance sensitivity matrix per-unit value, Represents the reactance sensitivity matrix The per-unit value.

[0083] Step 1.2.3: Considering that not all controllable resources directly correspond to node injection power, this embodiment further introduces a resource-node mapping matrix. , This is used to map the active and reactive power adjustments of each controllable resource to changes in node injected power. If the... One controllable resource access node If the value is 1, the corresponding element in the mapping matrix is ​​1; otherwise, it is 0. When multiple controllable resources are connected to the same node, multiple non-zero elements are allowed in the corresponding node row.

[0084] Based on the above resistance sensitivity matrix Reactance sensitivity matrix Per-unit values ​​and resource-node mapping matrix, linearizing the predicted node voltage. It can be represented as:

[0085] in, The linearized predicted node voltage vector is obtained from the linearized voltage sensitivity model. To balance the node voltage, It is a column vector of all 1s. , These represent the active load vector and reactive load vector of the nodes, respectively. , These represent the active power injection vector and reactive power injection vector for all controllable resources, respectively. , These are the active resource-node mapping matrix and the reactive resource-node mapping matrix, respectively. This is a linearization error compensation term used to compensate for the deviation between the linearized model and the actual power flow model.

[0086] The resource-node mapping matrix is ​​used to map resource-side power to node-injected power. If the... One controllable resource access node Then the corresponding mapping matrix is ​​the first Line 1 Column elements are 1 if they are otherwise 0; when multiple controllable resources are connected to the same node, multiple non-zero elements are allowed in the corresponding node row.

[0087] It should be noted that the resistance sensitivity matrix and reactance sensitivity matrix Once the network topology and line parameters are determined, they remain unchanged; during online operation, the linearization error compensation term is dynamically updated mainly based on the actual power flow results. This is to improve the accuracy of the linearized voltage prediction model in approximating the actual power flow results.

[0088] In this embodiment, a time is set. At the current moment, the active power injection vector that has actually been applied to each controllable resource at the current operating point is: The reactive power injection vector is Then the active power injection vector of all controllable resources at the current moment. Reactive injection vector They are represented as follows:

[0089]

[0090] in, For a moment The active target adjustment amount, For a moment The reactive power target adjustment amount.

[0091] Will , Substitution The expression yields the time. Linearized prediction of node voltage :

[0092] in, , Representing time respectively Active load vector and reactive load vector at nodes.

[0093] After sorting, we get:

[0094] Define the reference predicted voltage corresponding to the current operating point. for:

[0095] Therefore, we can conclude that:

[0096] This establishes a linear approximate relationship between the node injected power increment and the node voltage increment:

[0097] in, For a moment The node voltage increment vector; for The amount of active power injection adjustment at any given moment. for The amount of reactive power injection adjustment at any given moment.

[0098] Step 2: Based on the active distribution network physical model and linearized voltage sensitivity model established in Step 1, the real-time collaborative scheduling problem of the active distribution network is formalized into a distributed partially observable Markov decision process (Dec-POMDP). This transforms the real-time scheduling problem described by the physical model into a sequential decision problem suitable for multi-agent collaborative learning. The key elements are defined as follows: The Dec-POMDP formula can be represented as a 6-tuple. , It is a set of intelligent agents, where each agent can make decisions based on its perception of the environment. It is a set of states used to describe the current state of the environment. For the joint observation set of all agents, For the joint action set of all agents, It is the global reward for evaluating the decisions of all agents. As a discount factor, The following section provides a detailed introduction to Dec-POMDP.

[0099] Step 2.1.1: Considering the reality that distributed resources in actual power distribution systems are owned and managed by different entities, and the difficulty of large-scale global sharing of network information, this embodiment models each distributed resource (or its logical aggregation) as an independent intelligent agent. The specific intelligent agents are divided as follows:

[0100] in, Represents a set of intelligent agents; , , , and They represent the first One dispatchable thermal power unit intelligent agent, photovoltaic power generation unit intelligent agent, wind power generation unit intelligent agent, distributed energy storage system intelligent agent and flexible load intelligent agent; , , , and These represent the sets of dispatchable thermal power units, photovoltaic power generation units, wind power generation units, distributed energy storage systems, and flexible load access nodes, respectively. , , , and These represent the sets of intelligent agents for dispatchable thermal power units, photovoltaic power generation units, wind power generation units, distributed energy storage systems, and flexible loads, respectively. Simultaneously, based on geographical location or management authority, the entire distribution network's node set is also included. Divided into A set of regions consists of non-overlapping regions. During the online execution phase, each agent makes action decisions based solely on local observations of its own region, without relying on real-time observations from other regions. During the training phase, the Critic network can utilize global state and joint actions for centralized training to improve the stability of multi-agent collaborative learning. The region set partitioning satisfies the following conditions:

[0101] in, , They represent the first The, the Each region.

[0102] Step 2.1.2: Define the state space State space It integrates current load and renewable energy output, the output of each controllable resource from the previous moment, electricity price, and time step information. Global Status time It contains all the information needed to describe the environment, as follows:

[0103] in, Indicates load demand. Indicates electricity price and time. Indicates the status of energy storage planning. , Representing nodes respectively At any moment Active load and reactive load , They represent Injection node of thermal power unit at all times Active power and reactive power express Photovoltaic unit injection node active power, express Photovoltaic unit injection node reactive power, Indicates time Fan unit injection node active power, express Time Fan Unit Injection Node reactive power, express Injection node of the instantaneous energy storage system active power, For nodes Energy storage system in State of charge at time t, Indicates time Reducible flexible load injection node active power, This indicates the real-time electricity price on the main grid. , Each of the previous planning cycles The location and capacity of energy storage. At the same time, variables that need to be optimized here (such as...) , , , , , , Only the value of the previous moment can be observed, reflecting the sequential nature of decision-making.

[0104] Step 2.1.3: Define the observation space And uncertainty. In real-world systems, agents typically cannot obtain perfect global state information. Therefore, each agent is configured... Only its own region can be obtained Local observation Local observation It is the global state. A subset, including intelligent agents The system manages the voltage, load, local renewable energy output, local resource status, and electricity price of the nodes where the resources are located. Simultaneously, to simulate real-world uncertainties such as sensor errors and communication noise, Gaussian perturbations are introduced into the local observations. .

[0105]

[0106]

[0107] in, Indicates the first An intelligent agent at time The actual received noisy local observations, , This represents the total number of intelligent agents. Indicates the first An intelligent agent at time The global state; For at any time Zero-mean Gaussian white noise, i.e. , This is the noise covariance matrix, which depends on the accuracy of the measuring equipment.

[0108] Joint observation by all agents for:

[0109] in, This represents the total number of intelligent agents.

[0110] Step 2.1.4, at time All intelligent agents take joint action , This represents the action space. Each agent... action This corresponds to the setpoint of the distributed resources it controls. (Time) Action space Specifically, it includes:

[0111] in, These are the active power output setpoints and power factor setpoints for dispatchable thermal power units. For nodes The reactive power coefficient of the inverter connected to the photovoltaic unit. For nodes The reactive power coefficient of the inverter connected to the wind turbine unit. Indicates time Energy storage system injection node active power, Indicates the load reduction / transfer factor. Time period. Action space It is a high-dimensional continuous action space, where agents need to make collaborative decisions to achieve global optimization.

[0112] Step 2.1.5: Define state transitions and rewards. The agent executes joint actions. Subsequently, the environment (i.e., the distribution network physical model) transitions to the next state based on the power balance flow equations and energy storage dynamic equations (such as energy storage dynamics). and generate a global reward. The reward function : Incorporate power loss and time-based integrated operating costs into the reward function.

[0113] The power loss function is as follows:

[0114] in, For a moment The sum of active power loss and reactive power loss. Represents active network loss. Represents reactive power loss. For a moment branch road The current, branch road impedance, branch road The reactance.

[0115] The overall operating costs at any given time are as follows:

[0116] in, To consider the overall operating costs at all times, This is the set of all optimized time periods within a day. The set of nodes to which the schedulable thermal power units belong. This is the set of nodes where wind / solar curtailment occurs. express Injection node of thermal power unit at all times The active power (i.e., the active power of the thermal power unit at time) (meritorious contributions) For nodes exist The amount of load reduced / transferred in real time through demand response ( This indicates that the load has been reduced. (This indicates that the load has been increased). For nodes exist Active power that is constantly being discarded For nodes exist Reactive power that is constantly being discarded. , , For nodes The coefficient of the quadratic cost curve of DG. The unit load demand response compensation price, The unit price for curtailing active / reactive power can be set to the same value, or it can be assigned a value based on the system operator's assessment unit price for curtailing wind / solar power. Cost per unit of carbon emissions.

[0117] Simultaneously, to further guide the agent's learning, this invention introduces a counterfactual risk guidance mechanism: In each decision step, the safety projection layer uses its linearized model to pre-evaluate the "counterfactual" voltage risk after the execution of the original action (uncorrected). This risk value is incorporated into the reward function as a penalty term. Even if the safety projection layer subsequently corrects the original action, as long as the original action itself has a potential risk of exceeding the limit, the agent will still be subject to this counterfactual risk penalty, thereby prompting the agent to actively learn the physical safety boundary during training. The specific form of the counterfactual risk penalty is as follows:

[0118] in, This refers to the potential risk of voltage exceeding limits that may result from executing the original action (without correction by the safety projection layer). It is a set of nodes in a distributed resource-coordinated active distribution network system. To ensure the safety of the projection layer, a global linearized voltage sensitivity model is used to predict the voltage of the original action at the node. Voltage estimate, and These represent the lower and upper limits of the node voltage, respectively.

[0119] The reward function is designed as a composite form including operating cost, network loss, and counterfactual risk penalty. While optimizing economic and technical objectives, the counterfactual risk penalty term guides the agent to autonomously learn safe strategies. This penalty term characterizes the risk that the agent's original output action, without safety projection correction, will cause node voltage exceedance. This allows the agent to learn and generate low-risk original actions during training, while the hard guarantee of voltage safety constraints is handled by the subsequent safety projection layer, thus achieving a separation of responsibilities between objectives and constraints. Reward Function Specifically as follows:

[0120] in, To consider the overall operating costs at all times, For a moment The active network loss, This is to account for the potential risk of voltage exceeding limits after the original action is executed. , , These represent the cost weighting coefficient, network loss weighting coefficient, and counterfactual risk penalty coefficient, respectively. , , It is used to balance economic optimization with the learning of intelligent agents' autonomous safety awareness.

[0121] The core of this reward mechanism lies in "separating responsibility" from optimization objectives such as economic efficiency and network losses to strict physical safety constraints. The reward function primarily integrates incentives for optimizing economic indicators such as operating costs, active power losses, and renewable energy consumption. Meanwhile, the guarantee of hard safety constraints, such as node voltage exceeding limits, is handled by a dedicated safety layer designed in the subsequent third step. This avoids the problem in traditional methods where a single reward function struggles to balance soft penalties and hard constraints. To further guide agent learning, this invention innovatively introduces a counterfactual risk guidance mechanism: In each decision step, the safety layer uses its linearized model to pre-evaluate the "counterfactual" voltage risk after executing the original action (uncorrected). This risk value is included as a penalty in the reward function. Even if the safety layer ultimately corrects the action, the agent will still be penalized for outputting a high-risk original action, thus forcing the agent to autonomously explore the physical safety action boundaries during training.

[0122] Finally, there's the strategy and objective: for each agent... Corresponding policy network It is an intelligent agent The local observation vector is mapped to the action function, i.e. The policy network of all agents constitutes a joint policy. The goal of Dec-POMDP is to find an optimal joint strategy. To maximize the expected cumulative reward with discounts starting from the initial state distribution:

[0123] in, In order to implement joint strategies The cumulative discount reward that the next intelligent agent swarm expects to obtain. Indicates the joint strategy The expected value of the state-action trajectory generated under the action is obtained, and the joint strategy is applied. The set of policies of all agents , As a discount factor, Discount factor Used to weigh the importance of immediate rewards against future rewards. Indicates time state, Indicates the state Input Union Strategy The resulting combined action This indicates that the intelligent agent is performing a joint action. Then, the environment transitions to the next state based on the power balance flow equation and the energy storage dynamic equation. .

[0124] This invention models each controllable resource as an intelligent agent. And for each intelligent agent Equipped with the Actor Network. All agents share the Critic network. ,in, For intelligent agents The local observation vector, For intelligent agents The parameters of the corresponding Actor network, Indicates that the intelligent agent Local observation vector Input the original action obtained from the Actor network corresponding to this agent. Represents the joint state-action pair of all agents. For the parameters of the Critic network, This indicates that the Critic network represents the joint state-action pairs. Value estimate.

[0125] See Figure 2 This invention employs two sets of Critic networks, namely Critic Network 1 (corresponding to...) Figure 2 Critic1 and Critic2 (corresponding to) Figure 2 In Critic2, the corresponding Actor target network is set for the Actor network. For Critic Network Critic Network II Set up the Critic target network one respectively Critic Target Network II By training all networks, we obtain the trained Actor network, which is the policy network.

[0126] In step 2, the policy network is the trained Actor network. The training process of the policy network is described in detail below: First, initialize each agent. Actor Network Actor Target Network and Critic Network Critic Network II Critic Target Network Critic Target Network II and the parameters of the Actor network Parameters copied to the Actor target network The parameters of Critic network one Parameters copied to Critic target network 1 The parameters of Critic Network II Parameters copied to Critic target network 2 and initialize the experience replay buffer. .

[0127] Each intelligent agent The local observation vector is input to its corresponding Actor network to obtain the data for each agent. The original action, and the local observation vectors of all agents constitute the current state. The primitive actions of all agents constitute the primitive actions. By minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation (specifically a convex optimization problem, solved online using the numerical optimization solver Gurobi), the original action is... Corrected to safety action The specific correction process follows steps 3 to 5.

[0128] All Actor networks form a joint policy to enforce security actions. Receive rewards and the next state Experience tuples generated by the interaction between the agent and the environment. Stored in the experience replay buffer The data for network training comes from the experience replay buffer. From the experience replay buffer Randomly sample a batch of data ,in, Indicates the current state. Indicates a safety action. As a reward, Given the new global state transitioned to by the environment, the local observation vectors of all agents at the next time step are: Each intelligent agent The corresponding Actor target network is represented as , For intelligent agents The corresponding Actor target network parameters.

[0129] Step 2.2.1: Configure each agent Local observation vector at the next time step Input agent corresponding to the Actor target network The deterministic actions of the agents are obtained, and the deterministic actions of all agents constitute the target action. ; Step 2.2.2: Target Action and the environment transition to a new global state Input Critic target network one and Critic target network two respectively, and calculate the target value based on the minimum value between the outputs of Critic target network one and Critic target network two. :

[0130] in, for Global reward at any moment This is a discount factor used to balance immediate rewards with future cumulative rewards; in this embodiment, it is taken as... , The minimum value between the outputs of Critic target network one and Critic target network two. ,in, This indicates the new global state to which the environment has transitioned. This represents the deterministic action output by the Actor target network under the local observation vector at the next time step. , Indicates that the intelligent agent The local observation vector is input into the agent at the next time step. The corresponding Actor target network yields the deterministic actions of the agent. , This represents the output of Critic target network one. For the parameters of Critic target network one, This represents the output of the Critic target network 2. These are the parameters of Critic target network 2.

[0131] Step 2.2.3: Input the joint state-action pairs of all agents at the current moment into Critic Network 1 and Critic Network 2 respectively. Critic Network 1 outputs the current value. The Critic network outputs the current value. The parameters of Critic Network 1 and Critic Network 2 are updated by minimizing the time-series difference error, respectively. The calculation formula is as follows:

[0132]

[0133] in, and Let represent the loss functions of Critic Network 1 and Critic Network 2, respectively. Indicates the experience replay buffer Randomly sample state transition samples and calculate the expectation of the quantities within parentheses. and These represent the parameters of the two Critic networks; This represents the experience replay buffer; , , and Representing time respectively The global state, combined actions, global reward, and the global state at the next moment; Indicates the target value of the time-series difference; and Let represent the value estimates of the current state-action pair by the two Critic networks. The parameters of the two Critic networks are updated by minimizing the mean squared error between the target value and the current value estimate.

[0134] Step 2.2.4: For each agent... Local observation vector at the current moment Input Actor Network , For Actor Network The parameters are used to obtain new actions. : From the experience replay buffer A batch of global states and excluding agents are sampled. Historical actions of other intelligent agents Constructing joint actions , The current global state and joint actions Input Critic Network 1 and Critic Network 2 respectively, and Critic Network 1 will output the current value. The Critic network outputs the current value. Get the current value Current value minimum value Maximize the target value using gradient ascent method Update the parameters of the Actor network:

[0135] in, For intelligent agents The corresponding Actor network parameters, For the current Critic network to the agent The gradient of the action, The gradient of the Actor network with respect to the parameters. This represents the playback buffer. The average of the states and actions of other agents is calculated in batches.

[0136] The parameters of the Actor target network, Critic target network 1, and Critic target network 2 are kept stable through soft updates:

[0137]

[0138]

[0139] in, The smaller the value, the slower the target network changes, resulting in more stable training. For the parameters of the Actor network, For the parameters of Critic network one, For the parameters of Critic Network II, For the parameters of the Actor target network, For the parameters of Critic target network one, These are the parameters of Critic target network 2.

[0140] After training, the trained Actor network for each agent is obtained, which is also the policy network for each agent. All policy networks together constitute the joint policy. After fixing the parameters of all policy networks, they can be deployed in a real-world active distribution network system. During online operation, each agent's local observations are input into its corresponding policy network, and each agent generates its initial actions based on these real-time local observations. , original action The target active power injection vector corresponding to each controllable resource and target reactive power injection vector : and use it as input for subsequent security verification and security projection layer correction.

[0141] Step 3: Calculate the actual node voltage based on the original action and verify whether the actual node voltage meets the voltage safety constraints. If the actual node voltage meets the voltage safety constraints, the original action is directly used as the final execution instruction; if the actual node voltage does not meet the voltage safety constraints, proceed to Step 4.

[0142] During the online operation phase, the target active power injection vectors of each controllable resource corresponding to the original actions output by each agent in step 2 are injected. and target reactive power injection vector Input the data into high-fidelity power flow calculation software to perform real power flow calculations and obtain the actual node voltages under the original actions. Determine whether the actual node voltage meets the node voltage constraint:

[0143] in, , These represent the lower and upper limits of the node voltage, respectively. If all nodes meet the above voltage safety constraints, it means that the original action can be directly executed under the current operating conditions. and It is issued to the active distribution network system as the final execution instruction.

[0144] If at least one node does not meet the above voltage safety constraints, it means that the original action has the risk of causing the node voltage to exceed the limit. Instead of directly executing the original action, we proceed to step 4 and correct the original action online through the global linearized safety projection layer.

[0145] Step 4: The original action is corrected online through the global linearized safety projection layer. A convex optimization problem is constructed with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint to participate in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved jointly to obtain candidate safety actions.

[0146] When step 3 determines that the original action does not meet the voltage safety constraint, the original action will be... Corresponding target active power injection vector Target reactive power injection vector Input the safety projection layer. The safety projection layer uses the current actual operating point as a benchmark and employs incremental modeling to correct the original actions. Before the original actions are executed, the actual active and reactive power injection vectors of each controllable resource at the current operating point are denoted as follows: and , and They represent The active and reactive target adjustment quantities, mapped from the agent's original actions at each moment, are used to characterize the agent's original adjustment intention. The mapping relationship is as follows:

[0147]

[0148] The goal of the secure projection layer is to solve for the active power injection adjustment that satisfies grid security constraints while preserving as much of the agent's original adjustment intent as possible. Adjustment amount of active and reactive power injection .

[0149] Let the optimization variable to be solved for the safety projection layer be the active power injection adjustment. and reactive power injection adjustment amount and node voltage lower bound slack variables and upper limit slack variables Construct the following convex optimization problem:

[0150]

[0151]

[0152] in, for The amount of active power injection adjustment at any given moment. for The amount of reactive power injection adjustment at any given moment. This is the lower limit relaxation variable for the node voltage. This is a relaxation variable for the upper limit of the node voltage. , , These represent the weighting coefficients for active power regulation, reactive power regulation, and voltage slack, used to balance the penalty levels for these three parameters. Prioritize the use of reactive power resources to regulate voltage; This is a vector of all 1s, used to sum over slack variables; , These represent the lower and upper bound vectors of the node voltage, respectively.

[0153] Among them, the node voltage constraint is constructed as a linear constraint based on the global linearized voltage sensitivity model, the upper and lower limits of equipment output, the power factor constraint, the flexible load adjustment constraint and the energy storage state of charge constraint are constructed as linear constraints, and the apparent capacity constraint of the inverter of the photovoltaic power generation unit, the wind power generation unit and the distributed energy storage system is constructed as a second-order cone constraint, so that the safety projection convex optimization problem is constituted as a convex optimization problem in the form of second-order cone programming.

[0154] In the secure projection layer, at any time Linearized prediction of node voltage The expression is denoted as:

[0155]

[0156] in: for Linearized prediction of node voltage in the time-safe projection optimization problem (corresponding to) Figure 2 Medium voltage ); The reference predicted node voltage vector at the current time (corresponding to) Figure 2 Medium voltage ); , These are the active resource-node mapping matrix and the reactive resource-node mapping matrix (corresponding to...). Figure 2 In , ); and These are the per-unit regulated resistivity sensitivity matrix and reactance sensitivity matrix (corresponding to...) Figure 2 In , ); Represents the linearization error compensation term (corresponding to) Figure 2 In This is used to compensate for the deviation between the linearized model and the actual power flow model; and They are respectively The active power injection adjustment and reactive power injection adjustment at any given time; The voltage at the balancing node; 1 represents a column vector of all 1s; and Inject vectors for the actual active and reactive power of each controllable resource at the current operating point; and These are the active load vector and reactive load vector of the node at the current time, respectively.

[0157] To ensure that the safety projection problem can be solved quickly online, this invention does not directly solve the complete nonlinear power flow equation in the safety projection layer. Instead, it uses the current real power flow operating point as a benchmark and adopts a globally linearized voltage sensitivity model to describe the approximate relationship between the active and reactive power injection increments and the node voltage increments.

[0158] The resource operation constraints in the aforementioned optimization problem include active power output constraints and power factor constraints for dispatchable thermal power units; reactive power regulation constraints and apparent capacity constraints for photovoltaic and wind turbine inverters; charging and discharging power constraints, SOC constraints, and inverter capacity constraints for energy storage systems; and flexible load reduction or transfer coefficient constraints. Among these, the upper and lower limits of equipment output, the range of power factor, the range of flexible load reduction or transfer coefficients, and the safe range of energy storage SOC are represented by linear constraints; the apparent capacity circle constraints for photovoltaic, wind turbine, and energy storage inverters are represented by second-order cone constraints; and the node voltage constraints are represented by linear constraints based on a globally linearized voltage sensitivity model. Therefore, the problem solved by the safety projection layer is a convex optimization problem in the form of a second-order cone programming problem.

[0159] By solving the above convex optimization problem, the active power injection adjustment amount is obtained. Reactive power injection adjustment amount and node voltage relaxation variables , Then, the active power injection adjustment amount will be determined. Reactive power injection adjustment amount By superimposing the original action, candidate safe actions are obtained:

[0160]

[0161] in, , These are the target active power injection vector and the target reactive power injection vector corresponding to the original action, respectively. , These are the candidate secure active power injection vector and the candidate secure reactive power injection vector, respectively, obtained after correction by the secure projection layer.

[0162] It should be noted that the results obtained in this step and As a candidate safety action, it satisfies the safety constraints based on the linearized voltage sensitivity model. However, due to the nonlinear characteristics of the real power flow model, it is still necessary to proceed to step 5 for real power flow verification and error compensation.

[0163] Step 5: Perform real power flow verification based on candidate safety actions, and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, output the candidate safety action as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, substitute the updated linearization error compensation term back into the convex optimization problem in step 4, resolve the action correction amount, and iteratively execute steps 4 and 5 until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

[0164] Step 5.1: Set the maximum number of iterations Sum of deviation smoothing factors In this embodiment, Take 3. Set the value to 0.8. Initialize the number of iterations. The current candidate security action is , .

[0165] Step 5.2: Select the current candidate security action. and Input the data into high-fidelity power flow calculation software to perform real power flow calculations and obtain the actual node voltages. Determine the actual node voltage Does the node voltage constraint meet?

[0166] in, , Let these represent the lower and upper limits of the node voltage, respectively. If the actual node voltages of all nodes... All satisfied If the current candidate safe action meets the safety requirements under the real power flow model, the iteration is terminated, and the current candidate safe action is changed. and The final execution command is issued to the active distribution network system. If at least one node does not meet the above voltage safety constraints, it indicates that the current candidate safety action poses a risk of causing the node voltage to exceed the limit. Instead of directly executing the current candidate safety action, the original action intention is kept unchanged, and the updated linearized error compensation term is substituted back into the convex optimization problem in step 4 to resolve the active power injection adjustment and reactive power injection adjustment, and a new candidate safety action is obtained.

[0167] If the actual node voltage does not meet the voltage safety constraint and the current iteration number has not reached the maximum iteration number. Then, calculate the voltage prediction error between the actual power flow result (the actual node voltage corresponding to the current candidate safety action) and the linearized predicted node voltage:

[0168]

[0169] in, Indicates the first The actual node voltage in the next iteration With linearized prediction of node voltage The deviation between them The reference predicted node voltage vector at the current time; , They represent the first The active power injection adjustment amount and reactive power injection adjustment amount corresponding to each iteration; Indicates the first The linearization error compensation term used in the next iteration.

[0170] According to the The actual node voltage in the next iteration With linearized prediction of node voltage Deviation between Update the linearization error compensation term:

[0171] in, This represents the updated linearization error compensation term. This is the deviation smoothing factor. The updated linearization error compensation term is then substituted back into the safety projection convex optimization problem, and the linearized predicted node voltage is used as a voltage constraint in the solution. The new active power injection adjustment and reactive power injection adjustment are obtained, and the candidate safety actions are updated based on the new active power injection adjustment and reactive power injection adjustment until the actual node voltage corresponding to the candidate safety action satisfies the voltage safety constraint.

[0172] Identify the nodes with the greatest voltage exceedance in the current real power flow calculation results as the set of dangerous nodes. For low-voltage out-of-limit nodes, increase their lower voltage limit constraint when resolving the safe projection problem; for high-voltage out-of-limit nodes, decrease their upper voltage limit constraint when resolving the safe projection problem.

[0173] Specifically, for dangerous nodes The upper and lower voltage limits are set with additional voltage safety margins. : like Then the lower voltage limit in the next safe projection of that node. Adjusted to ; like Then the upper limit of voltage in the next safe projection of that node. Adjusted to .

[0174] Among them, among them, Indicates the first The actual node voltage in the next iteration. Small positive quantities, for example, can be taken ; and Indicates dangerous nodes The upper and lower voltage limits are defined. By applying additional margins to critical nodes, the correction strength of the safety projection layer for voltage-weak nodes can be improved.

[0175] Maintain the agent's original action intention , The updated linearization error compensation term remains unchanged. The tightened dangerous node voltage constraint is then resubmitted into the safety projection convex optimization problem in step 4, and a new active power injection adjustment is obtained by solving it again. and reactive power injection adjustment amount .

[0176] Then update the candidate security actions:

[0177]

[0178] make Then return to step 5 to continue the real trend verification.

[0179] If the maximum number of iterations is reached... If the actual node voltage still does not fully meet the voltage safety constraints, then node voltage relaxation variables are selected from the candidate safety actions generated in each iteration. , The candidate safety action with the smallest sum is selected as the final execution instruction, and a safety alarm message is output to indicate that there is a strong risk of exceeding limits or insufficient resource adjustment capability in the current operating state.

[0180] When the original action In step 3, the actual power flow is verified, or a candidate safety action is selected. After verifying the actual power flow in step 5, the corresponding active power injection vector will be... and reactive power injection vector As the final execution command, it is issued to the active distribution network system, thereby issuing the original action or the safety action after iterative correction that meets the voltage safety constraints as the final execution command to the active distribution network system.

[0181] After execution, the active distribution network transitions to the next state based on actual power flow results, energy storage SOC dynamic equations, flexible load response results, and actual renewable energy output. The system collects information such as node load, node voltage, distributed resource output, energy storage SOC, and main grid electricity price at the next time step, and uses this information as input observations for each agent at the next scheduling time, continuing to execute steps 2 to 5 of the online collaborative optimization process.

[0182] Through the collaboration of the above modules, this invention forms a closed-loop system structure of "model construction - agent decision-making - original action verification - safety projection correction - power flow verification and error compensation - execution output". This enables the multi-agent reinforcement learning strategy to be responsible for economic and collaborative optimization, the safety projection module to be responsible for action feasibility correction, and the power flow verification and error compensation module to be responsible for final safety assurance. In this way, while improving the economic efficiency of active distribution network operation, it ensures that hard safety constraints such as node voltage are met.

[0183] Therefore, this embodiment decouples the multi-agent reinforcement learning strategy from the physical security verification mechanism: the upper-layer multi-agent policy network is responsible for learning the laws of operational economy, network loss optimization, and distributed resource collaborative scheduling; the middle-layer security projection layer is responsible for correcting potentially unsafe original actions into candidate safe actions that meet the constraints in real time; and the lower-layer real power flow verification and error compensation mechanism is responsible for closed-loop correction of linearized model errors to ensure that the final control commands issued meet the operational safety requirements of the active distribution network.

[0184] The advantages of this invention are: (1) This invention innovatively proposes a hybrid driving paradigm of "data-driven decision-making - physical model verification", which systematically applies constrained multi-objective reinforcement learning to active distribution network collaborative optimization. This method can not only learn economic optimization strategies from high-dimensional and uncertain data, but more importantly, it breaks through the limitation of traditional reinforcement learning methods that can only approximate constraints through rewards and punishments after the fact by constructing a verifiable online safe execution layer, and realizes a paradigm advancement from "soft constraint learning" to "hard constraint guarantee".

[0185] (2) This invention addresses the physical feasibility problem of actions in a high-dimensional continuous action space by proposing an innovative hierarchical decision-making and safety projection architecture. At the top level, multi-agent reinforcement learning handles multi-objective optimization decisions, including economic efficiency and network loss. In the middle layer, a real-time safety layer with iterative closed-loop correction capability is innovatively introduced. This safety layer utilizes a topology-adaptive sensitivity model to project the agent's original actions into a feasible region satisfying all power grid safety constraints in real time through a PQ joint convex optimization problem. Furthermore, through mechanisms such as "dangerous node identification," "incremental modeling," and "deviation feedback iteration," it effectively overcomes linearization model errors, ensuring that actions can be corrected to an absolutely safe feasible region under any operating condition. In addition, this method pioneers a counterfactual risk guidance mechanism, incorporating the risk of the original actions into the reward function, achieving an evolution from "dependence on safety layer correction" to "agent autonomous safety," fundamentally solving the engineering risk of unsafe actions generated during reinforcement learning exploration.

[0186] (3) This invention achieves accuracy and efficiency in safety correction through refined and differentiated physical constraint modeling and incremental optimization mechanisms. The safety layer not only uniformly handles node voltage constraints, but also models power factor constraints, inverter capacity circle constraints (using a second-order cone form), and active and reactive power coupling relationships for different resource characteristics of DG, PV / WT, ESS, and FL. In particular, for energy storage systems, the safety layer jointly models their active power SOC dynamic constraints and apparent power capacity constraints, fully exploring their potential in voltage regulation. "Incremental modeling" avoids repeated state calculations, and introduces a "dangerous node" identification and additional margin application mechanism to adaptively strengthen voltage control of weak links. Combined with the iterative closed loop of "projection-power flow verification-deviation compensation", the linearization model error is effectively compensated, ensuring the final reliability of the safety correction results.

[0187] (4) This invention significantly improves the convergence stability and practicality of the algorithm by constructing a training environment that integrates high-fidelity simulation (Panda power) and data-driven models, and by adopting a stable multi-agent training algorithm (GRU-Actor and dual-commentator architecture). The introduction of the security layer decouples the complex physical constraint verification from the learning process, greatly reducing the invalid space of agent exploration, thereby accelerating training convergence. The proposed architecture achieves decoupling of decision-making and security, making the trained policy more interpretable and giving engineers confidence in its deployment, providing an efficient, robust and safe solution for online intelligent optimization of large-scale active distribution networks.

[0188] Example 2 The difference between this embodiment and Embodiment 1 is that this embodiment is a system embodiment and can be used to execute the method of Embodiment 1. For details not disclosed in the system embodiment of this invention, please refer to the method embodiment of this invention, which will not be repeated here. This embodiment provides a multi-agent reinforcement learning-based active distribution network cooperative optimization system, including: The model building module is used to establish the operation model of each distributed resource and the system constraints for an active distribution network system based on distributed resource collaboration, and to build a global linearized voltage sensitivity model. The agent decision-making module is used to treat each distributed resource as an independent agent, divide the active distribution network node into multiple non-overlapping regions, obtain the local observation vector of each agent in its own region, input the local observation vector of each agent into the corresponding policy network, and output the original action. The original action verification module is used to perform real power flow calculations based on the original actions to obtain the real node voltages, and to verify whether the real node voltages meet the voltage safety constraints. If they do, the original actions are directly used as the final execution instructions; otherwise, the module proceeds to the safety projection module. The safety projection module is used to correct the original actions online through a global linearized safety projection layer. It constructs a convex optimization problem with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved together to obtain candidate safety actions. The deviation compensation and iteration module is used to perform real power flow verification based on candidate safety actions and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, the candidate safety action is output as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, the updated linearization error compensation term is substituted back into the convex optimization problem in the safety projection module to solve the action correction amount again, and the safety projection module and the deviation compensation and iteration module are iteratively executed until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

[0189] This invention designs a hierarchical multi-agent reinforcement learning algorithm architecture based on a "decision-projection-verification" closed loop and a global linearized security layer, in order to overcome the shortcomings of existing methods that treat security constraints only as ex-post penalties.

[0190] The upper layer (distributed decision-making agents) treats each controllable resource as an agent. Based on its local observations, it is trained using an improved multi-agent deep deterministic policy gradient algorithm (the MATD3 framework integrating dual commentator networks, objective policy smoothing, and delayed updates), outputting raw control actions with cost and network loss as optimization objectives. Specifically, considering the observable and time-dependent nature of power systems, a single-layer gated recurrent unit is introduced at the front end of the policy and value networks. This unit uses historical observation sequences to capture the time-series dynamics of node voltage, renewable energy output, and load, and the output of the recurrent unit is normalized at the next layer to ensure training convergence.

[0191] Middle Layer (Globally Linearized Safety Projection Layer): This is the core innovative module. This layer receives the raw action instructions from all agents and converts them into P and Q injection increments for each node. Unlike existing technologies, this module adopts an incremental modeling approach, using the current network state as a baseline and optimizing only the changes brought about by the actions, avoiding repeated state calculations and improving solution efficiency. Utilizing the globally linearized sensitivity model constructed in the first step, a convex optimization problem (such as a second-order cone programming problem) is constructed with the objective of minimizing action corrections and satisfying all power grid safety constraints.

[0192] The solution process for this optimization problem incorporates a multi-round iterative closed-loop correction mechanism: First, "dangerous nodes" are dynamically identified based on historical limit violation records and the current voltage approximation level, and stricter voltage safety margins are applied to them during the optimization process. Then, the corrected instructions obtained from the solution are substituted into a high-fidelity power flow calculator for verification. If the result still violates the safety constraints, the deviation between the actual power flow and the linear model prediction is calculated. This deviation is used to update the linearized model, automatically triggering a new round of optimization until the safety requirements are met or the iteration limit is reached. This optimization problem refines the physical limits of different devices: including the power factor constraint of DG, the apparent power capacity circle constraint of PV / WT (using a second-order cone constraint), the active power SOC dynamic constraint and the joint constraint of apparent power capacity of ESS (allowing it to provide reactive power support within a safe range), and the reduction ratio constraint of FL, etc. After optimization, a safety-corrected solution is obtained. , instruction.

[0193] The bottom layer (physical simulation and iterative verification layer) applies the actions output by the safety layer to the high-fidelity power flow calculation software (Panda Power) for real power flow verification. If voltage exceedances still occur, the deviation between the actual voltage and the linearly predicted voltage is used as feedback to dynamically update the deviation term in the linear model and initiate an iterative optimization process until verification is passed or the iteration limit is reached, thus forming a reliable execution closed loop of "decision-safe projection-physical verification-deviation compensation". Finally, the trained agent policy and safety projection layer can be deployed in the actual system to achieve data-driven and safe verifiable real-time optimization scheduling.

[0194] This invention constructs a training environment that integrates high-fidelity power system simulators (OpenDSS, PandaPower) with data-driven models. This environment can dynamically simulate the system's operational effects based on the control actions of intelligent agents.

[0195] To accelerate training and improve sample efficiency, the agent is trained from simple to complex. In the initial stage, it is trained on a small network, in a deterministic scenario, with a single primary objective; subsequently, the network size is gradually increased, uncertainties in renewable energy and load are introduced, and more optimization objectives are added, allowing the agent to smoothly adapt to complex problems.

[0196] Finally, using the constructed environment and historical data, the designed agent is trained on a large scale offline until the policy achieves stable convergence in terms of multi-objective rewards and constraint violation rates. After training, the trained global coordination agent and the underlying local control agent model are solidified into a decision support system. In practical applications, the system can input state information based on real-time or predicted data of the target distribution network, and the agent can output recommended control strategies to achieve data-driven intelligent optimization.

[0197] Simulation Analysis 1 To preliminarily verify the training stability and optimization capability of the proposed hierarchical multi-agent reinforcement learning architecture (improved MATD3 combined with a globally linearized security layer), this embodiment conducted centralized offline training for 3000 episodes on the constructed multi-agent model based on an IEEE 33-node active distribution network environment. The simulation environment was based on an IEEE 33-node active distribution network constructed using Pandapower, with the topology as shown below. Figure 1 As shown. The system reference voltage is 12.66 kV, and the node voltage safety limit is 0.95 pu~1.05 pu. Distributed resources include: 2 dispatchable gas turbine units (nodes 11, 29), 3 photovoltaic units (nodes 19, 24, 32), 3 wind turbine units (nodes 5, 15, 30), 1 energy storage system (node ​​18), and 1 flexible load (node ​​10). Each controllable resource (DG, PV, WT, ESS, FL) is an independent intelligent agent, for a total of 9 intelligent agents. The observation space of each intelligent agent includes the voltage, load, electricity price, and its own status in the local area. The action space is defined as described in step 2.1.4. The load data uses a public dataset, which includes active and reactive power at one point every 15 minutes for 8760 hours throughout the year; renewable energy output is generated based on meteorological data. The electricity price adopts peak-valley time-of-use pricing, with a peak price of $123.4 / MWh and a valley price of $36.95 / MWh. The multi-agent reinforcement learning algorithm employs an improved MATD3 framework, with gated recurrent units (GRUs) introduced at the front end of the policy and value networks to handle time-series observations. A globally linearized security layer is constructed based on a voltage sensitivity model using LinDistFlow, and a second-order cone programming approach is employed to solve the security correction problem.

[0198] By analyzing the evaluation of network parameters and the evolution of physical performance indicators during the training process, the model was comprehensively validated from two dimensions: "training process" and "final performance".

[0199] Algorithm convergence and network parameter evaluation: As shown in Figure 4(a), in the early training stage (first 300 rounds), the agent is in a random exploration phase, and the "episode reward" is at a low level (approximately -2200 to -1600). As shown in Figure 4(b), as training progresses, the "critic loss" experiences a rapid initial surge followed by a rapid decline, converging to an extremely low level close to 0 after approximately 25,000 training steps, indicating that the value network can accurately evaluate the long-term benefits of the current state-action pair. As shown in Figure 4(c), the "actor loss" rises steadily and eventually stabilizes at around 450. The stable convergence of the network loss directly drives the "episode reward" to rapidly climb after 500 rounds and stabilize in the convergence range of approximately -1000, proving that the MATD3 algorithm with the GRU module has excellent convergence efficiency when dealing with high-dimensional partially observable states.

[0200] Boundary Learning Mechanism Guided by Safety Layer: The core advantage of this invention lies in the fact that the safety layer not only provides a safety net but also serves as a "teacher." As shown in Figures 5(a) and 5(b), the curves for "Safety Layer Usage" and "Action Difference" reveal that during the initial 300 rounds of blind exploration in training, the agent is prone to outputting dangerous actions. At this time, the safety layer intervenes frequently (peak usage reaches 6%-7%) and significantly corrects the original actions (peak action difference reaches 0.030). However, as training progresses, the agent gradually learns the physical safety boundaries of the power grid through interaction with the environment and the safety layer. After 500 rounds, both the safety layer usage and action difference rapidly decay and converge strictly to 0. This indicates that the agent ultimately learns spontaneously to output the optimal action within an absolutely safe physical feasible domain, achieving intelligent evolution from "relying on the safety layer for correction" to "autonomously satisfying hard constraints."

[0201] As shown in Figure 6(c), the number of voltage overruns remained strictly zero throughout the 3000 training cycles, which intuitively and powerfully demonstrates the absolute reliability of the "globally linearized safe projection layer" proposed in this invention. Even when the agent outputs a large number of random trial actions in the early stages of training, the safe layer can still intercept and correct dangerous instructions 100%, fundamentally avoiding the risks to power grid operation during the reinforcement learning trial and error process.

[0202] Under the premise of ensuring absolute system security, the agent efficiently achieves joint optimization of economic and technical aspects. Referring to Figures 6(a) and 6(b), after initial exploratory fluctuations, the two core indicators, "operating cost" and "system network loss," both show a rapid decline and stable convergence trend after about 500 rounds. Ultimately, the operating cost stabilizes in the optimal range of around 340, and the network loss also converges to an extremely low level of about 0.15. This evolution trajectory fully demonstrates that the architecture of this invention successfully decouples security constraints from economic optimization, enabling the agent to stably learn the optimal cooperative scheduling strategy that balances minimum cost and minimum network loss within a security boundary with zero risk of exceeding limits.

[0203] Simulation Analysis 2 To further verify the effectiveness of the multi-agent reinforcement learning-based active distribution network collaborative optimization method in Example 1 under actual continuous operation scenarios, this example conducts a 24-hour (96 time steps, 15 minutes per step) online scheduling simulation test based on the active distribution network environment in Example 1.

[0204] The following section presents the collaborative decision-making output of each distributed resource agent within the system, the overall operating status of the system, and the effectiveness and advancement of three verification models from the aspects of the feasibility and hard constraint guarantee of voltage safety control, agent economic arbitrage strategy and multi-source collaboration, and multi-objective comprehensive optimization.

[0205] First, regarding the feasibility and hard constraint guarantee of voltage safety control: As shown in the "average node voltage" curve of Figure 7(a), during the 24-hour dynamic scheduling cycle, despite the strong nonlinear fluctuations in photovoltaic and wind power output and the dynamic changes in load, the average node voltage of the entire network is always strictly and stably limited within the safe operating red line of 0.95 pu to 1.05 pu. This result fully demonstrates the effectiveness of the original global linearized safety projection layer of this invention. When the upper-layer agent explores the maximization of economic benefits and may produce actions that lead to voltage exceeding the limit, the middle-layer safety projection layer can accurately project these dangerous actions (such as improper reactive power regulation or excessive active power throughput) into the feasible domain that satisfies all physical safety constraints in real time based on the voltage sensitivity model. This realizes a paradigm shift from the "post-event soft punishment" of traditional reinforcement learning to "pre-event hard constraint guarantee", ensuring the absolute physical feasibility of the control strategy.

[0206] Second, regarding agent-based economic arbitrage strategies and multi-source collaboration: the hierarchical architecture of this invention enables multiple agents to perfectly learn and execute "buy low, sell high" arbitrage strategies that conform to market rules, achieving a high degree of synergy between economic efficiency and flexibility. The profit-seeking decision-making behavior of the agents at different times can be clearly observed by combining the electricity price curve in Figure 8(a): Precise arbitrage using energy storage systems (ESS): As shown in Figure 8(b) with "ESS charging and discharging power" and "ESS state of charge," the energy storage system actively performs charging during the midday period (approximately 10:00-14:00, when electricity prices are at their lowest and photovoltaic output reaches its peak, as shown in Figure 9(a)); while during the peak electricity price period from 16:00-19:00, the energy storage system quickly switches to discharging mode (power is positive and SOC decreases). This decision not only maximizes the economic arbitrage benefits of low-storage-high-discharge but also promotes the absorption of excess renewable energy during midday.

[0207] Peak shaving response of flexible load (FL): As shown in the flexible load reduction coefficient in Figure 8(c), during the high electricity price period from 16:00 to 19:00, the flexible load agent actively reduces the reduction coefficient to the range of 0.5-0.7, which significantly reduces the actual power consumption during the peak period (Figure 9(b)) and effectively avoids the high peak electricity purchase cost.

[0208] Peak substitution of dispatchable generators (DG): As shown in Figure 9(a), DG1 and DG2 significantly increased their active power output during peak electricity price periods, replacing expensive grid-purchased power with local generation. This directly led to a significant decrease in "grid-purchased power" in Figure 9(b) during peak electricity price periods, significantly reducing the overall operating cost of the system.

[0209] Third, in terms of multi-objective comprehensive optimization: while achieving the aforementioned economic arbitrage and voltage safety, the system also considers the optimization of network losses. As shown in the "Network Loss" curve in Figure 7(b), the network loss remains at a low level throughout the day (below 0.2MW for most of the time). This indicates that the improved MATD3 algorithm proposed in this invention, combined with the second-order cone safety projection, can find a Pareto optimal scheduling trajectory in a high-dimensional continuous action space that balances "lowest operating cost," "minimum network loss," and "absolute voltage safety."

[0210] In summary, the "decision-projection-verification" closed-loop architecture proposed in this invention not only endows multi-agent agents with keen economic arbitrage and multi-source collaboration capabilities, but also successfully resolves the inherent conflict between economic optimization and power grid physical security through real-time correction of the linearized security layer, possessing extremely high engineering practical value and deployment reliability.

[0211] The two simulations described above validated the core advantages of the method described in this invention: the globally linearized safety layer can perform real-time safety corrections on any action (including random exploration and immature strategies in the early stages of training), fundamentally preventing voltage exceedances; multi-agent reinforcement learning effectively learns economically optimized strategies, achieving an optimal balance between safety and economy in collaboration with the safety layer; the post-training strategy significantly reduces its dependence on the safety layer, but still retains the safety layer as a last line of defense, ensuring absolute safety during deployment. This invention provides a safe, efficient, and verifiable collaborative optimization solution for active distribution networks with a high proportion of renewable energy integration.

[0212] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-agent reinforcement learning-based active distribution network cooperative optimization method, characterized by: include: Step 1: Based on the active distribution network system of distributed resource collaboration, establish the operation model of each distributed resource and the system constraints, and construct a global linearized voltage sensitivity model; Step 2: Treat each distributed resource as an independent intelligent agent and divide the active distribution network node into multiple non-overlapping regions. Obtain the local observation vector of each intelligent agent in its respective region, input the local observation vector of each intelligent agent into the corresponding policy network, and output the original action. Step 3: Calculate the actual node voltage based on the original action and verify whether the actual node voltage meets the voltage safety constraints. If so, directly use the original action as the final execution instruction. Otherwise proceed to step 4; Step 4: The original action is corrected online through the global linearized safety projection layer. A convex optimization problem is constructed with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint to participate in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved together to obtain candidate safety actions. Step 5: Perform real power flow verification based on candidate safety actions, and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, output the candidate safety action as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, substitute the updated linearization error compensation term back into the convex optimization problem in step 4, resolve the action correction amount, and iteratively execute steps 4 and 5 until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

2. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: Distributed resources include dispatchable thermal power units, photovoltaic power generation units, wind power generation units, distributed energy storage systems, and flexible loads. The operation models for each distributed resource include: dispatchable thermal power units simultaneously satisfy active power output constraints and power factor constraints; for photovoltaic or wind power generation units, under a given active power output, the reactive power output of the photovoltaic or wind power generation units is controlled by adjusting the inverter power coefficient, and the operation model reflects the reactive power regulation capability and operating range; for distributed energy storage systems, a dynamic equation for energy storage based on the state of charge is established, considering charging and discharging efficiency and self-discharge rate; for flexible loads, a model for flexible load participation in demand response is established, characterizing the range of proportions of flexible loads that can be reduced or transferred. System constraints include: node power balance constraints, upper and lower limits of active power output of adjustable thermal power units, upper and lower limits of power factor of adjustable thermal power units, upper and lower limits of reactive power coefficient of inverters connected to photovoltaic or wind power generation units, upper and lower limits of output of distributed energy storage systems, upper and lower limits of state of charge of distributed energy storage systems, upper and lower limits of flexible load reduction or transfer coefficients, node voltage constraints, branch current constraints, and substation capacity constraints.

3. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: The process of constructing a globally linearized voltage sensitivity model includes: Calculate the reference impedance based on the reference voltage and reference power of the active distribution network system; The topology of the active distribution network system is analyzed using the shortest path algorithm in graph theory. The resistance sensitivity matrix and reactance sensitivity matrix of the node pairs are calculated and then converted into per-unit values. Establish a linear approximation relationship between the node injection power increment and the linearized predicted node voltage.

4. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 3, characterized in that: The linear approximation relationship between the node injection power increment and the linearized predicted node voltage is as follows: in, For a moment Linearized prediction of node voltage, To balance the node voltage, For a moment The active target adjustment amount, For a moment The reactive power target adjustment amount, , Representing time respectively Active load vector and reactive load vector at nodes , These represent the active power injection vector and reactive power injection vector that have been actually applied to each controllable resource, respectively. , These are the active resource-node mapping matrix and the reactive resource-node mapping matrix, respectively. For linearization error compensation term, It is a column vector of all 1s. Represents the resistance sensitivity matrix per-unit value, Represents the reactance sensitivity matrix The per-unit value.

5. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: The calculation method for convex optimization problems is as follows: in, for The amount of active power injection adjustment at any given moment. for The amount of reactive power injection adjustment at any given moment. This is the lower limit relaxation variable for the node voltage. This is a relaxation variable for the upper limit of the node voltage. , , These represent the weighting coefficients for active power adjustment, reactive power adjustment, and voltage slack, respectively. This is a vector of all 1s, used to sum over slack variables; , These represent the lower and upper bound vectors of the node voltage, respectively. and They represent The active and reactive target adjustment quantities obtained from the mapping of the agent's original actions at any given time. For a moment Linearized prediction of node voltage.

6. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: The process of updating the linearization error compensation term based on the deviation between the actual node voltage corresponding to the candidate safety action and the linearized predicted node voltage includes: Calculate the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. : in, Indicates the first The actual node voltage in the next iteration. Indicates the first Linearized prediction of node voltage in the next iteration; According to the The deviation between the actual node voltage and the linearized predicted node voltage in the next iteration Update the linearization error compensation term: in, This represents the updated linearization error compensation term. Indicates the first The linearization error compensation term used in the next iteration This is the deviation smoothing factor. .

7. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: During the iterative execution of steps 4 and 5, one or more nodes with the greatest voltage exceedance are identified as a set of dangerous nodes based on the actual power flow calculation results. and dangerous nodes The upper and lower voltage limits are set with additional voltage safety margins. : like Then the lower voltage limit in the next safe projection of that node. Adjusted to ; like Then the upper limit of voltage in the next safe projection of that node. Adjusted to ; in, Indicates the first The actual node voltage in the next iteration. For voltage safety margin, and Indicates dangerous nodes The upper and lower voltage limits.

8. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 1, characterized in that: The training process of the policy network includes: From the experience replay buffer Randomly sample a batch of data ,in, Indicates the current state. Indicates a safety action. As a reward, For the new global state transitioned to by the environment, each agent will... Local observation vector at the next time step Input agent corresponding to the Actor target network , Represents the Actor target network The parameters are used to obtain the deterministic actions of the agents, and the deterministic actions of all agents constitute the target action; The target action and the new global state transferred by the environment are input into Critic Target Network 1 and Critic Target Network 2, respectively. The minimum value between the outputs of Critic Target Network 1 and Critic Target Network 2 is then used. Calculate the target value; Input the joint state-action pairs of all agents at the current moment into Critic Network 1 and Critic Network 2 respectively. Critic Network 1 outputs the current value. The Critic network outputs the current value. The parameters of Critic Network 1 and Critic Network 2 are updated by minimizing the time difference error. Each intelligent agent Local observation vector at the current moment Input Actor Network , For Actor Network The parameters are used to obtain new actions. Based on new actions and the removal of intelligent agents Historical actions of other intelligent agents Constructing joint actions The current global state and joint actions Input Critic Network 1 and Critic Network 2, Critic Network 1 outputs the current value. The Critic network outputs the current value. Get the current value Current value minimum value Maximize the target value using gradient ascent method Update the parameters of the Actor network to obtain the policy network.

9. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 8, characterized in that: target value for: in, for Global rewards gained at any time As a discount factor, The minimum value between the outputs of Critic target network one and Critic target network two. ,in, This indicates the new global state to which the environment has transitioned. This represents the deterministic action output by the Actor target network under the local observation vector at the next time step. , Indicates that the intelligent agent The local observation vector is input into the agent at the next time step. The corresponding Actor target network yields the deterministic actions of the agent. , This represents the output of Critic target network one. For the parameters of Critic target network one, This represents the output of the Critic target network 2. These are the parameters of Critic target network 2.

10. The active distribution network cooperative optimization method based on multi-agent reinforcement learning according to claim 9, characterized in that: Global rewards gained at each moment for: in, For a moment Daily comprehensive operating cost For a moment The active network loss, This is to account for the potential risk of voltage exceeding limits after the original action is executed. , , These represent the cost weighting coefficient, network loss weighting coefficient, and counterfactual risk penalty coefficient, respectively; among them, the potential voltage over-limit risk after the original action is executed. for: in, It is a set of nodes in a distributed resource-coordinated active distribution network system. To ensure the safety of the projection layer, a global linearized voltage sensitivity model is used to predict the voltage of the original action at the node. Voltage estimate, and These are the lower and upper limits of the node voltage, respectively.

11. A multi-agent reinforcement learning-based active distribution network cooperative optimization system, characterized in that: include: The model building module is used to establish the operation model of each distributed resource and the system constraints for an active distribution network system based on distributed resource collaboration, and to build a global linearized voltage sensitivity model. The agent decision-making module is used to treat each distributed resource as an independent agent, divide the active distribution network node into multiple non-overlapping regions, obtain the local observation vector of each agent in its own region, input the local observation vector of each agent into the corresponding policy network, and output the original action. The original action verification module is used to perform real power flow calculations based on the original actions to obtain the real node voltage, and to verify whether the real node voltage meets the voltage safety constraints. If so, the original action is directly used as the final execution instruction. Otherwise, it will enter the secure projection module; The safety projection module is used to correct the original actions online through a global linearized safety projection layer. It constructs a convex optimization problem with the goal of minimizing the weighted sum of the action correction amount and the degree of voltage constraint violation. The linearized predicted node voltage is used as a voltage constraint in the solution. The active power injection adjustment amount, reactive power injection adjustment amount and node voltage relaxation variable are solved together to obtain candidate safety actions. The deviation compensation and iteration module is used to perform real power flow verification based on candidate safety actions and update the linearization error compensation term according to the deviation between the actual node voltage and the linearized predicted node voltage corresponding to the candidate safety action. If the actual node voltage meets the voltage safety constraint, the candidate safety action is output as the final execution instruction. If the actual node voltage does not meet the voltage safety constraint, the updated linearization error compensation term is substituted back into the convex optimization problem in the safety projection module to solve the action correction amount again, and the safety projection module and the deviation compensation and iteration module are iteratively executed until the node voltage meets the voltage safety constraint or the maximum number of iterations is reached.

Images