A Low-Carbon Dispatch Method for Distribution Networks with Multiple Microgrids Based on Multi-Agent Composite Security Learning

By constructing a hierarchical constraint system and an adaptive penalty scheduling strategy, a multi-agent composite safety learning method is developed, which solves the problems of insufficient constraint hierarchical processing and penalty weights in existing technologies, and achieves better economic benefits and carbon emission performance.

CN123073708APending Publication Date: 2026-11-10YANCHENG POWER SUPPLY CO STATE GRID JIANGSU ELECTRIC POWER CO
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611172208.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-04
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Existing secure multi-agent deep reinforcement learning methods suffer from insufficient constraint hierarchical processing, large gradient link errors, and fixed penalty weights in distribution network-microgrid scheduling, resulting in limited optimization space and insufficient security.

Method used

A multi-agent composite safety learning method is adopted to construct a hierarchical constraint system, eliminate gradient approximation error, and design a two-stage adaptive penalty coefficient scheduling strategy. Hard constraints are forced to be met through physical projection, and soft constraints are guided by penalties, so as to realize the individualized adjustment of penalty intensity by the agents.

Benefits of technology

While ensuring safety constraints, it improved economic benefits and carbon emission performance, reduced total microgrid carbon emissions by 13.3%, reduced the number of violations of original actions by 73%, and increased total rewards by 10.3%.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123073708A_ABST
    Figure CN123073708A_ABST
Patent Text Reader

Abstract

This invention proposes a low-carbon scheduling method for distribution networks with multiple microgrids based on multi-agent composite security learning, belonging to the field of distribution network operation optimization and control technology. Addressing the difficulty in coordinating security constraint assurance and operation optimization when applying existing security multi-agent deep reinforcement learning methods to distribution network-microgrid scheduling, this invention proposes an improved scheme. The method includes the following steps: First, constructing a two-layer low-carbon economic scheduling model hierarchically based on hard and soft constraints; second, designing an input correction mechanism for the evaluation network, enabling the evaluation network to directly learn value functions in the original action space to eliminate gradient approximation errors; finally, designing a two-stage adaptive penalty scheduling strategy, with global linear decay in the early stage to establish a safety baseline, and individualized exponential decay in the later stage to release optimization degrees of freedom. This invention achieves better economic benefits and carbon emission performance than existing methods while strictly ensuring the satisfaction of hard constraints.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of distribution network operation optimization and control technology, specifically to a low-carbon scheduling method for multiple microgrids in distribution networks based on multi-agent composite security learning. Background Technology

[0002] The global energy transition is driving the development of low-carbon power systems, with active distribution networks and microgrids forming the core architecture of distributed energy management. Multi-agent deep reinforcement learning, due to its distributed decision-making capabilities, has been introduced into this field, achieving collaborative optimization through a centralized training-distributed execution framework. However, existing secure multi-agent deep reinforcement learning methods still have shortcomings at three technical levels: constraint systems, gradient links, and penalty weights.

[0003] Specifically, existing active distribution network-multi-microgrid security scheduling methods suffer from the following technical bottlenecks: 1) At the constraint system level, system-level hard constraints and device-level soft constraints differ fundamentally in their violation consequences and recovery characteristics. Existing methods do not classify constraints according to their nature, leading to excessive constraints on the optimization space or ineffective penalties that fail to curb violations of dangerous constraints; 2) At the gradient link level, security projections involve non-differentiable operations, and their projection Jacobian is implicitly taken as the identity matrix, introducing gradient approximation errors that increase with action deviations. Existing methods have not yet provided theoretical analysis and correction schemes; 3) At the penalty weight level, the penalty coefficient remains fixed throughout the process, failing to consider the dynamic requirements of the training process stages and the heterogeneity between agents on the penalty intensity. In summary, existing technologies have significant shortcomings in constraint classification, gradient link accuracy, and penalty weight adaptability. Therefore, it is necessary to design a new method that can classify constraints according to their nature, eliminate gradient approximation errors, and adaptively adjust the penalty intensity according to the training process to meet practical needs. Summary of the Invention

[0004] This invention addresses the shortcomings of existing secure multi-agent deep reinforcement learning methods in low-carbon economic dispatching of active distribution networks and multi-microgrids, particularly in terms of constraint hierarchy, gradient links, and penalty weights. It constructs a multi-agent composite secure learning mechanism from three levels: constraint system, gradient correction, and penalty dispatching, aiming to achieve better economic benefits and carbon emission performance while ensuring that security constraints are met.

[0005] A low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning includes the following steps: Step 1: Establish an active distribution network-multi-microgrid dual-layer low-carbon economic dispatch model, construct a hierarchical constraint system, and divide the operation constraints into two levels, hard constraints and soft constraints, based on the severity of the consequences of violations and the feasibility of recovery. Hard constraints are forcibly satisfied through physical projection to ensure system safety, while soft constraints are added to the reward function in the form of penalty terms to guide the strategy to gradually avoid them. Step 2: Formalize the scheduling problem into a partially observable Markov game model, and modify the input to the evaluation network so that the evaluation network directly uses the original actions output by the agent's policy network as its input learning value function. Q ( s , a This approach incorporates the security projection into the environmental dynamics, eliminating the gradient approximation error introduced in the gradient links of the policy network due to the implicit taking of the projection Jacobian as the identity matrix. Step 3: Design a two-stage adaptive penalty coefficient scheduling strategy. In the first stage, a global linear decay is used to uniformly establish a safety baseline. In the second stage, it switches to individualized exponential decay, and the penalty coefficient is adjusted differently according to the cumulative number of violations of the original actions of each agent, so as to release the optimization freedom of agents with good safety performance. Step 4: Integrate the hierarchical constraint system, evaluate the network input correction and adaptive penalty coefficient scheduling strategy to form an improved multi-agent deep reinforcement learning algorithm, and perform day-ahead low-carbon economic dispatch on the target distribution network system.

[0006] Further: In step 1, the hard constraints of the hierarchical constraint system include: line overload constraints, node voltage over-limit constraints, and gas turbine output over-limit constraints, which are retained in the safety projection stage and forcibly satisfied through physical projection; the physical projection includes equipment action boundary trimming and feasible region correction based on AC power flow constraints, with the goal of minimizing the deviation between the projected action and the action after boundary trimming, to obtain safe actions that satisfy the hard constraints; the soft constraints include: energy storage state of charge edge over-limit constraints, gas turbine ramp rate over-limit constraints, and demand response daily cumulative deviation constraints, which are removed from the physical projection and added to the reward function in the form of penalty terms.

[0007] Furthermore: In step 1, the active distribution network-multi-microgrid dual-layer low-carbon economic dispatch model includes two layers: the active distribution network operator optimization model and the multi-microgrid operator optimization model; the active distribution network operator uses internal electricity price as the decision variable, and the optimization objective is the weighted sum of economic benefits and network cleanliness.

[0008] Furthermore: Each microgrid operator uses gas turbine output, energy storage charging and discharging power, and demand response load transfer volume as decision variables, with the optimization objective being the minimization of the sum of operating costs and carbon emission costs; the active distribution network and each microgrid are coupled with each other through internal electricity prices and trading power; carbon emission costs are calculated based on a spatiotemporal carbon intensity assessment method, which achieves fair allocation of carbon intensity through a distributed hierarchical calculation sequence: the electricity-selling microgrid calculates its own carbon intensity and reports it, the active distribution network synthesizes the system carbon intensity and broadcasts it to all microgrids, and the electricity-purchasing microgrid calculates its own carbon intensity based on the broadcast value.

[0009] Furthermore: In step 2, a partially observable Markov game model is defined as containingN A seven-tuple of agents: ,in N =5, corresponding to 1 active distribution network operator and 4 microgrid operators respectively; S For the global state space; For the first n The action space of each intelligent agent, the active distribution network corresponds to the one-dimensional electricity price adjustment, and each microgrid corresponds to a three-dimensional vector composed of gas turbine output, energy storage charging and discharging power and demand response load transfer. For the first n The local observation space of each intelligent agent includes the aggregated trading power at the point of common coupling, the power injected by the upper-level grid and carbon intensity and electricity price, and its own carbon intensity. The local observation of each microgrid includes the trading power at the point of common coupling, the output and load of the internal gas turbine and the state of charge of the energy storage, the internal electricity price, and the carbon intensity broadcast by the active distribution network. T This is the state transition function; For the first n The reward function for each agent; c This is the discount factor.

[0010] Furthermore: the mathematical principle of evaluating network input correction in step 2 is to define the corrected state transition function and the corrected reward function, and to project the safety data... Absorption is part of environmental dynamics, making the quaternion... This constitutes a standard Markov decision process.

[0011] Furthermore: In step 3, the two-stage adaptive penalty coefficient scheduling strategy is based on the total number of training rounds. T =800 midpoint T / 2=400 is the switching point; first stage t ≤ T / 2, using global linear decay; second stage t > T / 2, switch to individualized index decay.

[0012] Furthermore: the two-stage adaptive penalty coefficient scheduling strategy synergizes with the hierarchical constraint system, and the soft constraint penalty weights... , Using EAP penalty coefficient l The same two-stage scheduling logic.

[0013] Furthermore, in step 4, the improved multi-agent deep reinforcement learning algorithm adopts a centralized training-decentralized execution architecture. During training, the centralized dual evaluation network acquires the global state and global original actions for centralized learning. During execution, each agent only needs a local policy network to independently output the original actions with local observations as input. After being corrected by a safety projection containing only hard constraints, the actions are sent out for execution. The evaluation network does not participate in the execution stage.

[0014] Furthermore: The improved multi-agent deep reinforcement learning algorithm performs the following process in each round of the training phase: First, calculate the current adaptive penalty coefficient. l Value; then iterates through the 24-hour scheduling cycle, each agent outputs its original action based on local observations through the policy network, and adds exploration noise. The original action is corrected into a safe action by a safety projection containing only hard constraints and then sent to the environment for execution; the environment returns a reward and adds a soft constraint penalty term; transition tuple Store the data in the experience replay buffer; when the accumulated number of samples in the buffer reaches the preset batch size, sample a small batch to update the network, where the network is evaluated using the original actions. a Learning the timing difference error for input computation Policy networks Q The objective is to maximize the value and the weighted sum of EAP penalties, and the parameters are updated once every fixed number of steps using a deterministic policy gradient.

[0015] The beneficial effects of adopting the above technical solution in this invention are: 1. This invention addresses the lack of hierarchical constraint handling. It categorizes operational constraints into hard and soft constraints based on their violation consequences and recovery characteristics. Hard constraints are enforced to ensure safety, while soft constraints penalize and guide gradual optimization. This approach ensures system safety while allowing for greater flexibility in scheduling device-level constraints.

[0016] 2. Eliminates gradient approximation errors introduced by safe projection. This invention incorporates safe projection as part of the environmental dynamics, allowing the evaluation network to learn value functions directly in the original action space. The gradient path no longer contains non-differentiable projection links, fundamentally avoiding gradient bias caused by the implicit approximation of the projected Jacobian as an identity matrix.

[0017] 3. The penalty coefficient achieves dual adaptation to both the training process and individual agents. A global linear decay is used in the early stages to establish a uniform safety baseline, while individualized exponential decay is adjusted based on the differences in the number of violations of the original actions. This avoids the solidification of exploration noise bias into a fixed bias, while allowing agents with good safety performance greater optimization freedom.

[0018] 4. Significant overall technical benefits. The results of the implementation examples show that the number of violations of the original actions decreased by approximately 73%, the total carbon emissions of the microgrid decreased by approximately 13.3%, and the total rewards increased by approximately 10.3%. The combined effect of these three improvements is progressively stronger, achieving better economic benefits and carbon emission performance than existing methods while ensuring strict compliance with hard constraints. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the overall algorithm architecture and training-execution process of the method of the present invention.

[0020] Figure 2 This is a comparison chart of the total reward training curves under five different methods.

[0021] Figure 3 This is a comparison chart of the training curves for the number of violations of the original movements under five different methods.

[0022] Figure 4 This is a training curve diagram of the ADN reward for the complete method of this invention. Detailed Implementation

[0023] The following will refer to the appendices in the embodiments of the present invention. Figure 1-4 The technical solutions in the embodiments of the present invention will be clearly and completely described. Example 1

[0024] Figure 1 This is a schematic diagram of the overall algorithm architecture and training-execution process of the method of the present invention. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning in this embodiment includes: Step 1: Establish an active distribution network-multi-microgrid dual-layer low-carbon economic dispatch model, construct a hierarchical constraint system, and divide the operation constraints into two levels, hard constraints and soft constraints, based on the severity of the consequences of violations and the feasibility of recovery. Hard constraints are forcibly satisfied through physical projection to ensure system safety, while soft constraints are added to the reward function in the form of penalty terms to guide the strategy to gradually avoid them. Step 2: The scheduling problem is formalized into a partially observable Markov game model, providing a mathematical framework for multi-agent deep reinforcement learning. The input to the evaluation network is then corrected, allowing the evaluation network to directly use the original actions output by the agent's policy network as its input learning value function. Q ( s , a This approach incorporates the security projection as part of the environmental dynamics, thereby eliminating the gradient approximation error introduced in the policy network gradient links due to the implicit taking of the projection Jacobian as the identity matrix. Step 3: Design a two-stage adaptive penalty coefficient scheduling strategy. In the first stage, a global linear decay is used to uniformly establish a safety baseline. In the second stage, it switches to individualized exponential decay, and the penalty coefficient is adjusted differently according to the cumulative number of violations of the original actions of each agent, so as to release the optimization freedom of agents with good safety performance. Step 4: Integrate the hierarchical constraint system, evaluate the network input correction and adaptive penalty coefficient scheduling strategy to form an improved multi-agent deep reinforcement learning algorithm, and perform day-ahead low-carbon economic dispatch on the target distribution network system.

[0025] In step 1, the hard constraints of the hierarchical constraint system include line overload constraints, node voltage over-limit constraints, and gas turbine output over-limit constraints. Violations of these constraints are irreversible and directly endanger system operational safety; therefore, they are retained in the safety projection stage and forcibly satisfied through physical projection. Soft constraints include energy storage state-of-charge edge over-limit constraints, gas turbine ramp rate over-limit constraints, and daily cumulative deviation constraints in demand response. Temporary violations of these constraints can be recovered through subsequent periods and do not endanger system safety; therefore, they are removed from the physical projection and added to the reward function as penalty terms. (1) In the formula, This is a penalty item for violating soft constraints; Weighting for penalties when the state of charge of energy storage exceeds limits; Weighting of penalties for exceeding ramp limits on gas turbines; for t Constant energy storage state of charge; As the lower limit of the energy storage state of charge, it is taken as 10% of the rated capacity; The upper limit of the energy storage state of charge is taken as 90% of the rated capacity; This represents the change in gas turbine output. This is the limit for the gradeability of a gas turbine.

[0026] In this embodiment, the security projection is implemented using physical projection with hard constraints. The physical projection includes two stages: device action boundary pruning and AC power flow feasible region correction. First, the original joint actions output by each agent's policy network are... The preliminary corrective action is obtained by tailoring the parameters according to the allowable ranges of gas turbine output, energy storage charging and discharging power, and demand response adjustment. .

[0027] Based on the current system status The hard-constrained feasible region is composed of AC power balance constraints, node voltage upper and lower limits constraints, line capacity constraints, gas turbine output upper and lower limits constraints, and action boundaries. And solve the minimum motion deviation projection problem: (2) In the formula, Safety actions are obtained through physical projection; The combined actions to be solved; for and The square of the Euclidean distance between them is used to measure the adjustment magnitude of the physical projection to the initial correction action.

[0028] The aforementioned nonlinear programming problem with AC power flow constraints is solved iteratively using the primordial-dual interior-point method. When the solution satisfies the hard constraints, This was issued and executed as a safety measure.

[0029] If the projection problem fails to converge within the preset maximum number of iterations, the first time period adopts the safety action corresponding to the initial operating state of each device, and the subsequent time periods adopt the safety action of the previous time period, and the AC power flow verification is performed again; if the verification fails or the feasible domain of the hard constraint is empty, the current action is stopped and the projection failure is recorded.

[0030] The active distribution network-multi-microgrid two-layer low-carbon economic dispatch model comprises two layers: an active distribution network operator optimization model and a multi-microgrid operator optimization model. Active distribution network operators use internal electricity prices as decision variables, with the optimization objective being a weighted sum of economic benefits and network cleanliness. Each microgrid operator uses gas turbine output, energy storage charging / discharging power, and demand response load transfer volume as decision variables, with the optimization objective being the minimization of the sum of operating costs and carbon emission costs. The active distribution network and each microgrid are coupled through internal electricity prices and trading power. Carbon emission costs are calculated based on a spatiotemporal carbon intensity assessment method. This method employs a distributed hierarchical calculation sequence: the microgrid selling electricity calculates its own carbon intensity and reports it; the active distribution network synthesizes the system carbon intensity and broadcasts it to all microgrids; and the microgrid purchasing electricity calculates its own carbon intensity based on the broadcast value, achieving a fair allocation of carbon intensity. The active distribution network carbon intensity, after network loss correction, is used for subsequent optimization.

[0031] In the spatiotemporal carbon intensity assessment method adopted by the low-carbon economic dispatch model, the carbon intensity of the active distribution network is calculated by power-weighted averaging of the carbon intensity of the upstream power grid and each retail microgrid, and then corrected for network losses. (3) In the formula, Carbon intensity of active distribution network after network loss correction; Carbon intensity of the active distribution network (uncorrected); This refers to the power loss of the active distribution network. Injecting power into the upstream power grid; For electricity sales microgrid aggregation; For micro-network m Trading power with the active distribution network.

[0032] Based on the active distribution network carbon intensity after network loss correction, the network cleanliness is characterized by the reduction ratio of the active distribution network carbon intensity relative to the upstream grid carbon intensity. The calculation formula is as follows: (4) In the formula, for Network cleanliness during a given time period; Carbon intensity of the upper-level power grid.

[0033] The higher the network cleanliness, the lower the carbon intensity of the active distribution network relative to the carbon intensity of the upstream grid, and the higher the low-carbon and cleanliness level of the active distribution network.

[0034] Based on the aforementioned network cleanliness, active distribution network operators use internal electricity prices as decision variables and maximize the weighted sum of economic benefits and network cleanliness as their optimization objective. Their objective function is: (5) In the formula, This refers to the net profit from the price difference obtained by the active distribution network and various microgrids through power transactions; This serves as the benchmark for normalized economic returns. This is a trade-off coefficient between economic benefits and network cleanliness.

[0035] In this embodiment, both the normalized economic benefit term and the network cleanliness term are dimensionless. Considering both the economic efficiency and low-carbon operation of the active distribution network, and maintaining a slightly higher optimization weight for economic benefits, the following approach is adopted: The weight of the corresponding network cleanliness item is 0.45.

[0036] In step 2, a partially observable Markov game model is defined as containing N A seven-tuple of agents: ,in N =5, corresponding to 1 active distribution network operator and 4 microgrid operators respectively; S For the global state space; For the first n The action space of each intelligent agent, the active distribution network corresponds to the one-dimensional electricity price adjustment, and each microgrid corresponds to a three-dimensional vector composed of gas turbine output, energy storage charging and discharging power and demand response load transfer. For the first n The local observation space of each intelligent agent includes the aggregated trading power at the point of common coupling, the power injected by the upper-level grid and carbon intensity and electricity price, and its own carbon intensity. The local observation of each microgrid includes the trading power at the point of common coupling, the output and load of the internal gas turbine and the state of charge of the energy storage, the internal electricity price, and the carbon intensity broadcast by the active distribution network. T This is the state transition function; For the first n The reward function for each agent; c This is the discount factor.

[0037] The mathematical principle for evaluating network input correction is to define the corrected state transition function. and the revised reward function , to project security Absorption is part of environmental dynamics, making the quaternion... For a standard Markov decision process, the Bellman equation holds strictly in the original action space: (6) In the formula, For state s Execute the original action a Action value function; An instant reward after secure projection; c Discount factor; The next state is represented by π; π represents the deterministic strategy. T This is the state transition function; This refers to the safety action after being corrected by safety projection.

[0038] Under this design, the policy network loss function and gradient They are respectively: (7) (8) In the formula, Let the policy network loss function be used. f Here are the policy network parameters; E is the expectation. For parameters i Evaluation network; a The original action output by the policy network; a 'Safety actions after safe projection;' l EAP penalty coefficient; For parameters f The strategy network; o This is a local observation.

[0039] First item The evaluation network is accurately calculated through backpropagation. It is a standard multilayer perceptron, differentiable at every point, and the gradient path originates from the policy network output and passes through the original action. a To evaluate network output Q ( a The backward propagation of the projected Jacobian has no non-differentiable components, thus fundamentally eliminating the projection Jacobian. The gradient approximation error caused by being implicitly approximated as the identity matrix.

[0040] The two-stage adaptive penalty coefficient scheduling strategy described in step 3 is based on the total number of training rounds. T =800 midpoint T / 2=400 is the switching point; first stage t ≤ T / 2, using global linear decay: (9) In the formula, For the first t A uniform penalty coefficient for each training round; The initial penalty coefficient is set to 0.5. β The decay rate parameter is set to 0.8; T This represents the total number of training rounds.

[0041] Phase Two t > T / 2, switch to individualized exponential decay: (10) In the formula, For the first m The micro-network agent in the first t Individualized penalty coefficient for each training round; For the first m The cumulative number of violations of original actions by each micro-network intelligent agent, in order to As a criterion for a single violation; t The exponential decay time constant is set to 5; l ( T / 2)≈0.3 is the uniform penalty coefficient value at the switching time; the active distribution network intelligent agent has no EAP penalty item. .

[0042] The two-stage adaptive penalty coefficient scheduling strategy and the hierarchical constraint system work together to form a synergy, with soft constraint penalty weights. , Using EAP penalty coefficient l The same two-stage scheduling logic enables unified control of the safety guidance intensity.

[0043] In step 4, the improved multi-agent deep reinforcement learning algorithm adopts a centralized training-decentralized execution architecture. During training, the centralized dual evaluation network acquires the global state and global original actions for centralized learning. During execution, each agent only needs a local policy network to independently output the original actions with local observations as input. After being corrected by a safety projection containing only hard constraints, the actions are sent out for execution. The evaluation network does not participate in the execution stage.

[0044] The improved multi-agent deep reinforcement learning algorithm performs the following process in each round of the training phase: First, calculate the current adaptive penalty coefficient. l Value; then iterates through the 24-hour scheduling cycle, each agent outputs its original action based on local observations through the policy network, and adds exploration noise. The original action is corrected into a safe action by a safety projection containing only hard constraints and then sent to the environment for execution; the environment returns a reward and adds a soft constraint penalty term; transition tuple An experience replay buffer with a capacity of 50,000 records is stored; when the accumulated number of samples in the buffer reaches the preset batch size of 256, a small batch is sampled to update the network, in which the evaluation of the network is based on the original actions. a Learning the timing difference error for input computation Policy networks Q The objective is to maximize the weighted sum of values ​​and EAP penalties, updating the parameters every two steps via a deterministic policy gradient; the target network uses soft-update coefficients. Track online parameters. Example 2

[0045] To verify the effectiveness of the method of this invention, simulation analysis was conducted on a modified IEEE 33-node radial power distribution system. This system contains 32 branches, a voltage level of 12.66 kV, a total active power load of 3715 kW, and a total reactive power load of 2547 kvar. Four microgrids are connected to bus 17, bus 21, bus 24, and bus 32, respectively. Five ablation experiment configurations were set up: E0 was the MATD3 method without safety constraints; E1 was the EAP-MATD3 baseline method with a fixed penalty coefficient λ=0.5 and the evaluation network taking the projected safety action as input; E2 introduced a hierarchical constraint system based on E1; E3 superimposed evaluation network input correction on E2; and E4 superimposed a two-stage hybrid penalty scheduling on E3, which is the complete method of this invention. Each configuration was run independently five times, and the average result was taken as the final result to verify the independent effectiveness and superimposed gain of the three improvements.

[0046] Figure 2 This section compares the total reward convergence processes of the five methods. E0 has a final reward of approximately -1923, but its average number of violations is as high as 13.7, sacrificing safety and highlighting the indispensability of safety mechanisms. E1 relies on a unified safety projection to ensure stable convergence, with a final value of approximately -2151, but fixed penalties and gradient approximation errors constrain the optimization space. E2 introduces hierarchical constraints to release scheduling flexibility, improving the final value to approximately -2066. E3 eliminates gradient link errors by superimposing evaluation network input corrections, resulting in a final value of approximately -2013. E4 integrates three improvements, achieving a final value of approximately -1929, an improvement of approximately 10.3% compared to the E1 baseline.

[0047] Figure 3This section compares the average number of violations for the original actions of the five methods. E0, without safety constraints, has an average violation count as high as 13.7. E1, after unifying the physical projection, reduces the number of violations to 6.3, but the fixed penalty causes convergence stagnation in the later stages. E2, after introducing hierarchical constraints, reduces the number of violations to 4.1; E3, after overlaying the evaluation network input correction, reduces it to 3.7; and E4, after integrating two-stage adaptive penalty scheduling, reduces it to 1.7, a reduction of approximately 73% compared to the E1 baseline. This verifies the cumulative gain of the three improvements in satisfying safety constraints.

[0048] Figure 4 The diagram shows the reward training curves for the complete ADN method of this invention, illustrating the reward convergence of the active distribution network agent throughout the training process. The agent's reward fluctuates significantly in the early stages of training due to high exploration noise, but gradually converges to a stable level as training progresses. Furthermore, the reward convergence trend is consistent with the overall reward convergence trend, verifying the training stability and consistency of the method under multi-agent distributed decision-making.

[0049] The overall performance comparison is shown in Table 1. The method of this invention reduces total carbon emissions from microgrids by 21.8% compared to MATD3 and by 13.3% compared to EAP-MATD3; the number of violations of original actions is reduced by 87.6% compared to MATD3 and by 73.0% compared to EAP-MATD3.

[0050]

[0051] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning, characterized in that, Includes the following steps: Step 1: Establish an active distribution network-multi-microgrid dual-layer low-carbon economic dispatch model, construct a hierarchical constraint system, and divide the operation constraints into two levels: hard constraints and soft constraints. Hard constraints are forcibly satisfied through physical projection to ensure system safety, while soft constraints are added to the reward function in the form of penalty terms to guide the strategy to gradually avoid them. Step 2: Formalize the scheduling problem into a partially observable Markov game model, and modify the input to the evaluation network so that the evaluation network directly uses the original actions output by the agent's policy network as its input learning value function. Q ( s , a This approach incorporates the security projection into the environmental dynamics, eliminating the gradient approximation error introduced in the gradient links of the policy network due to the implicit taking of the projection Jacobian as the identity matrix. Step 3: Design a two-stage adaptive penalty coefficient scheduling strategy. In the first stage, a global linear decay is used to uniformly establish a safety baseline. In the second stage, it switches to individualized exponential decay, and the penalty coefficient is adjusted differently according to the cumulative number of violations of the original actions of each agent, so as to release the optimization freedom of agents with good safety performance. Step 4: Integrate the hierarchical constraint system, evaluate the network input correction and adaptive penalty coefficient scheduling strategy to form an improved multi-agent deep reinforcement learning algorithm, and perform day-ahead low-carbon economic dispatch on the target distribution network system.

2. The low-carbon scheduling method for multiple microgrids in distribution networks based on multi-agent composite security learning according to claim 1, characterized in that: In step 1, the hard constraints of the hierarchical constraint system include: line overload constraints, node voltage over-limit constraints, and gas turbine output over-limit constraints, which are retained in the safety projection stage and forcibly satisfied through physical projection. The physical projection includes equipment action boundary trimming and feasible region correction based on AC power flow constraints, with the goal of minimizing the deviation between the projected action and the action after boundary trimming, to obtain safe actions that satisfy the hard constraints. The soft constraints include: energy storage state of charge edge over-limit constraints, gas turbine ramp rate over-limit constraints, and demand response daily cumulative deviation constraints, which are removed from the physical projection and added to the reward function in the form of penalty terms.

3. The low-carbon scheduling method for multiple microgrids in distribution networks based on multi-agent composite security learning according to claim 1, characterized in that: In step 1, the active distribution network-multi-microgrid two-layer low-carbon economic dispatch model includes two layers: the active distribution network operator optimization model and the multi-microgrid operator optimization model. The active distribution network operator uses internal electricity price as the decision variable, and the optimization objective is the weighted sum of economic benefits and network cleanliness.

4. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 3, characterized in that: Each microgrid operator uses gas turbine output, energy storage charging and discharging power, and demand response load transfer as decision variables, with the optimization objective being the minimization of the sum of operating costs and carbon emission costs. The active distribution network and each microgrid are coupled with each other through internal electricity prices and trading power. Carbon emission costs are calculated based on a spatiotemporal carbon intensity assessment method, which achieves fair allocation of carbon intensity through a distributed hierarchical calculation sequence: the electricity-selling microgrid calculates its own carbon intensity and reports it, the active distribution network synthesizes the system carbon intensity and broadcasts it to all microgrids, and the electricity-purchasing microgrid calculates its own carbon intensity based on the broadcast value.

5. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 1, characterized in that: In step 2, a partially observable Markov game model is defined as containing N A seven-tuple of agents: ,in N =5, corresponding to 1 active distribution network operator and 4 microgrid operators respectively; S For the global state space; For the first n The action space of each intelligent agent, the active distribution network corresponds to the one-dimensional electricity price adjustment, and each microgrid corresponds to a three-dimensional vector consisting of gas turbine output, energy storage charging and discharging power and demand response load transfer. For the first n The local observation space of each intelligent agent includes the aggregated trading power at the point of common coupling, the power injected by the upper-level grid and carbon intensity and electricity price, and its own carbon intensity. The local observation of each microgrid includes the trading power at the point of common coupling, the output and load of the internal gas turbine and the state of charge of the energy storage, the internal electricity price, and the carbon intensity broadcast by the active distribution network. T This is the state transition function; For the first n The reward function for each agent; γ This is the discount factor.

6. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 1, characterized in that: The evaluation network input correction mentioned in step 2 involves defining a corrected state transition function and a corrected reward function, and then projecting the safety data onto the network. Absorption is part of environmental dynamics, making the quaternion... This constitutes a standard Markov decision process.

7. The low-carbon scheduling method for multiple microgrids in distribution networks based on multi-agent composite security learning according to claim 1, characterized in that: In step 3, the two-stage adaptive penalty coefficient scheduling strategy is based on the total number of training rounds. T midpoint T / 2 is the switching point; Phase 1 t ≤ T / 2, using global linear decay; second stage t > T / 2, switch to individualized index decay.

8. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 7, characterized in that: The two-stage adaptive penalty coefficient scheduling strategy and the hierarchical constraint system work together to form a synergy, with soft constraint penalty weights. , Using EAP penalty coefficient λ The same two-stage scheduling logic.

9. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 1, characterized in that: In step 4, the improved multi-agent deep reinforcement learning algorithm adopts a centralized training-decentralized execution architecture. During training, the centralized dual evaluation network acquires the global state and global original actions for centralized learning. During execution, each agent only needs a local policy network to independently output the original actions with local observations as input. After being corrected by a safety projection containing only hard constraints, the actions are sent out for execution. The evaluation network does not participate in the execution stage.

10. The low-carbon scheduling method for multiple microgrids in a distribution network based on multi-agent composite security learning according to claim 9, characterized in that: The improved multi-agent deep reinforcement learning algorithm performs the following process in each round of the training phase: First, calculate the current adaptive penalty coefficient. λ value; Subsequently, the scheduling cycle is traversed. Each agent outputs the original action based on local observations through the policy network and adds exploration noise. The original action is corrected into a safe action by a safe projection containing only hard constraints and then sent to the environment for execution. The environment returns a reward and then adds a soft constraint penalty term. Transition tuple Store the data in the experience replay buffer; once the number of accumulated samples in the buffer reaches the preset batch size, sample a small batch for network updates.