A Method for Steady-State Operation and Dynamic Decision-Making of Microgrids Based on Hierarchical Reinforcement Learning Architecture
By employing a hierarchical reinforcement learning architecture and federated learning methods, the problems of data privacy leakage, unstable policy updates, and low resource utilization efficiency in multi-microgrid systems are solved. This enables steady-state operation and dynamic decision-making across microgrids, improving overall system efficiency and equipment lifespan management.
Patent Information
- Application Number
- CN202611196185.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-07
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies in multi-microgrid systems suffer from problems such as data privacy leakage risks, low training efficiency, unstable policy updates, lack of constraints on energy trading mechanisms, and overuse of energy storage devices, resulting in low overall system resource utilization efficiency.
A hierarchical reinforcement learning architecture is adopted, which constructs a federated learning global coordination layer and a proximal policy optimization local control layer. By using partially observable Markov game modeling and proximal policy optimization algorithms, combined with the energy trading mechanism between microgrids and a three-objective reward function, privacy protection and policy stability across microgrids are achieved.
While protecting data privacy, it improves the resource utilization efficiency and steady-state operation level of multi-microgrid systems, prevents opportunistic bidding behavior, and takes into account both the long-term economic benefits of the system and the lifespan management of equipment.
Smart Images

Figure CN123117601A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of smart grid optimization scheduling technology, and in particular to a microgrid steady-state operation and dynamic decision-making method based on a hierarchical reinforcement learning architecture. Background Technology
[0002] With the deepening of the "carbon peaking and carbon neutrality" goals, the penetration rate of renewable energy, represented by wind power and solar photovoltaic, in the power system continues to increase. Microgrids, as a typical form of distributed energy system, typically include wind power units, photovoltaic power units, diesel generators, fuel cells, energy storage systems, and various loads, enabling local consumption and flexible dispatch of distributed resources. However, with a large number of distributed energy sources connected to the distribution network, the traditional centralized dispatch model faces severe challenges: on the one hand, the randomness and intermittency of renewable energy output increases the difficulty of power balancing; on the other hand, multiple microgrids belong to different management entities, and there is a real need for data privacy protection among them, making it difficult to achieve centralized optimization by sharing complete operational information. How to achieve coordinated dispatch and steady-state operation of multiple microgrid systems while ensuring data privacy has become a key issue that urgently needs to be solved in the field of smart grids.
[0003] Existing research has established a multi-timescale control system for microgrid dispatching: the day-ahead planning dispatch layer, based on short-term power forecasts, formulates the output plans of each distributed power source for the next day with the goal of minimizing economic and environmental costs; the intraday rolling correction layer, based on ultra-short-term power forecasts, introduces adjustment costs on top of economic and environmental costs through model predictive control methods to dynamically correct the day-ahead plan; and the real-time coordination control layer, on a second-level timescale, uses the power allocation strategy of a hybrid energy storage system to absorb forecast errors and smooth power fluctuations. While the above multi-timescale framework has achieved good results in single microgrid scenarios, the establishment of dispatching models at each level usually relies on accurate forecasts of renewable energy output and load demand, resulting in limited adaptability to uncertainties and an inability to effectively address the need for cross-entity coordination optimization when multiple microgrids are operating in parallel.
[0004] In multi-agent decision-making, the multi-agent deep deterministic policy gradient algorithm adopts a centralized training and decentralized execution architecture. It alleviates the environmental non-stationarity problem in multi-agent training by introducing a commentator network containing global state information and incorporates market dynamics signals into the commentator network to approximate a centralized Q-value function. However, this method requires the commentator network's input to encompass the state and action information of all agents. As the number of agents increases, the network input dimension expands dramatically, significantly reducing training efficiency. Furthermore, the centralized training framework requires aggregating complete operational data from each microgrid, facing the risk of data privacy leaks, making it difficult to implement in real-world scenarios where microgrids belong to different management entities.
[0005] Federated learning, as a distributed machine learning paradigm, allows multiple participants to jointly optimize a global model through an iterative process of local training, parameter uploading, and aggregate broadcasting, without sharing the original data. This provides an effective way to address the aforementioned privacy protection issues. Existing research combining federated learning with multi-agent reinforcement learning shows that parameter aggregation mechanisms can effectively improve the generalization ability of agents and overcome the problem of single microgrids getting trapped in local optima due to insufficient data diversity. However, such research generally uses deep deterministic policy gradient algorithms for self-training at the local execution level. This algorithm suffers from insufficient policy update stability and struggles to guarantee continuous and stable policy improvement in scenarios with drastic fluctuations in renewable energy output and load demand, resulting in a significant bottleneck in the decision-making quality of the local control layer.
[0006] Furthermore, most existing research on reinforcement learning scheduling of multiple microgrids focuses on the independent operation of each microgrid and its interaction with the external power grid, failing to fully explore the potential for energy complementarity and regulation among microgrids. When each microgrid operates in isolation, microgrids with surplus energy can only sell their excess power to the external distribution network at a lower feeder price, while microgrids with insufficient energy must purchase electricity from the external grid at a higher market price. This results in poor overall system economic efficiency, and the flexibility potential of distributed resources is not fully realized.
[0007] In summary, existing technologies suffer from the following main shortcomings: First, centralized multi-agent reinforcement learning methods face the risk of data privacy leakage, and training efficiency deteriorates sharply as the scale of agents increases. Second, existing federated multi-agent reinforcement learning methods generally use deterministic policy gradient algorithms such as DDPG / TD3 for local execution, resulting in insufficient policy update stability and a tendency to crash during training under scenarios with drastic fluctuations in renewable energy output and load demand. Third, existing methods lack effective constraints on trading prices in inter-microgrid energy trading mechanisms, failing to prevent opportunistic bidding behavior by participants and leading to low overall system resource utilization efficiency. Fourth, existing reward function designs mostly only consider operating costs and power balance, failing to incorporate the cycle life loss of energy storage systems into the decision-making objectives, resulting in excessive use of energy storage devices by scheduling strategies and affecting the long-term economic viability of the system. Summary of the Invention
[0008] The technical solution of this invention is a microgrid steady-state operation and dynamic decision-making method based on a hierarchical reinforcement learning architecture, comprising the following steps: S1. Construct a hierarchical control architecture: Deploy a two-level control system in a distributed system containing N microgrids. The set of microgrids is denoted as N={1,2,...,N}, the scheduling time domain is T={1,2,...,T}, and the length of each time step is Δt. The upper layer is the federated learning global coordination layer, which is responsible for the aggregation of model parameters and policy broadcasting across microgrids. The lower layer is the local control layer for local policy optimization of each microgrid. Each microgrid energy management center acts as an intelligent agent and independently outputs scheduling decisions based on real-time local observations. S2. Establish physical models for each microgrid and perform partially observable Markov game modeling: Mathematically describe the diesel generators, fuel cells, renewable energy generators, energy storage systems, and loads in each microgrid, and construct a complete constraint system including power balance constraints, equipment operation constraints, and energy storage operation constraints; Formulate the multi-microgrid collaborative optimization problem as a partially observable Markov game, and determine the observation space, action space, reward function, and joint state transition probability of each microgrid agent; S3. Each microgrid agent uses a near-end policy optimization algorithm for local training: In each federated training cycle, each microgrid agent uses the received global aggregation parameters as initial values and performs several rounds of local self-training in parallel; The parameters of the actor network and the critic network are updated by pruning loss function and generalized advantage estimation method respectively. The critic network introduces real-time transaction price signals between microgrids on the basis of local observation and action to stabilize multi-agent training and obtain the local optimal scheduling strategy. S4. Establish an inter-microgrid energy trading mechanism: Allow microgrids with surplus energy to sell their excess power to microgrids with insufficient energy using internal bids. Microgrids with insufficient energy should prioritize purchasing power from the participant with the lowest bid in the internal market. When the internal market cannot meet the demand gap, the remaining power should be supplemented from the external distribution network. This power surplus and deficit mitigation mechanism will reduce the overall system power purchase cost and improve the efficiency of distributed resource utilization. Internal bids must meet the price constraint of the feeder grid price as the lower limit and the external power purchase price as the upper limit to prevent opportunistic bidding behavior by participants. S5. Perform federated learning parameter aggregation and broadcasting: After each microgrid agent completes local training, it uploads the parameters of the actor network and the critic network to the central server. The server calculates a weighted average based on the scale of local training data of each microgrid, generates a global aggregated model, and broadcasts the aggregated parameters to all microgrid agents to replace their local parameters, thus starting the next round of federated training. S6. Achieve steady-state optimized operation of multi-microgrid systems: After multiple rounds of federated training iterations, each microgrid agent forms a scheduling strategy that takes into account both global coordination and local optimization; during the execution phase, each agent outputs control commands independently based on its own real-time local observations, without needing to communicate in real time with other microgrids or the central server, thereby achieving steady-state operation and dynamic decision-making of multi-microgrid systems under a privacy-preserving distributed architecture.
[0009] Preferably, the physical model of each microgrid in step S2 includes the following sub-models: Diesel generator model: The output of the diesel generator at time step t is denoted as PDE(t), and it must simultaneously satisfy the upper and lower limits of output constraints and the ramping constraint: ; ; in , These represent the upper and lower limits of the diesel generator's output. , These are the upper and lower limits of the maximum allowable climbing power within a single time step; The output of the fuel cell at time step t Similarly, the following must be satisfied: ; Energy storage system model: The dynamic equation of the state of charge of the energy storage system at time step t is: ; in Let be the state of charge at time step t. For energy storage charging and discharging efficiency, For the rated capacity of energy storage, This represents the charging and discharging power; a positive value indicates charging, and a negative value indicates discharging. The state of charge and charge / discharge power must meet capacity and power constraints: ; ; Power balance constraint: Microgrid i must satisfy power balance within time step t, while also taking into account power exchange with other microgrids and the external distribution network: ; in To meet load demand, , They are respectively powered by wind and solar power. This represents the net exchange power between microgrids and with the external power grid; purchases are positive, and sales are negative. The load shedding power; the power deviation of microgrid i at time step t. Defined as the difference between load demand and actual local power generation: ; Preferably, in step S2, the partially observable Markov game consists of a plural group (O1,...,O) N A1,...,A N , T,r1,...,r NThe description and definitions of each component are as follows: Local observations of each microgrid agent i at time step t It only contains local information that it can perceive, and does not include the internal state of other microgrids: ; in To meet the load demand of the previous time step, To contribute to the next stage of renewable energy, In the state of energy storage charge, The transaction price between the previous time step and the distribution network is t, which represents the current time period. Each microgrid agent i generates scheduling actions based on local observations at time step t. This includes the output of the diesel generator, the output of the fuel cell, and the charging and discharging power of the energy storage system, and ensures that the actions meet the operating constraints of each device through a tailoring function: ; ; ; The instantaneous reward obtained by each microgrid agent i at time step t It consists of three physical objectives: operating cost, energy self-sufficiency rate, and energy storage lifetime penalty. ; in ∈[0,1]、 ∈[0,1]、 ∈[0,1] represents the weighting coefficients for balancing the three scalar items; , , These are the operation and maintenance costs for diesel generators, fuel cells, and energy storage systems, respectively. For fuel costs, The real-time electricity price for purchasing electricity from the distribution network at time step t; the energy self-sufficiency rate term measures the economic cost of purchasing electricity from outside the network by multiplying the power deviation by the real-time electricity price, and is in monetary terms consistent with the operating cost term. To account for the energy storage lifespan loss cost, the charging and discharging power of the energy storage system at time step t is calculated using the rainflow counting method. This is mapped to an equivalent cyclic depreciation amount, which is incorporated into rewards and penalties in the form of depreciation costs, thereby endogenizing the long-term lifespan management objectives of energy storage devices in scheduling decisions; the optimization objective of each microgrid agent is to maximize the cumulative discount reward within the scheduling cycle. ; in ∈[0,1] is the discount factor.
[0010] The overall reward of a multi-microgrid system is the sum of the rewards for each agent: ; Preferably, in step S3, the commentator network uses real-time inter-microgrid transaction price signals during the training phase. Introducing a Q-value function allows for approximate centralized evaluation without exposing the complete state and operational information of other microgrids: ; The actor network loss function employs a pruning form optimized using a near-end strategy: ; in and These are the old and new strategies updated in round k, respectively. For trimming parameters, The advantage function is calculated using the generalized advantage estimation method.
[0011] The critic network uses a loss function Conduct training: ; Actor Network Parameters With commentator network parameters Update each loss function based on its own gradient: ; in and These are the learning rates for the actor network and the critic network, respectively. By pruning the loss function, the policy update magnitude is limited to a reasonable range, effectively preventing training collapse caused by drastic policy changes. This ensures continuous and stable policy improvement even in scenarios with drastic fluctuations in renewable energy output and load demand.
[0012] Preferably, the energy trading mechanism between microgrids in step S4 stipulates that, at time step t, the microgrid i with insufficient energy will preferentially offer the lowest bid to the energy-surplus participant in the internal market. Electricity purchase: ; in For the microgrid l, the internal quote at time step t, To identify variables, if microgrid l has no electricity to sell, it is set to infinity; to prevent opportunistic bidding behavior by participants, internal bids must meet price constraints: ; in The feeder electricity price constitutes the lower limit of the internal quotation; The external electricity purchase price forms the upper limit for bids; the above constraints ensure that the internal transaction price always lies between the feeder grid price and the external electricity purchase price, which both incentivizes microgrids with surplus energy to participate in internal market transactions and prevents bids from exceeding the external electricity purchase price, thus harming the interests of buyers and maintaining the long-term stable operation of the internal market; when all participants in the internal market are unable to meet their electricity purchase demand, the shortfall is covered by the external distribution network at market prices. To supplement: the power interaction between microgrids must simultaneously satisfy conservation constraints, transmission line capacity constraints, and upper and lower limit constraints for the operation of each device.
[0013] Preferably, the parameter aggregation mechanism of federated learning in step S5 is defined as follows: after each microgrid agent j completes the e-th round of local training, it aggregates the parameters... Uploaded to a central server, which then determines the size of each participant's local training data. Calculate the weighted average to generate global aggregation parameters: ;
[0014] After aggregation is complete, the server will The model parameters are broadcast to all microgrid agents and their local parameters are replaced, serving as the initial parameters for the next federated training cycle. The entire process only transmits model parameters and does not share any raw operational data, thereby protecting the data privacy of each microgrid.
[0015] Compared with the prior art, the beneficial effects of the present invention include: This invention discloses a microgrid steady-state operation and dynamic decision-making method based on a hierarchical reinforcement learning architecture. By constructing a two-level hierarchical architecture—upper-level federated learning for global coordination and lower-level near-end policy optimization for local control—it achieves cross-microgrid knowledge sharing and policy generalization while protecting the data privacy of each microgrid, effectively overcoming the problem of local optima caused by insufficient data diversity in a single microgrid. The lower layer uses a near-end policy optimization algorithm to replace the traditional deep deterministic policy gradient algorithm for local self-training. By pruning the loss function to balance policy exploration and utilization, and combining it with the generalized advantage estimation method, the stability and convergence quality of policy updates are improved. Even in scenarios with highly uncertain renewable energy output and load demand, the decision-making effect of the local control layer is still guaranteed. The commentator network introduces real-time inter-microgrid trading price signals, enabling perception of the dynamic environment of multiple agents without needing to obtain complete state information from other agents. This protects the data privacy of each microgrid and stabilizes the multi-agent training process. The established complete constraint system integrates the ramping constraints of diesel generators and fuel cells, the state of charge constraints of energy storage, and power balance constraints into the scheduling decision framework, ensuring the engineering feasibility of the generated scheduling scheme. The introduced inter-microgrid energy trading mechanism achieves power surplus and deficit mitigation through internal market prices. This allows microgrids with surplus energy to obtain internal revenue exceeding the feeder price, while microgrids with insufficient energy can obtain the necessary electricity at a price lower than the external grid purchase price. This significantly reduces the overall system operating cost, improves the utilization efficiency of distributed resources, and ensures the steady-state operation of the multi-microgrid system as a whole. The designed three-objective reward function internalizes the energy storage lifetime depreciation cost as a scheduling decision constraint, avoiding excessive use of energy storage devices and balancing short-term system operating costs with long-term equipment economics. The inter-microgrid energy trading mechanism introduces a price constraint range with the feeder price as the lower limit and the external purchase price as the upper limit, ensuring that the internal trading price always lies between these two values. This effectively prevents opportunistic bidding behavior by participants and guarantees the long-term stable operation of the internal market. Attached Figure Description
[0016] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0017] Figure 1 This is a schematic diagram of the microgrid steady-state operation and dynamic decision-making method based on a hierarchical reinforcement learning architecture, as described in this invention.
[0018] Figure 2 This is a flowchart illustrating the steady-state operation and dynamic decision-making method for microgrids based on a hierarchical reinforcement learning architecture, as described in this invention. Detailed Implementation
[0019] The present invention will be explained in detail through the following embodiments. The purpose of disclosing the present invention is to protect all technical improvements within the scope of the present invention. In the description of the present invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," etc., indicating orientation or positional relationship, are only for the purpose of describing the present invention and are not intended to indicate or imply that the device or element referred to must have a specific orientation.
[0020] Combined with appendix Figure 1 and attached Figure 2 A method for steady-state operation and dynamic decision-making of microgrids based on a hierarchical reinforcement learning architecture is proposed, and the specific implementation steps are as follows: S1. Construct a hierarchical control architecture.
[0021] As attached Figure 1 As shown, the distributed multi-microgrid system established by this invention comprises N microgrids connected to the distribution network, denoted as N={1,2,...,N}, with a scheduling time domain of T={1,2,...,T} and a time step length of Δt. Each microgrid consists of wind turbines, photovoltaic generators, diesel generators, fuel cells, energy storage systems, and conventional loads. Its energy management center acts as an intelligent agent to control the dispatchable resources.
[0022] This invention deploys a two-level control architecture in the aforementioned multi-microgrid system. The upper layer is a federated learning global coordination layer: a central server acts as the aggregator, responsible for collecting local model parameters uploaded by each microgrid agent, generating a global aggregated model through weighted averaging, and broadcasting the aggregated parameters to all microgrid agents. This layer does not access the original operating data of each microgrid, only transmitting model parameters, thus protecting the data privacy of each microgrid. The lower layer is a local control layer for near-end policy optimization in each microgrid: after receiving the global parameters broadcast from the upper layer, each microgrid agent uses these as initial values, independently executes scheduling decisions based on its own real-time local observations, and performs local self-training. The updated local parameters are periodically uploaded to the upper-layer server for aggregation. During the execution phase, agents do not need to communicate with other microgrids in real time, demonstrating good feasibility for distributed deployment.
[0023] S2. Establish physical models for each microgrid and perform partial observable Markov game modeling.
[0024] The physical model of each microgrid consists of the following sub-models: The output of the diesel generator at time step t is denoted as It must simultaneously satisfy both the upper and lower limits of output and the ramping constraint: ; ; in , These represent the upper and lower limits of the diesel generator's output. , These are the upper and lower limits of the maximum allowable climbing power within a single time step, respectively.
[0025] The output of the fuel cell at time step t Similarly, the following must be satisfied: ; The dynamic equation of the state of charge of the energy storage system at time step t is: ; in Let be the state of charge at time step t. For energy storage charging and discharging efficiency, For the rated capacity of energy storage, This represents the charging and discharging power; a positive value indicates charging, and a negative value indicates discharging. The state of charge and charge / discharge power must meet capacity and power constraints: ; ; Microgrid i must achieve power balance within time step t, while also considering power exchange with other microgrids and the external distribution network: ; in To meet load demand, , They are respectively powered by wind and solar power. This represents the net exchange power between microgrids and with the external power grid; purchases are positive, and sales are negative. The load shedding power; the power deviation of microgrid i at time step t. Defined as the difference between load demand and actual local power generation: ; Based on the above physical model, this invention models the multi-microgrid collaborative optimization problem as a partially observable Markov game, consisting of a tuple of (O1,...,O) N A1,...,A N , T, r1,...,r N ) description, where O i A i r i Let represent the observation space, action space, and reward function of microgrid i, respectively, and T be the joint state transition probability. The local observations of each microgrid agent i at time step t are... It only contains local information that it can perceive, and does not include the internal state of other microgrids: ; in To meet the load demand of the previous time step, To contribute to the next stage of renewable energy, In the state of energy storage charge, The transaction price between the previous time step and the distribution network is t, which is the identifier for the current time period.
[0026] Each microgrid agent i generates scheduling actions based on local observations at time step t. This includes the output of the diesel generator, the output of the fuel cell, and the charging and discharging power of the energy storage system, and ensures that the actions meet the operating constraints of each device through a tailoring function: ; ; ; The instantaneous reward obtained by each microgrid agent i at time step t It consists of three physical objectives: operating cost, energy self-sufficiency rate, and energy storage lifetime penalty. ; in ∈[0,1]、 ∈[0,1]、 ∈[0,1] represents the weighting coefficients for balancing the three scalar items; , , These are the operation and maintenance costs for diesel generators, fuel cells, and energy storage systems, respectively. For fuel costs, The real-time electricity price for purchasing electricity from the distribution network at time step t; the energy self-sufficiency rate term measures the economic cost of purchasing electricity from outside the network by multiplying the power deviation by the real-time electricity price, and is in monetary terms consistent with the operating cost term. To account for the energy storage lifespan loss cost, the charging and discharging power of the energy storage system at time step t is calculated using the rainflow counting method. This is mapped to an equivalent cyclic depreciation amount, which is incorporated into rewards and penalties in the form of depreciation costs, thereby endogenizing the long-term lifespan management objectives of energy storage devices in scheduling decisions; the optimization objective of each microgrid agent is to maximize the cumulative discount reward within the scheduling cycle. ; in ∈[0,1] is the discount factor.
[0027] The overall reward of a multi-microgrid system is the sum of the rewards for each agent: ; S3. Each microgrid agent is trained locally using a near-end strategy optimization algorithm.
[0028] As attached Figure 2 As shown, within each federated training cycle, each microgrid agent uses the aggregated parameters received from the upper global coordination layer as initial values and performs several rounds of local self-training in parallel. Local training employs an actor-commentator architecture: the actor network... Based on local observation Input, output scheduling action ; Critics Network Value assessment of network-generated motions by actors, and introduction of real-time inter-microgrid transaction price signals based on local observations and motions. This approximates a centralized Q-value function without exposing the complete state and operational information of other microgrids. ; The actor network parameter update adopts a pruning loss function optimized by a near-end strategy: ; in and These are the old and new strategies updated in round k, respectively. For trimming parameters, The advantage function is calculated using the generalized advantage estimation method.
[0029] The critic network uses a loss function Conduct training: ; Actor Network Parameters With commentator network parameters Update each loss function based on its own gradient: ; in and These are the learning rates for the actor network and the critic network, respectively. By pruning the loss function, the policy update magnitude is limited to a reasonable range, effectively preventing training collapse caused by drastic policy changes. This ensures continuous and stable policy improvement even in scenarios with drastic fluctuations in renewable energy output and load demand.
[0030] S4. Establish an energy trading mechanism between microgrids.
[0031] This invention allows microgrids to engage in peer-to-peer energy trading within their internal markets. When microgrid i's local power generation exceeds its load demand at time step t, its surplus power can enter the internal trading market; microgrids with insufficient energy prioritize bidding against the energy surplus participant with the lowest bid in the internal market. Electricity purchase: ; in For the microgrid l, the internal quote at time step t, To identify variables, if microgrid l has no electricity to sell, it is set to infinity; to prevent opportunistic bidding behavior by participants, internal bids must meet price constraints: ; in The feeder electricity price constitutes the lower limit of the internal quotation; The external electricity purchase price forms the upper limit for bids; the above constraints ensure that the internal transaction price always lies between the feeder grid price and the external electricity purchase price, which both incentivizes microgrids with surplus energy to participate in internal market transactions and prevents bids from exceeding the external electricity purchase price, thus harming the interests of buyers and maintaining the long-term stable operation of the internal market; when all participants in the internal market are unable to meet their electricity purchase demand, the shortfall is covered by the external distribution network at market prices. To supplement: the power interaction between microgrids must simultaneously satisfy conservation constraints, transmission line capacity constraints, and upper and lower limit constraints for the operation of each device.
[0032] S5 performs federated learning parameter aggregation and broadcasting.
[0033] After each microgrid agent completes its local training, the parameters of the actor network and the critic network are combined. Uploaded to the central server. The server adjusts the size of the local training data from each participant. Calculate the weighted average to generate global aggregation parameters: ; After aggregation is complete, the server will The data is broadcast to all microgrid agents, replacing their local parameters, which are then used as the initial parameters for the next federated training cycle. Microgrids with larger data volumes contribute more to the global model, and the weighted averaging mechanism ensures that the global model reasonably reflects the data distribution of each microgrid.
[0034] S6. Achieve steady-state optimized operation of multi-microgrid systems.
[0035] Repeat steps S3 to S5. Each microgrid agent collects experience such as state, action, and reward through continuous interaction with the environment and stores it in an experience replay buffer. During each local training session, a batch of experience is randomly sampled from the buffer to form a training sample set T={<s0,a0,r0,s1> ,<s1,a1,r1,s2> The parameters of the actor network and the critic network are updated using the ,...}. After multiple rounds of federated training iterations, each microgrid agent forms a scheduling strategy that balances global coordination and local optimization.
[0036] After training, the policy network parameters are fixed. During the execution phase, each agent outputs scheduling commands based solely on its own real-time local observations through a single forward propagation of the agent network, with execution time on the order of milliseconds, meeting the response requirements of real-time scheduling in microgrids. Each agent makes independent decisions without needing real-time communication with other microgrids or a central server, thus achieving steady-state operation and dynamic decision-making of multi-microgrid systems within a privacy-preserving distributed architecture.
[0037] The parts of this invention not described in detail are prior art. It will be apparent to those skilled in the art that this invention is not limited to the details of the above exemplary embodiments, and that the invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention; therefore, the embodiments should be regarded in all respects as exemplary and non-limiting, and are intended to encompass all changes falling within the meaning and scope of equivalents within this invention.
Claims
1. A method for steady-state operation and dynamic decision-making of microgrids based on a hierarchical reinforcement learning architecture, characterized in that, Includes the following steps: S1. Construct a hierarchical control architecture: Deploy a two-level control system in the multi-microgrid system. The upper layer is a federated learning global coordination layer, and the lower layer is a local control layer for near-end policy optimization of each microgrid. Each microgrid is equipped with an independent agent. The multi-microgrid collaborative operation problem is modeled through partially observable Markov games. Each agent can only observe the equipment operating status, load demand, renewable energy output and energy storage charge status within its own microgrid, and cannot directly obtain internal information of other microgrids. S2. Establish physical models for each microgrid and perform partial observable Markov game modeling: Mathematically describe the diesel generators, fuel cells, renewable energy generators, energy storage systems, and loads within each microgrid, and construct a complete constraint system including power balance constraints, equipment operation constraints, and energy storage operation constraints; determine the observation space, action space, and reward function of each microgrid agent. The action output includes the controllable generator output adjustment and the energy storage system charging and discharging power. The pruning function ensures that the actions are always within the allowable operating range of each device. The microgrid operating cost, energy self-sufficiency rate, and energy storage lifetime penalty are used as three physical objectives of the reward function. The operating cost term quantifies the actual power generation cost of the controllable generator and energy storage system. The energy self-sufficiency rate term measures the economic cost of net electricity purchase from the external grid by multiplying the power deviation by the real-time electricity price. The energy storage lifetime penalty term maps the charging and discharging behavior into equivalent cycle loss cost based on the rainflow counting method. The three objectives are normalized by weighting coefficients. S3. Each microgrid agent uses a near-end policy optimization algorithm for local training: In each federated training cycle, each microgrid agent uses the received global aggregation parameters as initial values and performs several rounds of local self-training in parallel; The parameters of the actor network and the critic network are updated by pruning loss function and generalized advantage estimation method respectively. The critic network introduces real-time transaction price signals between microgrids on the basis of local observation and action to stabilize multi-agent training and obtain the local optimal scheduling strategy. S4. Establish an inter-microgrid energy trading mechanism: Allow microgrids to conduct point-to-point energy transactions within their internal markets. Microgrids with surplus energy can sell their excess power to other microgrids at internal bids, while microgrids with insufficient energy will prioritize purchasing power from the participant with the lowest bid. Only when the internal market cannot meet the gap will they purchase power from the external distribution network. This mechanism enables power surplus and shortage mutual assistance among microgrids, reducing the need for each microgrid to purchase power from external high-priced grids and improving overall resource utilization efficiency. Internal bids must meet price constraints with the feeder grid price as the lower limit and the external purchase price as the upper limit to prevent opportunistic bidding behavior by participants. Interactive power must meet power conservation constraints, transmission line capacity constraints, and upper and lower limit constraints for the operation of each device. S5. Perform federated learning parameter aggregation and broadcasting: After each microgrid agent completes several rounds of autonomous training locally, it uploads its local model parameters to the central server. The server calculates a weighted average based on the scale of local training data of each microgrid to generate a global aggregated model. The aggregated global parameters are broadcast to all microgrid agents to replace their local parameters and serve as the starting model for the next round of training. The entire process only transmits model parameters and does not share any original operating data to protect the data privacy of each microgrid. S6. Achieve steady-state optimized operation of multi-microgrid systems: After multiple rounds of federated training iterations, each microgrid agent forms a scheduling strategy that takes into account both global coordination and local optimization; during the execution phase, each agent independently outputs control commands based on real-time local observations, without the need for real-time communication with other microgrids or the central server, thereby achieving steady-state operation and dynamic decision-making of multi-microgrid systems under a distributed architecture.
2. The method as described in claim 1, characterized in that, In step S1, the partially observable Markov game model is described by a multivariate set, including the observation space, action space, reward function, and joint state transition probability of each microgrid agent. At each scheduling step, each agent independently selects an action based on its local observation. After the joint action of all agents acts on the environment, each agent obtains its local observation and current reward for the next step. The optimization objective of each agent is to maximize the cumulative discount reward within the scheduling period through training the scheduling strategy.
3. The method as described in claim 1, characterized in that, In step S2, the power deviation is defined as the difference between the load demand and actual power generation of each microgrid at the current time step. The actual power generation includes the algebraic sum of the output of controllable generators, the output of renewable energy, and the charging and discharging power of the energy storage system. The energy self-sufficiency rate term in the reward function reflects the economic loss from purchasing electricity from outside by multiplying the deviation by the real-time electricity price, thereby transforming the physical power imbalance into a monetary dimension that can be directly compared with the operating cost, ensuring the uniformity of the dimensions of each item in the reward function. The energy storage lifetime penalty term maps the energy storage charging and discharging power to the equivalent cyclic depreciation cost through the rainflow counting method, and incorporates it into the reward and penalty in the form of depreciation cost. This internalizes the long-term lifetime management target of the energy storage equipment in the dispatch decision, taking into account both the short-term operating cost of the system and the long-term equipment economy.
4. The method as described in claim 1, characterized in that, In step S3, each microgrid agent is trained locally using a near-end policy optimization algorithm. The strategy exploration and utilization are balanced by pruning the loss function. The generalized advantage estimation method is used to evaluate the action advantage. The discount factor and hyperparameters are used to adjust the trade-off between estimation bias and variance. The actor network and the critic network update their parameters independently through their respective loss functions. The actor network adjusts the policy distribution based on the advantage estimation, and the critic network updates the state value estimation with the goal of minimizing the temporal difference error.
5. The method as described in claim 1, characterized in that, In step S4, energy transactions between microgrids prioritize the energy surplus participant with the lowest internal market bid as the power source; when the power generation of a microgrid exceeds the local load demand, the surplus energy can participate in internal transactions; only when all participants in the internal market cannot meet the power purchase demand will the shortfall be made up by the external distribution network at the market price; internal bids must meet the price constraint of the feeder grid price as the lower limit and the external power purchase price as the upper limit, ensuring that the internal transaction price always lies between the two; during the transaction process, the power exchange between microgrids must meet the conservation constraint of algebraic sum of zero and be limited by the maximum transmission capacity of the transmission lines, and all types of power generation and energy storage equipment must operate within the allowable upper and lower power limits.
6. The method as described in claim 1, characterized in that, In step S5, the aggregate weights of federated learning are determined based on the data scale used by each microgrid for local training. Microgrids with larger data scales have a correspondingly higher contribution weight to the global model. After the server completes the weighted average calculation of the parameter list and generates the global model in each round of federated training, it immediately broadcasts the global model parameters to all participants. Each microgrid agent replaces its local parameters with the received global parameters and carries out local autonomous training for the next training cycle based on this.
7. The method as described in claim 1, characterized in that, In the hierarchical reinforcement learning architecture, the upper federated learning global coordination layer is responsible for knowledge sharing and policy generalization across microgrids. Through parameter aggregation, it enables each microgrid agent to utilize the data distribution information of other participants, overcoming the problem of a single microgrid falling into local optima due to insufficient data diversity. The lower near-end policy optimization local control layer is responsible for the independent decision-making and rapid response of each microgrid based on real-time local observations. The two layers work together to enable the system to achieve global policy optimization and local autonomous control simultaneously without sharing the original operating data.
