A multi-agent bi-layer optimization method for microgrids with seasonal hydrogen energy storage islands

Abstract

This invention proposes a multi-agent, two-layer optimization method for islanded microgrids with seasonal hydrogen energy storage. The upper layer uses NSGA-II for capacity configuration optimization, targeting annualized total cost, power shortage rate, and power curtailment rate. The lower layer employs a near-end strategy optimization algorithm to train three agents—energy storage battery, electrolyzer, and fuel cell—for real-time scheduling. The core of this invention lies in modeling the lower-layer operation model as a multi-agent Markov decision process and designing seasonal transition and annual closure constraints, as well as a three-layer safety mechanism—action pruning, ramp projection, and business rules—to ensure the physical feasibility of the reinforcement learning strategy. The upper-layer model calls the pre-trained lower-layer MADRL model for high-speed inference, replacing the traditional time-consuming MILP solution, thus achieving rapid and robust collaborative optimization of capacity planning and operation strategies for islanded microgrids.

Description

Technical Field

[0001] This invention belongs to the field of integrated energy system optimization technology, and relates to a multi-agent two-layer optimization method for islanded microgrids with seasonal hydrogen energy storage, and particularly to a collaborative solution method for capacity planning and operation optimization of islanded microgrids. Background Technology

[0002] Isolated microgrids, as an important solution for power supply in remote areas and islands, face multiple challenges in capacity planning and operation optimization, including large fluctuations in renewable energy, strong load uncertainty, and complex energy storage configuration. Traditional capacity planning methods typically employ mixed-integer linear programming (MILP) or heuristic optimization algorithms, simplifying the operation optimization problem into a deterministic model or typical scenario, which fails to accurately characterize the randomness and complex constraints in actual operation. In particular, with the introduction of seasonal hydrogen energy storage, the system needs to achieve energy balance on intraday, interseasonal, and annual scales. Traditional optimization methods suffer from high computational complexity, long solution time, and insufficient robustness when dealing with coupling constraints across time scales.

[0003] In recent years, deep reinforcement learning (DRL) has demonstrated powerful decision-making capabilities in the optimal scheduling of energy systems. It can learn optimal strategies through interaction with the environment, adapt to uncertain environments, and handle complex nonlinear constraints. However, existing research on DRL-based microgrid optimization mainly focuses on operation scheduling at a single time scale, lacking systematic research on the coordinated solution of capacity planning and operation optimization. Furthermore, traditional DRL methods typically employ a single-agent architecture when dealing with multi-device collaborative control, merging the control actions of all devices into a high-dimensional action space, leading to problems such as high training difficulty, slow convergence speed, and weak policy generalization ability.

[0004] Multi-agent deep reinforcement learning (MADRL) decomposes complex systems into multiple agents, each responsible for controlling one or a class of devices, effectively reducing the dimension of the action space and improving training efficiency and policy quality. Furthermore, MADRL supports information sharing and collaborative decision-making among agents, better aligning with the physical characteristics of multi-device collaborative operation in real-world microgrids. However, the application of MADRL in microgrid optimization still faces the following challenges: (1) How to design reasonable state and action spaces to ensure agents can perceive the global state of the system and output physically feasible control commands; (2) How to handle the cross-timescale constraints of seasonal hydrogen energy storage to achieve intraday, inter-seasonal, and annual energy balance; (3) How to effectively couple the MADRL model with the upper-level capacity planning model to achieve efficient solutions for bi-layer optimization. Summary of the Invention

[0005] In view of the above, this invention proposes a multi-agent two-layer optimization method for islanded microgrids with seasonal hydrogen energy storage, which realizes efficient collaborative solution of capacity planning and operation optimization of islanded microgrids.

[0006] This invention is achieved through the following technical solution. A multi-agent bi-layer optimization method for a microgrid with seasonal hydrogen energy storage, comprising three subsystems: a power generation end, a dispatch end, and a power consumption end. The power generation end includes wind turbines and photovoltaics, responsible for electricity production; the dispatch end includes energy storage batteries, an electrolyzer, seasonal hydrogen energy storage (i.e., a hydrogen storage tank), and a fuel cell, responsible for the production, storage, conversion, and dispatch of electrical and hydrogen energy; the power consumption end includes electrical loads, responsible for electricity consumption. The electrolyzer converts surplus electrical energy generated at the power generation end into hydrogen energy and stores it in the seasonal hydrogen energy storage. The fuel cell converts the hydrogen energy in the seasonal hydrogen energy storage into electrical energy and feeds it back to the system. Combined with the charging and discharging of the energy storage battery, this achieves bi-directional conversion of electrical and hydrogen energy and energy balance across time scales. The multi-agent bi-layer optimization method for the microgrid with seasonal hydrogen energy storage includes the following steps: Step 1: Construct the upper-level capacity configuration model: Minimize the annualized total cost (F1), minimize the power shortage rate (F2), and minimize the power curtailment rate (F3) as the objective functions for the economic, stability, and environmental aspects of the islanded microgrid system, respectively; select the upper-level decision variables as a six-dimensional capacity configuration combination, specifically including wind power capacity, photovoltaic capacity, energy storage battery capacity, electrolyzer capacity, hydrogen storage tank capacity, and fuel cell capacity; Step 2: Construct the lower-level operation simulation model: The microgrid optimization operation problem is modeled as a multi-agent Markov decision process (MDP), and one or more policy agents are obtained through offline training using a deep reinforcement learning algorithm. These agents are used to output operation control actions based on the current environmental state. Step 3: Perform two-layer optimization: The upper-layer capacity configuration model uses the Non-Dominated Sorting Genetic Algorithm (NSGA-II) to generate a first-generation population, with each individual representing a set of equipment capacity configuration combinations. The capacity configuration parameters represented by each individual are passed to the lower-layer operation simulation model. The lower-layer model loads the offline-trained policy agent and executes a complete cycle of operation simulation to obtain the corresponding annualized operating cost, power shortage, and power abandonment. These results are returned to the upper-layer model as fitness values ​​for calculating the three optimization objectives (F1, F2, F3). The NSGA-II algorithm generates the next-generation population based on the fitness values ​​of all individuals through selection, crossover, and mutation operations. The above simulation and evaluation process is repeated until the maximum number of iterations or the convergence condition is reached, finally outputting a set of Pareto optimal equipment capacity configuration combinations.

[0007] Further preferred, the lower-level operation simulation model is modeled as a MADRL model containing three agents: the energy storage battery agent (BAT-Agent), the electrolyzer agent (ELEC-Agent), and the fuel cell agent (FC-Agent). The state space of the MDP is a 16-dimensional vector, including time-coded information, energy storage state information, resource information, equipment state information, and prediction information, specifically defined as follows: ; ; ; ; in: Given the current hour, the hour is mapped onto the unit circle using sine and cosine functions to ensure the periodic continuity of time. Given the current season number, the sine and cosine functions are used to map the season number onto the unit circle, ensuring the periodic continuity of the seasons; This refers to the state of charge of the energy storage battery. This refers to the hydrogen storage status of the hydrogen storage tank; Let t represent the photovoltaic power generation during time period t; Let t represent the wind power generation capacity during time period t; Let t be the electrical load demand during time period t; The electrolytic cell operating power during time period t-1; This represents the operating power of the fuel cell during the t-1 time period; and These represent the on / off states of the electrolyzer and fuel cell during time period t-1, where 1 indicates on and 0 indicates off. This provides forward-looking information on photovoltaic power generation for the t+1 time period; This provides forward-looking information on wind power generation capacity for the t+1 time period; This provides forward-looking information on electricity load demand for the t+1 time period; The state space of each agent is defined as follows: ; ; ;

[0008] Further optimization involves defining the action space of each agent as a normalized continuous action space, represented as follows: ; in: , and These are the action spaces for the energy storage battery intelligent agent, the electrolyzer intelligent agent, and the fuel cell intelligent agent, respectively. t represents the action value output by the energy storage battery intelligent agent during time period t. Its positive and negative signs are used to distinguish the charging and discharging modes. A positive value indicates discharging and a negative value indicates charging. Its absolute value represents the ratio of charging and discharging power to rated power. and These are the action values ​​output by the electrolyzer and fuel cell intelligent agents during time period t, respectively, representing the operating load rate of their respective devices. The larger the value, the greater the operating power.

[0009] Further optimization involves performing the following action mapping and safety correction steps after each agent outputs its action value: First, physical mapping of actions is performed, mapping normalized action values ​​to physical power commands for each device based on the capacity parameters of each device: ; in, and These are discharge and charge power commands, respectively. The rated capacity of the energy storage battery, This refers to the maximum charge / discharge rate of the energy storage battery. and These are the rated capacities of the electrolyzer and the fuel cell, respectively. Secondly, the aforementioned physical power commands are sequentially input into the three-layer safety module for verification and correction, and the final execution action is output, including the following steps: (1) SOC boundary protection: Based on the current state of charge (SOC) of the battery and hydrogen storage tank, the power command that violates the physical boundary is forcibly set to 0: ; in: The state of charge of the energy storage battery during time period t; and These are the upper and lower limits of the state of charge of the energy storage battery, respectively. The state of charge of the hydrogen storage tank; and These are the upper and lower limits of the state of charge of the hydrogen storage tank, respectively. (2) Climbing constraint projection: Calculate the difference between the commanded power of the equipment at the current moment and the actual power at the previous moment. If it exceeds the maximum allowable climbing rate of the equipment, then project the commanded power onto the allowable climbing boundary: ; (3) Business rule restrictions: When the system's renewable energy and energy storage discharge are insufficient to meet the electrical load, a command to forcibly reduce or cut off the hydrogen production power of the electrolyzer is issued: First, calculate the remaining power of the system available for hydrogen production at the current time t. The formula is as follows: ; Secondly, the commanded power of the electrolytic cell Based on remaining power The corrected operating power is calculated using the following formula: ;

[0010] Further optimization involves simulating the entire operational cycle based on 24-hour data from four typical days across spring, summer, autumn, and winter, totaling 96 time steps. During the simulation, seasonal transition processing and annual closure constraint checks are performed to simulate cross-seasonal energy shifts and achieve sustainable annual cycles. The seasonal transition processing ensures that at the end of each typical day's simulation, the initial hydrogen storage capacity of seasonal hydrogen storage on that typical day is equal to the accumulated hydrogen storage capacity of the previous quarter, as shown in the following formula: ; in: This represents the initial hydrogen storage capacity on a typical day in the current season (s). This represents the initial hydrogen storage capacity for a typical day of s-1 in the previous season. and These represent the initial and final states of hydrogen energy storage on a typical day of s-1 in the previous season, respectively. This represents the total number of typical days in the current season (s); s represents the typical number of days in the corresponding season. The annual closed-loop constraint check refers to the situation where, at the end of the 96-time-step simulation, the amount of hydrogen stored under constraint during the winter is equal to the initial hydrogen storage on a typical day in the spring, as shown in the following formula: ;

[0011] Furthermore, the reward function of the MADRL model is designed to minimize the immediate running cost, and its calculation formula is as follows: ; in: , , , , and These are operation and maintenance costs, degradation costs, power shortage penalties, power abandonment penalties, start-up and shutdown costs, and constraint violation costs.

[0012] Furthermore, the MADRL model is trained using the Proximal Policy Optimization (PPO) algorithm, and each agent contains a policy network (Actor). And a value network (Critic) Furthermore, both the policy network and the value network adopt a multilayer perceptron structure; (1) Network parameter update: The update objective function of the policy network (Actor) The relevant definitions are as follows: ; in: The importance ratio of the old and new strategies; Generalized advantage estimation (GAE); For time-series difference residuals; For the clipping factor, Here is the entropy regularization coefficient. For policy entropy; The instantaneous reward at time t. As a discount factor, For GAE smoothing parameters; The update objective function of the value network (Critic) is: ; in: For target value; The overall loss function of the agent is: ; in: These are the loss weight coefficients for the value function. (2) Network structure configuration: The structure of the policy network includes an input layer, at least two fully connected hidden layers and an output layer; the hidden layers use the ReLU activation function; the output layer is configured with an activation function according to the agent type: for battery agents, the hyperbolic tangent activation function Tanh is used to output the charging and discharging action value in the interval [-1,1]; for electrolyzer agents and fuel cell agents, the Sigmoid activation function is used to output the load rate action value in the interval [0,1]; the output layer of the value network contains a neuron used to output the value estimate of the current state; the parameter updates of the policy network and the value network both use the adaptive moment estimation Adam optimizer.

[0013] Further preferred, the training process of the multi-agent deep reinforcement learning model adopts a multi-agent collaborative architecture based on shared observations, specifically including: during the training phase, constructing multiple parallel environments to simultaneously collect trajectory data, and storing the trajectory data in a fixed-length experience replay buffer; the three agents share a global observation vector containing the time encoding information, the energy storage status information, the resource information, the device status information, and the prediction information during both the training and execution phases, so as to achieve unified perception and collaborative decision-making of the system status in the isolated operation mode; when the experience replay buffer is full, based on the collected trajectory data, the network parameters of each agent are updated in multiple rounds using the mini-batch gradient descent method.

[0014] Further optimization, to enhance the robustness of the strategy under uncertain environments, a domain randomization mechanism is introduced during training. This mechanism includes environmental perturbation processing and initial state randomization processing. The environmental perturbation processing involves adding Gaussian-distributed random noise to the historical data of photovoltaic power, wind power, and electrical load at the beginning of each training round to generate diverse training scenarios. The calculation formula is as follows: ; in: , and These are training data for photovoltaic, wind power, and load, respectively. , and These are the original historical data for photovoltaic, wind power, and load, respectively; It is a random noise variable; It follows a Gaussian distribution; , and These are the noise standard deviations for photovoltaic, wind power, and load, respectively; the initial state randomization process refers to the uniform distribution and random initialization of the initial state of charge of the battery and hydrogen storage tank within a preset interval at the beginning of each training round.

[0015] Further optimization involves using the Non-Dominated Sorting Genetic Algorithm-II (NSGA-II) to solve the upper-level capacity allocation model. This algorithm employs an elite retention strategy, and its initial population generation utilizes the Latin Hypercube Sampling (LHS) method. The three objective functions set for the upper-level capacity allocation model are minimizing the annualized total cost F1, minimizing the power shortage rate F2, and minimizing the power curtailment rate F3, and their calculation formulas are as follows: ; in: A collection of devices; The discount rate; For equipment lifespan; Unit investment cost; Configure for capacity; Let t be the total operating cost at time t.

[0016] This invention achieves lower-level operational optimization by constructing a three-agent MADRL model, which effectively reduces the action space dimension and improves training efficiency and policy quality compared to the traditional single-agent architecture. By designing seasonal transition and annual closure constraint mechanisms, it achieves energy balance for seasonal hydrogen storage at intraday, interseasonal, and annual scales. The introduction of a three-layer safety mechanism—action pruning, ramp projection, and business rules—ensures the physical feasibility of the reinforcement learning strategy. Domain randomization enhances the robustness of the strategy under uncertain environments. Through collaborative solution with the upper-level NSGA-II capacity planning model, it achieves rapid and robust optimization of capacity configuration and operation strategies for islanded microgrids, significantly reducing computation time and improving solution efficiency compared to traditional MILP methods. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the isolated electric-hydrogen integrated energy system structure of the present invention; Figure 2 This is a flowchart of the two-layer optimization method of the present invention; Figure 3 This invention provides a collaborative training framework for isolated hydrogen microgrids based on the multi-agent PPO algorithm. Figure 4 This is a flowchart of the specific PPO algorithm of the present invention. Detailed Implementation

[0018] To describe the present invention in more detail, the technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] The structure of an isolated microgrid integrated energy system for electricity and hydrogen is as follows: Figure 1 As shown, the system comprises three subsystems: a power generation unit, a dispatching unit, and a power consumption unit. The power generation unit includes wind turbines and photovoltaic panels, responsible for generating electricity. The dispatching unit includes energy storage batteries, an electrolyzer, seasonal hydrogen storage (i.e., hydrogen tanks), and fuel cells, responsible for the production, storage, conversion, and dispatch of both electricity and hydrogen. The power consumption unit includes electrical loads, responsible for consuming electricity. The electrolyzer converts surplus electricity generated at the power generation unit into hydrogen energy and stores it in the seasonal hydrogen storage. The fuel cell converts the hydrogen energy in the seasonal hydrogen storage into electricity and feeds it back to the system. Combined with the charging and discharging of the energy storage batteries, this achieves bidirectional conversion of electricity and hydrogen energy and energy balance across time scales.

[0020] Capacity planning and operation optimization of isolated microgrid systems is a typical two-level optimization problem. The upper-level capacity configuration model aims to minimize the annualized total cost, the power shortage rate, and the power curtailment rate. The decision variables are the capacity configuration combinations of wind power generation units, photovoltaic power generation units, energy storage battery units, hydrogen storage tanks, electrolyzers, and fuel cells. The lower-level operation simulation model simulates the system's operation over a complete cycle (96 time steps in typical days of the four seasons) under given capacity configuration conditions, calculates the corresponding annualized operating cost, power shortage, and power curtailment, and returns these results to the upper-level model as the basis for fitness evaluation.

[0021] The constraints of the multi-agent bi-layer optimization model for a microgrid with seasonal hydrogen energy storage islands include power balance constraints, energy storage battery constraints, hydrogen storage tank constraints, electrolyzer constraints, fuel cell constraints, seasonal transition constraints, and annual closure constraints.

[0022] (1) Power balance constraint: ; in: and These represent the photovoltaic and wind power outputs during time period t, respectively. and These represent the discharge and charging power of the energy storage battery during time period t, respectively. and These represent the power of the fuel cell and the electrolyzer during time period t, respectively. For electrical load; and These represent power shortages and wasted power, respectively. (2) Constraints of energy storage batteries: ; In the formula: and These are the charging and discharging efficiencies, respectively. This refers to the rated capacity of the energy storage battery. For time step; This represents the maximum charge / discharge rate. (3) Constraints of hydrogen storage tanks: ; In the formula: and These are the efficiencies of the electrolyzer and the fuel cell, respectively. This refers to the rated capacity of the hydrogen storage tank. (4) Electrolytic cell constraints: ; In the formula: This refers to the rated capacity of the electrolytic cell; This refers to the ramp-up efficiency of the electrolytic cell. (5) Constraints of fuel cells: ; In the formula: This refers to the rated capacity of the fuel cell; This refers to the ramp-up efficiency of fuel cells. (6) Seasonal transition constraints: ; in: This represents the initial hydrogen storage capacity on a typical day in the current season (s). This represents the initial hydrogen storage capacity for a typical day of s-1 in the previous season. and These represent the initial and final states of hydrogen energy storage on a typical day of s-1 in the previous season, respectively. represents the total number of typical days in the current season s; s represents the typical number of days in the corresponding season. (7) Annual Closure Constraints: ;

[0023] Reference Figure 3 and Figure 4 This invention models the lower-level operational simulation model as a multi-agent Markov decision process (MDP) and solves it using the multi-agent deep reinforcement learning (MADRL) method. The multi-agent MDP consists of nine tuples. Definition, where For state space, For the action space, For the number of agents, For the reward function, Let be the state transition probability. As a discount factor, For agent policy, For observation space, This is a joint strategy.

[0024] This invention constructs three intelligent agents: a battery storage agent (BAT-Agent), an electrolyzer agent (ELEC-Agent), and a fuel cell agent (FC-Agent). Each agent is responsible for controlling the operating power of a device and learns the optimal control strategy through interaction with the environment. The three agents share global observation information to achieve unified perception and collaborative decision-making of the system state in islanded operation mode.

[0025] The state space designed in this invention is a 16-dimensional vector, including time-coded information, energy storage state information, resource information, equipment state information, and prediction information. The state space of each agent is defined as follows: ; ; ; in: , and These are the state space sets for the intelligent agents of energy storage batteries, electrolyzers, and fuel cells, respectively. and These represent the on / off states of the electrolyzer and fuel cell during time period t-1, respectively. , and These are forward-looking information for wind power, solar power, and load, respectively.

[0026] This invention defines the action space of each agent as a normalized continuous action space, and the action space of each agent is represented as follows: ; in: , and These are the action spaces for the energy storage battery intelligent agent, the electrolyzer intelligent agent, and the fuel cell intelligent agent, respectively. t represents the action value output by the energy storage battery intelligent agent during time period t. Its positive and negative signs are used to distinguish the charging and discharging modes. A positive value indicates discharging and a negative value indicates charging. Its absolute value represents the ratio of charging and discharging power to rated power. and These are the action values ​​output by the electrolyzer and fuel cell intelligent agents during time period t, respectively, representing the operating load rate of their respective devices. The larger the value, the greater the operating power.

[0027] Action Mapping and Safety Correction: This invention designs a three-layer safety mechanism to ensure that the action values ​​output by the intelligent agent can be mapped to physically feasible control commands. After each intelligent agent outputs its action value, the following action mapping and safety correction steps are performed: First, physical mapping of actions is performed, mapping normalized action values ​​to physical power commands for each device based on the capacity parameters of each device: ; in, and These are discharge and charge power commands, respectively. The rated capacity of the energy storage battery, This refers to the maximum charge / discharge rate of the energy storage battery. and These are the rated capacities of the electrolyzer and the fuel cell, respectively. Secondly, the aforementioned physical power commands are sequentially input into the three-layer safety module for verification and correction, and the final execution action is output, including the following steps: (1) SOC boundary protection: Based on the current state of charge (SOC) of the battery and hydrogen storage tank, the power command that violates the physical boundary is forcibly set to 0: ; in: The state of charge of the energy storage battery during time period t; and These are the upper and lower limits of the state of charge of the energy storage battery, respectively. The state of charge of the hydrogen storage tank; and These represent the upper and lower limits of the state of charge of the hydrogen storage tank. (2) Climbing constraint projection: Calculate the difference between the commanded power of the equipment at the current moment and the actual power at the previous moment. If it exceeds the maximum allowable climbing rate of the equipment, then project the commanded power onto the allowable climbing boundary: ; (3) Business rule restrictions: When the system's renewable energy and energy storage discharge are insufficient to meet the electrical load, a command to forcibly reduce or cut off the hydrogen production power of the electrolyzer is issued: First, calculate the remaining power of the system available for hydrogen production at the current time t. The formula is as follows: ; Secondly, the commanded power of the electrolytic cell Based on remaining power The corrected operating power is calculated using the following formula: ;

[0028] Reward Function Design: The reward function designed in this invention aims to minimize immediate operating costs, including six cost items: maintenance costs, degradation costs, power outage penalties, power curtailment penalties, start-up and shutdown costs, and constraint violation costs. The calculation formula for the reward function is as follows: ; in: , , , , and These are operation and maintenance costs, degradation costs, power shortage penalties, power abandonment penalties, start-up and shutdown costs, and constraint violation costs.

[0029] PPO algorithm implementation: (Refer to...) Figure 3 and Figure 4 This invention employs the Proximal Policy Optimization (PPO) algorithm to train the policy network and value network of each agent. The PPO algorithm is a reinforcement learning algorithm based on policy gradients, which limits the magnitude of policy updates through a pruning mechanism to ensure training stability. (1) Policy Network (Actor): Policy Network Receive status As input, output action The probability distribution. For a continuous action space, the policy network typically outputs the mean of the actions. and standard deviation Then from Gaussian distribution Actions are obtained through sampling. The update objective function and related definitions of the policy network are: ; in: The importance ratio of the old and new strategies; Generalized advantage estimation (GAE); For time-series difference residuals; For the clipping factor, Here is the entropy regularization coefficient. For policy entropy; The instantaneous reward at time t. As a discount factor, For GAE smoothing parameters. (2) Value Network (Critic): Value Network Receive status As input, the output state is a value estimate. The update objective function of the value network is: ; in: For target value; (3) The overall loss function of the agent is the weighted sum of the policy network loss and the value network loss: ; in: These are the loss weight coefficients for the value function.

[0030] Network Structure Design: Both the policy network and the value network in this invention adopt a multilayer perceptron (MLP) structure, as detailed below: (1) Policy network structure: Input layer: Receives information from the state space. The dimension depends on the type of agent (12-dimensional for energy storage battery agents, 12-dimensional for electrolyzer agents, and 10-dimensional for fuel cell agents). Hidden layers: Two fully connected hidden layers, each with 256 neurons, using the ReLU activation function; Output layer: Activation functions are configured according to the agent type. Energy storage battery agents use the Tanh activation function, outputting charge / discharge action values ​​within the interval [-1,1]; electrolyzer and fuel cell agents use the Sigmoid activation function, outputting load rate action values ​​within the interval [0,1]. (2) Value network structure: Input layer: Receives information from the state space; its dimensions are the same as those of the policy network. Hidden layers: Two fully connected hidden layers, each with 256 neurons, using the ReLU activation function; Output layer: One neuron, no activation function, outputs the value estimate of the current state; (3) Parameter update: The parameters of both the policy network and the value network are updated using the Adaptive Moment Estimator (Adam) optimizer, with a learning rate of 3×10⁻ 4 .

[0031] like Figure 3 The collaborative training framework based on the multi-agent PPO algorithm shown on the left illustrates the physical layer of an islanded microgrid system, primarily comprising wind turbines, photovoltaics, PEM electrolyzers, hydrogen storage tanks, fuel cells, and energy storage batteries. The right side demonstrates the MADRL training loop. The system inputs a global state vector containing states (power, SOC, hydrogen storage) into the multi-agent policy network (Actor), with each agent outputting actions (charging / discharging, hydrogen production / use). The resulting interactive data is stored in an experience replay buffer. During training, data is sampled from the buffer and fed into the loss function calculation module, which is responsible for calculating the PPO loss and value loss. Subsequently, the Adam optimizer performs gradient updates based on the calculated gradients, updating the policy network and the value network (Critic). To further illustrate the internal logic of this update process, Figure 4 The PPO algorithm's specific computational flow is shown in detail, including the following key computational modules: First, based on the old strategy... and new strategies The importance sampling ratio is obtained by performing ratio calculation. Secondly, the pruning loss of the policy network is calculated using generalized advantage estimation (GAE), which introduces a pruning mechanism to limit the update magnitude. Simultaneously, the value loss between the predicted value and the target value is calculated. Finally, the iterative optimization of the agent's policy is achieved by minimizing the aforementioned loss function. The specific steps are as follows: (1) Initialization: Initialize the policy network and value network parameters of the three agents, create an experience replay buffer, and set the training hyperparameters; (2) Parallel Environment Acquisition: Sixteen parallel environments were constructed, each independently running a complete simulation cycle (96 time steps for typical days in all four seasons). At each time step, the three agents, based on the current state... Output actions separately , and After action mapping and safety correction, the final action is obtained, and the environment calculates the next state based on the executed action. and instant rewards .Will Stored in the experience replay buffer; (3) Experience replay buffer full judgment: When the experience replay buffer is full, the parameter update stage is entered; (4) Calculate GAE and target value: For each experience in the experience replay buffer, use the current value network. Calculate time series differential residuals Then calculate GAE and target value. ; (5) Mini-batch gradient descent. The data in the experience replay buffer is randomly shuffled and divided into multiple mini-batches (64 experiences per batch). Multiple rounds of parameter updates are performed on each mini-batch. In each round of updates, the policy network loss is calculated. and value network loss Then, the Adam optimizer is used to update the network parameters; (6) Clear the experience replay buffer. After the parameters are updated, clear the experience replay buffer and return to step 2 to continue collecting new experience; (7) Convergence judgment. Repeat steps 2-6 until the maximum number of training rounds is reached or the policy converges.

[0032] Domain randomization mechanism: To enhance the robustness of the strategy under uncertain environments, a domain randomization mechanism is introduced during training. This mechanism includes environmental perturbation processing and initial state randomization processing. The environmental perturbation processing refers to adding random noise following a Gaussian distribution to the historical data of photovoltaic power, wind power, and electrical load at the beginning of each training round to generate diverse training scenarios. The calculation formula is as follows: ; in: , and These are training data for photovoltaic, wind power, and load, respectively. , and These are the original historical data for photovoltaic, wind power, and load, respectively; It is a random noise variable; It follows a Gaussian distribution; , and These are the noise standard deviations for photovoltaic, wind power, and load, respectively; the initial state randomization process refers to the uniform distribution and random initialization of the initial state of charge of the battery and hydrogen storage tank within a preset interval at the beginning of each training round.

[0033] Upper-level NSGA-II algorithm: Refer to Figure 2 The upper-level capacity allocation model of this invention is solved using the Non-Dominated Sorting Genetic Algorithm (NSGA-II). This algorithm employs an elite retention strategy, and its initial population generation uses the Latin Hypercube Sampling (LHS) method. The three optimization objective functions set by the upper-level capacity allocation model are minimizing the annualized total cost F1, minimizing the power shortage rate F2, and minimizing the power curtailment rate F3, and their calculation formulas are as follows: ; in: A collection of devices; The discount rate; For equipment lifespan; Unit investment cost; Configure for capacity; Let t be the total operating cost at time t.

[0034] Two-layer optimization collaborative solution: The two-layer optimization method of this invention achieves effective coupling of capacity planning and operation optimization through the collaborative solution of the upper and lower layer models. The upper-layer NSGA-II algorithm is responsible for searching the optimal combination of capacity configurations, while the lower-layer MADRL model is responsible for simulating the system operation process and calculating the operating cost under a given capacity configuration. The two-layer optimization method flow is as follows: Figure 2 As shown, the specific steps are as follows: (1) Initialization of the upper-layer NSGA-II. A 6-dimensional capacity scheme generated by LHS sampling is used as the initial population, and the optimization objectives are annualized cost, power shortage rate, and power curtailment rate; (2) MADRL agent simulation operation. The capacity scheme generated by the upper layer is passed to the lower layer, and the BAT / ELEC / FC three agents (i.e., energy storage battery, electrolyzer and fuel cell agents) are loaded. The simulation of typical days in four seasons is performed based on environmental data. (3) Calculate core indicators. Based on the simulation results, calculate the system's operating cost + power shortage rate + power curtailment rate; (4) Feedback indicators as fitness. The indicators calculated above are fed back to the upper-level model as the basis for evaluating the merits of the current capacity scheme; (5) Upper-level genetic operation iteration. Based on the fitness value, perform non-dominated sorting + selection / crossover / mutation operations to generate the next generation population; (6) Determine if the termination condition has been met. If not, return to step 2 and continue the loop; if yes, output the Pareto front optimal set and end the process. The method of this invention achieves lower-level operation optimization by constructing a three-agent MADRL model, and designs seasonal transition and annual closure constraint mechanisms to handle energy balance across time scales. At the same time, a three-layer security mechanism is introduced to ensure the physical feasibility of the strategy. Finally, it works in conjunction with the upper-level NSGA-II capacity planning model to achieve rapid and robust optimization of capacity configuration and operation strategy for islanded microgrids. Through this two-layer optimization and collaborative solution method, the present invention achieves deep coupling between capacity planning and operation optimization, and can search for the optimal capacity configuration scheme in terms of economy, reliability and environmental protection, while taking into account actual operation constraints.

[0035] Technical effects of the present invention: (1) High efficiency: Compared with the traditional MILP method, the present invention uses the MADRL method to train the lower layer running strategy offline. After training, high-speed inference can be performed. The fitness evaluation of each capacity configuration scheme only takes a few seconds, which greatly reduces the computation time of the two-layer optimization. (2) Robustness: Through the domain randomization mechanism, the strategy trained by this invention has stronger generalization ability and can cope with unseen scenarios and disturbances in practical applications; (3) Feasibility: Through a three-layer security mechanism, this invention ensures that the action values ​​output by the reinforcement learning strategy can satisfy all physical constraints, thus avoiding the problem of infeasible solutions; (4) Cross-timescale optimization: Through seasonal connection and annual closure constraint mechanism, this invention realizes the energy balance of seasonal hydrogen energy storage on the intraday, inter-seasonal and annual scales, giving full play to the long-term energy storage advantages of hydrogen energy storage; (5) Multi-objective optimization: Through the NSGA-II algorithm, this invention can search for the Pareto optimal solution set, providing decision-makers with a variety of capacity configuration schemes to meet different economic, reliability and environmental protection requirements.

[0036] The above description merely illustrates preferred embodiments of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make modifications or alterations to the above-disclosed content to create equivalent embodiments. However, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A multi-agent bi-level optimization method for a seasonal hydrogen energy storage islanded microgrid, characterized in that, The islanded microgrid with seasonal hydrogen energy storage comprises three subsystems: a generation end, a dispatch end, and a consumption end. The generation end includes wind turbines and photovoltaics, responsible for electricity production. The dispatch end includes energy storage batteries, electrolyzers, seasonal hydrogen energy storage (i.e., hydrogen storage tanks), and fuel cells, responsible for the production, storage, conversion, and dispatch of electricity and hydrogen. The consumption end includes electricity loads, responsible for electricity consumption. The electrolyzer converts surplus electricity generated at the generation end into hydrogen energy and stores it in the seasonal hydrogen energy storage. The fuel cell converts the hydrogen energy in the seasonal hydrogen energy storage into electricity and feeds it back to the system. Combined with the charging and discharging of the energy storage batteries, this achieves bidirectional conversion of electricity and hydrogen energy and energy balance across time scales. The multi-agent two-layer optimization method for the islanded microgrid with seasonal hydrogen energy storage includes the following steps: (1) Constructing an upper-level capacity configuration model: Minimizing the annualized total cost F1, minimizing the power shortage rate F2, and minimizing the power curtailment rate F3 are selected as the objective functions for the economic, stability, and environmental aspects of the islanded microgrid system, respectively; the upper-level decision variables are selected as a six-dimensional capacity configuration combination, specifically including wind power capacity, photovoltaic capacity, energy storage battery capacity, electrolyzer capacity, hydrogen storage tank capacity, and fuel cell capacity; (2) Constructing a lower-level operation simulation model: The optimization operation problem of the microgrid is modeled as a multi-agent Markov decision process (MDP), and multiple policy agents are obtained through offline training using a deep reinforcement learning algorithm. The agents are used to output operation control actions based on the current environmental state. (3) Perform two-layer optimization: The upper-layer capacity configuration model uses the non-dominated sorting genetic algorithm NSGA-II to generate a first-generation population, with each individual representing a set of equipment capacity configuration combinations; the capacity configuration parameters represented by each individual are passed to the lower-layer operation simulation model; the lower-layer operation simulation model loads the offline-trained policy agent and performs a complete cycle of operation simulation to obtain the corresponding annualized operating cost, power shortage and power abandonment; the above results are returned to the upper-layer model as the fitness values ​​for calculating the three optimization objectives F1, F2 and F3; the NSGA-II algorithm generates the next generation population through selection, crossover and mutation operations based on the fitness values ​​of all individuals; the above simulation and evaluation process is repeated until the maximum number of iterations or the convergence condition is reached, and finally a set of Pareto optimal equipment capacity configuration combinations are output.

2. The multi-agent bi-level optimization method for the seasonal hydrogen energy storage islanded microgrid according to claim 1, wherein, The lower-level operation simulation model is modeled as a MADRL model containing three agents: BAT-Agent (energy storage battery), ELEC-Agent (electrolyzer), and FC-Agent (fuel cell). The state space of the MDP is a 16-dimensional vector, including time-coded information, energy storage state information, resource information, equipment state information, and prediction information, specifically defined as follows: ； ； ； ； ； in: Given the current hour, the hour is mapped onto the unit circle using sine and cosine functions to ensure the periodic continuity of time. Given the current season number, the sine and cosine functions are used to map the season number onto the unit circle, ensuring the periodic continuity of the seasons; This refers to the state of charge of the energy storage battery. This refers to the hydrogen storage status of the hydrogen storage tank; Let be the photovoltaic power generation during time period t; Let t represent the wind power generation capacity during time period t; Let t be the electrical load demand during time period t; This represents the operating power of the electrolytic cell during time period t-1. This represents the operating power of the fuel cell during the t-1 time period; and These represent the on / off states of the electrolyzer and fuel cell during time period t-1, where 1 indicates on and 0 indicates off. This provides forward-looking information on photovoltaic power generation for the t+1 time period; This provides forward-looking information on wind power generation capacity for the t+1 time period; This provides forward-looking information on electricity load demand for the t+1 time period; Therefore, the state space of each agent is defined as follows: ； ； 。 3. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 2, characterized in that, The action space of each agent is defined as a normalized continuous action space, and the action space of each agent is represented as follows: ； in: , and These are the action spaces for the energy storage battery intelligent agent, the electrolyzer intelligent agent, and the fuel cell intelligent agent, respectively. t represents the action value output by the energy storage battery intelligent agent during time period t. Its positive and negative signs are used to distinguish the charging and discharging modes. A positive value indicates discharging and a negative value indicates charging. Its absolute value represents the ratio of charging and discharging power to rated power. and These are the action values ​​output by the electrolyzer and fuel cell intelligent agents during time period t, respectively, representing the operating load rate of their respective devices. The larger the value, the greater the operating power.

4. The multi-agent two-layer optimization method for microgrids with seasonal hydrogen energy storage islands as described in claim 3, after each agent outputs its action value, performs the following action mapping and safety correction steps: First, the physical mapping of actions is executed. Based on the capacity parameters of each device, the charging power of the energy storage battery will be calculated according to the following formula. Energy storage battery discharge power Electrolytic cell power and fuel cell power The normalized action values ​​are mapped to the device's physical power commands: ； in, and These are discharge and charge power commands, respectively. This refers to the rated capacity of the energy storage battery. This refers to the maximum charge / discharge rate of the energy storage battery. and These are the rated capacities of the electrolyzer and the fuel cell, respectively. Secondly, the aforementioned physical power commands are sequentially input into the three-layer safety module for verification and correction, and the final execution action is output, including the following steps: (1) SOC boundary protection: Based on the current state of charge (SOC) of the battery and hydrogen storage tank, power commands that violate physical boundaries will be forcibly set to 0 or limited to the allowable range: ； in: The state of charge of the energy storage battery during time period t; and These are the upper and lower limits of the state of charge of the energy storage battery, respectively. The state of charge of the hydrogen storage tank; and These are the upper and lower limits of the state of charge of the hydrogen storage tank, respectively. (2) Climbing constraint projection: Calculate the difference between the commanded power of the equipment at the current moment and the actual power at the previous moment. If it exceeds the maximum allowable climbing rate of the equipment, then project the commanded power onto the allowable climbing boundary: ； (3) Business rule restrictions: When the system's renewable energy and energy storage discharge are insufficient to meet the electrical load, a command to forcibly reduce or cut off the hydrogen production power of the electrolyzer is issued: First, calculate the remaining power of the system for hydrogen production at the current time t. The formula is as follows: ； Secondly, the commanded power of the electrolytic cell Based on remaining power The corrected operating power is calculated using the following formula: 。 5. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 1, characterized in that, The complete cycle simulation is based on 24-hour data from four typical days in spring, summer, autumn, and winter, totaling 96 time steps. During the simulation, seasonal transition processing and annual closure constraint checks are performed to simulate cross-seasonal energy shifts and achieve sustainable annual cycles. The seasonal transition processing means that at the end of each typical day's simulation, the initial hydrogen storage capacity of seasonal hydrogen storage on that typical day is equal to the accumulated hydrogen storage capacity of the previous quarter, as shown in the following formula: ； in: This represents the initial hydrogen storage capacity on a typical day in the current season (s). This represents the initial hydrogen storage capacity for a typical day of s-1 in the previous season. and These represent the initial and final states of hydrogen energy storage on a typical day of s-1 in the previous season, respectively. represents the total number of typical days in the current season s; s represents the typical number of days in the corresponding season. The annual closed-loop constraint check refers to the situation where, at the end of the 96-time-step simulation, the amount of hydrogen stored under constraint during the winter is equal to the initial hydrogen storage on a typical day in the spring, as shown in the following formula: 。 6. The method according to claim 1, characterized in that, The reward function of the MADRL model is designed to minimize the immediate running cost, and its calculation formula is as follows: ； in: , , , , and These are operation and maintenance costs, degradation costs, power shortage penalties, power abandonment penalties, start-up and shutdown costs, and constraint violation costs.

7. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 1, characterized in that, The MADRL model is trained using the Proximal Policy Optimization (PPO) algorithm, and each agent contains a policy network (Actor). A value network Critic Furthermore, both the policy network and the value network adopt a multilayer perceptron structure; (1) Network parameter update: The update objective function of the policy network Actor The relevant definitions are as follows: ； in: The importance ratio of the old and new strategies; Generalized advantage estimation (GAE); For time-series difference residuals; For the clipping factor, Here is the entropy regularization coefficient. For policy entropy; The instantaneous reward at time t. As a discount factor, For GAE smoothing parameters; The update objective function of the value network Critic is: ； in: For target value; The overall loss function of the agent is: ； in: The loss weight coefficients for the value function; (2) Network structure configuration: The structure of the policy network includes an input layer, at least two fully connected hidden layers and an output layer; the hidden layers use the ReLU activation function; the output layer is configured with an activation function according to the agent type: for battery agents, the hyperbolic tangent activation function Tanh is used to output the charging and discharging action value in the interval [-1,1]; for electrolyzer agents and fuel cell agents, the Sigmoid activation function is used to output the load rate action value in the interval [0,1]; the output layer of the value network contains a neuron used to output the value estimate of the current state; the parameter updates of the policy network and the value network both use the adaptive moment estimation Adam optimizer.

8. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 7, characterized in that, The training process of the multi-agent deep reinforcement learning model adopts a multi-agent collaborative architecture based on shared observations, specifically including: during the training phase, multiple parallel environments are constructed to simultaneously collect trajectory data, and the trajectory data is stored in a fixed-length experience replay buffer; during the training and execution phases, the three agents share a global observation vector containing the time encoding information, the energy storage status information, the resource information, the device status information, and the prediction information, so as to achieve unified perception and collaborative decision-making of the system status in the isolated operation mode; when the experience replay buffer is full, based on the collected trajectory data, the network parameters of each agent are updated in multiple rounds using the mini-batch gradient descent method.

9. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 1, characterized in that, To enhance the robustness of the strategy under uncertain environments, a domain randomization mechanism is introduced during training. This mechanism includes environmental perturbation processing and initial state randomization processing. The environmental perturbation processing involves adding random noise following a Gaussian distribution to the historical data of photovoltaic power, wind power, and electrical load at the beginning of each training round to generate diverse training scenarios. The calculation formula is as follows: ； in: , and These are training data for photovoltaic, wind power, and load, respectively. , and These are the original historical data for photovoltaic, wind power, and load, respectively. It is a random noise variable; It follows a Gaussian distribution; , and These are the noise standard deviations for photovoltaic, wind power, and load, respectively; the initial state randomization process refers to the uniform distribution and random initialization of the initial state of charge of the battery and hydrogen storage tank within a preset interval at the beginning of each training round.

10. The multi-agent bi-layer optimization method for microgrids containing seasonal hydrogen energy storage islands according to claim 1, characterized in that, The upper-level capacity allocation model is solved using the non-dominated sorting genetic algorithm NSGA-II, which employs an elite retention strategy and uses the Latin hypercube sampling (LHS) method for initial population generation. The three objective functions set for the upper-level capacity allocation model are minimizing the annualized total cost F1, minimizing the power shortage rate F2, and minimizing the power curtailment rate F3, and their calculation formulas are as follows: ； in: A collection of devices; The discount rate; For equipment lifespan; Unit investment cost; Configure for capacity; Let t be the total operating cost at time t.

Images