Hydrogen-containing micro-grid energy scheduling method based on distributed federal reinforcement learning

Abstract

The invention relates to the technical field of micro-grid energy optimization, in particular to a distributed federal reinforcement learning-based hydrogen-containing micro-grid energy scheduling method, which comprises the steps of constructing a multi-region hydrogen-containing micro-grid system model, designing a state space, an action space and a reward function of an intelligent agent, constructing an Actor-Critic network and an experience pool, and completing environment initialization. The intelligent agent inputs the operation state of the equipment into the Actor network, updates the state of the equipment according to the output action, verifies the constraint and outputs a reward value; tuples are extracted from the experience pool to update local network parameters, and the exploration rate is updated regularly; when a federation interaction period is reached, exchanging Critic network parameters and updating federation parameters; and when the training round arrives, outputting an equipment operation plan, deploying the model to the local hydrogen-containing micro-grid in the island mode, and outputting an equipment output value. According to the scheme, strategy sharing and learning collaboration are realized through neighborhood collaboration and local communication among the regional intelligent agents, so that dependence on a central node is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of micro-grid energy optimization, and particularly relates to a hydrogen-containing micro-grid energy scheduling method based on distributed federated reinforcement learning. BACKGROUND

[0002] A micro-grid is a local energy system that integrates distributed energy sources, renewable power generation, energy storage systems, and loads, and has the ability of autonomous control and flexible interconnection with the main grid. With the increasing demand for clean energy development, hydrogen energy, as a zero-carbon, high-density, and long-term storage secondary energy, is gradually introduced into the micro-grid system to improve energy regulation capacity and system flexibility. The hydrogen energy system forms a deep coupling with the electrical energy system through the processes of water electrolysis, hydrogen storage, and fuel cell power generation, and shows significant advantages in dealing with renewable energy volatility and grid load peaks and valleys. However, the integration of hydrogen and electricity brings problems such as complex device dynamic characteristics, diverse energy flow paths, and strong coupling of scheduling variables, and it is urgent to build a more efficient and intelligent coordination control and energy scheduling method to achieve stable, safe, and economic operation of the system.

[0003] In view of the problems of high-dimensional state space, strong coupling nonlinearity, and dynamic uncertainty in the hydrogen-containing micro-grid system, reinforcement learning, as an optimization method based on the interaction between agents and the environment, can continuously optimize the scheduling strategy through the trial-and-error mechanism without relying on accurate system models, has strong environmental adaptability and online decision-making ability, and has been widely applied to control optimization tasks in complex energy systems. However, traditional reinforcement learning usually relies on a single agent for training in a centralized environment, which is difficult to deal with data privacy, communication overhead, and heterogeneity problems in multi-region micro-grids, and is prone to convergence performance degradation due to low sample efficiency or data distribution differences. In contrast, federated reinforcement learning can protect local data privacy while improving strategy sharing and generalization ability through multi-agent collaborative learning, making it more suitable for energy scheduling scenarios in multi-region and multi-energy coordination.

[0004] It is worth noting that in existing federated reinforcement learning frameworks, a centralized architecture is mostly adopted, i.e., a central server is responsible for uniformly aggregating local model parameters of agents in each region and issuing a global strategy. However, this centralized approach has many limitations in multi-region micro-grid systems: on the one hand, the central node as the only coordinator may cause the learning process to be interrupted if it fails, which poses a significant single-point failure risk; on the other hand, as the system size expands, centralized communication and computation will cause communication bottlenecks and resource overload, affecting model training efficiency and scheduling response speed.

[0005] To overcome the above problems, it is urgent to build a distributed federal reinforcement learning mechanism to realize strategy sharing and learning cooperation through neighborhood cooperation and local communication between regional agents, without relying on a central node, so as to reduce the communication burden while ensuring system stability. SUMMARY

[0006] The present application provides a distributed federal reinforcement learning energy scheduling strategy with privacy protection capability through a multi-region hydrogen-containing micro-grid system, which realizes strategy sharing and learning cooperation through neighborhood cooperation and local communication between regional agents, breaks away from the dependence on the central node, reduces the communication burden while ensuring the stability of the system.

[0007] The technical scheme provided by the present application is: a hydrogen-containing micro-grid energy scheduling method based on distributed federal reinforcement learning, the method comprising: A multi-region hydrogen-containing micro-grid system model is constructed according to the multi-region equipment parameters and electricity prices, the state space, action space and reward function of the agent are designed, the Actor-Critic network and experience pool are constructed, and the environment initialization is completed; Each regional agent inputs the device operating state observation into the Actor network, and updates the device state at the next time according to the output action; The agent inputs the current and next state parameters, checks the constraint conditions, and outputs the reward value according to the constraint judgment result and the total operating cost; The agent extracts tuples from the experience pool, updates the local network parameters according to the loss function, updates the exploration rate every preset training round, and generates new experience tuples by re-exploring, and replaces the old data in the experience pool; When the federal interaction period is reached, the agent exchanges the Critic network parameters with the neighborhood agents and updates the federal parameters; When the training round reaches the maximum value, the device operation plan is output, the trained scheduling network model is deployed to the local hydrogen-containing micro-grid in island mode, and the device output value is output according to the real-time wind, light and load power values.

[0008] Preferably, the specific process of the environment initialization is as follows: In the multi-area hydrogen-containing micro-grid system model, each area contains wind turbines, photovoltaic power generation systems, energy storage batteries, water electrolysis hydrogen production devices, fuel cells, hydrogen storage tanks, generators and electrical loads, and allows each area to trade electricity with the main grid; the state space designed for each agent includes device power, energy storage state, electricity purchase and sale price and device power change amount, the action space includes device output adjustment and main grid interaction power adjustment, and the reward function considers economy and environmental protection; the interconnection topology between micro-grid areas is determined, and the double random matrix elements are defined based on this to realize dynamic weight distribution when sharing federal parameters; an Actor network, a Critic network and their respective target networks are constructed for each agent, an independent experience pool is established and a priority experience replay sampling mechanism is defined.

[0009] Preferably, the specific process of updating the device state at the next time is as follows: The device operating state observations collected by the agent include wind turbine and photovoltaic power generation system output power, energy storage battery and hydrogen storage tank energy state, generator output, water electrolysis hydrogen production device and fuel cell power, and main grid interaction power; after inputting the observations into the Actor network, Gaussian noise is introduced to the actions output by the network; based on the actions with added noise, the operating parameters of each device are adjusted, including energy storage battery charge and discharge power, water electrolysis hydrogen production device input power, fuel cell output power and main grid interaction power, and then the device state at the next time is updated.

[0010] Preferably, the specific process of the constraint condition checking and the reward value calculation is as follows: The constraint conditions include power balance constraint, energy storage device energy state constraint, power conversion constraint, climbing constraint, power upper and lower limit constraint and mutual exclusion constraint, which respectively ensure power supply and demand matching, energy storage device safe operation, energy conversion efficiency, device power stable change, device power in a safe range and energy storage device not charging and discharging at the same time; whether the device state parameters at the next time meet each constraint condition is checked one by one, if all meet, the cost function including generator, water electrolysis hydrogen production device, fuel cell operation cost and regional power transaction net cost is calculated as the reward value; if any constraint condition is not met, the product of the constraint violation degree and the corresponding weight is calculated as the penalty function, and the sum of the cost function and the penalty function is taken as the reward value.

[0011] Preferably, the specific process of the local network parameter update is as follows: The agent extracts a batch of experience tuples from the local experience pool, and the experience tuples include a current state, an action, a reward value, a next state and a termination flag; a target actor network outputs a corresponding action according to the state at the next moment, two target critic networks respectively evaluate the Q value of the state-action pair, and the minimum value of the two target critic networks is taken to calculate the target Q value; the critic network parameters are updated by minimizing the mean square error between the predicted Q value of the critic network and the target Q value; the actor network is updated every preset number of steps, and the critic network output Q value is maximized; the parameters of the target network and the corresponding original network are periodically synchronized by using a soft update mechanism according to a fixed proportion.

[0012] Preferably, the specific process of the federal parameter update is as follows: A fixed federal interaction cycle is preset, when the cycle is reached, the agent only establishes a communication connection with the neighborhood agent determined based on the regional connection topology; each agent sends its critic network parameters to the neighborhood agent, and receives the critic network parameters sent by the neighborhood agent; the weight coefficients are calculated according to the double random matrix determined in the initialization stage, and the critic network parameters of the agent and the neighborhood agent are weighted and aggregated to obtain new federal parameters; the original critic network parameters of the agent are replaced by the federal parameters, and the federal parameter update is completed.

[0013] Preferably, the specific process of the model deployment and device output is as follows: The trained agent scheduling network model is loaded into the control unit of the local hydrogen-containing micro-grid in island mode; the control unit collects real-time data of local wind turbine power, photovoltaic power generation system power and electrical load power; the collected real-time data are input into the scheduling network model, and the model outputs the charging and discharging power of the energy storage battery, the input power of the water electrolysis hydrogen production device, the output power of the fuel cell, the output power of the generator and the interactive power with the main grid, realizing online distribution of device output in island mode; the scheduling result is fed back to the monitoring module, realizing real-time monitoring and abnormal adjustment of the scheduling process.

[0014] The application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the hydrogen-containing micro-grid energy scheduling method based on distributed federal reinforcement learning.

[0015] The application has the following advantages: 1. The scheme adopts a fully decentralized federal reinforcement learning architecture, without relying on a central server, and each regional agent only interacts with the Critic network parameters and aggregates the policy through local communication and neighborhood cooperation. Compared with traditional centralized federal reinforcement learning, the learning interruption problem of the entire scheduling system caused by the failure of the central node can be effectively avoided, and without increasing the centralized communication and computing burden, the robustness and scalability of the system in complex scenarios in multiple regions are significantly improved.

[0016] 2. Each regional agent only shares the Critic network model parameters, without transmitting raw data such as wind turbine output, load demand, and device operating status, and the experience pool is independently owned by each agent, avoiding centralized storage or cross-regional transmission of raw data. Compared with the global data sharing mode of traditional deep reinforcement learning and the centralized federal learning mode relying on a central server, the risk of data privacy leakage can be minimized, meeting the demand for data privacy protection of each subject in the multi-region hydrogen-containing microgrid.

[0017] 3. The trained agent scheduling network model can be directly deployed to the local hydrogen-containing microgrid in island mode. When disconnected from the main grid, it does not require external communication or central node support, and can independently complete decisions such as energy storage battery charging and discharging, water electrolysis hydrogen production device and fuel cell power regulation, and generator output distribution, based on real-time acquisition of wind, light, and load power data. The scheduling strategy can achieve an economic operation mode of "hydrogen production and charging during low valley period, fuel cell power generation and energy storage discharging during peak period", which ensures stable power supply in island mode while reducing daily operating costs, balancing stability and economy. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 A flowchart of a hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning; Figure 2 A scheduling process flowchart of a hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning; Figure 3 A comparison chart of convergence of a distributed federal reinforcement learning algorithm and a centralized federal reinforcement learning algorithm of a hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning and a traditional deep reinforcement learning algorithm in the same environment; Figure 4 A daily energy scheduling diagram obtained in a hydrogen-containing microgrid of a hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning. DETAILED DESCRIPTION

[0019] The following description is presented to enable any person skilled in the art to practice the application as claimed. Preferred embodiments are presented in the following description only as examples and modifications thereto can be made by those skilled in the art upon reading the present disclosure without departing from the spirit of the application. The present application is well suited to achieving this object with the objects hereinbefore described and others that will become apparent in the course of the following description.

[0020] It can be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of one element can be one, and in another embodiment, the number of the element can be multiple, and the term "one" cannot be understood as a limitation on the number.

[0021] As shown in Figure 1 and Figure 2 In order to overcome the single point failure risk and communication bottleneck problem of traditional centralized federated reinforcement learning method in multi-region hydrogen-containing microgrid scheduling, the embodiment provides a distributed federated reinforcement learning hydrogen-containing multi-microgrid energy online scheduling method based on reinforcement learning algorithm and combined with distributed federated framework. The embodiment first completes the initialization environment, including system topology configuration, device parameter setting and initial state variable loading and other operations. Subsequently, each regional agent inputs the local actor network based on the local observation state to generate scheduling actions, and accordingly promotes the system state transition and reward calculation; the agent evaluates the feasibility of the scheduling action according to the local operation constraint, obtains the reward signal for training feedback, and packs and stores the current state, action, reward, next state and termination flag and other information into the local experience pool. Each agent updates the actor and critic networks using the historical interaction data in the experience pool, and improves the sample utilization efficiency and training stability by combining the priority experience replay mechanism. During the training process, the exploration rate will be dynamically adjusted according to the round, and the timeliness of the experience pool will be maintained through periodic experience replacement; within the preset federated synchronization period, each agent performs local sharing and strategy aggregation of the critic network parameters based on the topology structure and double random weight matrix, realizing decentralized federated learning. After deploying the network model trained to a certain round to each hydrogen-containing microgrid agent, the hydrogen-containing microgrid independent energy scheduling decision under the "island mode" is realized.

[0022] Specifically, the distributed federated reinforcement learning hydrogen-containing multi-microgrid energy online scheduling method of the embodiment includes the following steps: First, environmental initialization is crucial for the stable operation of subsequent scheduling algorithms. This initialization must be carried out across five core dimensions: system model construction, variable definition, spatial design, topology and network setup, and experience pool configuration. This ensures comprehensive preparation of all elements required for hydrogen-containing microgrid scheduling. A multi-regional hydrogen-containing microgrid system model is established. Each region is configured based on its specific application scenario (e.g., industrial parks, islands, remote mountainous areas) according to its energy endowment and load characteristics. This model includes wind turbines (WT), photovoltaic (PV) systems, energy storage batteries (ESS), water electrolysis hydrogen production units (EHP), fuel cells (FC), hydrogen storage tanks (HT), generators (DG), and necessary electrical loads (EL). For each type of equipment, a refined mathematical model is established, combining physical characteristics and engineering practice. This model covers operating power, energy conversion efficiency, energy charging and discharging limits, and power ramp-up limits. It also incorporates dynamic factors such as equipment aging coefficients and environmental disturbances (e.g., the impact of temperature on PV efficiency and the impact of wind speed fluctuations on WT output) to ensure the completeness and practicality of operational constraints. Each region is allowed to connect to the main grid for electricity trading through flexible interfaces. When there is a surplus of renewable energy output or a large load gap in the region, the power imbalance can be balanced through power interaction with the main grid (purchasing or selling electricity) to avoid problems such as voltage fluctuations and frequency deviations caused by local power imbalance.

[0023] To facilitate subsequent algorithm calculations and logical derivations, At this moment The power definitions for each device in the microgrid area are as follows: the hourly output power of the wind turbine and the photovoltaic power generation system are respectively... and The demand-side load power is The energy states of the energy storage battery and the hydrogen storage tank are respectively and The energy charging and discharging power of each of their devices are respectively , and , The output power of the generator is expressed as: The input electrical power and the output hydrogen power of the water electrolysis hydrogen production unit are respectively... , The input hydrogen power and the converted electrical power in the fuel cell are respectively , The power exchange between each microgrid area and the main grid is: The value is negative when the microgrid sells excess electricity to the main grid and positive when it purchases the required electricity.

[0024] For each microgrid agent, a local state space, scheduling action space, and reward function that comprehensively considers economic and environmental benefits are designed. Time, region State value of the agent Set as shown below; its action Set as shown below; ; ; In the formula, indicates the purchase and sale price of electricity between the th micro-grid and the main grid; , , The change amount of the generator output power, the water electrolysis hydrogen production device input power and the fuel cell output power between the current time and the last time .

[0025] Determine the connection topology structure between the micro-grid areas, and determine the element definition of the double random matrix according to the following formula from the structure, in order to realize the dynamic weight distribution in the federal parameter sharing process.

[0026] ; An Actor network (for generating scheduling actions) and two Critic networks (for evaluating the value function of state-action pairs) are constructed for each agent to alleviate the high bias problem in Q value estimation. At the same time, each network needs to be equipped with a target network with the same structure to realize the delayed update strategy. Among them, the target of the Critic network is to estimate the Q value function , the corresponding two Q network parameters are respectively denoted as , , and the optimization objective function is determined by minimizing the mean square error between the Q value output by the current Critic network and the target Q value, which is calculated by the immediate reward, the discount factor and the minimum value of the Q value output by the two target Critic networks. The double Q learning mechanism is adopted to take the minimum value of the Q value of the two target Critic networks to reduce the estimation bias. The target of the Actor network is to optimize the strategy by maximizing the Q value, and its network parameter is denoted as , and the loss function is determined by maximizing the Q value output by the Critic network, that is, minimizing the negative value of the Q value. The structure of the target network is completely consistent with the original network, which is composed of an input layer, a hidden layer and an output layer, and the difference is that the soft update mechanism is used to reduce the estimation variance, and the parameters are respectively denoted as and ( =1,2).

[0027] ; ; ; In the formula, is the action policy; is the state subject to the distribution; is the target value; usually calculated by the target network from the above formula; is the immediate reward; is the discount factor; is the next state; is the next action given by the target policy network.

[0028] An independent experience pool is established for each agent to store experience tuples such as state, action, reward, next state and termination flag generated during the interaction with the environment. When training the network model of the agent, the tuples from the experience pool are always used for parameter update to improve sample utilization efficiency and enhance training stability. In addition, a sampling mechanism of priority experience replay is defined, so that key experiences with larger TD errors are more likely to be selected for training, thereby accelerating the convergence speed of the policy and improving the learning performance.

[0029] In the environment exploration phase, the regional agent inputs the observation of the current time device operating state to the Actor network, and updates the device state at the next time according to the output action. In order to increase the exploration of the policy and avoid falling into local optimum, a part of Gaussian noise is introduced to the action.

[0030] The regional agent makes a constraint condition judgment according to the state quantity at the current time and the action taken, and if the device state quantity at the next time satisfies the constraint condition, the cost function is calculated as the reward value, otherwise the penalty function is additionally calculated, and the sum of the two is taken as the reward value. The constraint conditions designed in this embodiment are as follows: Power balance constraint: in order to ensure that the total power supply of power generation, energy storage, power purchase and the like is consistent with the total power consumption of load demand, charging and the like, the power balance constraint of the microgrid region at the time is expressed as: ; Energy state constraint of energy storage device: the energy state of the energy storage battery and the hydrogen storage tank in the region at the time is expressed by the following constraint: ; ; In the formula, , indicates the energy charging and discharging efficiency of the energy storage battery; ,​​​​ represents the energy charging and discharging power in the hydrogen storage tank, , represents the energy charging and discharging efficiency in the hydrogen storage tank.

[0031] Power conversion constraint: since there is an efficiency loss when the energy in the water electrolysis hydrogen production device and the fuel cell of each micro-grid is converted between different media, the power conversion constraint is defined; ; ; In addition, the hydrogen power generated by the water electrolysis hydrogen production device is directly stored in the hydrogen storage tank, and the hydrogen consumed by the fuel cell is directly provided by the hydrogen storage tank, so there are the following formulas.

[0032] ; ; In the formula, , are the energy conversion efficiency of the water electrolysis hydrogen production device and the power generation efficiency of the fuel cell, respectively.

[0033] Ramp constraint: in order to avoid damage to equipment due to too fast power fluctuation or affect the stability of the power system, it is necessary to define the ramp constraint for the water electrolysis hydrogen production device and the fuel cell in the system to limit the power change rate of the equipment. The ramp constraint of each device at time is defined as follows in this embodiment: ; ; In the formula, , are the maximum and minimum ramp power of the input electric power of the water electrolysis hydrogen production device; , are the maximum and minimum ramp power of the output electric power of the fuel cell; Upper and lower power limit constraint: in order to limit the output or input power of the equipment within a safe and controllable range, the upper and lower limit constraints of each device are defined as follows in this embodiment: ; ; ; ; ; ; In the formula, , Pmax, in and Pmax, out are the upper limits of input power of water electrolysis device and output power of fuel cell, respectively. , Pmax, in and Pmax, out are the upper limits of input power of water electrolysis device and output power of fuel cell, respectively. , and , Pmax, in and Pmax, out are the upper limits of input power of water electrolysis device and output power of fuel cell, respectively.

[0034] Mutual exclusion constraint: to ensure that the energy storage device will not be charged and discharged at the same time, therefore the device power at time t should satisfy the following formula.

[0035] ; ; In the mathematical model of the multi-region microgrid system established in this embodiment, the objective function is mainly composed of device operation cost and net power transaction cost. The operation cost of power output devices such as generators, water electrolysis devices, fuel cells, etc. is calculated by the following formula, respectively. ; ; ; In the formula, , , are cost calculation coefficients. The net power transaction cost of each hydrogen-containing microgrid region is given by the following formula.

[0036] ; Therefore, the total operation cost of the hydrogen-containing microgrid region at time t can be expressed as: ; The reward function in this embodiment is composed of two parts: one is the normal cost function calculated when the operation constraints are met, and the other is the penalty term applied when the constraints are violated. When all scheduling actions meet the system constraints, the reward function is calculated according to the following formula, which represents the total operation cost of the microgrid region at time t ; when the constraints are violated, the reward function is given by the following formula, in which the penalty term is replaced, and it is taken as the local reward value at the current time.

[0037] ; ​ ; wherein, is the region At time the total penalty term; is the degree of violation of the th constraint; is the corresponding weight factor. The experience tuples obtained from the environment exploration are constantly filled into the experience pool established in step 15 for subsequent training.

[0038] In the local parameter update phase, each region takes out a batch of experience tuple data from the local exclusive experience pool, and the target Actor network outputs the next time action value according to the next time state After that, the two target Critic networks respectively give their own Q value evaluation of the next time state action pair, and take the minimum value of the two. The target Q value is calculated by the target Q value calculation method (i.e. immediate reward + discount factor x minimum value of two target Q values), and the model is optimized and updated to minimize the mean square error loss of the Critic network. The Actor network is updated every fixed number of steps (such as 5 steps) to maximize the Q value given by the Critic network. In addition, all target networks will be periodically soft-updated, and the specific parameter update formula is as follows: ; ; ; ; wherein, , is the learning rate of the Critic and Actor networks; , indicates the gradient of each parameter; the loss function , is determined by the optimization goal of the Critic network and the Actor network respectively; is the soft update coefficient of the target network update.

[0039] To achieve a dynamic balance between exploration and utilization and avoid outdated experience, the exploration rate is updated using an exponentially decaying method every 20 training rounds, and a specific number of explorations in the environment are performed to generate new experience tuples and replace them in the experience pool.

[0040] determines whether the federal interaction cycle is reached ​If it enters the federal update training round, each agent will exchange its Critic network parameters with all agents in its neighborhood agent set according to the following formula, and replace the original parameters with the new federal parameters. The weight coefficient in the formula is calculated by a double random matrix, which has been defined in the initialization.

[0041] ; It is judged whether the training period reaches the maximum value. If the maximum training round has been reached, all regions output the operation plan optimization results of the respective schedulable devices at the end of the round, otherwise step 4 is returned to continue execution.

[0042] The trained agent scheduling network model is deployed to the local hydrogen-containing microgrid system in island mode, and reasonable device output values are given according to the wind, light and load power values at each time during microgrid operation.

[0043] In another embodiment, the effectiveness of the hydrogen-containing multi-microgrid energy scheduling method proposed in this embodiment is further illustrated through specific experiments: Under the environment of python3.6, the multi-region hydrogen-containing microgrid system model used in this experiment is written according to the content described in step 1, and the constraint conditions of each device are designed according to the description in step 3. After completing the environment building, the =1.5×10-4、 =1×10-4、 =3×105、 =1×10-3、 =100, the number of rounds is 4000, the batch size is 8400, and the training parameters of each round contain 24 time points. The agents in each hydrogen-containing microgrid are trained in the simulation environment according to steps 1-7. And the centralized federal reinforcement learning algorithm (CFRL) and the deep reinforcement learning algorithm (MATD3 algorithm) are taken as the control algorithm for experiment.

[0044] As shown in Figure 3 From the experimental results, it can be seen that although the convergence speed of CFRL is slightly faster than that of the Dis-FRL method proposed in this application, Dis-FRL shows a smoother reward growth curve in the early stage of convergence, with smaller fluctuations, indicating that its learning process is more stable. In addition, the Dis-FRL method shows a faster convergence speed within about 1500 rounds, compared with MATD3, which has slower convergence speed and stability. After more than 2000 rounds, it reaches convergence, and the reward fluctuation is larger in the later training period. It is shown that the distributed federal reinforcement learning method proposed in this application effectively coordinates the strategy update of each agent through the decentralized double random matrix coordination mechanism, thereby improving the stability and convergence speed of the convergence process.

[0045] And as can be seen from the following table, the MATD3 algorithm shares historical data through experience replay and adopts a centralized training and decentralized execution architecture, and the privacy is also low. CFRL adopts a distributed training and decentralized execution architecture, and achieves a certain degree of privacy protection through model parameter sharing, but since it depends on a central server, the privacy depends on the security of the server. In contrast, Dis-FRL completely abandons the central server, and the interaction between agents is only through parameter interaction to complete optimization, avoiding the sharing and centralized storage of raw data, and significantly improving the level of privacy protection. Finally, after deploying the trained scheduling optimization network model according to step 8, the energy scheduling simulation of any hydrogen-containing micro-grid system model for one day is performed, and the scheduling results are as shown in Figure 4 It can be seen from the results that the start-stop and output of various devices have obvious coordination and economy in the time dimension. In order to utilize EHP for hydrogen production preferentially in the period of 0-8 when the electricity price is relatively low, and at the same time to mobilize ESS for charging operation to achieve the goal of peak shaving and valley filling. The period of 9-11 and 14-20 is the peak period of electricity price, the system calls DG and FC to output power together to meet the higher net load, supplemented by energy storage discharge and main grid purchase strategy, to ensure load supply and minimize operation cost. In particular, the electricity price reaches the highest at 20 o'clock, the system greatly reduces the main grid purchase amount, fully mobilizes the output of local energy facilities, and avoids high-priced electricity; and in some medium and low electricity price periods (such as around 12 o'clock), the system appropriately sells redundant electricity to the main grid, takes advantage of the time difference between low purchase and high sale to obtain economic benefits, and embodies good economic scheduling ability. It is proved that the method proposed in the application can effectively realize the low-cost optimal scheduling of the hydrogen-containing micro-grid system model.

[0046] For example, taking a hydrogen-containing micro-grid system of 3 adjacent counties under the jurisdiction of a certain prefecture-level city in East China as the application object, the system covers a chemical industrial park (region A), a suburban residential area (region B), and an ecological tourist area (region C), which are interconnected through 20 kV distribution lines and 5G private networks for power interconnection and data interaction, and need to complete online energy scheduling through distributed federated reinforcement learning, while adapting to the "island mode" operation demand under extreme weather. Each region configures core devices according to energy endowment and load characteristics, and the specific parameters are as follows: region A (chemical park) is equipped with 2.5 MW wind turbine ( = 3.2 m / s, = 14 m / s, = 26 m / s), 2 MW photovoltaic system ( = -0.0042 / ℃, 2500 kWh energy storage battery = 0.91, = 0.89, = 0.0055 / day], 1000 kW water electrolysis hydrogen production device ( = 0.76], 600 kW fuel cell ( = 0.56], 3500 kWh hydrogen storage tank ( = 0.96, = 0.94, = 0.0022 / day], 1200 kW diesel generator ( = 240 kW), daily load 2200-3800 kW, industrial continuous production load ratio 90%; region B (suburban residential area) is configured with 1.2 MW wind turbine (same wind speed parameter as region A), 1.2 MW photovoltaic system (same temperature coefficient as region A), 1200 kWh energy storage battery ( = 0.9, = 0.88], 600 kW water electrolysis hydrogen production device ( = 0.75], 350 kW fuel cell ( = 0.55], 2200 kWh hydrogen storage tank (same efficiency parameter as region A), 700 kW diesel generator ( = 140 kW), daily load 400-1800 kW, peak load ratio 50% during 18-22 hours in the evening; region C (tourist area) is configured with 1.8 MW wind turbine ( = 3 m / s], 0.9 MW photovoltaic system ( = -0.0039 / °C], 1800 kWh energy storage battery ( = 0.89, = 0.87], 700 kW water electrolysis hydrogen production device ( = 0.77], 450 kW fuel cell ( = 0.57], 2800 kWh hydrogen storage tank ( = 0.95, = 0.93], 900 kW diesel generator ( = 180 kW), daily load 700-2500 kW, peak load during holiday tourism increased by 60%. The trading electricity price of the three regions with the main grid is set according to the time period: valley time (0-8 hours) 0.52 yuan / kWh, flat time (8-11 hours, 14-18 hours, 22-24 hours) 0.83 yuan / kWh, peak time (11-14 hours, 18-22 hours) 1.25 yuan / kWh, the upper limit of the interactive power of the main grid is , to avoid impact on the main grid voltage and frequency.

[0047] In the environmental initialization stage, the system model and variable system are first constructed, and the definitions of the core parameters of each region at time t are determined: the hourly output power of the wind turbine in region A , the photovoltaic output , demand-side load power , energy state of energy storage battery and charging / discharging power / , energy state of hydrogen storage tank and hydrogen charging power 、 , diesel generator output power , water electrolysis hydrogen production device input power and hydrogen production power , fuel cell input hydrogen power and output power , interaction power with main grid (purchase power is positive, and sale power is negative), the variables of region B and C are defined in the same way. Then, the state space and action space of the agent of each region are designed. The state of the agent of region A at time t is , , which covers 13 core dimensions, among which is the real-time purchase / sale power price, and the like are the power change amounts of the equipment; the action space is , and each action variable satisfies the safety constraint, such as kW ( ), 240 kW 1200 kW. The reward function adopts a unified form of “cost + penalty”, , wherein , = 105 yuan / kW (penalty weight of power balance constraint), = 52 yuan / kWh (penalty weight of energy storage energy constraint), is the constraint violation degree, such as when the power is unbalanced.

[0048] The connection topology of the three regions is determined to be a linear structure of “region A-region B-region C”, the neighborhood set of region A is ( = 1), the neighborhood set of region B is ( = 2), and the neighborhood set of region C is ( = 1). The elements of the double random matrix are calculated according to the formula: , , ; , , ; , , , and the matrix is It satisfies the double randomness property that both row sum and column sum are 1. A "1 Actor + 2 Critics + Target Network" architecture is constructed for each region's agent. The Actor network has a 13-dimensional input layer, 64 / 32 hidden nodes (ReLU activation), and a 6-dimensional output layer (tanh activation). The Critic network has a 19-dimensional input layer (state + action), 64 / 32 hidden nodes, and a 1-dimensional output layer (Linear activation). The target network uses soft update coefficients. =0.005, discount factor =0.95, Critic learning rate Actor learning rate Establish capacity The independent experience pool of each item adopts a priority experience replay mechanism, TD error Sampling probability Importance sampling weight =0.4 (increases to 1.0 after 2000 training rounds), re-explore 10 rounds every 20 rounds, and replace 10% of the old experience in the experience pool.

[0049] Entering the environmental exploration phase, agents in each region generate actions based on real-time states and introduce Gaussian noise ( For example, region A in =3 hours (valley time, wind speed 7 m / s, irradiance 200 W / m², load 2000 kW), Actor network output =800kW =600kW =0, update device status: kWh (not exceeding 0.9) 2500 = 2250 kWh, Correction: Initial =1800kWh, kWh, exceeding the limit, triggering penalty. Yuan, reward experience tuple Store in the experience pool.

[0050] During the constraint verification phase, the six types of constraints in the scheme are strictly enforced: power balance constraints are implemented according to... Calculate, region B in =12 o'clock (peak hours, PV output 900kW, WT output 600kW, load 1600kW, =0、 =400kW =300kW =0), kW, meeting electricity purchase demand; under energy storage constraints, region C It needs to be at 0.2 1800 = 360 kWh to 0.9 1800 = 1620 kWh, At 0.1 2800 = 280 kWh to 0.95 2800 = 2660 kWh; power conversion constraints are = 0.76 , = 0.56 Execute, region = 800 kW, = 608 kW, all stored in hydrogen storage tanks ( = 608 kW); ramping constraints limit kW / min, kW / min; in the power upper and lower limit constraints, kW, kW; mutual exclusion constraints ensure , avoid energy inefficiencies.

[0051] Local parameter update phase, every 2 steps from the experience pool to sample 256 experiences, region A agent output by the target Actor network corresponding , two target Critic network output = 2800, = 2750, target Q value , Critic network loss , updated by Adam optimizer; Actor network is updated every 5 steps, loss , gradient ascent optimization strategy; soft update is performed every 10 steps: , . Exploration rate is updated according to , = 100 episodes = 0.27, = 1000 episodes = 0.11, balancing exploration and exploitation.

[0052] Arrive at the federal interaction cycle = 100 episodes, each agent encrypts and shares Critic parameters, region B aggregates parameters according to , replace the local Critic network parameters and continue training. When training to 4000 episodes, the reward value is stable at -820 yuan / day (region A), -510 yuan / day (region B), -630 yuan / day (region C), with a fluctuation range of <±25 yuan, meeting the convergence condition, and outputting the 24-hour device output plan for each region.

[0053] After the model is deployed, in the "island mode" caused by typhoon leading to the main network interruption, the regional A agent collects the WT output 1200kW, PV output 500kW, load 2800kW, =1500kWh, =2200kWh, input Actor network output =600kW, =500kW, =500kW, total supply 1200+500+600+500=2800kW, meet the load demand; region B cooperates with FC and ESS to ensure the power supply of key loads such as lighting and refrigerator; region C preferentially calls the hydrogen energy stored in HT to avoid frequent start of DG and realize 72 hours continuous power supply.

[0054] The experimental comparison shows that the convergence speed of the scheme (Dis-FRL) is 25% faster than that of MATD3, the reward fluctuation amplitude is 40% smaller than that of CFRL, the privacy leakage risk level is "low" (only the neighborhood shares the encryption parameters), the load power supply reliability in island mode reaches 99.6%, the daily average operation cost is 8.2% lower than that of CFRL and 14.5% lower than that of MATD3, the hydrogen energy utilization rate reaches 86%, and the renewable energy consumption rate reaches 93%, which fully verifies the advantages of the scheme in economy, stability and privacy protection.

[0055] The processes described above with reference to the flowcharts can be implemented as computer software programs in accordance with embodiments of the present disclosure. Embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program comprising program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication section, and / or installed from a detachable medium. When the computer program is executed by a central processing unit (CPU), the above-described functions defined in the methods of the present application are performed. It should be noted that the computer readable medium of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but not limited to, be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device. In the present application, the computer readable signal medium can include a data signal carried in a baseband or as part of a carrier wave, in which the computer readable program code is carried. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium that can send, propagate or transfer the program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to, wireless, wire, optical cable, RF, or any suitable combination of the above.

[0056] The computer program product of the present application can be a computer program product comprising a computer-readable medium bearing computer program code embodied therein for use with a computer. The computer program code can be code defining and / or implementing the present application. The computer program code can be organized into one or more computer program components, including but not limited to, program components for implementing the features of the present application. The one or more computer program components can each include the code implementing the features of the present application. The one or more computer program components can each be embodied on the computer-readable medium. The computer-readable medium can be a tangible computer-readable medium. In operation, the computer program code is provided on the computer-readable medium to a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the code can be executed via the The present application can be a system, a method, an apparatus and / or a computer program product at any possible technical detail level of integration. The present application can also take the form of an apparatus including a computer program product and / or a computer program for executing the processes of the present application on computing devices in the system.

[0057] The embodiments of this application can be implemented by computer software executable by a computer or processor, such as an ASIC, an FPGA, or the like; computer software, such as firmware, or the like; hardware, or a combination of any of the above. Embodiments of the application can also be implemented as or integrated in hardware, such as ASIC, FPGA, or the like, which can be generally configured to perform the techniques of the present application on suitable hardware computer platforms.Those skilled in the art will appreciate that the application described herein is susceptible to variations in absence of departing from the spirit and scope of the application. Any reference signs in the claims should not be construed as limiting the scope of the claims.

Claims

1. A hydrogen-containing microgrid energy scheduling method based on distributed federated reinforcement learning, characterized in that, The method comprises: According to the multi-region device parameter and the electricity price, a multi-region hydrogen-containing micro-grid system model is constructed, the state space, action space and reward function of the intelligent agent are designed, the Actor-Critic network and experience pool are constructed, and the environment initialization is completed; Each regional intelligent agent inputs the device operating state observation into the Actor network, and updates the device state at the next time according to the output action; The intelligent agent inputs the current and next time state parameters, checks the constraint condition, and outputs the reward value according to the constraint judgment result and the total operation cost; The intelligent agent extracts tuples from the experience pool, updates the local network parameters according to the loss function, updates the exploration rate every preset training round and generates new experience tuples by re-exploring, and replaces the old data in the experience pool; When the federal interaction cycle is reached, the intelligent agent exchanges the Critic network parameters with the neighbor intelligent agent and updates the federal parameters; When the training round reaches the maximum value, the device operation plan is output, the trained scheduling network model is deployed to the local hydrogen-containing micro-grid in island mode, and the device output value is output according to the real-time wind, light and load power values.

2. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 1, characterized in that, The specific process of the environment initialization is as follows: In the multi-region hydrogen-containing micro-grid system model, each region contains wind turbines, photovoltaic power generation systems, energy storage batteries, water electrolysis hydrogen production devices, fuel cells, hydrogen storage tanks, generators and electrical loads, and allows each region to trade electricity with the main grid; the state space designed for each intelligent agent includes device power, energy storage state, purchase and sale price and device power change, the action space includes device output adjustment and main grid interaction power adjustment, and the reward function considers economy and environmental protection; the connection topology structure between the micro-grid regions is determined, and the double random matrix elements are defined to realize dynamic weight distribution when sharing federal parameters; the Actor network, Critic network and respective target network are constructed for each intelligent agent, an independent experience pool is established, and a priority experience playback sampling mechanism is defined.

3. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 2, characterized in that, The specific process of updating the device state at the next time is as follows: The device operating state observation collected by the intelligent agent includes the output power of the wind turbine and the photovoltaic power generation system, the energy state of the energy storage battery and the hydrogen storage tank, the generator output, the water electrolysis hydrogen production device and the fuel cell power, and the main grid interaction power; after inputting the observation into the Actor network, the action output by the network is introduced into the Gaussian noise; according to the action with noise, the operating parameters of each device are adjusted, including the energy storage battery charging and discharging power, the water electrolysis hydrogen production device input power, the fuel cell output power and the main grid interaction power, and then the device state at the next time is updated.

4. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 3, characterized in that, The specific process of the constraint condition checking and the reward value calculation is as follows: The constraints include power balance constraints, energy storage device energy state constraints, power conversion constraints, ramping constraints, power upper and lower limit constraints, and mutual exclusion constraints, which respectively ensure power supply and demand matching, safe operation of the energy storage device, energy conversion efficiency, stable change of device power, device power within a safe range, and simultaneous charging and discharging of the energy storage device; whether the device state parameters at the next moment meet each constraint condition is checked one by one, if all meet, a cost function including the operating cost of the generator, the water electrolysis hydrogen production device, the fuel cell, and the net cost of regional power trading is calculated as a reward value; if any constraint condition does not meet, the product of the constraint violation degree and the corresponding weight is calculated as a penalty function, and the sum of the cost function and the penalty function is taken as the reward value.

5. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 4, characterized in that, The specific process of the local network parameter update is as follows: The agent extracts a batch of experience tuples from the local experience pool, including the current state, action, reward value, next state, and termination flag; the target Actor network outputs the corresponding action according to the next moment state, and the two target Critic networks respectively evaluate the Q value of the state-action pair, and the minimum value of the two is taken to calculate the target Q value; the Critic network parameters are updated to minimize the mean square error between the Critic network prediction Q value and the target Q value; the Actor network updates the policy parameters every preset number of steps to maximize the Critic network output Q value; periodically, the soft update mechanism is used to synchronize the parameters of the target network and the corresponding original network at a fixed proportion.

6. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 5, characterized in that, The specific process of the federal parameter update is as follows: A fixed federal interaction period is preset, and when the period is reached, the agent only establishes a communication connection with the neighborhood agents determined based on the regional connection topology; each agent sends its Critic network parameters to the neighborhood agents and receives the Critic network parameters sent by the neighborhood agents; the weight coefficients are calculated according to the double random matrix determined in the initialization stage, the Critic network parameters of the agent and the neighborhood agents are weighted and aggregated to obtain new federal parameters; the federal parameters replace the original Critic network parameters of the agent to complete the federal parameter update.

7. The hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning according to claim 5, characterized in that, The specific process of the model deployment and device output is as follows: The trained agent scheduling network model is loaded into the control unit of the local hydrogen-containing microgrid in island mode; the control unit collects real-time data of local wind turbine power, photovoltaic power generation system power, and electrical load power; the collected real-time data is input into the scheduling network model, and the model outputs the energy storage battery charging and discharging power, the water electrolysis hydrogen production device input power, the fuel cell output power, the generator output power, and the interaction power with the main grid, realizing online allocation of device output in island mode; the scheduling result is fed back to the monitoring module to realize real-time monitoring and abnormal adjustment of the scheduling process.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the hydrogen-containing microgrid energy scheduling method based on distributed federal reinforcement learning in any one of claims 1-7.

Images