A deep reinforcement learning-based optimization scheduling method for electric-thermal-hydrogen integrated energy systems

Abstract

The application provides an electricity-heat-hydrogen comprehensive energy system optimization scheduling method based on deep reinforcement learning, and belongs to the technical field of optimization scheduling of comprehensive energy systems. The method firstly constructs mathematical models of various unit devices, and mathematically describes the optimization scheduling problem of the electricity-heat-hydrogen comprehensive energy system by taking economy as a target; secondly, the problem is expressed as a Markov decision process framework, and the state space, action space and reward function of the system are reasonably designed; then, a deep reinforcement learning algorithm is used for training until a stable optimization scheduling strategy model is obtained. After the training is completed, the model can make scheduling decisions according to the system state, and output unit scheduling instructions. The method can perform optimal scheduling according to the load demand and renewable energy output of the system, can realize the operation economy of the system, and has good scheduling performance.

Description

Technical Field

[0001] This invention belongs to the field of optimization scheduling technology for integrated energy systems, specifically relating to an optimization scheduling method for an integrated energy system of electricity, heat, and hydrogen based on deep reinforcement learning. Background Technology

[0002] With the continuous development of wind power and photovoltaic power generation technologies, the installed capacity of renewable energy has achieved rapid growth. However, the intermittency and randomness of renewable energy bring many challenges to the safe and stable operation of the power grid. Integrated energy systems have proven to be a promising solution, effectively integrating and utilizing various energy devices to improve energy efficiency and the absorption capacity of renewable energy. Hydrogen energy, as a clean energy source, possesses advantages such as zero pollution, high energy density, long storage period, and diverse utilization forms. It can be coupled with other energy forms in integrated energy systems to construct a more flexible energy system. Therefore, the electricity-heat-hydrogen integrated energy system can not only meet diversified energy needs but also alleviate the grid absorption pressure brought about by renewable energy grid connection through hydrogen storage and hydrogen energy conversion, further promoting the efficient utilization of the energy system.

[0003] Currently, model-driven optimization methods are widely used in the field of integrated energy system optimization and scheduling. However, these methods heavily rely on accurate model construction and struggle to handle the nonlinear coupling relationships prevalent in multi-energy systems. Furthermore, the complex energy flows, renewable energy sources, and uncertainties within the system further increase the difficulty of solving the optimization problem, significantly raising computational complexity. In recent years, artificial intelligence technology has rapidly emerged, and reinforcement learning based on Markov decision processes has gained considerable attention in the field of integrated energy systems due to its powerful decision-making capabilities. Reinforcement learning maximizes the cumulative reward obtained by the agent from the environment through continuous interaction and iterative optimization strategies between the agent and the environment. Deep reinforcement learning models combining deep neural networks and reinforcement learning possess stronger adaptive learning capabilities and can efficiently solve nonlinear optimization problems. The electric-thermal-hydrogen integrated energy system constructed in this invention faces challenges such as a multidimensional continuous action space, strong coupling of multiple energy flows, and multiple uncertainties. Therefore, it is urgent to propose an optimization and scheduling method for the electric-thermal-hydrogen integrated energy system based on deep reinforcement learning to improve the system's adaptive decision-making capabilities under complex operating conditions. Summary of the Invention

[0004] To address the optimization scheduling problem of an integrated energy system combining electricity, heat, and hydrogen, this invention provides an optimization scheduling method for such systems based on deep reinforcement learning.

[0005] The technical solution of this invention is as follows:

[0006] An optimal scheduling method for an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning, comprising:

[0007] First, mathematical models of each unit in the integrated energy system of electricity, heat and hydrogen are constructed, and the optimization scheduling problem of the integrated energy system is mathematically described with the economic efficiency of system operation as the objective.

[0008] Secondly, the optimization scheduling problem is formulated as a Markov decision process, and the state space, action space and reward function of the system are constructed.

[0009] Then, the SAC algorithm is used to make economic scheduling decisions, and a stable optimized scheduling strategy model is obtained after training.

[0010] Finally, the system's current state is acquired in real time, the trained optimized scheduling strategy model is input, and the scheduling instructions for each device at each time step are output.

[0011] The optimization scheduling problem of the described integrated electricity-heat-hydrogen energy system is a sequential decision problem. Its scheduling objective is to determine the output power of each unit at each time step while minimizing the system's daily operating cost. The system environment includes the electrical, thermal, and hydrogen flows within the integrated energy system, as well as the conversion and transmission between these energy sources. An environmental model of the integrated energy system is constructed based on the various energy devices and the energy supply-demand balance. This environment refers to the integrated energy system environment that interacts with the intelligent agent. It is a Markov decision process where the agent observes the real-time state of the environment, makes scheduling decisions, and generates corresponding actions for each unit to adjust the power output of each device. The environment then transitions to the next state, and the agent uses this new state as the new state to update the actions of each unit.

[0012] The integrated electricity-heat-hydrogen energy system primarily incorporates three heterogeneous energy forms: electricity, heat, and hydrogen. It meets diverse energy demands within the system through various energy conversion devices. The system comprises a four-layer structure: energy supply, conversion, storage, and demand. The energy supply layer integrates the upstream power grid, wind power, and photovoltaic power generation. The energy conversion layer integrates combined heat and power (CHP) units, proton exchange membrane electrolyzers, fuel cells, electric boilers, and gas-fired boilers for coordinated energy conversion. The energy storage layer mainly consists of batteries and hydrogen storage tanks; these storage devices maintain supply and demand balance through storage and release. The energy demand layer includes the user-side electrical and thermal loads. Through multi-energy flow coupling, the system fully utilizes the coordinated operation of various devices to achieve efficient energy distribution and supply-demand balance.

[0013] The model construction for each energy device includes:

[0014] (1) Modeling of cogeneration units:

[0015] Combined heat and power (CHP) units generate electricity and heat by consuming natural gas. They are important energy flow hubs in CHP scenarios, and their physical model is as follows:

[0016] ;

[0017] ;

[0018] In the formula, P CHP,t H CHP,t and G CHP,t Represent the electrical power, thermal power, and input power of the combined heat and power unit at time t, respectively; η CHP_e and η CHP_h These represent the power generation efficiency and heating efficiency of the combined heat and power unit, respectively.

[0019] (2) Modeling of gas-fired boilers:

[0020] Gas-fired boilers generate heat energy by consuming natural gas to meet heat load demands. Their physical model is as follows:

[0021] ;

[0022] In the formula, η GB G represents the conversion efficiency of a gas-fired boiler. GB,t H represents the input gas power of the gas-fired boiler. GB,t This represents the output thermal power of the gas-fired boiler;

[0023] (3) Modeling of electric boiler:

[0024] Electric boilers generate heat energy by consuming electrical energy to meet heat load demands. Their physical model is as follows:

[0025] ;

[0026] In the formula, η EB P represents the conversion efficiency of the electric boiler. EB,t H represents the input electrical power of the electric boiler. EB,t This represents the output thermal power of the electric boiler.

[0027] (4) Modeling of proton exchange membrane electrolyzer:

[0028] The proton exchange membrane electrolyzer converts electrical energy into hydrogen gas, and the mass flow rate of the produced hydrogen gas is:

[0029] ;

[0030] In the formula, η PEME It is the conversion efficiency of the proton exchange membrane electrolyzer; P PEME,t M represents the input electrical power of the proton exchange membrane electrolyzer; PEME,t It is the hydrogen production mass flow rate; LHV is the lower heating value of hydrogen.

[0031] (5) Fuel cell modeling:

[0032] During fuel cell operation, electricity and waste heat are generated by consuming hydrogen. The calculation formula is as follows:

[0033] ;

[0034] In the formula, η HFC_e and η HFC_h These represent the electrical efficiency and thermal efficiency of the fuel cell, respectively; P HFC,t and H HFC,t These represent the output electrical power and thermal power of the fuel cell, respectively.

[0035] (6) Energy storage device modeling:

[0036] Energy storage devices include batteries and hydrogen storage tanks. The State of Charge (SOC) at each time t can be expressed as:

[0037] ;

[0038] ;

[0039] In the formula, SOC BESS,t and SOC HST,t These represent the SOC state of the battery and the hydrogen storage tank, respectively; P cha,t , P dis,t ,η cha_e,t and η dis_e,t Represents the charging / discharging power and efficiency of the battery; M cha,t and M dis,t E represents the mass of hydrogen gas supplied to / discharged from the hydrogen storage tank. BESS and E HST These represent the rated capacity of the battery and the hydrogen storage tank, respectively.

[0040] The constructed integrated electricity-heat-hydrogen energy system also includes the following constraints:

[0041] (1) Power balance constraint:

[0042] The electrical power generated by photovoltaic and wind power in the system, the electrical power generated / consumed by each unit, and the energy exchange power of the energy storage device must be balanced with the system's electrical load demand. The energy balance equation is as follows:

[0043] ;

[0044] In the formula, P Load,t P represents the electrical load demand of the system; grid,t This refers to the electricity purchased by the system from the upstream power grid; P PV,t and P WT,t Represents photovoltaic power output and wind power output; PHFC,t P represents the electrical power output of the fuel cell; CHP,t P represents the electrical power output of a combined heat and power (CHP) unit. PEME,t P represents the electrical power consumed by the proton exchange membrane electrolyzer; EB,t P represents the electrical power consumed by the electric boiler. dis,t P represents the electrical power released by the battery. cha,t This represents the electrical power stored in the battery.

[0045] (2) Thermal power balance constraint:

[0046] ;

[0047] In the formula, H Load,t Represents the system's heat load requirements; H HFC,t H represents the thermal power output of the fuel cell; CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. EB,t This represents the thermal power output of the electric boiler.

[0048] (3) Power constraints:

[0049] ;

[0050] ;

[0051] ;

[0052] ;

[0053] ;

[0054] ;

[0055] ;

[0056] In the formula, P grid,max P PEME,max P HFC,max G GB,max G CHP,max P EB,max P represents the power purchased from the upper-level power grid and the upper limit of the power output of each generating unit, respectively. grid,min P PEME,min P HFC,min G GB,min G CHP,min and P EB,min These represent the power purchase limits from the upstream power grid and the lower power limits of each generating unit; SOC n,max and SOC n,minThe upper and lower limits of SOC for the two types of energy storage devices;

[0057] (4) Climbing constraint:

[0058] ;

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] In the formula, ΔP PEME,up ΔP HFC,up ΔG GB,up ΔG CHP,up ΔP EB,up ΔP represents the maximum ramp power of each unit's equipment. PEME,down ΔP HFC,down ΔG GB,down ΔG CHP,down and ΔP EB,down These represent the lower limit of the ramp power of each unit's equipment.

[0064] The objective function for the optimal scheduling problem of the integrated electricity-heat-hydrogen energy system is:

[0065] ;

[0066] In the formula, F represents the total operating cost of the system, and C... om,t For the operation and maintenance costs of each unit, C grid,t The cost of purchasing electricity from the upper-level power grid.

[0067] The deep reinforcement learning model for optimizing the operation of an integrated electricity-heat-hydrogen energy system is constructed as follows:

[0068] This invention employs the SAC algorithm for scheduling training. The SAC algorithm, as a method incorporating maximum entropy theory, has an architecture comprising one Actor network, two Critic networks, and two corresponding target Critic networks. In this network, the Actor network samples actions based on the current state and interacts with the environment to obtain immediate rewards and the next-time state. The two Critic networks are responsible for evaluating the value of state-action pairs, and the two target Critic networks generate stable value estimates through soft updates. This approach enables better exploration of scheduling strategies in complex integrated energy systems.

[0069] (1) The SAC algorithm can explore complex action spaces, prevent the model from getting trapped in local optima, and thus find a better policy, improving the robustness of the system. The optimal policy function of the SAC algorithm can be expressed as:

[0070] ;

[0071] In the formula, π* represents the strategy function that maximizes the expected cumulative return; s t It is the environmental state at time t; a t It represents the action at time t; α is the temperature coefficient, used to balance the trade-off between expected reward and entropy; It is the policy entropy term;

[0072] The modified Bellman equation can be expressed as:

[0073] ;

[0074] ;

[0075] In the formula, Q(s) t ,a t V(s) represents the state-action value function; t ) represents the soft-state value function; p represents the state transition probability; γ represents the discount factor, which takes values ​​in the range [0,1], indicating the importance of future rewards; r(s) t ,a t ) represents the instantaneous reward value at time step t;

[0076] In the process of agent policy improvement, the difference between the two distributions is minimized by minimizing the KL divergence, which is expressed as:

[0077] ;

[0078] In the formula, D KL () represents the KL divergence; Π is the policy set; Z πold (s t It is usually represented by a standard Gaussian distribution, which is used to normalize the distribution.

[0079] Q-value networks achieve adaptive parameter updates by minimizing residuals, as described below:

[0080] ;

[0081] ;

[0082] ;

[0083] In the formula, J Q(θ) represents the loss function of the Q-network, and θ and Let represent the parameters of the value network and its corresponding target network, respectively. During training, these parameters are assigned using the following update method:

[0084] ;

[0085] In the formula, k represents the update parameter.

[0086] The policy network is updated as follows:

[0087] ;

[0088] In the formula, ϕ is the coefficient of the policy network.

[0089] (2) State-space design:

[0090] The electric-thermal-hydrogen integrated energy system of this invention uses a 1-hour scheduling cycle and takes the electrical load, thermal load, wind power output, photovoltaic power output, time-of-use electricity price, and the SOC of energy storage as state information for deep reinforcement learning. Its state space at time t is defined as follows:

[0091] ;

[0092] In the formula, P PV,t and P WT,t Represents photovoltaic power output and wind power output; C grid,t It is time-of-use electricity pricing; H Load,t P represents the system's heat load demand; Load,t Represents the electrical load demand of the system; SOC BESS,t and SOC HST,t These represent the SOC state of the battery and hydrogen storage tank, respectively.

[0093] (3) Motion space design:

[0094] The motion space design of this invention is as follows:

[0095] ;

[0096] In the formula, P PEME,t P represents the electrical power consumed by the proton exchange membrane electrolyzer; EB,t P represents the electrical power consumed by the electric boiler. grid,t This refers to the electricity purchased by the system from the upstream power grid; H CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. HFC,t P represents the thermal power output of the fuel cell; BESS,t P represents the charge / discharge power of the BESS at time t; HST,t This represents the charge / discharge power of HST at time t.

[0097] (4) Reward function design:

[0098] The goal of the integrated electricity-heat-hydrogen energy system is to minimize the operating costs and electricity purchase costs of each unit while meeting electricity and heat load demands. The agent reward function designed in this invention comprises three parts: operating cost, electricity purchase cost, and constraint penalty, and its formula is as follows:

[0099] ;

[0100] In the formula, C om,t F1 is the operating cost of each unit in the system; β1 is the penalty cost for thermal imbalance; F2 is the penalty cost for electrical imbalance; β2 is the penalty coefficient for electrical imbalance.

[0101] The technical effects and advantages of this invention are as follows:

[0102] 1. This invention addresses the integrated electric-thermal-hydrogen energy system comprising multiple devices and energy flows, constructing a deep reinforcement learning scheduling model that considers multi-energy flow coordination. With the objective of minimizing the overall system operating cost, a reasonable continuous action space, state space, and reward function are designed. Simulation results demonstrate that this method can effectively coordinate the operational output of multiple devices, thereby significantly reducing the overall system operating cost.

[0103] 2. The proposed deep reinforcement learning-based optimization scheduling method for an integrated electricity-heat-hydrogen energy system enables unified and continuous optimization control of various devices involved in the three energy flows (electricity, heat, and hydrogen). This method effectively avoids the insufficient accuracy caused by complex modeling in traditional model-based optimization methods, as well as their inability to adapt to actual dynamic changes such as random fluctuations in source and load. Even in complex environments with fluctuating renewable energy output and load demand, this method maintains efficient and stable solution capabilities.

[0104] 3. This invention integrates electricity and heat load demand with renewable energy output information into the state space, enabling the deep reinforcement learning model to adaptively learn scheduling patterns under different operating scenarios. After training, the model can solve for the optimal scheduling strategy without retraining for new loads or renewable energy output scenarios, demonstrating good generalization performance and engineering practicality. Attached Figure Description

[0105] Figure 1 This is a diagram of the integrated electric-thermal-hydrogen energy system of the present invention;

[0106] Figure 2 This is a graph showing the results of power dispatching;

[0107] Figure 3 This is a graph showing the results of thermal power scheduling;

[0108] Figure 4 This is a graph showing the hydrogen power scheduling results;

[0109] Figure 5 This is a flowchart of the optimized scheduling method of the present invention. Detailed Implementation

[0110] The following will combine Figures 1-5 The technical solution of the present invention will be further illustrated through specific embodiments.

[0111] Example 1:

[0112] Description of the test scenario;

[0113] To verify the proposed... Figure 5 The effectiveness of the deep reinforcement learning-based optimal scheduling method for the integrated electric-thermal-hydrogen energy system is demonstrated. Figure 1 The simulation analysis is conducted using the example of an integrated electric-thermal-hydrogen energy system. A dataset containing monthly electricity load, heat load, photovoltaic power, and wind power data was used to train the algorithm. The system's equipment operating parameters are shown in Table 1, and the time-of-use electricity price is shown in Table 2. The deep reinforcement learning model constructed in this invention uses the Adam optimizer for both the Actor and Critic networks to update network weights. The training iterations are set to 2000 epochs, with the learning rate for both the Actor and Critic layers set to 5e-4 and 1e-3 respectively. Furthermore, to account for future rewards, the discount factor is set to 0.99.

[0114] The system's equipment operating parameters are shown in Table 1.

[0115] Table 1

[0116]

[0117] The time-of-use electricity prices are shown in Table 2.

[0118] Table 2

[0119]

[0120] Results analysis:

[0121] To verify the effectiveness of the proposed deep reinforcement learning-based optimization scheduling method for an integrated electric-thermal-hydrogen energy system, historical datasets were used for training and testing to analyze the overall operating cost of the system.

[0122] Scheduling result analysis:

[0123] After training the algorithm network using the constructed historical dataset, the trained network is used for dynamic economic optimization scheduling of the system. To verify the effectiveness of the proposed method, this invention randomly selects a day in the test set as validation data.

[0124] Figure 2 The results of hourly power balance over 24 hours under the optimized scheduling strategy proposed in this invention are presented. Figure 2 As can be seen, the system's electrical load is mainly supplied by wind power, photovoltaic power, combined heat and power (CHP) units, and hydrogen fuel cells. Simultaneously, battery energy storage systems, electric boilers, and proton exchange membrane electrolyzers participate in the regulation and conversion of system power. During nighttime, the overall user-side electrical load remains low, and photovoltaic units are not outputting power, with wind power output dominating. To avoid wind curtailment during periods of high wind power output, the battery energy storage system switches to charging mode to store excess electrical energy. Meanwhile, the electric boiler and proton exchange membrane electrolyzer, as key equipment in the electro-thermal-hydrogen coupling, actively absorb excess renewable electricity within the system, converting it into heat and hydrogen for storage, effectively solving the problem of excess electricity during off-peak hours. During daytime load increases and periods of high renewable energy output, user electricity demand significantly increases, and the combined wind and solar power supply capacity reaches its peak. During this phase, the battery energy storage system switches to discharging mode to release the electrical energy stored overnight, working in conjunction with CHP units and hydrogen fuel cells to smooth out peak loads and fill power supply gaps. When the system still experiences power shortages, a small amount of electricity is purchased from the external power grid to meet the demand and ensure reliable power supply. The entire system achieves real-time and precise power balance on a 24-hour scale, significantly reducing dependence on the main power grid while effectively improving the overall operational economy and comprehensive energy utilization efficiency of the system.

[0125] Figure 3The figure illustrates the hourly heat power balance results of the system under the optimized scheduling strategy proposed in this invention. As can be seen from the figure, the overall heat load throughout the day is mainly optimized through the collaborative efforts of the cogeneration units and gas-fired boilers as the main energy supply units, while hydrogen fuel cells and electric boilers participate in system balancing as auxiliary regulating heat sources. During the nighttime hours, the overall heat load remains high with relatively gentle fluctuations. To effectively cope with the high heating demand at night, the gas-fired boilers and cogeneration units bear the core heat load, with both contributing the dominant output. The electric boilers effectively absorb some surplus wind power at night, injecting a small amount of power stably into the heating network. This not only improves the utilization rate of renewable energy but also provides auxiliary heat energy without increasing natural gas consumption. During the daytime hours, as the system's heat load demand decreases, the heat output of the gas-fired boilers and cogeneration units also decreases at midday. Due to abundant wind and solar resources during the day, the electric boilers maintain a stable power injection during this period, coordinating and sharing the system's basic heat load, effectively mitigating the heat fluctuations caused by the reduced output of the cogeneration units and gas-fired boilers. The proposed scheduling strategy fully utilizes the main regulation advantages of cogeneration and gas-fired boilers, as well as the auxiliary flexibility of electric boilers and hydrogen fuel cells. While meeting the real-time balance of heat supply and demand throughout the day, the system significantly improves the operational economy and energy efficiency of the integrated energy system on the heat side.

[0126] Figure 4This paper presents the hourly hydrogen power balance results of the system under the optimized scheduling strategy proposed in this invention, over a 24-hour period. As shown in the figure, the system adopts an integrated closed-loop operation mode for hydrogen supply and demand, encompassing production, storage, and utilization. The proton exchange membrane electrolyzer is responsible for hydrogen production, while the hydrogen fuel cell consumes hydrogen as the end-use energy device. The hydrogen storage tank system acts as a regulating hub for energy transfer across time periods. During nighttime, due to the fluctuations in wind power output and the absence of photovoltaic power, the system primarily relies on the hydrogen storage tanks operating in a hydrogen release mode to meet the basic operating needs of the hydrogen fuel cells. At this time, the proton exchange membrane electrolyzer is in a low-power or shutdown state, indicating that the system effectively avoids the economic disadvantage of hydrogen production during off-peak electricity periods by fully utilizing stored hydrogen energy. During peak daytime periods with abundant wind and solar resources, the combined output of photovoltaic and wind power reaches its peak, significantly increasing the surplus electricity within the system. To absorb this unbalanced electricity, the proton exchange membrane electrolyzer quickly responds and upgrades to a high-efficiency hydrogen production state. Since hydrogen consumption by the hydrogen fuel cell remains low during this period, the majority of the hydrogen produced by the electrolyzer is stored in the hydrogen storage tank, achieving the conversion of surplus electrical energy into hydrogen energy and cross-period storage. Between 2:00 PM and 8:00 PM, as photovoltaic output decreases and thermal load increases, the hydrogen fuel cell, in conjunction with the cogeneration unit for auxiliary regulation, experiences increased hydrogen consumption. Limited by the upper limit of real-time hydrogen production capacity, the current output of the proton exchange membrane electrolyzer alone cannot directly meet the current hydrogen demand. At this time, the hydrogen storage tank promptly switches its operating mode from hydrogen filling mode to hydrogen releasing mode, releasing the stored green hydrogen energy in a timely manner, working in conjunction with the real-time hydrogen production of the electrolyzer to fill the hydrogen supply gap during peak hours. In summary, through the real-time multi-energy conversion of the electrolyzer and the cross-period scheduling of the hydrogen storage tank, the entire hydrogen energy system achieves dynamic balance. This not only improves the absorption rate of intermittent renewable energy, but also effectively smooths the fluctuations of multi-energy loads through the time-shifting effect of hydrogen energy, greatly enhancing the scheduling flexibility and operational economy of the integrated energy system under complex conditions.

[0127] The above descriptions are merely specific embodiments of the present invention, and are intended only to explain the principles of the invention, and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without any creative effort, and these embodiments will all fall within the scope of protection of the present invention.

Claims

1. A method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning, characterized in that, include: S1. First, construct mathematical models of each unit in the integrated energy system of electricity-heat-hydrogen, and mathematically describe the optimization scheduling problem of the integrated energy system with the economic efficiency of system operation as the objective. S2. Next, the optimization scheduling problem is formulated as a Markov decision process, and the state space, action space and reward function of the system are constructed. S3. Then, the SAC algorithm is used to make economic scheduling decisions, and a stable optimized scheduling strategy model is obtained after training. S4. Real-time acquisition of the current system status, input of the trained optimized scheduling strategy model, and output of the scheduling instructions for each device at each time step.

2. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 1, characterized in that: The optimization scheduling problem of the aforementioned integrated electric-thermal-hydrogen energy system is a sequential decision problem. Its scheduling objective is to determine the output power of each unit at each time step while minimizing the daily operating cost of the system.

3. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 1, characterized in that: The integrated electric-thermal-hydrogen energy system includes: The energy supply layer integrates the upstream power grid, wind power generation, and photovoltaic power generation. The energy conversion layer integrates a combined heat and power unit, a proton exchange membrane electrolyzer, a fuel cell, an electric boiler, and a gas-fired boiler for coordinated energy conversion. The energy storage layer consists of batteries and hydrogen storage tanks, which meet the supply and demand balance through storage and release; The energy demand layer includes the electrical load and heat load on the user side.

4. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 3, characterized in that: The model building for each device includes: Cogeneration unit model: ； ； In the formula, P CHP,t H CHP,t and G CHP,t Represent the electrical power, thermal power, and input power of the combined heat and power unit at time t, respectively; η CHP_e and η CHP_h These represent the power generation efficiency and heating efficiency of the combined heat and power unit, respectively. Gas boiler model: ； In the formula, η GB G represents the conversion efficiency of a gas-fired boiler. GB,t H represents the input gas power of the gas-fired boiler. GB,t This represents the output thermal power of the gas-fired boiler; Electric boiler model: ； In the formula, η EB P represents the conversion efficiency of the electric boiler. EB,t H represents the input electrical power of the electric boiler. EB,t This represents the output thermal power of the electric boiler; Proton exchange membrane electrolyzer model: ； In the formula, η PEME It is the conversion efficiency of the proton exchange membrane electrolyzer; P PEME,t M represents the input electrical power of the proton exchange membrane electrolyzer; PEME,t It is the hydrogen production mass flow rate; LHV is the lower heating value of hydrogen. Fuel cell model: ； In the formula, η HFC_e and η HFC_h These represent the electrical efficiency and thermal efficiency of the fuel cell, respectively; P HFC,t and H HFC,t These represent the output electrical power and thermal power of the fuel cell, respectively. Energy storage device model: ； ； In the formula, SOC BESS,t and SOC HST,t These represent the SOC state of the battery and the hydrogen storage tank, respectively; P cha,t , P dis,t , η cha_e,t and η dis_e,t Represents the charging / discharging power and efficiency of the battery; M cha,t and M dis,t E represents the mass of hydrogen gas supplied to / discharged from the hydrogen storage tank. BESS and E HST These represent the rated capacity of the battery and the hydrogen storage tank, respectively.

5. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 3, characterized in that: The aforementioned integrated electric-thermal-hydrogen energy system is also subject to the following system constraints: Electric power balance constraints: The electrical power generated by photovoltaic and wind power in the system, the electrical power generated / consumed by each unit, and the energy exchange power of the energy storage device must be balanced with the system's electrical load demand. The energy balance equation is as follows: ； In the formula, P Load,t P represents the electrical load demand of the system; grid,t This refers to the electricity purchased by the system from the upstream power grid; P PV,t and P WT,t Represents photovoltaic power output and wind power output; P HFC,t P represents the electrical power output of the fuel cell; CHP,t P represents the electrical power output of a combined heat and power (CHP) unit. PEME,t P represents the electrical power consumed by the proton exchange membrane electrolyzer; EB,t P represents the electrical power consumed by the electric boiler. dis,t P represents the electrical power released by the battery. cha,t Represents the electrical power stored in the battery; Thermal power balance constraint: ； In the formula, H Load,t Represents the system's heat load requirements; H HFC,t H represents the thermal power output of the fuel cell; CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. EB,t This represents the thermal power output of the electric boiler; Power constraints: ； ； ； ； ； ； ； In the formula, P grid,max P PEME,max P HFC,max G GB,max G CHP,max P EB,max P represents the power purchased from the upper-level power grid and the upper limit of the power output of each generating unit, respectively. grid,min P PEME,min P HFC,min G GB,min G CHP,min and P EB,min These represent the power purchase limits from the upstream power grid and the lower power limits of each generating unit; SOC n,max and SOC n,min The upper and lower limits of SOC for the two types of energy storage devices; Climbing constraints: ； ； ； ； ； In the formula, ΔP PEME,up ΔP HFC,up ΔG GB,up ΔG CHP,up ΔP EB,up ΔP represents the maximum ramp power of each unit's equipment. PEME,down ΔP HFC,down ΔG GB,down ΔG CHP,down and ΔP EB,down These represent the lower limit of the ramp power of each unit's equipment.

6. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 3, characterized in that: The objective function for the optimal scheduling problem of the integrated electric-thermal-hydrogen energy system is as follows: ； In the formula, F represents the total operating cost of the system, and C... om,t For the operation and maintenance costs of each unit, C grid,t The cost of purchasing electricity from the upper-level power grid.

7. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 1, characterized in that: The architecture of the SAC algorithm includes one Actor network, two Critic networks, and two corresponding target Critic networks. The Actor network samples actions based on the current state and interacts with the environment to obtain immediate rewards and the state at the next moment. The two Critic networks are responsible for evaluating the value of state-action pairs, and the two target Critic networks generate stable value estimates through soft updates.

8. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 7, characterized in that: The SAC algorithm updates the strategy as follows: The optimal policy function of the SAC algorithm is expressed as: ； In the formula, π* represents the strategy function that maximizes the expected cumulative return; s t It is the environmental state at time t; a t It represents the action at time t; α is the temperature coefficient, used to balance the trade-off between expected reward and entropy; It is the policy entropy term; The modified Bellman equation is expressed as follows: ； ； In the formula, Q(s) t ,a t V(s) represents the state-action value function; t ) represents the soft-state value function; p represents the state transition probability; γ represents the discount factor, taking values ​​in the range [0,1], indicating the importance of future rewards; r(s) t ,a t ) represents the instantaneous reward value at time step t; Minimizing the difference between two distributions by minimizing the KL divergence is expressed as follows: ； In the formula, D KL () represents the KL divergence; Π is the policy set; Z πold (s t The distribution is represented by the standard Gaussian distribution and is used to analyze the distribution. Perform normalization; Q-value networks achieve adaptive parameter updates by minimizing residuals, as described below: ； ； ； In the formula, J Q (θ) represents the loss function of the Q-network, and θ and Let represent the parameters of the value network and its corresponding target network, respectively. During training, these parameters are assigned using the following update method: ； In the formula, k represents the update parameter; The policy network is updated as follows: ； In the formula, ϕ is the coefficient of the policy network.

9. The method for optimal scheduling of an integrated electric-thermal-hydrogen energy system based on deep reinforcement learning according to claim 1, characterized in that: The state space is defined as: ； In the formula, P PV,t and P WT,t Represents photovoltaic power output and wind power output; C grid,t It is time-of-use electricity pricing; H Load,t P represents the system's heat load demand; Load,t Represents the electrical load demand of the system; SOC BESS,t and SOC HST,t These represent the SOC state of the battery and hydrogen storage tank, respectively. Action space is defined as: ； In the formula, P PEME,t P represents the electrical power consumed by the proton exchange membrane electrolyzer; EB,t P represents the electrical power consumed by the electric boiler. grid,t This refers to the electricity purchased by the system from the upstream power grid; H CHP,t H represents the thermal power output of a combined heat and power unit. GB,t H represents the thermal power output of a gas-fired boiler. HFC,t P represents the thermal power output of the fuel cell; BESS,t P represents the charge / discharge power of the BESS at time t; HST,t This represents the charge / discharge power of HST at time t; The reward function is: ； In the formula, C om,t F1 is the operating cost of each unit in the system; β1 is the penalty cost for thermal imbalance; F2 is the penalty cost for electrical imbalance; β2 is the penalty coefficient for electrical imbalance.

Images