Scheduling method for electricity-hydrogen-heat comprehensive energy system

Abstract

The invention provides an electricity-hydrogen-heat comprehensive energy system scheduling method, and belongs to the technical field of comprehensive energy system intelligent scheduling. In order to solve the problems that the response speed cannot meet the scheduling requirement, the anti-interference capability is weak, a dynamic cooperation mechanism is lacked and the quantification precision is low in the scheduling method for the electric-hydrogen-thermal coupling comprehensive energy system, the method comprises the following steps: S1, establishing an electric-hydrogen-thermal coupling power distribution network real-time scheduling model; s2, constructing a constrained Markov decision process aiming at the problem characteristics of the electro-hydrogen thermal coupling power distribution network; s3, building a security reinforcement learning algorithm to train the constrained Markov decision process, and obtaining a strategy neural network; and S4, outputting the optimal strategy for real-time scheduling of the power distribution network by using the strategy neural network to realize real-time scheduling of the power distribution network.

Description

Technical Field

[0001] This invention relates to a scheduling method for an integrated energy system combining electricity, hydrogen, and heat, belonging to the field of intelligent scheduling technology for integrated energy systems. Background Technology

[0002] Driven by the global energy transition and the "dual carbon" goal, the power system is undergoing profound changes. The large-scale integration of distributed energy resources, especially high-proportion renewable energy sources such as wind and solar power, has become a key characteristic of the modern energy system. However, the inherent intermittency, volatility, and uncertainty of renewable energy pose unprecedented challenges to the real-time balancing capabilities of the power system. Traditional dispatching methods, relying on deterministic prediction and offline optimization, are ill-suited to rapid power fluctuations occurring on a minute or even second basis, resulting in insufficient system resilience and increased operational risks.

[0003] Meanwhile, urban energy supply models are evolving from traditional single-energy supply to integrated energy systems with deep coupling and complementarity of multiple energy flows such as electricity, hydrogen, and heat. While this multi-energy coupling improves energy utilization efficiency and flexibility, it also significantly increases the complexity of system modeling and scheduling. The dynamic characteristics of different energy carriers vary greatly, and their conversion, transmission, and storage links are mutually coupled, making the scheduling problem a complex optimization problem that requires simultaneously satisfying multidimensional physical constraints and economic objectives, and is characterized by high dimensionality, nonlinearity, and strong coupling.

[0004] Hydrogen energy, as a clean and efficient secondary energy carrier, plays an increasingly important role in integrated energy systems, especially in long-term energy storage, transportation decarbonization, and industrial carbon reduction. However, the introduction of hydrogen energy also brings unique and stringent safety challenges: hydrogen has physicochemical properties such as low density, wide explosion limits, and easy leakage. This necessitates that systems achieve both high economic efficiency and safety requirements.

[0005] Zhang Haotian et al. from Shanghai University of Electric Power proposed a multi-timescale optimization scheduling strategy for hydrogen-electric hybrid energy storage systems. In the day-ahead scheduling phase, the strategy aims to minimize operating costs and voltage deviation rate, providing a reference for intraday optimization scheduling. In the intraday rolling optimization scheduling phase, the strategy focuses on economic efficiency, fine-tuning equipment output and correcting power fluctuations caused by prediction errors. Xie Yingbiao et al. from Wuhan University of Technology proposed a load recovery strategy for an integrated electric-gas energy system considering mobile hydrogen energy storage. By introducing second-order cone constraints, the proposed model is transformed into a mixed-integer second-order cone model, and a solver is used to solve it, significantly reducing the losses of the integrated electric-gas energy system under typhoon disasters. Zhang Zishang et al. from Three Gorges University used the Adaptive Step Alternating Directional Multiplier Method (ADMM) to solve the distributed optimization problem of electric-carbon-hydrogen coupled multi-energy microgrid-distribution network, improving the system's economic efficiency and its ability to absorb renewable energy.

[0006] While current research on scheduling methods for integrated energy systems coupled with hydrogen and heat has made some progress, it still faces the following key challenges and significant shortcomings: First, mathematical programming methods suffer from excessively long solution times due to computational dimensionality explosion when dealing with highly volatile renewable energy and multi-energy coupled systems. Their hourly response speed cannot meet minute-level scheduling requirements, their anti-interference capability is weak, and they struggle to smooth out instantaneous fluctuations in source and load. Second, the electric-hydrogen-thermal system contains a massive number of heterogeneous safety constraints. Mainstream methods transform these constraints into manually set penalty coefficients, resulting in strong reliance on experience and a lack of dynamic coordination mechanisms. They cannot accurately quantify risks such as voltage exceeding limits and hydrogen safety thresholds, easily leading to agents exceeding safety boundaries for economic reasons or operating excessively conservatively. Summary of the Invention

[0007] This invention addresses the problems of insufficient response speed, weak anti-interference capability, lack of dynamic coordination mechanism, and low quantification accuracy in scheduling methods for integrated energy systems coupled with hydrogen and electricity. Therefore, it proposes a scheduling method for integrated energy systems coupled with hydrogen and electricity.

[0008] The technical solution adopted by the present invention to solve the above problems is as follows: The present invention includes the following steps: Step 1: Based on time-of-use pricing, the location of different types of distributed fuel cells, and the load demand of fuel cells, establish a real-time scheduling model for the electricity-hydrogen-thermal coupled power distribution network; Step 2: Based on the established real-time scheduling model of the electric-hydrogen-thermal coupled distribution network, the electric-hydrogen-thermal coupled distribution network scheduling system is modeled as a constrained Markov decision process; Step 3: Based on the constrained Markov decision process, establish a policy neural network and train the policy neural network using a secure reinforcement learning method; Step 4: Calculate the real-time optimal scheduling strategy for the electric-hydrogen-thermal coupled distribution network based on the trained policy neural network.

[0009] Furthermore, the real-time scheduling model of the electric-hydrogen-thermal coupled distribution network in step 1 includes a power flow unit, a fuel cell unit, an alkaline electrolyzer unit, a hydrogen storage tank constraint unit, a wind turbine unit, an electric boiler unit, and an inverter unit, each of which meets preset physical constraints and safety thresholds.

[0010] Furthermore, step 2 specifically includes: Define the state space, action space, state transition probabilities, and reward function of a constrained Markov decision process; The state space includes the current grid electricity price, hydrogen storage tank status, the difference between the hydrogen storage tank capacity and the minimum hydrogen storage target, wind farm wind resources, active and reactive loads of each node, and heat load of each region. The operating space includes the current density and reactive power output of the fuel cell, the current density of the electrolyzer, the reactive power output of the wind turbine transformer, and the heat gap missing after the fuel cell provides heating through the electric boiler unit. The state transition probability is the probability calculated by taking action a after the electric-hydrogen-thermal coupled distribution network scheduling system takes action a from the state at time t in the real-time scheduling model of the electric-hydrogen-thermal coupled distribution network, based on the data collected from each unit. Constrained Markov decision processes optimize strategies Maximize the cumulative rewards while limiting the cumulative number of constraint violations to a predetermined range; The expression for the optimization problem of a constrained Markov decision process is: (1); (2); In formulas (1) and (2), For the reward function, For constraint functions, The cost threshold for violating the constraint. As a discount factor, Expected reward for this strategy

[0011] The state expression of the electro-hydrogen-thermal coupled distribution network dispatching system at time t is: (3); In formula (3), Indicates the current time. express The electricity price at any given time express The current state of the hydrogen storage tank at any given moment. express The difference between the current hydrogen storage tank capacity and the minimum hydrogen storage target at any given time. and express The wind resources of the two wind farms at all times. and Indicates the first Each node Active and reactive loads at any given time. Indicates the area exist The heat load at any time, This represents the active load of the 33rd node at time t. This represents the reactive load of the first node at time t. This represents the reactive load of the 33rd node at time t. Indicates that region 1 is in The heat load at any time, Indicates region 4 in The heat load at any given time; The action expression of the electro-hydrogen-thermal coupled distribution network dispatching system at time t is: (4); In formula (4), and Indicates fuel cell exist Current density and reactive power output at any given time. Indicates that the electrolytic cell is in Current density at time t, and This indicates that the two wind turbine transformers are in Effortless efforts at every moment.

[0012] Furthermore, the reward function is used to decouple the system reward from the system constraint of the electric-hydrogen-thermal coupled distribution network dispatch system. The constraint function is solved through the power flow unit, alkaline electrolyzer unit, hydrogen storage tank constraint unit and inverter unit, and the system constraint is obtained through the constraint function. The expression for the reward function is: (5); In formula (5), The system's cost of purchasing electricity from the grid. t represents the total number of time steps, and t represents the current time step. The expression for the constraint function is: (6) (7) (8); (9); (10); In formulas (6)-(10), This indicates a violation of node voltage constraints. The number of nodes and These are the upper and lower limits of the voltage, respectively; This indicates a violation of HTO constraints in the electrolytic cell. This indicates the limit on the hydrogen content in oxygen; This indicates the limitations and constraints of the hydrogen storage tank. and These are the upper and lower limits for safe hydrogen storage in hydrogen storage tanks, respectively. This indicates the output constraints of the inverter, including inverters for all wind turbines and fuel cells. The number of inverters. This indicates the maximum apparent power of the inverter. express The active power output of the device at any given time. express The reactive power output of this device at any given time; , , , These represent the violation coefficient factors for node voltage violation constraints, HTO violation constraints in the electrolyzer, hydrogen storage tank limitation constraints, and inverter output constraints, respectively.

[0013] Furthermore, the secure reinforcement learning methods in step 3 include the maximum entropy policy update method, the constraint policy based on primal duality, and the constraint weight update based on cross-entropy. The maximum entropy policy update method employs the SAC algorithm and introduces the maximum entropy pair to optimize the policy. Update; The constraint weight update based on cross-entropy includes: when the cumulative constraint violation amount exceeds the warning threshold, standardizing the cumulative constraint violation amounts of different safety constraints and mapping them uniformly to the [0,1] interval to form a comparable risk index. ; within the time window Within, comparable risk indicators will be available. Convert to probability distribution And calculate the information entropy of the penalty index. Information entropy based on penalty indicators Establish constraint weights ; The formula for calculating maximum entropy is: (11); In formula (11), For strategy In system state The entropy below, For the expected value, In system state Take action below The logarithmic probability, Representative strategy, Representing state Execute action a at that time; Optimal Strategy The update formula is: (12); In formula (12), , Discount factor For the reward function; Comparable risk indicators The calculation formula is: (13); In formula (13), Represented as constraints The original violation value, This indicates the maximum value of constraint violation within the sliding time window. It is a local minimum; Information entropy The calculation formula is: (14); (15); In formulas (14) and (15), Information entropy, representing the penalty indicator, is used to measure the dispersion of its risk distribution. This indicates a relatively high risk, requiring a stronger weighting. This indicates that the risk is relatively diversified, thus mitigating the punishment. Constraint weights The calculation formula is: (16); In formula (16), This is the lower limit of the weight, used to prevent the corresponding constraint from being ignored due to the weight being too low.

[0014] Furthermore, based on the constraint strategy of primal duality, a Lagrange relaxation framework is adopted, which introduces Lagrange multipliers. , The problem of finding the optimal strategy in a constrained Markov decision process is transformed into an unconstrained saddle point optimization problem. Lagrange multipliers Combined with the SAC algorithm, a SAC algorithm based on primal-dual is constructed. A scheduling problem is established based on the primal-dual SAC algorithm, and in each iteration, a fixed... Then, based on the original policy type adopted by the system, the corresponding policy gradient update is performed, and the dual variable is updated simultaneously. To maximize the cumulative expected reward while minimizing the result from the Lagrange multipliers Weighted cumulative constraint violation; The established expression for the scheduling problem is: (17); In formula (17), Values ​​that violate system constraints; The expression for policy update is: (18); In formula (18), This indicates the policy gradient update step size. For strategy parameters, The strategy parameters are for the k-th iteration. The value function of the strategy, It is a Lagrange function; Dual variables The update formula is: (19); In formula (19), This represents the update compensation for the dual field. On the dual space , Let be the threshold of the i-th constraint. For strategy The cumulative violation of the i-th constraint. The current value of the dual variable. This is the updated value of the dual variable; The expression for minimizing the cumulative constraint violations is: (20); (twenty one); In formulas (20) and (21), To take action at time t Post-system status The reward To take action at time t Post-system status The resulting constraint costs, To constrain the threshold, Discount factor

[0015] Furthermore, in step 3, the policy neural network adopts the Actor-Critic framework, which includes a decision network and two sets of value networks. The decision network is used for action decision evaluation, and the value network is used for state-action value evaluation. The input to the decision network is the current state of the system, and the output is the probability distribution parameters of the actions in the current state. The probability distribution parameters include the mean. and standard deviation During training, reparameterization techniques are used, combined with mean values. and standard deviation The final action of the calculation system in its current state ; The two sets of value networks include a reward value network and a constraint value network. The reward value network is used to evaluate the system in a given state. Next action Subsequently, the expected cumulative discount reward is obtained; the constrained value network is used to evaluate the system in state. Next action Subsequently, the expected cumulative discount cost, reward value network and constraint value network both include a dual Q-learning network structure, with each Q-learning network connected to the corresponding target network.

[0016] Furthermore, the training of the value network specifically includes: Step 3.1: Calculate the target Q value using a soft update mechanism through the target network corresponding to the Q-learning network structure; Step 3.2: Pair the tuple { The empirical replay buffer D is stored at each time step, where, The reward at time step t. The cost at time step t, and For system status, Perform actions for the system; Step 3.3 Randomly sample small batches of data from the empirical replay buffer D, and update the optimal strategy. By combining the Bellman equation to obtain the current state-action value and the expected values ​​of all possible next states and corresponding actions, and minimizing the mean squared error of the obtained current state-action value and the expected values ​​of all possible next states and corresponding actions, the action-value function in time step t is obtained. Approximate values ​​are used to alternately update the reward value network parameters, Lagrange multipliers, and constraint coefficient factors; Step 3.4: Repeat step 3.3 to obtain the cost value function at time step t. Approximate values ​​are obtained by alternately updating the constrained value network parameters, Lagrange multipliers, and constraint coefficient factors.

[0017] The beneficial effects of this invention are: 1. This invention integrates a secure reinforcement learning framework with a dynamic evaluation mechanism based on constrained entropy weights, effectively addressing the bottlenecks of traditional mathematical programming methods, such as computational complexity and slow response in highly volatile and multi-energy coupled scenarios. Through online autonomous decision-making capabilities, the scheduling decision time is compressed from hours to seconds, significantly improving the system's real-time response capability and anti-interference performance in the face of renewable energy fluctuations, meeting the rapid scheduling requirements at the minute or even second level.

[0018] 2. This invention innovatively introduces the constraint entropy weight method to dynamically quantify and assess the risk distribution and importance differences of multi-dimensional safety constraints such as electricity, hydrogen, and heat, replacing the simplistic traditional method that relies on manual experience to set fixed penalty coefficients. This method can autonomously perceive system operational risks, dynamically adjust constraint weights, accurately balance the contradiction between safety and economy, and significantly reduce the risk of intelligent agents actively violating key safety constraints in pursuit of economic efficiency, especially ensuring the high-risk safety requirements of hydrogen energy systems.

[0019] 3. This invention, through constrained Markov decision process modeling and combining maximum entropy strategy optimization with a primal dual optimization framework, achieves global optimization of the scheduling strategy while ensuring that all key safety constraints (such as node voltage, hydrogen-in-oxygen safety threshold HTO, hydrogen storage tank capacity, inverter capacity, etc.) are strictly met. Compared to existing reinforcement learning methods, it significantly reduces system operating costs at the same safety level, solving the problems of traditional methods sacrificing economy due to conservative strategies or the high constraint violation rate caused by safety mechanism defects in existing reinforcement learning. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a scheduling method for an integrated electric-hydrogen-thermal energy system. Figure 2 This is a schematic diagram showing the simulation results of wind power resources and electrical and thermal load on a typical day; Figure 3 This is a schematic diagram illustrating the simulation results of a typical daily electricity dispatch strategy; Figure 4 This is a schematic diagram of the simulation results for a typical daily thermal energy dispatch strategy; Figure 5 This is a schematic diagram of the simulation results for the voltage of a typical daily scheduling node; Figure 6 This is a schematic diagram of typical daily HTO changes. Detailed Implementation

[0021] like Figure 1 As shown, the steps of the integrated energy system scheduling method for electricity, hydrogen, and heat described in this embodiment include: S1: Establish a real-time scheduling model for an electric-hydrogen-thermal coupled power distribution network; In this embodiment, the integrated energy system of electricity, hydrogen, and heat needs to consider time-of-use electricity pricing, the location of each distributed fuel cell and the load demand in its vicinity, and rationally adjust the output of the electrolyzer, fuel cell, and electric boiler to ensure the safety of the fuel cell while balancing the conversion relationship between electricity, heat, and hydrogen to meet the energy demand of the power grid. The real-time scheduling model of the electricity-hydrogen-heat coupled power distribution network includes a power flow unit, a fuel cell unit, an alkaline electrolyzer unit, a hydrogen storage tank constraint unit, a wind turbine unit, an electric boiler unit, and an inverter unit. Each unit meets preset physical constraints and safety thresholds, as detailed below: (1) Power Flow of the Power Grid A multi-node distribution network, nodes The active power balance constraints and reactive power balance constraints at the location are: (1); (2); In formulas (1) and (2), and These are active power injection and reactive power injection at the nodes, respectively. and They represent nodes respectively Transmit to node The active power flow and reactive power flow; and Connect respectively Nodes and The resistance and reactance of the node circuit, yes Time Node To the node The square of the current, where M is the set of branches connecting all nodes; node The voltage square difference between them is defined as: (3); (4); In formulas (3) and (4), yes Time Node To the node The square of the current, Let be the voltage at node j at time t. Let be the voltage at node i at time t; node The voltage constraint at the point is: (5); In formula (5), and These are the minimum and maximum allowable voltage amplitudes, respectively.

[0022] (2) Fuel cell A proton exchange membrane (PEM) fuel cell is a complex electrochemical and thermodynamic system. The voltage of a single PEM fuel cell can be expressed as: (6); In formula (6), This represents the Nernst electromotive force of a single PEM fuel cell under standard conditions. , , These represent the activation polarization voltage, ohmic polarization voltage, and concentration polarization voltage, respectively. This represents the voltage of a single PEM fuel cell.

[0023] The combined output voltage of a series stack of fuel cells can be expressed as: (7); In formula (7), for The combined output voltage of a series stack of fuel cells; The current in a stack of fuel cells connected in parallel can be expressed as: (8); In formula (8), The rate at which hydrogen is fed into the fuel cell stack. Here is the molar mass of hydrogen. It is Faraday's constant. for The current of a fuel cell stack connected in parallel; During the operation of a fuel cell, voltage losses cause some energy to be released as heat instead of electrical energy. The mathematical expressions for the electrical power and thermal power of a fuel cell are: (9); (10); In formulas (9) and (10), For the electrical power of the fuel cell, This represents the thermal power of the fuel cell.

[0024] (3) Alkaline electrolytic cell The mass of hydrogen produced during electrolysis is directly proportional to the amount of charge passing through the electrolyzer: (11); In formula (11), It represents the amount of charge; is the molar mass of hydrogen gas; The number of electrons transferred; It is Faraday's constant. The charge in the electrolytic cell; In actual electrolyzer production, due to factors such as gas permeation and side reactions, the actual hydrogen production is generally lower than the theoretical value. (12); In formula (12), Theoretical hydrogen production For actual hydrogen production, The current efficiency can be expressed as: (13); In formula (13), and The constant is used for fitting; the production unit is converted from g / s to kg / h, and the calculation of the hydrogen production capacity of the electrolyzer can be expressed as: (14); The active power on the DC side of the electrolyzer can be calculated using the following formula: (15); (16); In formulas (15) and (16), Theoretically reversible voltage, For ohmic overvoltage, To activate overvoltage. The rate of change of HTO concentration in the electrolyzer can be calculated using the following formula: (17); In formula (17), This is the volume fraction of impurities in the hydrogen. The volume fraction of impurities in the output hydrogen is determined by the current HTO value and electrolysis current, and the calculation formula is as follows: (18); In formula (18), Let be the ideal gas constant. For separator temperature, For separator pressure, For separator volume, and These are the current efficiency and the Faraday constant, respectively.

[0025] Discretize the continuous-time equation (time step) ): (19); Substitute : (20); The explosion limit of hydrogen in oxygen is 4%, and in industry, the safety threshold is usually set at 2% to allow for a margin. Therefore, the HTO constraint for the electrolyzer is set as follows: (twenty one); If HTO exceeds the allowable value, the system will shut down and be immediately purified with nitrogen.

[0026] (4) Constraints of hydrogen storage tanks The hydrogen storage tank is the energy storage device in the system, and its energy storage and release characteristics are as follows: (twenty two); In formula (20), For the first The remaining hydrogen mass in the hydrogen storage tank at the end of the cycle; For energy storage systems in The remaining hydrogen storage mass at the end of the period; and These represent the hydrogen production and consumption rates of the electrolyzer and fuel cell, respectively. The unit period is the length of time.

[0027] Hydrogen storage tanks should be designed with a minimum hydrogen storage capacity to prevent negative pressure or air backflow, while also having a maximum hydrogen storage capacity to avoid overpressure. Therefore, the real-time capacity must be limited to a safe range. (twenty three); In formula (23), and These are the minimum and maximum hydrogen storage capacities of the hydrogen storage tank, respectively. Meanwhile, to ensure the system can operate in the long term, the amount of hydrogen stored at the end of each day is... It must exceed the daily minimum hydrogen storage target. : (twenty four); (5) Fan The power output characteristics of a wind turbine can be represented by a piecewise linear function: (25); In formula (25), This refers to the power generation capacity of the wind turbine. This refers to the rated power of the wind turbine. The inlet wind speed of the wind turbine; Rated wind speed; To cut the wind speed; The wind speed is at the height of the wind turbine hub.

[0028] (6) Electric boiler An electric boiler is an energy coupling device that converts electrical energy into heat energy. Its model is related to the boiler efficiency and the converter efficiency. (26); In formula (26), For electric boilers The heat generated at all times; For the efficiency of the electric boiler converter; The electro-thermal conversion efficiency of electric boilers; for The power absorbed by the electric boiler from the busbar at any given time; The input power must meet the capacity limit of the electric boiler: (27); The electric boiler is used to supplement the heat gap after the fuel cell provides heating. It only needs to be controlled according to rules and does not require reinforcement learning algorithms to control the node voltage. The reactive power provided by the active power / inverter of each node device is jointly controlled.

[0029] (7) Inverter By adjusting the control strategies of grid-connected inverters for distributed energy sources such as fuel cells, wind, and solar power, the reactive power injected into the grid can be flexibly, quickly, and effectively controlled. This achieves the goal of regulating the voltage level at the grid connection point and supporting voltage stability in neighboring areas. The regulation must meet the following requirements: (28); (29); In formulas (28) and (29), and They represent the first The active and reactive power generated by each inverter and Indicates the upper and lower limits of the inverter's reactive power. This indicates the apparent power of the inverter.

[0030] S2: Construct a constrained Markov decision process for the characteristics of the electric-hydrogen-thermal coupled power distribution network problem; To address the real-time and safety requirements of the electro-hydrogen-thermal coupling system, this implementation model its scheduling problem as a constrained Markov decision process (CMDP). The goal of CMDP is to find an optimal strategy. To maximize the cumulative reward while limiting cumulative constraint violations to a predetermined range, the standard optimization problem can be formulated as: (30); (31); In formulas (30) and (31), For the reward function, For constraint functions, This is a threshold for the cost of violating constraints, requiring that the cumulative cost of the policy does not exceed this value. As a discount factor, Expected reward for this strategy

[0031] The constrained Markov decision process is constructed as follows: State space: States are used by an agent to take actions after observing the environment. They need to fully describe the current environment so that the agent can make accurate decisions. The system state at time t is defined as follows: (32); In formula (32), Indicates the current time. express The electricity price at any given time express The current state of the hydrogen storage tank at any given moment. express The difference between the current hydrogen storage tank capacity and the minimum hydrogen storage target at any given time. and express The wind resources of the two wind farms at all times. and Indicates the first Each node Active and reactive loads at any given time. Indicates the area exist The heat load at any time, This represents the active load of the 33rd node at time t. This represents the reactive load of the first node at time t. This represents the reactive load of the 33rd node at time t. Indicates that region 1 is in The heat load at any time, Indicates region 4 in Heat load at any given time.

[0032] Action space: An action is the action performed by an agent after observing a state. By controlling various devices in the electro-hydrogen-thermal coupling system, the agent maximizes the reward obtained. The action at a given moment is defined as: (33); In formula (33), and Indicates fuel cell exist Current density and reactive power output at any given time. Indicates that the electrolytic cell is in Current density at time t, and This indicates that the two wind turbine transformers are in Effortless efforts at every moment.

[0033] State transition probability: The state transition probability is the probability that the agent is in a given state. Use action a For the coordination and optimization problem of distribution networks, resource quantity, node load, etc. are not affected by the decision, so the state transition probability directly corresponds to the collected data.

[0034] Reward function: In constrained Markov decision-making, the system reward is decoupled from the system constraint, which reduces the negative impact of the constraint function on the convergence of the Critic network and improves the training efficiency of the algorithm. This invention utilizes constrained Markov decision process modeling, combined with maximum entropy policy optimization and a primal dual optimization framework. It achieves global optimization of the scheduling strategy while ensuring strict satisfaction of all key safety constraints (such as node voltage, the hydrogen-in-oxygen safety threshold (HTO), hydrogen storage tank capacity, and inverter capacity). Compared to existing reinforcement learning methods, it significantly reduces system operating costs at the same safety level, resolving the issues of traditional methods sacrificing economy due to conservative strategies or the high constraint violation rate caused by safety mechanism defects in existing reinforcement learning.

[0035] In this implementation, the reward function is defined as: (34); In formula (34), The system's cost of purchasing electricity from the grid. t represents the total number of time steps, and t represents the current time step. The constraint function is defined as: (35) (36) (37); (38); (39); In formulas (35)-(39), This indicates a violation of node voltage constraints. The number of nodes and These are the upper and lower limits of the voltage, respectively; This indicates a violation of HTO constraints in the electrolytic cell. This indicates the limit on the hydrogen content in oxygen; This indicates the limitations and constraints of the hydrogen storage tank. and These are the upper and lower limits for safe hydrogen storage in hydrogen storage tanks, respectively. This indicates the output constraints of the inverter, including inverters for all wind turbines and fuel cells. The number of inverters. This indicates the maximum apparent power of the inverter. express The active power output of the device at any given time. express The reactive power output of this device at any given time; , , , These represent the violation coefficient factors for node voltage violation constraints, HTO violation constraints in the electrolyzer, hydrogen storage tank limitation constraints, and inverter output constraints, respectively.

[0036] S3: Build a secure reinforcement learning algorithm to train a constrained Markov decision process and obtain a policy neural network; S301: Establish a maximum entropy strategy update method; In terms of reinforcement learning updates, this implementation establishes an Actor-critic framework. The Actor-critic framework has wide applications in reinforcement learning; algorithms such as DDPG, PPO, and A3C all employ it. However, the DDPG algorithm is highly sensitive to hyperparameters, making parameter tuning difficult, while PPO and A3C suffer from low sample efficiency and insufficient exploration. To address these issues, the SAC algorithm introduces the concept of maximum entropy. By combining entropy with the value function, it achieves better exploration and avoids falling into suboptimal conditions. The entropy is calculated as follows: (40); In formula (40), For strategy In system state The entropy below, For the expected value, In system state Take action below The logarithmic probability, Representative strategy, Representing state Execute action a at that time; Introducing maximum entropy enhances the randomness of the strategy during the optimization process, preventing excessive concentration on specific actions and thus improving the algorithm's exploration capability and stability. Optimal Strategy Updated to: (41); In formula (41), , Discount factor For reward function

[0037] S302: Construct a constraint strategy based on primal duality; The central idea of ​​Constrained Markov Decision Making (CDM) is to treat problem-related constraints as a penalty, using the sum of reward and constraint terms as a new reward function, while using constraint factors as weighting coefficients to achieve a balance between maximizing the agent's reward and satisfying the constraints. This implementation uses the Lagrange relaxation framework, introducing Lagrange multipliers... The constrained optimization problem is transformed into an unconstrained saddle point optimization problem, with the goal of maximizing the cumulative expected reward while minimizing the result of Lagrange multipliers. Weighted cumulative constraint violation: (42); (43); In formulas (42) and (43), To take action at time t Post-system status The reward To take action at time t Post-system status The resulting constraint costs, To constrain the threshold, Discount factor

[0038] By introducing the Lagrangian operator By combining with the classic SAC algorithm, a SAC algorithm based on primal-dual can be constructed to solve the scheduling problem: (44); In formula (44), Values ​​that violate system constraints; The primal-dual method, abbreviated as PD, is the standard method for solving the unconstrained minimax problem. In each iteration, it updates the primal policy sequentially. With dual variables First, fix it. Then, based on the policy type adopted by the agent, the corresponding policy gradient update is performed, and the update formula is as follows: (45); In formula (45), This indicates the policy gradient update step size. For strategy parameters, The strategy parameters are for the k-th iteration. The value function of the strategy, It is a Lagrange function; Then fixed Simultaneously perform dual updates Dual update function We employ dual gradient ascent, i.e.: (46); In formula (46), This represents the update compensation for the dual field. On the dual space , Let be the threshold of the i-th constraint. For strategy The cumulative violation of the i-th constraint. The current value of the dual variable. This is the updated value of the dual variable; S303: Construct constraint weight updates based on cross-entropy; The electro-hydrogen-thermal multi-energy coupling system has multiple safety constraints, and the degree and amount of violation of these constraints vary. To achieve a balance among these constraints and enable the system to more proactively manage high-reward risks, the core objective is to perceive the dynamic risk distribution within each constraint through a data-driven approach before the system approaches the safety boundary. This invention employs the entropy weight method to achieve precise matching between the penalty mechanism and the system's risk level. The specific steps are as follows: When the total number of constraint violations exceeds the warning threshold, the violation amounts of different safety constraints are first standardized: (47); In formula (47), Represented as constraints The original violation value, This indicates the maximum value of constraint violation within the sliding time window. This is a minimum value. This operation maps all dimensions of risk to the [0,1] interval, forming a comparable risk indicator. .

[0039] This invention innovatively introduces the constraint entropy weight method to dynamically quantify and assess the risk distribution and importance differences of multi-dimensional safety constraints such as electricity, hydrogen, and heat, replacing the simplistic traditional method that relies on manual experience to set fixed penalty coefficients. This method can autonomously perceive system operational risks, dynamically adjust constraint weights, and accurately balance the contradiction between safety and economy. It significantly reduces the risk of intelligent agents actively violating key safety constraints in pursuit of economic efficiency, especially ensuring the high-risk safety requirements of hydrogen energy systems.

[0040] In the time window Internally, the risk indicators are converted into probability distributions and information entropy is calculated: (48); (49); In formulas (48) and (49), Information entropy, representing the penalty indicator, is used to measure the dispersion of its risk distribution. This indicates a relatively high risk, requiring a stronger weighting. This indicates that the risk is relatively diversified, thus mitigating the punishment. Information entropy based on penalty index Establish constraint weights : (50); In formula (50), This is the lower bound of the weights, used to prevent the weights from being too low and thus ignoring the overall weight of the corresponding constraints. and They are inversely proportional.

[0041] By dynamically adapting the penalty mechanism through real-time perception of system operational risks, when a certain type of constraint is frequently exceeded, its risk distribution information entropy is calculated. A sharp drop triggers the weighting factor. The autonomous improvement leads to the reinforcement of this penalty; conversely, when the constraint stabilizes, the entropy value... Drive weight decay, free up optimization space, and improve model convergence speed.

[0042] S304: Establishment of the policy neural network; The algorithm of this invention employs a deep neural network and learns the optimal security policy through an alternating optimization training process. The algorithm's network structure includes a policy network and two sets of value networks. The policy network is called the Actor, and the value network is called the Critic, which are responsible for action decision-making and state-action value evaluation, respectively.

[0043] Policy Network A stochastic strategy is employed to encourage full exploration of the environment. Its input is the current state. The output is the probability distribution parameters of the actions in this state, i.e., the mean. and standard deviation To enable the gradient to propagate back through the sampling process, the algorithm employs a reparameterization trick. Specifically, the agent's final action... Generate in the following way: (51); In formula (51), The noise is sampled from a standard normal distribution. The tanh function smoothly restricts the output action value to a symmetrical interval of [-1, 1], which facilitates normalization. This represents the Hadama product.

[0044] To evaluate the strategy's performance in terms of economy (reward) and safety (cost) separately, the algorithm sets up two independent value networks: a reward value network and a constraint value network. Used to assess the state Next action Subsequently, the expected cumulative discount rewards that can be obtained constrain the value network. Used to assess the state Next action After that, the expected cumulative discount cost that can be obtained.

[0045] To mitigate the problem of overestimation of Q-values ​​and improve training stability, each value network employs a dual-network structure, known as Double Q-learning. This involves training two Q-networks simultaneously and taking the smaller of the two when calculating the target Q-value. Furthermore, each Q-network is paired with a corresponding target network, whose parameters are slowly updated to track the main network, further enhancing the stability of the training process.

[0046] S305: Training of the policy neural network; Policy Network: To enable gradients to backpropagate through the sampling process, the algorithm employs a reparameterization technique. During training, this technique is combined with the mean. and standard deviation The final action of the calculation system in its current state .

[0047] Reward value network: In the Lagrange function, Indicates the strategy in state s The product of the expected discount reward after taking action a and the corresponding entropy multiplier will be the tuple { The network parameters, Lagrange multipliers, and constraint coefficients are alternately updated in a mini-batch of data randomly sampled from D at each time step for training.

[0048] This implementation uses two sets of neural networks to approximate the action-value function at time step t. Using the Bellman equation, we can... To approximate the current state-action value with the expectation of all possible next states and corresponding actions, that is: (52); (53); In formulas (52) and (53), For the target network of the reward function, use soft updates for updates: (54); In formula (54), This represents the update rate. The goal of training is to minimize the mean squared error (MSE). (55); Constrained Value Network: The state-value function of the constraint function is designed in a similar way. This implementation uses two sets of networks to approximate the cost-value function. Similarly, the current cost value can be approximated as: (56); (57); In formulas (56) and (57), The target network is a constraint function that updates its parameters by minimizing the loss: (58); The constrained value network also adopts the same soft update method as the reward value network, alternately updating the constrained value network parameters, Lagrange multipliers, and constraint coefficient factors.

[0049] S4: Utilize a policy neural network to output the optimal real-time dispatch strategy for the distribution network, thereby achieving real-time dispatch of the distribution network.

[0050] In summary, this invention integrates a secure reinforcement learning framework with a dynamic evaluation mechanism based on constrained entropy weights, effectively addressing the bottlenecks of traditional mathematical programming methods, such as computational complexity and slow response in highly volatile and multi-energy coupled scenarios. Through online autonomous decision-making capabilities, the scheduling decision time is compressed from hours to seconds, significantly improving the system's real-time response capability and anti-interference performance in the face of renewable energy fluctuations, meeting the rapid scheduling requirements at the minute or even second level.

[0051] Example To verify the technical effect of the present invention, the following simulation verification was performed in this embodiment: The following experiment uses an improved model of the IEEE-33 node distribution network as the illustrative example. Based on the standard IEEE-33 node distribution network, node 1 of the system is connected to the upper-level grid, and nodes 10 and 18 are respectively connected to two wind farms with an installed capacity of 3MW; the electrolyzer is installed at node 2 with a maximum power of 5MW; nodes 3, 7, 14, and 30 are respectively connected to a combined heat and power unit, each of which includes a 1MW fuel cell and a 1MW electric boiler; on the heating network, the combined heat and power units are respectively connected to a centralized heat supply point in one area for providing regional heating.

[0052] The algorithm was trained using real wind power resources, electricity load, and heat load data from a certain location over two years. One year was used as the training set and the other year as the test set. All algorithms were trained for 60,000 episodes for evaluation. After training, 10 days of data were randomly selected from the test set for testing. NSGA-II was also selected as a traditional optimization method for comparison. The test results are shown in Table 1.

[0053] Table 1

[0054] The analysis in the table above shows that, compared to the NSGA-II method, the method of this invention reduces the decision-making time to 1 / 5000, demonstrating high practical value. Furthermore, compared to the SAC and Cliff-SAC methods, the method of this invention can better balance rewards and constraint violations, significantly improving economic efficiency when constraint violations meet the limit threshold.

[0055] The multi-stage scheduling method was verified by randomly selecting a typical day's operation from the test set. The simulation results of the system are as follows: Figure 2-6 As shown. Figure 2 The data illustrates wind power resources and electricity / heat load on a typical day. It can be seen that the system can adjust energy production and distribution strategies in real time according to fluctuations in renewable energy and changes in load demand, proving that the invention is not based on an ideal, stable scenario, but rather designed for the renewable energy fluctuations in the actual power grid, demonstrating strong scenario adaptability. Figure 3 and Figure 4 The scheduling strategies for electrical and thermal energy were demonstrated respectively. Figure 3 and Figure 4 The system was verified to be able to adjust the device power in real time with low decision latency. Figure 5 This indicates that the system can maintain the voltage level within a safe range during the scheduling process, ensuring the stable operation of the power grid. Figure 6 This demonstrates the system's control over hydrogen purity during electrolysis, ensuring the safety of hydrogen production. Through... Figure 2-6The simulation results verify the technical effectiveness of this invention in real-time scheduling, security assurance, economic improvement, anti-interference performance, and multi-energy flow collaborative optimization. These results demonstrate that this invention can effectively address the challenges of high-proportion renewable energy fluctuations, deep coupling of multiple energy flows, and the special security requirements of hydrogen energy.

[0056] These charts show how the system optimizes energy allocation based on real-time data to meet electricity and heat demands.

[0057] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent substitutions, and improvements made to the above embodiments without departing from the scope of the present invention, based on the technical essence of the present invention and within the spirit and principles of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for scheduling an integrated energy system of electro-hydrogen-thermal, characterized in that, The application relates to a real-time scheduling method for an electricity-hydrogen-heat coupled power distribution network. The method comprises the following steps: Step 1: based on time-of-use electricity price, the positions of different types of distributed fuel cells and the load demand of the fuel cells, a real-time scheduling model of the electricity-hydrogen-heat coupled power distribution network is established; Step 2: based on the established real-time scheduling model of the electricity-hydrogen-heat coupled power distribution network, a scheduling system of the electricity-hydrogen-heat coupled power distribution network is modeled as a constrained Markov decision process; Step 3: based on the constrained Markov decision process, a strategy neural network is established, and a safe reinforcement learning method is used to train the strategy neural network; 2.The method of claim 1, wherein, Step 4: based on the trained strategy neural network, a real-time optimal scheduling strategy of the electricity-hydrogen-heat coupled power distribution network is calculated. 3.The method of claim 2, wherein, The real-time scheduling model of the electricity-hydrogen-heat coupled power distribution network in step 1 comprises a power grid power flow unit, a fuel cell unit, an alkaline electrolytic cell unit, a hydrogen storage tank constraint unit, a fan unit, an electric boiler unit and an inverter unit, and each unit satisfies preset physical constraints and safety thresholds. Step 2 specifically comprises the following steps: A state space, an action space, a state transition probability and a reward function of the constrained Markov decision process are defined; The state space comprises the current time electricity price, the hydrogen storage tank state, the difference between the hydrogen storage tank capacity and the minimum hydrogen storage target, the wind power field wind resource, the active load and the reactive load of each node and the heat load of each region; The action space comprises the current density and the reactive output of the fuel cell, the current density of the electrolytic cell and the reactive output of the fan transformer, and the heat gap of the fuel cell after the fuel cell heat supply is supplemented by the electric boiler unit; Constrained markov decision processes by optimizing policies maximize cumulative reward while limiting cumulative constraint violations to a predetermined range The state transition probability is the probability that the scheduling system of the electricity-hydrogen-heat coupled power distribution network is transferred from the state at t time to the state at t+1 time after the action a is adopted, which is calculated by collecting the data of each unit in the real-time scheduling model of the electricity-hydrogen-heat coupled power distribution network; （1）； （2）； In Equations (1) and (2), is a reward function, is a constraint function, is a threshold of the cost of violating the constraint, requiring that the cumulative cost of the policy does not exceed this value, is a discount factor, is the expected reward obtained by the policy; The expression of the optimization problem of the constrained Markov decision process is: （3）； In formula (3), Indicates the current time. express The electricity price at any given time express The current state of the hydrogen storage tank at any given moment. express The difference between the current hydrogen storage tank capacity and the minimum hydrogen storage target at any given time. and express The wind resources of the two wind farms at all times. and Indicates the first Each node Active and reactive loads at any given time. Indicates the region exist The heat load at any time, This represents the active load of the 33rd node at time t. This represents the reactive load of the first node at time t. This represents the reactive load of the 33rd node at time t. Indicates that region 1 is in The heat load at any time, Indicates region 4 in The heat load at any given time; The state expression of the scheduling system of the electricity-hydrogen-heat coupled power distribution network at t time is: （4）； In equation (4), With denotes the fuel cell At the current density and the reactive power output at time denotes the current density of the electrolyzer at time t, With denotes the reactive power output of the two fan transformers at time t.

4. The method of claim 3, wherein, The action expression of the scheduling system of the electricity-hydrogen-heat coupled power distribution network at t time is: The reward function is used for decoupling the system reward and the system constraint of the scheduling system of the electricity-hydrogen-heat coupled power distribution network, the constraint function is solved through the power grid power flow unit, the alkaline electrolytic cell unit, the hydrogen storage tank constraint unit and the inverter unit, and the system constraint is obtained through the constraint function; （5）； In equation (5), is the cost of electricity purchased from the grid by the system, is the total number of time steps, t is the current time step; The expression of the reward function is: （6） （7） （8）； （9）； （10）； In equations (6)-(10), represents the node voltage violation constraint, is the number of nodes, and are the upper and lower limits of voltage, respectively; represents the HTO violation constraint in the electrolyzer, represents the limit of hydrogen content in oxygen; represents the limit constraint of the hydrogen storage tank, and are the upper and lower limits of safe hydrogen storage of the hydrogen storage tank, respectively; represents the output constraint of the inverter, including all the wind turbines and fuel cells, is the number of inverters, represents the maximum apparent power of the inverter, represents the active output of the device at the moment, represents the reactive output of the device at the moment; , , , represent the violation coefficient factors of the node voltage violation constraint, the HTO violation constraint in the electrolyzer, the limit constraint of the hydrogen storage tank, and the output constraint of the inverter, respectively.

5. The method of claim 1, wherein, The expression of the constraint function is: The maximum entropy policy updating method introduces the maximum entropy into the optimal policy by adopting the SAC algorithm is updated; The constraint weight update based on cross-entropy includes: when the cumulative constraint violation amount exceeds the warning threshold, standardizing the cumulative constraint violation amounts of different safety constraints and mapping them uniformly to the [0,1] interval to form a comparable risk index. ; within the time window Within, comparable risk indicators will be available. Convert to probability distribution And calculate the information entropy of the penalty index. Information entropy based on penalty indicators Establish constraint weights ; The safe reinforcement learning method in step 3 comprises a maximum entropy strategy updating method, a constraint strategy based on an original dual and a constraint weight updating based on cross entropy; （11）； In formula (11), For strategy In system state The entropy below, For the expected value, In system state The logarithmic probability of taking action a. Representative strategy, Representing state Execute action a at that time; optimal strategy The update formula is: （12）； In equation (12), a temperature coefficient to control the trade-off between entropy and reward, for determining the proportion of entropy in the reward, a discount factor, a reward function; Comparative risk indicator The formula for calculating the comparative risk indicator is: （13）； In equation (13), The original violation value is represented as a constraint The original violation value is represented as a constraint The constraint violation maximum within the sliding time window is represented as is a minimum value; Information entropy The formula for calculating the information entropy is: （14）； （15）； In formulas (14) and (15), The information entropy representing the penalty index is used to measure the dispersion degree of the risk distribution, It indicates that the risk is relatively severe, and the weight needs to be strengthened, It indicates that the risk is relatively dispersed, and the penalty needs to be weakened; Constraint weight The calculation formula is: （16）； In equation (16), is the lower weight limit, which is used to prevent the weight from being too low to ignore the corresponding constraint.

6. The method of claim 5, wherein, The constraint strategy based on the original dual adopts a Lagrange relaxation framework, by introducing Lagrange multipliers , , transforms the constraint optimization optimal strategy solving problem in the constrained Markov decision process into an unconstrained saddle point optimization problem; a Lagrange multiplier and the SAC algorithm, a primal-dual based SAC algorithm is constructed, and a scheduling problem is established according to the primal-dual based SAC algorithm, in each iteration process, the is fixed, and a corresponding policy gradient update is performed according to the original strategy type adopted by the system, and the dual variable is updated at the same time, while maximizing the cumulative expected reward, the cumulative constraint violation weighted by the Lagrange multiplier is minimized; The calculation formula of the maximum entropy is: （17）； In equation (17), is a system constraint violation value; The expression of the established scheduling problem is: （18）； In equation (18), denotes the policy gradient update step, is the policy parameter, is the policy parameter at the kth iteration, is the value function of the policy, is the Lagrangian function; dual variable The update formula for is （19）； In equation (19), denotes the update compensation of the dual domain, denotes the cumulative violation of the ith constraint under the policy , is the threshold value of the ith constraint, is the cumulative violation of the ith constraint under the policy , is the current value of the dual variable, is the updated value of the dual variable; The expression of the strategy updating is: （20）； （21）； In Equations (20) and (21), the reward for taking action at time t, the reward for taking action at time t, the reward for taking action at time t, the constraint threshold, the discount factor.

7. The method of claim 1, wherein, The expression of the minimum cumulative constraint violation amount is: The input to the decision network is the current state of the system, and the output is the probability distribution parameters of the actions in the current state. The probability distribution parameters include the mean. and standard deviation During training, reparameterization techniques are used, combined with mean values. and standard deviation The final action of the calculation system in its current state ; The two sets of value networks include a reward value network and a constraint value network, the reward value network is used to evaluate the expected cumulative discounted reward obtained by the system after performing an action in a state The constraint value network is used to evaluate the expected cumulative discounted cost obtained by the system after performing an action in a state The reward value network and the constraint value network each include a double Q-learning network structure, and each Q-learning network is connected to a corresponding target network. 8.The method of claim 7, wherein, The strategy neural network in step 3 adopts an Actor-Critic framework, comprises a decision network and two groups of value networks, the decision network is used for action decision evaluation, and the value networks are used for state-action value evaluation; The training of the value network specifically comprises the following steps: Step 3.1: the target Q value is calculated by using a soft updating mechanism through a target network corresponding to a Q-learning network structure; Step 3.2: Store the tuple in the experience replay buffer D at each time step, where, is the reward at time step t, is the cost at time step t, and is the system state, is the system performing the action; Step 3.3 randomly sample mini-batch data from the experience replay buffer D, pass through the updated optimal policy , obtain the current state-action value and the expectation of all possible next states and corresponding actions by combining the Bellman equation, and minimize the mean square error of the obtained current state-action value and the expectation of all possible next states and corresponding actions, to obtain an approximation of the action-value function in time step t, alternately update the reward value network parameters, the Lagrange multiplier and the constraint coefficient factor; Step 3.4: Repeat Step 3.3 to obtain an approximation of the cost value function at time step t, updating the constraint value network parameters, Lagrange multipliers, and constraint coefficient factors alternately. Step 3.4: Repeat Step 3.3 to obtain an approximation of the cost value function at time step t, updating the constraint value network parameters, Lagrange multipliers, and constraint coefficient factors alternately.

Images