Electricity-hydrogen coupling system risk scheduling method based on robust safety deep reinforcement learning

Abstract

The invention discloses an electricity-hydrogen coupling system risk scheduling method based on robust safety deep reinforcement learning, and the method comprises the following steps: 1), considering the Faraday efficiency loss effect of an electrolytic cell, and constructing an electricity-hydrogen hybrid energy storage dynamic operation model; 2) considering the wind power uncertainty, and establishing an electricity-hydrogen coupling system risk scheduling model considering the dynamic operation of electricity-hydrogen hybrid energy storage; 3) constructing a robust constraint Markov decision architecture; 4) parameterizing the robust constraint Markov decision architecture by using a flexible action device-evaluator baseline algorithm; and 5) solving the risk scheduling model of the electricity-hydrogen coupling system by using the parameterized robust constraint Markov decision process architecture to obtain a risk scheduling scheme of the electricity-hydrogen coupling system. According to the method, the new safety deep reinforcement learning algorithm is provided for the important requirements of energy low-carbon green transformation and energy safety, efficient and strict management and control of the electric hydrogen coupling system risk are achieved, and the reliability and safety of system operation are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of risk scheduling of an electric-hydrogen coupling system, and in particular to a risk scheduling method of an electric-hydrogen coupling system based on robust and safe deep reinforcement learning. Background Art

[0002] In order to alleviate the problems of environmental pollution and energy depletion, it is particularly urgent to transition to an electric energy structure dominated by high-penetration clean energy. This also exacerbates the risk of highly random new energy sources such as wind and light to the operation of the power grid. As an emerging energy carrier, hydrogen has the advantages of green zero carbon, high energy density, and strong flexibility, which can smooth out the randomness of new energy and reduce operational risks. Among them, the risk scheduling of the electric-hydrogen coupling system, as a key technology to support the reliable development and overall integrated coordination of electric-hydrogen energy, starts from the perspective of an integrated management system, and deeply analyzes the two dimensions of operation over-limit probability and severity, aiming to fully explore the application potential of hydrogen energy in ensuring the sustainable and safe advancement of renewable energy plans.

[0003] In the electric-hydrogen hybrid system, the electrolyzer is one of the core devices for realizing the electric-hydrogen coupling. From the perspective of risk scheduling, reasonable modeling of the electrolyzer is the cornerstone and key to accurately quantify and strictly manage the electric-hydrogen hybrid energy system. Existing studies have established a physical model of the electrolyzer based on the electrochemical reaction process of water electrolysis. For example, some literature considers the fixed efficiency operation of the electrolyzer in the day-ahead economic scheduling model of the integrated energy system, ignoring the changes in the efficiency of the electrolyzer during the actual operation, resulting in large deviations in actual applications; there are also literatures based on the polarization relationship inside the electrolyzer, establishing an operation model that considers the nonlinear relationship between the hydrogen production of the electrolyzer and the power consumption, in order to analyze the impact of the electrolyzer characteristics on the economic operation of the system. However, the establishment of the above electrolyzer models all ignores the impact of the change in Faraday efficiency on the operating efficiency of the electrolyzer, resulting in a rough modeling process and difficulty in accurately describing the dynamic efficiency operating characteristics of the electrolyzer. Therefore, in the actual power grid, there may be significant deviations between the dispatching behavior and the actual operation, resulting in unreasonable results.

[0004] The safe and reliable clean energy solution based on risk scheduling of the electric-hydrogen coupling system faces the following significant challenges: a) The complex heterogeneous energy balance relationship in the electric-hydrogen coupling system and the dynamic physical characteristics of the electrolyzer make the operation model of the electric-hydrogen coupling system mathematically classified as a nonlinear non-convex high-dimensional mixed integer programming problem, especially when considering the change of Faraday efficiency. b) The risk term of branch over-limit is usually modeled as an integral term of the probability and severity of branch overload. These risk terms originate from the uncertainty of renewable energy distributed through the affine characteristics of the power flow. This poses an extremely difficult challenge to the existing solution methods. c) In addition, dealing with the volatility of renewable energy is a difficult task. The prediction error of new energy has been shown to obey a non-Gaussian distribution, which shows that in the perspective of optimization, forward-looking scheduling places high demands on the perception and adaptability of complex uncertain environments.

[0005] Existing work mainly focuses on scenario-based methods to deal with the risk scheduling problem of complex electric-hydrogen coupled systems. Scenario-based methods attempt to simplify the model through linearization and convexity assumptions, while using methods such as scenario discretization to deal with risk terms caused by complex uncertainties. However, in practical applications, the effectiveness of the trade-off between the number of scenarios and uncertainty perception ability under the non-Gaussian distribution of uncertainty has not been verified, which suggests that the need to increase the number of scenarios may be more urgent. Therefore, although this method sacrifices model accuracy to a certain extent, it does not significantly improve the solution efficiency.

[0006] Some recent exploratory studies have focused on the deployment and application of deep reinforcement learning in the operation of electric-hydrogen coupling systems. This is due to the superior performance of deep reinforcement learning in dealing with uncertainty, nonlinear dynamic optimization problems and online decision-making. However, the electric-hydrogen coupling system is subject to the physical constraints of energy equipment and the limitations of available resources. Its requirements for high safety and complexity significantly increase the demand and difficulty of deep reinforcement learning for constraint satisfaction performance. Therefore, designing a deep reinforcement learning safety paradigm that can meet these constraints has become the key to ensuring the reliable operation of the electric-hydrogen coupling system. The existing deep reinforcement learning safety paradigms mainly include the following three categories:

[0007] 1) Incorporate the constraint violation signal as a penalty augmentation term into the reward function. For example, existing literature has proposed an energy optimization management method for the electric-hydrogen coupling system based on DQN. Some other literatures have proposed hybrid energy storage electric-hydrogen coupling system scheduling strategies based on SAC and DDPG. However, the design of the penalty factor is quite challenging, especially when dealing with the risk scheduling problem of the electric-hydrogen coupling system, because the penalty factor involves a large number of constraints of different units. In this case, it is necessary to perform an extremely complicated and time-consuming combined parameter adjustment process, which leads to severe challenges in training performance and stability.

[0008] 2) Corrective actions in the physical safety layer. The safety layer correction method makes minimal adjustments to the output actions by designing an optimization model or projection algorithm to ensure that the system operates within the safety domain. A literature has proposed an optimized operation algorithm for a deep reinforcement learning electric-hydrogen coupling system with an integrated adaptive safety module, which effectively ensures the constraint satisfaction of the hydrogen storage state and voltage amplitude. However, the optimization and correction process of the safety layer significantly increases the training computational burden. In addition, this correction process independent of the training environment hinders the agent's learning of the problem itself, which may lead to the generation of suboptimal strategies.

[0009] 3) Introduce additional constrained training mechanisms. Pioneering work is to guide the agent to conduct safety surveys by establishing constrained Markov processes, including C-SAC, PD-DDPG, IPO, etc. The constrained Markov process regards the constraint violation as the safety survey cost of the agent, characterizes the satisfaction of the constraint with the expectation of the cost, and performs additional training with the goal of limiting it to the cost threshold. This provides a suitable idea for solving the operation of the electric-hydrogen coupled system. However, the constrained Markov process only relies on the cost expectation as a safety indicator, which means that this method may have limitations when there are some unexpected operational risks, especially when the state space is too large after the introduction of uncertainty.

[0010] To address these challenges, it is necessary to explore alternative deep reinforcement learning methods that incorporate more robust constraint learning mechanisms that can develop personalized strategies based on the different risk requirements of high-safety-demand electric-hydrogen coupled systems. Summary of the invention

[0011] The object of the present invention is to provide a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning, comprising the following steps:

[0012] 1) Considering the Faraday efficiency loss effect of the electrolyzer, a dynamic operation model of electric-hydrogen hybrid energy storage is constructed;

[0013] 2) Taking into account the uncertainty of wind power, a risk dispatch model for the electric-hydrogen coupling system taking into account the dynamic operation of electric-hydrogen hybrid energy storage is established;

[0014] 3) Construct a robust constrained Markov decision framework;

[0015] 4) Parameterize the robust constrained Markov decision architecture using the flexible actor-critic baseline algorithm;

[0016] 5) The parameterized robust constrained Markov decision process architecture is used to solve the risk scheduling model of the electric-hydrogen coupling system and obtain the risk scheduling plan of the electric-hydrogen coupling system.

[0017] Furthermore, the dynamic operation model of electric-hydrogen hybrid energy storage is as follows:

[0018]

[0019] Where, HHV H is the higher calorific value of hydrogen under standard conditions; η e,t The operating efficiency of electric-hydrogen hybrid energy storage; The power consumption and hydrogen production of the electric-hydrogen hybrid energy storage electrolyzer;

[0020] Among them, the power consumption of the electrolytic cell and hydrogen production As shown below:

[0021]

[0022] In the formula, is the hydrogen production of the electrolyzer at time t; I cell,t =A cell i cell,t is the operating current of the electrolytic cell; N E is the number of electrolytic cells in series, which is related to the maximum capacity of the electrolytic cell; k1 represents the conversion coefficient; η f,t is the Faraday efficiency;

[0023] Electric-hydrogen hybrid energy storage operating voltage V cell,t As shown below:

[0024]

[0025] Where t is the scheduling interval; V rev is the reversible voltage; T is the operating temperature; R and F are the rational gas constant and Faraday constant respectively; and are the partial pressures of hydrogen and oxygen, respectively; is the activity of H2O; R cell is the internal equivalent ohmic resistance; α an and α cat are the anode and cathode charge transfer coefficients respectively; i an and i cat are the anode and cathode exchange current densities, respectively; i cell,t is the current density;

[0026] Faraday efficiency η f,t As shown below:

[0027]

[0028] In the formula, is the oxygen permeability in the diffusion mechanism; is the hydrogen permeability in the diffusion mechanism; are the cathode hydrogen partial pressure and the anode oxygen partial pressure in the catalyst layer, respectively; A H2 ,A O2 are the fitting parameters for the increase in hydrogen and oxygen partial pressures in the cathode catalyst layer, respectively; is the hydrogen permeability caused by pressure difference;

[0029] Furthermore, the risk dispatch model of the electricity-hydrogen coupling system is as follows:

[0030]

[0031] sth(x,y,ζ)=0 (9)

[0032] Ax+Dy+Mζ+e≤0 (10)

[0033] g(x,y,ζ)≤0 (11)

[0034] u(x,y,ζ)≤0 (12)

[0035]

[0036]

[0037] Where, the superscript t represents the time period; T is the scheduling period; the vector x represents the decision variables including the scheduling strategy; the vector y represents other state variables in the HEES; the vector ζ represents the uncertain prediction error; A, D and M are coefficient matrices, respectively, and e is the coefficient vector; the subscript l represents the lth branch; l represents the branch set; the symbol '-' represents the upper limit of each variable; (·)+ represents max{·,0}; Represents the branch overload degree; d B is the risk threshold of branch overload; Pr{x}=1-F(x) represents the probability of x, F() is the cumulative distribution function; P l,t Indicates branch flow; d B represents the risk threshold; f t (x) represents the operating cost function; h(x, y, ζ) = 0 represents the AC power flow equation and the energy balance equation.

[0038] Furthermore, the robust constrained Markov decision architecture Including action network, reward judgment network, safety judgment network, Lagrange multiplier, and target network; represents the state space, represents the action space, is the state transfer function, r represents the state s t Next, perform action a t The reward obtained, c represents the reward in state s t Next, perform action a tThe obtained safe exploration cost, d is the crossing threshold, γ is the discount factor; γ∈[0,1];

[0039] The action network encapsulates the action strategy distribution π with the neural network parameter θ, and establishes a mapping from the state space to the action space

[0040] The reward judgment network is used to evaluate the operating status of HEES t Next regulatory decision t The reward soft Q function (s t ,a t );

[0041] Safety evaluation network is used to evaluate the operating status of HEES t Next regulatory decision t The security assessment function Γ π (s t ,a t ,β);

[0042] Lagrange multipliers are used to adjust the dual multipliers related to entropy and cross-limit cost weights in the dual update process;

[0043] The target network is used to enhance the training stability of the reward value discrimination network and the cost value discrimination network modules.

[0044] Furthermore, the state space As shown below:

[0045]

[0046] Where P L,t 、m H2,t , P W,t , P G,t-1 , E B,t-1 They respectively represent the electric load prediction vector, the hydrogen load prediction vector, the new energy prediction value, the first-order derivative of the new energy prediction value with respect to time, the thermal power output vector at the previous moment, the hydrogen storage capacity of the hydrogen storage tank at the previous moment, and the charge state of the electric energy storage at the previous moment; the subscripts L, W, G, H, and B represent the load, wind farm, thermal power unit, hydrogen storage tank, and battery collection, respectively.

[0047] Among them, the first-order derivative of the new energy prediction value with respect to time is as follows:

[0048]

[0049] Where Δt is the time difference; P W,t-1represents the predicted value of new energy at time t-1;

[0050] The action space As shown below:

[0051]

[0052] ΔP G,t =a G,t Θ[-T 60 D G ,T 60 U G ] (18)

[0053]

[0054] rd G,t =a D,t Θ[0,min(P G,t - P G ,T 10 D G )] (20)

[0055]

[0056] In the formula, the subscripts E and FC represent the electrolyzer set, fuel cell set, and energy storage device set, respectively. The subscript '_' represents the lower limit of the corresponding variable; represent the energy storage charging / discharging power respectively, which are combined into a decision P B,t To reduce the action space; U G With D G Respectively represent the climbing rate and descending rate of the unit. 60 With T 10 Refers to time intervals of one hour and ten minutes respectively; a G,t 、a U,t 、a D,t 、a E,t 、a F,t 、a B,t 、a H,t They represent the output increment action of thermal power unit, the upper rotation standby action of thermal power unit, the lower rotation standby action, the operation current density action of electrolyzer, the power generation power action of fuel cell, the charge / discharge power action of electric energy storage, and the hydrogen output action of hydrogen storage tank respectively; G,t ,rd G,t Respectively represent the upper and lower spinning reserve capacities; i cell,E,t The current density that represents the operating limit of the electrolytic cell; P B,t Indicates the power of electrical energy storage.

[0057] Furthermore, the state transition function of the robust constrained Markov decision architecture is as follows:

[0058] P G,t =P G,t-1 +ΔP G,t (25)

[0059]

[0060] In the formula, represents the hydrogen inlet vector of the hydrogen storage tank; η H,in , η H,out The efficiency vectors of hydrogen input and output of the hydrogen storage tank respectively; η cha,B , η dis,B Efficiency vectors of charging and discharging of energy storage respectively.

[0061] Furthermore, the cost function c π,ρ (s t ,a t ,ζ) is as follows:

[0062] c π,ρ (s t ,a t ,ζ)=[ε e c e,t +ε b c b,t +ε pf c pf,t +ε pb c pb,t +ε ph c ph,t ]|π,ρ (31)

[0063]

[0064]

[0065] Where π represents the distribution of agent control decision strategies; ε e , ε b , ε pf , ε pb With ε ph Respectively represent scaling; c e,t Indicates that the capacity safety constraint of the hydrogen storage tank exceeds the limit; c b,t Indicates that the safety constraint of the electric energy storage capacity exceeds the limit; c pf,t Indicates that the branch overload risk constraint exceeds the limit; c pb,t Indicates the unbalanced amount of HEES electric power; c ph,t Indicates the hydrogen energy imbalance; P e,t Represents the power consumption of the e-th PEM electrolyzer at time t.

[0066] The reward function is as follows:

[0067] r t =-(f t +f C,t )=-(f G,t +f Q,t +f L,t +f HESS,t +f C,t ) (37)

[0068]

[0069] f HESS,t =λ h (f HS,om,t +f HS,loss,t +f B,om,t +f B,loss,t ) (41)

[0070] f C,t =λ c (c π,ρ (s t ,a t ,ζ)-d) + (42)

[0071]

[0072] In the formula, f G,t represents the operating cost of thermal power; f Q,t 、f L,t They represent the penalty items for abandoning wind and solar power and shedding load respectively; f HESS,t represents the operating cost of electric-hydrogen hybrid energy storage; f C,t represents the penalty fee for exceeding the constraint limit; a g , b g 、c g 、u g ,d g The constant, primary, secondary, upward spinning reserve and downward spinning reserve of the thermal power unit are represented respectively; g , q , l , h , c represents the cost coefficient; represents the power imbalance caused by uncertainty at the i-th node; f HS,om,t 、f B,om,t are the operation and maintenance costs of hydrogen energy storage and electric energy storage respectively; f HS,loss,t 、f B,loss,t are the energy consumption costs of hydrogen energy storage and electric energy storage respectively; C e,inv , C fc,inv are the investment and construction costs of the electrolyzer e and fuel cell fc respectively; T e,life , Tfc,life are the maximum operating time of the electrolyzer e and the fuel cell fc respectively; B,om is the unit operation and maintenance cost coefficient of electric energy storage; ep The transmission and distribution price.

[0073] Furthermore, the security assessment function Γ π (s t ,a t ,β) is as follows:

[0074]

[0075] Where φ(·) and Φ(·) represent the probability density distribution function and cumulative probability distribution function of the standard normal distribution, respectively; F π,ρ Represents C π,ρ The probability density function of

[0076] Among them, C π The second central moment of As shown below:

[0077]

[0078] Furthermore, when the robust constrained Markov decision architecture is parameterized using the flexible actor-critic baseline algorithm, the objective function and constraints are as follows:

[0079]

[0080] In the formula, Represents the entropy threshold.

[0081] Furthermore, when parameterizing the robust constrained Markov decision architecture, the loss function of the safety evaluation network is As shown below:

[0082]

[0083] In the formula, Indicates experience playback;

[0084] The loss function of the reward judgment network is as follows:

[0085]

[0086] The loss function J of the action network π (θ) is as follows:

[0087]

[0088] The update equation of the Lagrange multiplier is as follows:

[0089]

[0090] Where α and κ are Lagrange multipliers; ω is the learning rate.

[0091] The technical effect of the present invention is unquestionable. Aiming at the major needs of low-carbon green transformation of energy and energy security, the present invention proposes a risk scheduling method for an electric-hydrogen coupling system based on a robust flexible actor-evaluator, so as to achieve efficient and strict control of the risks of the electric-hydrogen coupling system and ensure the reliability and safety of the system operation.

[0092] This paper proposes a risk dispatch model for an electric-hydrogen coupling system that considers the dynamic efficiency characteristics of the electrolyzer. The model takes into account the Faraday efficiency loss, effectively reduces the deviation in practical applications, and manages the branch overload risk caused by the propagation of renewable energy uncertainty through AC power flow to ensure the safe operation of the electric-hydrogen coupling system.

[0093] The present invention develops a robust constrained Markov decision process framework for the risk scheduling problem of the electric-hydrogen coupling system. In this framework, the constraint violation risk is regarded as the safe exploration cost of the intelligent agent. By using the conditional risk value of this cost as the safety judgment criterion, instead of the expected value used in the traditional constrained Markov decision process, the present invention establishes a risk-averse constraint learning mechanism to improve the constraint satisfaction of the risk scheduling of the electric-hydrogen coupling system. In addition, a safety judgment function based on the second-order distributed Bellman operator is designed for efficient estimation of the conditional risk value.

[0094] The present invention proposes a more robust and safe deep reinforcement learning algorithm, the robust flexible actor-evaluator. The algorithm uses a deep neural network to parameterize the robust constrained Markov decision process framework, so that the decision expression and strategy learning capabilities of the intelligent agent in the risk scheduling of the electric-hydrogen coupling system can be realized. This process incorporates entropy and cost constraints into the actor-evaluator framework. The primal-dual optimization method is introduced to solve the constrained optimization training and drive the robust constrained Markov decision process framework towards maximum entropy adaptive learning. The proposed robust flexible actor-evaluator algorithm ensures global optimality and constraint satisfaction in the formulation of the risk scheduling scheduling strategy of the electric-hydrogen coupling system, and has online decision-making capabilities.

[0095] In terms of usage effect, the robust flexible actor-evaluator algorithm framework proposed in the present invention demonstrates significantly improved convergence performance and safety exploration capabilities. Compared with the traditional constrained Markov decision process framework and the algorithm based on the Markov decision process framework, the proposed robust constrained Markov decision process framework improves the convergence efficiency by at least 45.03%, while strictly ensuring the satisfaction of the operating constraints; compared with the existing optimization-based methods, the proposed algorithm shows excellent uncertainty handling capabilities. Compared with the scenario-based method, the robust flexible actor-evaluator algorithm has a more comprehensive uncertainty perception and enhanced risk resistance, and can effectively cope with the extreme operating conditions of the electric-hydrogen coupling system; compared with the scenario-based method, the electric-hydrogen coupling system scheduling strategy formulated by the method proposed in the present invention shows a more reasonable trade-off between safety and economic efficiency. It not only reliably meets the load demand, but also makes more effective use of the electric-hydrogen hybrid energy storage to effectively manage operational risks; for the deployment time of the electric-hydrogen coupling system, the robust flexible actor-evaluator algorithm achieved a solution time of only 0.412 seconds in the IEEE-118 node system, which is more than 15,907 times higher than other risk-based scheduling optimization methods. The algorithm proposed in this invention demonstrates the great potential of promoting the application and deployment of next-generation artificial intelligence technology in promoting the safe operation of green energy systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] Figure 1 is the graph of electrolyzer operating efficiency and hydrogen production;

[0097] Figure 2 This is the architecture diagram of the electric-hydrogen coupling system;

[0098] Figure 3 It is the framework diagram of the proposed robust constrained Markov decision process;

[0099] Figure 4 It is a schematic diagram of the proposed robust constrained Markov decision process and the traditional constrained Markov decision process in the constraint learning stage;

[0100] Figure 5 This is the schematic diagram of the proposed robust flexible actor-judge algorithm;

[0101] Figure 6 This is the training process of the robust flexible actor-critic. It includes (a) reward, (b) cost, (c) cost rate, (d) action network loss function, (e)-(f) reward-critic network loss function, (g)-(h) cost-critic network loss function, and (e) strategy entropy value.

[0102] Figure 7 This is a comparison chart of rewards and costs for various deep reinforcement learning training processes;

[0103] Figure 8 is the load rate distribution of the heavy-load branch under uncertainty conditions. Among them, Q0 represents the lower critical value, Q1 represents the first quartile, Q2 is the median, Q3 is the third quartile, and Q4 is the upper critical value;

[0104] Fig. 9 It is based on the proposed robust flexible actor-evaluator method and the scheduling strategy of the electric-hydrogen coupling system formulated based on the optimization method;

[0105] Fig.10 It is the curve of changes in hydrogen storage capacity of hydrogen storage tank, battery state of charge and electrolyzer efficiency under the scheduling strategy of electric-hydrogen coupling system formulated based on the proposed robust flexible actor-evaluator method and the optimization method. DETAILED DESCRIPTION

[0106] The present invention is further described below in conjunction with the embodiments, but it should not be understood that the above subject matter of the present invention is limited to the following embodiments. Without departing from the above technical ideas of the present invention, various substitutions and changes are made according to the common technical knowledge and customary means in the art, which should all be included in the protection scope of the present invention.

[0107] Embodiment 1:

[0108] A risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning comprises the following steps:

[0109] 1) Considering the Faraday efficiency loss effect of the electrolyzer, a dynamic operation model of electric-hydrogen hybrid energy storage is constructed;

[0110] 2) Taking into account the uncertainty of wind power, a risk dispatch model for the electric-hydrogen coupling system taking into account the dynamic operation of electric-hydrogen hybrid energy storage is established;

[0111] 3) Construct a robust constrained Markov decision framework;

[0112] 4) Parameterize the robust constrained Markov decision architecture using the flexible actor-critic baseline algorithm;

[0113] 5) The parameterized robust constrained Markov decision process architecture is used to solve the risk scheduling model of the electric-hydrogen coupling system and obtain the risk scheduling plan of the electric-hydrogen coupling system.

[0114] Embodiment 2:

[0115] A risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning, the technical content is the same as that of Example 1, and further, the dynamic operation model of the electric-hydrogen hybrid energy storage is as follows:

[0116]

[0117] Where, HHVH is the higher calorific value of hydrogen under standard conditions; η e,t The operating efficiency of electric-hydrogen hybrid energy storage; The power consumption and hydrogen production of the electric-hydrogen hybrid energy storage electrolyzer;

[0118] Among them, the power consumption of the electrolytic cell and hydrogen production As shown below:

[0119]

[0120] In the formula, is the hydrogen production of the electrolyzer at time t; I cell,t =A cell i cell,t is the operating current of the electrolytic cell; N E is the number of electrolytic cells in series, which is related to the maximum capacity of the electrolytic cell; k1 represents the conversion coefficient; η f,t is the Faraday efficiency;

[0121] Electric-hydrogen hybrid energy storage operating voltage V cell,t As shown below:

[0122]

[0123] Where t is the scheduling interval; V rev is the reversible voltage; T is the operating temperature; R and F are the rational gas constant and Faraday constant respectively; and are the partial pressures of hydrogen and oxygen, respectively; is the activity of H2O; R cell is the internal equivalent ohmic resistance; α an and α cat are the anode and cathode charge transfer coefficients respectively; i an and i cat are the anode and cathode exchange current densities, respectively; i cell,t is the current density;

[0124] Faraday efficiency η f,t As shown below:

[0125]

[0126] In the formula, is the oxygen permeability in the diffusion mechanism; is the hydrogen permeability in the diffusion mechanism; are the cathode hydrogen partial pressure and the anode oxygen partial pressure in the catalyst layer, respectively; A H2 ,A O2 are the fitting parameters for the increase in hydrogen and oxygen partial pressures in the cathode catalyst layer, respectively; is the hydrogen permeability caused by pressure difference;

[0127] Embodiment 3:

[0128] A risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning, the technical content of which is the same as any one of Embodiments 1-2, and further, a risk scheduling model for an electric-hydrogen coupling system is as follows:

[0129]

[0130] st h(x,y,ζ)=0 (9)

[0131] Ax+Dy+Mζ+e≤0 (10)

[0132] g(x,y,ζ)≤0 (11)

[0133] u(x,y,ζ)≤0 (12)

[0134]

[0135] Where, the superscript t represents the time period; T is the scheduling period; the vector x represents the decision variables including the scheduling strategy; the vector y represents other state variables in the HEES; the vector ζ represents the uncertain prediction error; A, D and M are coefficient matrices, respectively, and e is the coefficient vector; the subscript l represents the lth branch; l represents the branch set; the symbol '-' represents the upper limit of each variable; (·)+ represents max{·,0}; Represents the branch overload degree; d B is the risk threshold of branch overload; Pr{x}=1-F(x) represents the probability of x, F() is the cumulative distribution function; P l,t Indicates branch flow; d B represents the risk threshold; f t (x) represents the operating cost function; h(x, y, ζ) = 0 represents the AC power flow equation and the energy balance equation.

[0136] Embodiment 4:

[0137] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content is the same as any one of embodiments 1-3, further, a robust constrained Markov decision architecture Including action network, reward judgment network, safety judgment network, Lagrange multiplier, and target network; represents the state space, represents the action space, is the state transfer function, r represents the state s t Next, perform action at The reward obtained, c represents the reward in state s t Next, perform action a t The obtained safe exploration cost, d is the crossing threshold, γ is the discount factor; γ∈[0,1];

[0138] The action network encapsulates the action strategy distribution π with the neural network parameter θ, and establishes a mapping from the state space to the action space

[0139] The reward judgment network is used to evaluate the operating status of HEES t Next regulatory decision t The reward soft Q function (s t ,a t );

[0140] Safety evaluation network is used to evaluate the operating status of HEES t Next regulatory decision t The security assessment function Γ π (s t ,a t ,β);

[0141] Lagrange multipliers are used to adjust the dual multipliers related to entropy and cross-limit cost weights in the dual update process;

[0142] The target network is used to enhance the training stability of the reward value discrimination network and the cost value discrimination network modules.

[0143] Embodiment 5:

[0144] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of embodiments 1-4, further, the state space As shown below:

[0145]

[0146] Where P L,t 、m H2,t , P W,t , P G,t-1 , E B,t-1 They respectively represent the electric load prediction vector, the hydrogen load prediction vector, the new energy prediction value, the first-order derivative of the new energy prediction value with respect to time, the thermal power output vector at the last moment, the hydrogen storage capacity of the hydrogen storage tank at the last moment, and the charge state of the electric energy storage at the last moment;

[0147] Among them, the first-order derivative of the new energy prediction value with respect to time is as follows:

[0148]

[0149] In the formula, the subscripts L, W, G, H, and B represent the load, wind farm, thermal power unit, hydrogen storage tank, and battery set, respectively. Δt is the time difference; P W,t-1 represents the predicted value of new energy at time t-1;

[0150] The action space As shown below:

[0151]

[0152] ΔP G,t =a G,t Θ[-T 60 D G ,T 60 U G ] (18)

[0153]

[0154] rd G,t =a D,t Θ[0,min(P G,t - P G ,T 10 D G )] (20)

[0155]

[0156] In the formula, the subscripts E and FC represent the electrolyzer set, fuel cell set, and energy storage device set, respectively. The subscript '_' represents the lower limit of the corresponding variable; represent the energy storage charging / discharging power respectively, which are combined into a decision P B,t To reduce the action space; U G With D G Respectively represent the climbing rate and descending rate of the unit. 60 With T 10 Refers to time intervals of one hour and ten minutes respectively; a G,t 、a U,t 、a D,t 、a E,t 、a F,t 、a B,t 、a H,t They represent the output increment action of thermal power unit, the upper rotation standby action of thermal power unit, the lower rotation standby action, the operation current density action of electrolyzer, the power generation power action of fuel cell, the charge / discharge power action of electric energy storage, and the hydrogen output action of hydrogen storage tank respectively; G,t ,rdG,t Respectively represent the upper and lower spinning reserve capacities; i cell,E,t The current density that represents the operating limit of the electrolytic cell; P B,t Indicates the power of electrical energy storage.

[0157] Embodiment 6:

[0158] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of Embodiments 1-5, and further, the state transfer function of the robust constrained Markov decision architecture is as follows:

[0159] P G,t =P G,t-1 +ΔP G,t (25)

[0160]

[0161] In the formula, represents the hydrogen inlet vector of the hydrogen storage tank; η H,in , η H,out The efficiency vectors of hydrogen input and output of the hydrogen storage tank respectively; η cha,B , η dis,B Efficiency vectors of charging and discharging of energy storage respectively.

[0162] Embodiment 7:

[0163] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of embodiments 1-6, further, the cost function c π,ρ (s t ,a t ,ζ) is as follows:

[0164] c π,ρ (s t ,a t ,ζ)=[ε e c e,t +ε b c b,t +ε pf c pf,t +ε pb c pb,t +ε ph c ph,t ]|π,ρ (31)

[0165]

[0166] Where π represents the distribution of agent control decision strategies; ε e , ε b , ε pf , ε pbWith ε ph Respectively represent scaling; c e,t Indicates that the capacity safety constraint of the hydrogen storage tank exceeds the limit; c b,t Indicates that the safety constraint of the electric energy storage capacity exceeds the limit; c pf,t Indicates that the branch overload risk constraint exceeds the limit; c pb,t Indicates the unbalanced amount of HEES electric power; c ph,t Indicates the hydrogen energy imbalance; P e,t Represents the power consumption of the ethPEM electrolyzer at time t.

[0167] The reward function is as follows:

[0168] r t =-(f t +f C,t )=-(f G,t +f Q,t +f L,t +f HESS,t +f C,t ) (37)

[0169]

[0170] f HESS,t =λ h (f HS,om,t +f HS,loss,t +f B,om,t +f B,loss,t ) (41)

[0171] f C,t =λ c (c π,ρ (s t ,a t ,ζ)-d) + (42)

[0172]

[0173] In the formula, f G,t represents the operating cost of thermal power; f Q,t 、f L,t They represent the penalty items for abandoning wind and solar power and shedding load respectively; f HESS,t represents the operating cost of electric-hydrogen hybrid energy storage; f C,t represents the penalty fee for exceeding the constraint limit; a g 、b g 、c g 、u g d g The constant, primary, secondary, upward spinning reserve and downward spinning reserve of the thermal power unit are represented respectively; g , q, l , h , c represents the cost coefficient; represents the power imbalance caused by uncertainty at the i-th node; f HS,om,t 、f B,om,t are the operation and maintenance costs of hydrogen energy storage and electric energy storage respectively; f HS,loss,t 、f B,loss,t are the energy consumption costs of hydrogen energy storage and electric energy storage respectively; C e,inv , C fc,inv are the investment and construction costs of the electrolyzer e and fuel cell fc respectively; T e,life , T fc,life are the maximum operating time of the electrolyzer e and the fuel cell fc respectively; B,om is the unit operation and maintenance cost coefficient of electric energy storage; ep The transmission and distribution price.

[0174] Embodiment 8:

[0175] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of embodiments 1-7, and further, the safety assessment function Γ π (s t ,a t ,β) is as follows:

[0176]

[0177] Where φ(·) and Φ(·) represent the probability density distribution function and cumulative probability distribution function of the standard normal distribution, respectively; F π,ρ Represents C π,ρ The probability density function of

[0178] Among them, C π The second central moment of As shown below:

[0179]

[0180] Embodiment 9:

[0181] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of Examples 1-8. Further, when the robust constrained Markov decision architecture is parameterized using a flexible actor-judge baseline algorithm, the objective function and the constraint conditions are respectively as follows:

[0182]

[0183] In the formula, Represents the entropy threshold.

[0184] Embodiment 10:

[0185] A risk scheduling method for an electric-hydrogen coupling system based on robust safety deep reinforcement learning, the technical content of which is the same as any one of Examples 1-9. Further, when the robust constrained Markov decision architecture is parameterized, the loss function of the safety evaluation network is as follows:

[0186]

[0187] In the formula, Indicates experience playback;

[0188] The loss function of the reward judgment network is as follows:

[0189]

[0190] The loss function of the action network is as follows:

[0191]

[0192] The update equation of the Lagrange multiplier is as follows:

[0193]

[0194] Where α and κ are Lagrange multipliers; ω is the learning rate.

[0195] Embodiment 11:

[0196] The verification of a risk scheduling method for electric-hydrogen coupling system based on robust and safe deep reinforcement learning is as follows:

[0197] This embodiment is executed on an improved IEEE-118 node test system, and four wind farms are connected to nodes 4, 8, 15 and 40, respectively. The total installed wind power capacity is 6400MW, and the penetration rate is about 39%. The hybrid electric hydrogen energy storage system operates in conjunction with the wind farm on the same node, where the hydrogen storage configuration capacity is set to 5-10% of the total installed capacity. The maximum electric load and hydrogen load are 8860W and 11t, respectively. The historical renewable energy and load data are derived from the actual power grid data of a city in southern China. The allowable branch overload range is 0.1 times the rated capacity, and the confidence level β is set to 0.05. The remaining parameter details are shown in Table 1. Technical analysis and performance evaluation are both performed using the TensorFlow framework in Python, and the background support includes IntelCore i9-10900KGPU, NVIDIA GeForce RTX 3090GPU and lightweight 8-core 16G 18M server.

[0198] Table 1 Simulation technical parameters

[0199]

[0200] The embodiments of the present invention are as follows:

[0201] 1) Establish an electrolyzer operation model taking into account Faraday efficiency loss

[0202] The relationship between PEM operating voltage and current density is shown below:

[0203]

[0204] Where t is the scheduling interval; V rev is the reversible voltage; T is the operating temperature; R and F are the rational gas constant and Faraday constant respectively; P H2 and P O2 are the partial pressures of hydrogen and oxygen respectively; a H2O is the activity of H2O; R cell is the internal equivalent ohmic resistance; α an and α cat are the anode and cathode charge transfer coefficients respectively; i an and i cat are the anode and cathode exchange current densities, respectively; i cell,t is the current density.

[0205] Power consumption of electrolytic cell and hydrogen production As shown below:

[0206]

[0207] In the formula, is the hydrogen production of the electrolyzer at time t; I cell,t =A cell i cell,t is the operating current of the electrolytic cell; N E is the number of electrolytic cells in series, which is related to the maximum capacity of the electrolytic cell; k1 represents the conversion coefficient; η f is the Faraday efficiency, which is related to the production of hydrogen and oxygen in the electrolyzer. Due to the permeability of PME, there is diffusion of hydrogen and oxygen during the mass transfer process. Especially at low current density, the gas crossover phenomenon is more obvious, which will cause Faraday loss. The PEM dynamic efficiency operation model considering Faraday loss can be expressed as a function of membrane flux density and hydrogen production density, as shown below:

[0208]

[0209] In the formula, Γ H2 is the hydrogen production density; Φ enH2 is the hydrogen permeation flux density; Φ enO2 is the oxygen permeation flux density.

[0210] According to Faraday's law, the hydrogen production density is as follows:

[0211]

[0212] The diffusion of hydrogen on the membrane during PEM electrolysis is caused by the thermal motion of molecules and the concentration difference in the anode and cathode catalyst layers. Therefore, the hydrogen permeation flux density is the sum of the diffusion flux density and the permeation flux density caused by the pressure difference. Based on Fick's law, the hydrogen permeation flux density function is as follows:

[0213]

[0214] In the formula, is the hydrogen permeability in the diffusion mechanism; is the hydrogen permeability caused by pressure difference; are the cathode hydrogen partial pressure and the anode oxygen partial pressure in the catalyst layer, respectively, which are defined in (7)-(8); d is the membrane thickness.

[0215]

[0216] In the formula, A H2 ,A O2 are the fitting parameters for the increase in hydrogen and oxygen partial pressures in the cathode catalyst layer, respectively, which depend on the structure, thickness and permeability of the catalyst layer.

[0217] The oxygen permeation in PEM mainly comes from the diffusion mechanism. According to Fick's law, the oxygen permeation flux density is as follows:

[0218]

[0219] In the formula, is the oxygen permeability in the diffusion mechanism.

[0220] Based on equations (5)-(7), (4) can be further deduced to obtain the dynamic Faraday efficiency:

[0221]

[0222] In order to compare the influence of dynamic Faraday efficiency and fixed Faraday efficiency (0.9) on the operating characteristics of PEM electrolyzer. Considering the relationship between PEM input, output and energy conversion, PEM operating efficiency η e,t As shown below:

[0223]

[0224] Where, HHV H It is the high calorific value of hydrogen under standard conditions.

[0225] 2) Establish a risk dispatch model for the electric-hydrogen coupling system taking into account the dynamic operation of electric-hydrogen hybrid energy storage

[0226] The established HEES risk dispatch model (Risk-based dispatch for HEES, HEES-RD) is described in compact form as in formula (12).

[0227]

[0228] sth(x,y,ζ)=0(12b)

[0229] Ax+Dy+Mζ+e≤0(12c)

[0230] g(x,y,ζ)≤0(12d)

[0231] u(x,y,ζ)≤0(12e)

[0232] Wherein, the superscript t represents the time period; T is the scheduling period; the vector x represents the decision variables including the scheduling strategy; the vector y represents other state variables in the HEES; the vector ζ represents the uncertain prediction error, which follows the distribution ρ; A, D and M are coefficient matrices, respectively, and e is the coefficient vector.

[0233]

[0234] In the formula, the subscript l represents the lth branch; l represents the branch set; the symbol '-' represents the upper limit of each variable; (·)+ represents max{·,0}; represents the branch overload degree, which is a function of ζ; d B is the risk threshold of branch overload; Pr{x}=1-F(x) represents the probability of x, where F() is the cumulative distribution function; P l,t represents the branch flow. Formula (13) defines the branch overload rate to measure the severity of the overload; constraint (14) limits the expected value of the overload risk to keep it within a given risk threshold d B the following.

[0235] 3) Establish a robust constrained Markov decision process for the risk scheduling model of the electricity-hydrogen coupling system

[0236] The present invention establishes a robust constrained Markov decision process (R-CMDP) including the core elements of R-CMDP and a risk aversion training mechanism. For the R-CMDP of the HEES-RD problem, its core elements are defined as in represents the state space, represents the action space, is the state transfer function, r represents the state Next action The reward obtained, c represents the reward in state s t Next, perform action a t The obtained safety exploration cost, d is the crossing threshold, and γ (γ∈[0,1]) is the discount factor. The proposed R-CMDP adopts the iterative interaction mode of “state perception-decision making-posteriori simulation-feedback learning” and combines it with the risk avoidance training mechanism to explore the optimal action strategy.

[0237] 3.1) The core elements of establishing R-CMDP are as follows:

[0238] State Space

[0239] The state space specifically includes: the electric load prediction vector P L,t ; Hydrogen load prediction vector m H2,t ; New energy forecast value P W,t ; The first-order derivative of the new energy prediction value with respect to time P (1)W,t , used to capture the time series variation characteristics of new energy; the output vector P of the thermal generator at the last moment G,t-1 ; Hydrogen storage capacity of the hydrogen storage tank at the last moment The state of charge E of the energy storage at the last moment B,t-1 .

[0240]

[0241] Among them, the first-order derivative of the new energy prediction value with respect to time is calculated as follows:

[0242]

[0243] In the formula, the subscripts L, W, G, H, and B represent the set of load, wind farms, units, hydrogen storage tanks, and batteries, respectively.

[0244] Action Space

[0245] The action space includes: thermal power unit output increment action a G,t ; Thermal power unit up / down rotation standby action a U,t , a D,t ; Electrolytic cell operation current density action a E,t ; Fuel cell power generation action a F,t ;Electric energy storage charging / discharging power action a B,t ; Hydrogen output of hydrogen storage tank action a H,tIn order to project the action space into the actual physical space taking into account the operating limitations of the electric hydrogen equipment, the linear scaling operator Θ is introduced, as shown in equations (18)-(24). Equation (18) maps the increment ΔP of the thermal power unit to H,t , to ensure that the ramp constraint is met. Equations (19) and (20) map the upper and lower spinning reserve capacities, ru G,t With rd G,t , to meet the actual unit capacity limit. Equation (21) reflects the current density i that meets the electrolyzer operation limit cell,E,t Formula (22) calculates the power generation power P of the fuel cell FC,t Equation (23) maps the power P of the electric energy storage B,t , to meet the charge and discharge power constraints. Equation (24) maps the hydrogen output of the hydrogen storage tank To the hydrogen output limit.

[0246]

[0247] ΔP G,t =a G,t Θ[-T 60 D G ,T 60 U G ] (18)

[0248]

[0249] rd G,t =a D,t Θ[0,min(P G,t - P G ,T 10 D G )] (20)

[0250]

[0251] In the formula, the subscripts E and FC represent the electrolyzer set, fuel cell set, and energy storage device set, respectively. The subscript '_' represents the lower limit of the corresponding variable; represent the energy storage charging / discharging power respectively, which are combined into a decision P B,t To reduce the action space; U G With D G Respectively represent the climbing rate and descending rate of the unit. 60 With T 10 Refers to time intervals of one hour and ten minutes respectively.

[0252] State transfer function

[0253] The state transfer function is shown in equations (25)-(29). Among them, equation (25) calculates the unit output; equation (26) represents the energy conversion relationship of the gas storage tank at adjacent moments; equation (27) represents the charge state relationship of the electric energy storage at adjacent moments; equations (28)-(29) respectively represent the start-stop state relationship of the electrolyzer and fuel cell equipment.

[0254] P G,t =P G,t-1 +ΔP G,t (25)

[0255]

[0256]

[0257] In the formula, It represents the hydrogen inlet vector of the hydrogen storage tank. In the hybrid electric hydrogen energy storage, it is equal to the hydrogen output of the electrolyzer. Therefore, it can be obtained from the electrolyzer current density i cell,E,t and is calculated by equations (3) and (10). H,in , η H,out The efficiency vectors of hydrogen input and output of the hydrogen storage tank respectively; η cha,B , η dis,B The efficiency vectors of charging and discharging of energy storage are respectively; in addition, in order to ensure that the thermal power units meet the capacity constraints, the thermal power output decision space is restricted to formula (30).

[0258]

[0259] Cost Function

[0260] The cost function is defined as shown in formula (31).

[0261] c π,ρ (s t ,a t ,ζ)=[ε e c e,t +ε b c b,t +ε pf c pf,t +ε pb c pb,t +ε ph c ph,t ]|π,ρ (31)

[0262] Where π represents the distribution of agent control decision strategies; ε e , ε b , ε pf , ε pb With ε ph They represent the scaling, so they normalize the cost terms to the same dimension; c e,tIndicates that the capacity safety constraint of the hydrogen storage tank exceeds the limit, which is defined in formula (32); c b,t Indicates that the safety constraint of the electric energy storage capacity exceeds the limit, which is defined in formula (33); c pf,t Indicates that the branch overload risk constraint exceeds the limit, which is defined in formula (34); c pb,t represents the unbalanced power of HEES, which is defined in equation (35); c ph,t represents the hydrogen energy imbalance and is defined in equation (36).

[0263]

[0264] Where P e,t The power consumption of the PEM electrolyzer at time t is represented by the electrolyzer current density i cell,E,t and is calculated by equations (1) and (2).

[0265] Reward Function

[0266] The reward function is defined as shown in Equation (37).

[0267] r t =-(f t +f C,t )=-(f G,t +f Q,t +f L,t +f HESS,t +f C,t ) (37)

[0268] In the formula, f G,t represents the operating cost of thermal power, see (38) for details; f Q,t 、f L,t They represent the penalty items for wind and solar power abandonment and load shedding, respectively. They are calculated under network security constraints by taking into account the response capability of the spinning reserve of thermal power units to uncertain wind power. They are defined in (39)-(40); f HESS,t represents the operating cost of the electric-hydrogen hybrid energy storage, see formula (41); f C,t represents the constraint violation penalty cost, which is used to improve the efficiency of agent training, see formula (42) for details.

[0269]

[0270]

[0271] f HESS,t =λ h (f HS,om,t +f HS,loss,t +f B,om,t +f B,loss,t ) (41)

[0272] f C,t =λ c (c π,ρ (s t ,a t ,ζ)-d) + (42)

[0273] In the formula, a g , b g 、c g 、u g ,d g The constant, primary, secondary, upward spinning reserve and downward spinning reserve of the thermal power unit are represented respectively; g , q , l , h , c represents the cost coefficient; represents the power imbalance caused by uncertainty at the i-th node, as shown in formula (43). HS,om,t 、f B,om,t are the operation and maintenance costs of hydrogen energy storage and electric energy storage respectively; f HS,loss,t 、f B,loss,t are the energy consumption costs of hydrogen energy storage and electric energy storage, respectively, and are defined as shown in formula (44).

[0274]

[0275] In the formula, C e,inv , C fc,inv are the investment and construction costs of the electrolyzer e and fuel cell fc respectively; T e,life , T fc,life are the maximum operating time of the electrolyzer e and the fuel cell fc respectively; B,om is the unit operation and maintenance cost coefficient of electric energy storage; ep The transmission and distribution price.

[0276] 3.2) The training mechanism of the proposed R-CMDP is as follows:

[0277] The R-CMDP proposed in this paper introduces a risk aversion constraint learning mechanism, which is no longer limited to constraint satisfaction and low safety exploration efficiency. R-CMDP replaces the CMDP-based safety exploration index based on expectation with the risk control safety exploration index based on CVaR (conditional value at risk), as defined in formula (45) and in Figure 4(b). By adjusting the confidence level β, R-CMDP introduces an infeasible region within the controllable range, thereby achieving constraint violation identification under risk control. R-CMDP aims to learn a more robust cost objective to ensure stricter constraint satisfaction and be able to cope with unexpected operating conditions in the HEES-RD system. This mechanism quantifies and limits the risk aversion of constraint violations from a safety perspective.

[0278]

[0279] 4) Establish a security assessment function based on the second-order distributed Bellman operator

[0280] The established security assessment function based on the second-order distributed Bellman operator is as follows:

[0281]

[0282] In the formula, F π,ρ Represents C π,ρ The probability density function of , which is often considered to obey the normal distribution in the exploration space of deep reinforcement learning, that is, here, Represents C π The second central moment of is as follows:

[0283]

[0284] Distributed Bellman operator is defined in (49). Based on this, The second-order distributed Bellman operator is derived in Eq. (50), which implements a temporal difference estimate, avoiding the burden of full trajectory sampling.

[0285]

[0286] Based on the above second-order distributed Bell operator, we establish C π,ρ (s t ,a t ,ζ)CVaR safety assessment function Γ with confidence level α π (s t ,a t ,β) is as follows:

[0287]

[0288] Where φ(·) and Φ(·) represent the probability density distribution function and cumulative probability distribution function of the standard normal distribution, respectively.

[0289] 5) Propose a risk scheduling method for the electric-hydrogen coupling system based on a robust flexible actor-judge

[0290] The present invention proposes a robust soft actor-critic (R-SAC) algorithm, which is built on the basis of the SAC baseline neural network and parameterizes the R-CMDP architecture of HEES-RD in the manner described in formula (51). This process combines entropy constraints and over-limit cost constraints. In order to effectively solve this constrained optimization problem, a strategy search method based on primal-dual optimization (PDO) is introduced. R-SAC demonstrates efficient adaptive learning and powerful global exploration capabilities, ensuring global optimality and constraint satisfaction in the HEES-RD problem.

[0291]

[0292] In the formula, Represents the entropy threshold.

[0293] 5.1) Training of the evaluation network

[0294] For the loss function of RCN, first establish the reward soft Q function Distributed Bellman operator As shown below:

[0295]

[0296] Further, based on The loss function of the reward value discrimination network is as follows:

[0297]

[0298] In the formula, Represents the experience replay pool, which uses an asynchronous strategy training method to ensure full utilization of safe exploration experience to improve training efficiency.

[0299] For the loss function of SCN, based on the distributed Bellman operator and The training loss function is established as shown in equations (54)-(55).

[0300]

[0301] 5.2) Training of Action Networks and Lagrange Multipliers

[0302] In order to find the optimal action strategy, the PDO strategy exploration method is introduced. This method transforms the constraint maximization problem (51) into a dual problem by constructing a Lagrangian function (as shown in Equation (56)). By alternating between the original strategy update and the dual Lagrangian multiplier update, this method drives the action network to perform maximum entropy adaptive learning.

[0303]

[0304] Based on the above Lagrangian function, the loss function of the action network is defined as follows:

[0305]

[0306] Furthermore, the primal-dual update process of the kth generation is derived as follows:

[0307] For the update of the original strategy, the Lagrange multiplier α (k) and κ (k) is fixed, and the policy gradient descent is performed with ω as the learning rate, as follows:

[0308]

[0309] For the dual Lagrange multiplier, the strategy is fixed and performs dual gradient ascent as follows:

[0310]

[0311] 7) Verification of the effectiveness of the method proposed in the present invention

[0312] The following five advanced deep reinforcement learning algorithms and optimization methods are compared to demonstrate the performance of the R-SAC algorithm proposed in the present invention in terms of training performance, economy, safety, and solution efficiency in solving the HEES-RD problem.

[0313] The network structure and parameters of the R-SAC algorithm proposed in the present invention are shown in Table 2;

[0314] The C-SAC algorithm based on the CMDP framework adopts the soft Q function based only on the expected cost as the safety criterion of the learning constraint. In addition, the network structure and parameter settings of C-SAC are exactly the same as those of R-SAC.

[0315] The SAC algorithm based on the MDP framework is optimized by integrating the penalty function into the reward function. Except for the SCN and LM modules, its network structure and parameters are consistent with the R-SAC algorithm.

[0316] The scenario-based approach (SA) handles uncertainty by sampling scenarios and linearizes the HEES-RD model using piecewise linearization and the Big M method;

[0317] Chance constrained programming (CCP) models risk constraints separately from a probability perspective, uses the sample average approximation method to deal with uncertainty, and the confidence level of constraint satisfaction is set to 0.95.

[0318] Table 2 R-SAC neural network architecture and parameters of IEEE-118 test system

[0319]

[0320] 7.1) Training Performance Analysis

[0321] In order to verify the convergence of the proposed R-SAC algorithm and its safe exploration ability, the key steps in the training process of the present invention are shown in Figure 6 . Figure 7 The convergence performance of this algorithm in terms of reward and cost is compared with that of advanced deep reinforcement learning algorithms C-SAC and SAC. The detailed training performance indicators are shown in Table 3.

[0322] Figure 6 The average reward and cost changes for each iteration during the training process are presented, clearly demonstrating the excellent stable convergence characteristics of the R-SAC algorithm. Figure 6 The cost rate indicator of the entire training process is depicted. This indicator is the average cost of each stage and intuitively shows the efficiency of safe exploration. The rapid decline in the cost rate in the figure means that it performs well in the constraint learning of the HEES-RD problem. Figure 6 Shown is the change of the loss function of the actor network over the training epochs. Figure 6 The losses of the two reward-evaluation networks are plotted separately. Figure 6 Evaluate the loss of the network's displayed Q-values ​​around the cost value. Figure 6 We focus on the loss of the second-order center distance of the crossing cost. It can be seen that the RCN and SCN networks converged at about the 800th round, showing a very high convergence efficiency of the evaluation network. Figure 6 The change of entropy in R-SAC is shown. The convergence process of entropy value indicates that the intelligent agent is exploring the global optimal strategy and is safe.

[0323] Table 3 Indicators of various deep reinforcement learning algorithms in the training process

[0324]

[0325] Figure 4(a)-(b) and Table 3 show that compared with C-SAC and SAC, the R-SAC algorithm has the fastest reward increase, achieving convergence in only 3694 rounds, and its convergence efficiency has increased by more than 45.03%. Moreover, R-SAC outperforms other deep reinforcement learning algorithms in the convergence values ​​of the final reward and cost, showing the best comprehensive operation cost and the lowest constraint violation risk in the HEES-RD problem. This outstanding performance is attributed to the proposed risk-averse training mechanism based on the CVaR safety criterion, which puts forward stricter requirements on constraint learning, thereby achieving more effective safety exploration and risk avoidance performance during the training process. In contrast, the expectation-based safety criterion adopted by C-SAC leads to a conservative and less robust constraint learning method. SAC lacks an effective learning operation constraint mechanism and only relies on avoiding high penalties, which results in low training efficiency and is prone to falling into local optimality. The above research results show that the proposed R-SAC algorithm shows excellent convergence performance and safety exploration capabilities on the HEES-RD problem. This conclusion provides strong technical support for the optimal scheduling of the electric-hydrogen coupled system, and also broadens the thinking for the application of deep reinforcement learning in complex systems.

[0326] 7.2) Economic analysis

[0327] This section evaluates the economic efficiency of HEES-RD strategies formulated by different methods by comparing their comprehensive operating costs. The specific data are shown in Table 4. The proposed R-SAC outperforms C-SAC and SAC in terms of operating costs, mainly due to its superior constraint satisfaction ability, which avoids the huge penalties caused by constraint violations. Although the operating cost of R-SAC is 0.79% higher than that of SA, this is mainly due to its more effective use of hybrid electric hydrogen energy storage and backup capacity to cope with uncertainty. Subsequent analysis will show that this slight increase in cost has a positive impact on the rationality of safety and scheduling strategies. At the same time, CCP incurs higher costs than R-SAC. In fact, this depends on the tail distribution of uncertainty. This shows that in the distribution in this specific case, the tail probability of controlling constraint violations is more expensive than the risk of managing violations.

[0328] Table 4 Running costs of various algorithms

[0329]

[0330] 7.3) Safety Analysis

[0331] This section compares the risk resistance of various methods in scheduling strategies under uncertainty. To quantify the operational risk of HEES, we conduct a posteriori tests on the scheduling strategies of each method under uncertainty scenarios. In addition, we introduce the overload probability δ P , expected overload δ E, and the maximum overload level δ M The three risk assessment indicators are shown in formulas (61)-(63) respectively. Figure 8 The results of the a posteriori tests of the strategies developed by different methods are shown in Table 5. The results of the risk assessment indicators are detailed in Table 5.

[0332] δ P =n v / (N s ×T)×100% (61)

[0333]

[0334] Where n v N is the number of times the branch exceeds the limit; s is the number of random scenes.

[0335] Table 5 Risk metrics for each algorithm

[0336]

[0337] Depend on Figure 8 It can be seen that the load factors of all methods are greater than 1, which indicates that high penetration of renewable energy increases the risk of branch overload. Compared with C-SAC and SAC, the overall distribution range of the load factor of R-SAC (Q0~Q4) is [0.28~1.16], which is significantly closer to the safety boundary. As shown in Table 5, the risk assessment index of R-SAC is significantly lower, indicating that the scheduling strategy it formulates has higher safety. This advantage comes from the proposed R-CMDP safety paradigm, which adopts a more robust risk-averse constraint learning mechanism, enhances the constraint satisfaction ability, and ensures the safety and feasibility of the risk scheduling strategy.

[0338] The load rate distribution of SA is relatively compact, with an overall distribution range of [0.48~1.23]. Although SA effectively handles uncertainty through multi-scenario expectations, its overload probability δ P is 6.95%, the expected overload δ E is 0.082, the maximum overload degree δ M is 0.281, which are all higher than the corresponding indicators of R-SAC (5.08%, 0.064 and 0.158, respectively). This shows that the proposed R-SAC method develops a safer and more reliable scheduling strategy to cope with the uncertainty brought by high penetration of renewable energy. This is because the limited number of scenarios of SA fails to fully capture the distribution information required to hedge uncertainty. In contrast, R-SAC interacts with the uncertain environment through the process of "state perception-decision-making-posteriori simulation-feedback learning", thereby fully perceiving uncertainty and developing strong risk resistance.

[0339] CCP focuses on the probabilistic satisfaction of operating constraints, and its main load rate distribution range is [0.25~0.97]. Compared with R-SAC, its over-limit probability is lower, only 3.42%. However, CCP has a large number of high-severity outliers (exceeding the upper neighbor value Q4), and the expected overload δ E is 0.181, the maximum overload degree δ M The R-SAC has a higher probability of exceeding the limit than the high-risk CCP, but it has a significantly superior risk resistance and can reliably handle extreme scenarios.

[0340] 7.4) Analysis of the rationality of scheduling strategy

[0341] This section evaluates the rationality of the scheduling strategy for the HEES-RD problem by comparing the commonly used risk-based scheduling method SA with the proposed R-SAC algorithm. Fig. 9 The scheduling strategies formulated by R-SAC and SA are shown. Fig.10 The status changes of hydrogen storage tanks and batteries, as well as the average operating efficiency of electrolyzers under different scheduling strategies are shown. For visualization purposes, the hydrogen quality has been converted to electrical power using the higher heating value (HHVH).

[0342] Fig. 9 It shows that the overall trends of the strategies formulated by R-SAC and SA are similar. During periods of excess renewable energy, excess electricity is absorbed through electrolyzer hydrogen production and battery charging; and during peak load periods, such as noon and evening, power shortages are alleviated through fuel cell and battery discharge. This synergy of electricity and hydrogen energy storage ensures the reliability of power supply and increases the absorption rate of renewable energy, proving the feasibility of R-SAC in hybrid energy storage to fill snow peaks. Compared with SA, R-SAC starts charging earlier and for a longer period of time before the load peak, such as Fig.10 The change in battery state of charge during the 3:00-9:00 and 17:00 periods is shown in Figure 2. During the peak load periods of 10:00-12:00 and 19:00-20:00, R-SAC achieves a higher net discharge from the hybrid electric-hydrogen energy storage system than SA, thereby reducing the output required from the thermal power units and releasing additional spinning reserves to cope with uncertainty. Fig. 9 The lower power consumption of the electrolyzer during these periods and Fig.10 As a trade-off, the electrolyzer under R-SAC operates less efficiently during these peak hours, e.g. Fig.10 The time periods are shown as 10:00-12:00 and 19:00-20:00.

[0343] Based on the above economic and safety analysis, R-SAC is more inclined to safety in the trade-off between economic efficiency and safety, and has developed a reasonable and safe dispatch strategy with its strong uncertainty hedging ability. These research results show that R-SAC is able to formulate a dispatch strategy that meets load demand, and at the same time, in comparison with SA, it makes better use of the hybrid electric hydrogen energy storage system to deal with uncertainty.

[0344] 7.5) Computational efficiency analysis

[0345] This section evaluates the computational efficiency of the proposed R-SAC algorithm by comparing the solution time, and the relevant results are shown in Table 6. The results show that for complex HEES-RD problems, optimization-based methods such as SA and CCP take hours to complete the solution. In contrast, R-SAC completed the solution in only 0.412 seconds, and the computational efficiency was improved by more than 99.994% (i.e., 15907 times). This significant difference stems from the high dimensionality, non-convexity, and nonlinear probabilistic optimization characteristics of the HEES-RD problem itself. Existing commercial optimization solvers are limited by the requirements of linearization, convexification, and scenario discretization, and introduce a large number of auxiliary variables, resulting in a long solution time. The proposed R-SAC method formulates the scheduling strategy through the forward propagation of the neural network, replacing the traditional modeling and iterative optimization process. Therefore, R-SAC significantly improves the efficiency of risk management while ensuring the feasibility and safety of the HEES scheduling strategy.

[0346] Table 6 Solution time of each algorithm

[0347]

Claims

1. A risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning, characterized in that: The following steps are involved: 1) Considering the Faraday efficiency loss effect of the electrolyzer, a dynamic operation model of electric-hydrogen hybrid energy storage is constructed; 2) Taking into account the uncertainty of wind power, a risk dispatch model for the electric-hydrogen coupling system taking into account the dynamic operation of electric-hydrogen hybrid energy storage is established; 3) Construct a robust constrained Markov decision framework; 4) Parameterize the robust constrained Markov decision architecture using the flexible actor-critic baseline algorithm; 5) The parameterized robust constrained Markov decision process architecture is used to solve the risk scheduling model of the electric-hydrogen coupling system and obtain the risk scheduling plan of the electric-hydrogen coupling system.

2. According to claim 1, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: The dynamic operation model of electric-hydrogen hybrid energy storage is as follows: Where, HHV H is the higher calorific value of hydrogen under standard conditions; η e,t The operating efficiency of electric-hydrogen hybrid energy storage; The power consumption and hydrogen production of the electric-hydrogen hybrid energy storage electrolyzer; Among them, the power consumption of the electrolytic cell and hydrogen production As shown below: P t e =N E V cell,t I cell,t (2) In the formula, I cell,t is the operating current of the electrolytic cell; N E is the number of electrolytic cells in series, which is related to the maximum capacity of the electrolytic cell; k1 represents the conversion coefficient; η f,t is the Faraday efficiency; F is the Faraday constant; Electric-hydrogen hybrid energy storage operating voltage V cell,t As shown below: Where t is the scheduling interval; V rev is the reversible voltage; T is the operating temperature; R and F are the rational gas constant and Faraday constant respectively; and are the partial pressures of hydrogen and oxygen, respectively; is the activity of H2O; R cell is the internal equivalent ohmic resistance; α an and α cat are the anode and cathode charge transfer coefficients respectively; i an and i cat are the anode and cathode exchange current densities, respectively; i cell,t is the current density; Faraday efficiency η f,t As shown below: In the formula, is the oxygen permeability in the diffusion mechanism; is the hydrogen permeability in the diffusion mechanism; are the cathode hydrogen partial pressure and the anode oxygen partial pressure in the catalyst layer, respectively; A H2 ,A O2 are the fitting parameters for the increase in hydrogen and oxygen partial pressures in the cathode catalyst layer, respectively; is the hydrogen permeability caused by pressure difference.

3. According to claim 1, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: The risk dispatch model of the electric-hydrogen coupling system is as follows: st h(x,y,ζ)=0 (9) Ax+Dy+Mζ+e≤0 (10) g(x,y,ζ)≤0 (11) u(x,y,ζ)≤0 (12) Where, the superscript t represents the time period; T is the scheduling period; the vector x represents the decision variables including the scheduling strategy; the vector y represents other state variables in the HEES; the vector ζ represents the uncertain prediction error; A, D and M are coefficient matrices, respectively, and e is the coefficient vector; the subscript l represents the lth branch; l represents the branch set; the symbol '-' represents the upper limit of each variable; (·)+ represents max{·,0}; Represents the branch overload degree; d B is the risk threshold of branch overload; Pr{x}=1-F(x) represents the probability of x, F() is the cumulative distribution function; P l,t Indicates branch flow; d B represents the risk threshold; f t (x) represents the operating cost function; h(x, y, ζ) = 0 represents the AC power flow equation and the energy balance equation.

4. According to claim 1, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: Robust Constrained Markov Decision Architecture Including action network, reward judgment network, safety judgment network, Lagrange multiplier, and target network; represents the state space, represents the action space, is the state transfer function, r represents the state s t Next, perform action a t The reward obtained, c represents the reward in state s t Next, perform action a t The obtained safe exploration cost, d is the crossing threshold, γ is the discount factor; γ∈[0,1]; The action network encapsulates the action strategy distribution π with the neural network parameter θ, and establishes a mapping from the state space to the action space The reward judgment network is used to evaluate the operating status of HEES t Next regulatory decision t The reward soft Q function (s t ,a t ); Safety evaluation network is used to evaluate the operating status of HEES t Next regulatory decision t The security assessment function Γ π (s t ,a t ,β); Lagrange multipliers are used to adjust the dual multipliers related to entropy and cross-limit cost weights in the dual update process; The target network is used to enhance the training stability of the reward value discrimination network and the cost value discrimination network modules.

5. According to claim 4, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: The state space As shown below: Where P L,t 、m H2,t , P W,t , P G,t-1 , E B,t-1 They represent the electric load prediction vector, hydrogen load prediction vector, new energy prediction value, the first-order derivative of the new energy prediction value with respect to time, the thermal power output vector at the last moment, the hydrogen storage capacity of the hydrogen storage tank at the last moment, and the charge state of the electric energy storage at the last moment respectively; the subscripts L, W, G, H, and B represent the load, wind farm, thermal power unit, hydrogen storage tank, and battery set respectively; Among them, the first-order derivative of the new energy prediction value with respect to time is as follows: Where Δt is the time difference; P W,t-1 represents the predicted value of new energy at time t-1; The action space As shown below: ΔP G,t =a G,t Θ[-T 60 D G ,T 60 U G ] (18) rd G,t =a D,t Θ[0,min(P G,t -P G ,T 10 D G )] (20) In the formula, the subscripts E and FC represent the electrolyzer set, fuel cell set, and energy storage device set, respectively. The subscript '_' represents the lower limit of the corresponding variable; represent the energy storage charging / discharging power respectively, which are combined into a decision P B,t To reduce the action space; U G With D G Respectively represent the climbing rate and descending rate of the unit. 60 With T 10 Refers to time intervals of one hour and ten minutes respectively; a G,t 、a U,t 、a D,t 、a E,t 、a F,t 、a B,t 、a H,t They represent the output increment action of thermal power unit, the upper rotation standby action of thermal power unit, the lower rotation standby action, the operation current density action of electrolyzer, the power generation power action of fuel cell, the charge / discharge power action of electric energy storage, and the hydrogen output action of hydrogen storage tank respectively; G,t ,rd G,t Respectively represent the upper and lower spinning reserve capacities; i cell,E,t The current density that represents the operating limit of the electrolytic cell; P B,t Indicates the power of electrical energy storage.

6. According to claim 4, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: The state transition function of the robust constrained Markov decision architecture is as follows: P G,t =P G,t-1 +ΔP G,t (25) In the formula, Represents the hydrogen inlet vector of the hydrogen storage tank; η H,in , η H,out The efficiency vectors of hydrogen input and output of the hydrogen storage tank respectively; η cha,B , η dis,B Efficiency vectors of charging and discharging of energy storage respectively.

7. According to claim 4, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: The cost function c π,ρ (s t ,a t ,ζ) is as follows: c π,ρ (s t ,a t ,g)=[e e c e,t +e b c b,t +e pf c pf,t +e pb c pb,t +e ph c ph,t ]|π,ρ (31) Where π represents the distribution of agent control decision strategies; ε e , ε b , ε pf , ε pb With ε ph Respectively represent scaling; c e,t Indicates that the capacity safety constraint of the hydrogen storage tank exceeds the limit; c b,t Indicates that the safety constraint of the electric energy storage capacity exceeds the limit; c pf,t Indicates that the branch overload risk constraint exceeds the limit; c pb,t Indicates the unbalanced amount of HEES electric power; c ph,t Indicates the hydrogen energy imbalance; P e,t Represents the power consumption of the e-th PEM electrolyzer at time t. The reward function is as follows: r t =-(f t +f C,t )=-(f G,t +f Q,t +f L,t +f HESS,t +f C,t ) (37) f HESS,t =λ h (f HS,om,t +f HS,loss,t +f B,om,t +f B,loss,t ) (41) f C,t =λ c (c π,ρ (s t ,a t ,g)-d) + (42) In the formula, f G,t represents the operating cost of thermal power; f Q,t 、f L,t They represent the penalty items for abandoning wind and solar power and shedding load respectively; f HESS,t represents the operating cost of electric-hydrogen hybrid energy storage; f C,t represents the penalty fee for over-limit constraint; a g 、b g 、c g 、u g d g The unit costs of constant, primary, secondary, upward spinning reserve and downward spinning reserve of thermal power units are respectively indicated; λ g , q , l , h , c represents the cost coefficient; represents the power imbalance caused by uncertainty at the i-th node; f HS,om,t 、f B,om,t are the operation and maintenance costs of hydrogen energy storage and electric energy storage respectively; f HS,loss,t 、f B,loss,t are the energy consumption costs of hydrogen energy storage and electric energy storage respectively; C e,inv , C fc,inv are the investment and construction costs of the electrolyzer e and fuel cell fc respectively; T e,life 、T fc,life are the maximum operating time of the electrolyzer e and the fuel cell fc respectively; B,om is the unit operation and maintenance cost coefficient of electric energy storage; λ ep The transmission and distribution price.

8. According to claim 4, a risk scheduling method for an electric-hydrogen coupling system based on robust and safe deep reinforcement learning is characterized in that: Security Assessment Function Γ π (s t ,a t ,β) is as follows: Where φ(·) and Φ(·) represent the probability density distribution function and cumulative probability distribution function of the standard normal distribution, respectively; F π,ρ Represents C π,ρ The probability density function of Among them, C π The second central moment of As shown below:

9. The method for risk scheduling of an electric-hydrogen coupling system based on robust and safe deep reinforcement learning according to claim 1 is characterized in that: When parameterizing the robust constrained Markov decision architecture using the flexible actor-critic baseline algorithm, the objective function and constraints are as follows: In the formula, Represents the entropy threshold.

10. The method for risk scheduling of an electric-hydrogen coupling system based on robust and safe deep reinforcement learning according to claim 1 is characterized in that: Loss function of the safety evaluation network when parameterizing the robust constrained Markov decision architecture As shown below: In the formula, Indicates experience playback; The loss function of the reward judgment network is as follows: The loss function J of the action network π (θ) is as follows: The update equation of the Lagrange multiplier is as follows: Where α and κ are Lagrange multipliers; ω is the learning rate.

Images