Wind, light, electricity, heat, cold air and hydrogen optimization scheduling method for off-grid zero-carbon park

Abstract

The invention discloses a wind, light, electricity, heat, cold air and hydrogen optimal scheduling method for an off-grid zero-carbon park, and the method comprises the steps: firstly, building an off-grid zero-carbon park system model, the system comprises energy supply equipment, energy conversion equipment and energy storage equipment, the energy source is renewable energy wind energy and light energy, and the load types comprise an electric load, a heat load and a cold load; establishing a mathematical model and operation constraint conditions according to the system model, and establishing an optimization objective function for the system; describing an optimized scheduling reinforcement learning framework of the off-network zero-carbon park system as a Markov decision process; on the basis of an improved flexible actor-commentator algorithm, advantage learning, transfer learning and an SAC algorithm are combined to solve the model. The advantages of multi-energy complementation are fully utilized, and the economical efficiency and the environmental protection property of the system are improved; and an improved flexible actor-commentator algorithm is adopted, so that the training time is shortened, the algorithm iteration speed is accelerated, and the flexible and efficient scheduling of the integrated energy system is realized.

Description

Technical Field

[0001] The present invention belongs to the field of new energy technology, and in particular relates to a method for optimizing the dispatching of wind, solar, thermal, cooling and hydrogen for an off-grid zero-carbon park. Background Art

[0002] Integrated energy systems (IES) usually include multiple types of energy such as electricity, heat, cold, gas and hydrogen. Through energy conversion equipment such as electricity to hydrogen, hydrogen to gas, electric cooling and heating, and heat to cold, the mutual coupling and conversion utilization of heterogeneous energy sources can be achieved. While meeting the diverse load demands of energy users, formulating reasonable and efficient collaborative operation and scheduling strategies is of great significance to ensuring the reliable and stable operation of the system and improving the system's economy and low-carbon environmental protection.

[0003] At present, the traditional method for studying the coordinated operation and optimal scheduling of systems is to use optimization methods in mathematics, such as nonlinear programming, second-order cone programming, and mixed integer programming, and then use mature commercial solvers such as Yalmip, Cplex, Gurobi, and Mosek to solve the problem directly. However, as the scale of the problem increases, the calculation time will increase sharply, and it may even become difficult to solve large-scale problems.

[0004] Currently, many researchers have applied intelligent optimization algorithms such as particle swarm optimization (PSO), genetic algorithm (GA) and simulated annealing algorithm (SA) to solve multi-objective problems in integrated energy systems. The quality of the solution results of these algorithms is closely related to whether the static parameters are reasonably set. They are very dependent on the user's experience. At the same time, they are prone to fall into local optimal solutions when solving multi-objective optimization problems, which leads to the scheduling results that cannot guarantee the efficient, stable and economical operation of the system.

[0005] With the introduction of various energy conversion devices into the integrated energy system and the diversification of back-end requirements, the structure and coupling relationship of the system have become more complex. For such a complex, nonlinear, and high-dimensional system, the agent in deep reinforcement learning (DRL) continuously learns to obtain the optimal scheduling strategy by taking a series of actions in the environment, and the environment feedbacks reward signals based on the agent's actions. The deep Q network algorithm (DQN) can only handle discrete action spaces, and it is prone to dimensionality explosion problems for complex integrated energy systems; although the deep deterministic policy gradient algorithm (DDPG) can handle continuous action spaces, it also has problems such as overestimation, unstable policy network updates, and insufficient exploration; the double-delayed deep deterministic policy gradient algorithm (TD3) has shown good performance in handling problems in continuous action spaces, but it is very sensitive to the choice of hyperparameters, and coupled with a more complex model, this will result in huge computer resources consumption. Summary of the invention

[0006] In view of the deficiencies in the prior art, the present invention provides an optimized scheduling method for wind, solar, thermal, cooling and hydrogen in an off-grid zero-carbon park, in the hope that when the source and load fluctuate and electricity and gas cannot be purchased from the outside, the demand on the load side can still be met as much as possible, ultimately achieving stable and economical operation of the system.

[0007] The invention discloses an off-grid zero-carbon park wind, solar, thermal, cold and hydrogen optimization scheduling method, comprising the following steps:

[0008] Step 1: Establish an off-grid zero-carbon park system model, which includes energy supply equipment, energy conversion equipment, and energy storage equipment. The energy source is renewable energy wind energy and light energy, and the load types include electrical load, thermal load, and cooling load.

[0009] The energy supply equipment includes wind turbines WT and photovoltaic panels PV; the energy conversion equipment includes the combined cooling, heating and power system CCHP, gas boiler GB, electrolyzer EL, fuel cell HFC, methane reactor MR, carbon capture equipment CCUS, electric refrigerator EC and electric boiler EB; the energy storage equipment includes batteries BAT, hydrogen storage tanks HST, heat storage devices HES and cold storage devices CES.

[0010] The combined cooling, heating and power system consists of a gas turbine GT, a waste heat recovery device WHR, an absorption chiller AC and a waste heat boiler WHB.

[0011] Step 1.1: Based on the above off-grid zero-carbon park system model, construct a mathematical model and operating constraints.

[0012] (1) Mathematical model of wind turbine WT and operating constraints.

[0013]

[0014] 0≤P WT (t)≤P WT,max (t) (2)

[0015] Where P WT,max (t) is the maximum power output of the wind turbine generator set at time t; N WT is the number of fans; P WT,rated is the rated power of a single fan; ν w (t) is the wind speed of the wind turbine at the time t at the height of the wind turbine; ν rated is the rated wind speed of the wind turbine; v in is the cut-in wind speed of the wind turbine; v out P is the cut-out wind speed of the wind turbine; WT (t) is the actual output power of the wind turbine generator set at time t.

[0016] (2) PV mathematical model and operating constraints of photovoltaic generator sets.

[0017]

[0018] Where P PV,max (t) is the maximum power output of the photovoltaic generator set at time t; f PV is the attenuation coefficient of the photovoltaic panel; P PV,rated is the rated power of a single photovoltaic panel; S a (t) is the actual light intensity at time t; S stc is the light intensity under standard environment; k PV is the power temperature coefficient; T pv (t) is the actual temperature of the photovoltaic panel surface at time t; T stc The temperature of the photovoltaic panel surface under standard conditions.

[0019] The actual temperature T of the photovoltaic panel surface pv The calculation formula of (t) is:

[0020]

[0021] 0≤P PV (t)≤P PV,max (t) (5)

[0022] Where, T amb is the ambient temperature; NOCT is the nominal operating temperature of the photovoltaic module, provided by the photovoltaic manufacturer; P PV (t) is the actual power output of the photovoltaic generator set at time t.

[0023] (3) Mathematical model and operating constraints of CCHP system.

[0024]

[0025] Where P g,GT (t) is the natural gas power input to GT at time t, i.e., the operating power of GT; P GT,e (t) and P GT,h (t) are the electrical power and thermal power output of GT at time t; η GT is the energy conversion efficiency of GT; and They represent the upper and lower limits of the adjustable thermal-to-electric ratio of GT respectively; and The upper and lower limits of the natural gas power input to the GT unit; and These are the upper and lower limits of the GT unit's climbing power.

[0026]

[0027] In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6 J; M is the calorific value of natural gas, which indicates the heat released by the combustion of unit volume of natural gas; g,GT (t) is the mass of natural gas consumed by the GT unit at time t; The mass of CO2 produced by complete combustion of unit mass of natural gas; is the mass of carbon dioxide produced by the GT unit at time t.

[0028]

[0029] Where P h,AC (t) is the thermal power input to AC at time t, and the operating power of AC; P AC,c (t) is the cooling power output by AC at time t; The conversion efficiency of AC from heat to cold; and The upper and lower limits of AC operating power; and These are the upper and lower limits of the AC unit's ramp-up power.

[0030]

[0031] Where P h,WHB (t) is the thermal power input to WHB at time t, and the operating power of WHB; P WHB,h (t) is the thermal power output of WHB at time t; is the WHB heat-to-heat conversion efficiency; and The upper and lower limits of WHB operating power; and These are the upper and lower limits of the WHB unit’s climbing power.

[0032] (4) Mathematical model of gas boiler GB and operating constraints.

[0033]

[0034] Where P g,GB (t) is the natural gas power input to GB at time t, and the operating power of GB; P GB,h (t) is the thermal power output by GB at time t; is the conversion efficiency of GB natural gas to heat; and The upper and lower limits of GB operating power; and These are the upper and lower limits of the GB unit’s climbing power.

[0035]

[0036] In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6 J; M is the calorific value of natural gas, which indicates the heat released by the combustion of unit volume of natural gas; g,GB (t) is the volume of natural gas consumed by GB unit at time t; The mass of CO2 produced by complete combustion of unit mass of natural gas; is the mass of carbon dioxide produced by unit GB at time t.

[0037] (5) Mathematical model of EL electrolyzer and operating constraints.

[0038]

[0039] Where P e,EL (t) is the natural gas power input to EL at time t, and the operating power of EL; P EL,H (t) is the hydrogen power output by EL at time t; is the conversion efficiency of EL to hydrogen; and is the upper and lower limits of EL operating power; and These are the upper and lower limits of the EL unit's climbing power.

[0040] (6) Fuel cell HFC mathematical model and operating constraints.

[0041]

[0042] Where P H,HFC (t) is the hydrogen power input to HFC at time t, i.e., the operating power of HFC; P HFC,e (t) and P HFC,h (t) are the electrical power and thermal power output of HFC respectively; η HFC is the energy conversion efficiency of HFC; and They represent the upper and lower limits of the adjustable heat-to-electricity ratio of HFC respectively; and The upper and lower limits of the hydrogen power input to the HFC unit; and These are the upper and lower limits of the HFC unit's ramp-up power.

[0043] (7) Mathematical model of methane reactor MR and operating constraints.

[0044]

[0045] Where P H,MR (t) is the hydrogen power input to MR at time t, and the operating power of MR; P MR,g (t) is the natural gas power output by MR at time t; The conversion efficiency of hydrogen to natural gas for MR production; and is the upper and lower limits of hydrogen power input to the MR unit; and These are the upper and lower limits of the MR unit's climbing power.

[0046]

[0047] In the formula, is the mass of natural gas generated by MR at time t; The mass of carbon dioxide required to produce a unit mass of natural gas; is the mass of carbon dioxide consumed by MR at time t.

[0048] (8) Carbon capture CCUS mathematical model and operating constraints.

[0049]

[0050] Where η CCUS The amount of electricity required to absorb a unit mass of carbon dioxide; P e,CCUS (t) is the mass of carbon dioxide absorbed by CCUS at time t.

[0051] (9) Mathematical model of electric refrigeration EC and operating constraints.

[0052]

[0053] Where P e,EC (t) is the electric power input to the EC at time t, and the operating power of the EC; P EC,c (t) is the cooling power output by the EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of the EC operating power; and These are the upper and lower limits of the EC unit’s climbing power.

[0054] (10) Electric boiler EB mathematical model and operating constraints.

[0055]

[0056] Where P e,EB (t) is the electric power input to EB at time t, and the operating power of EB; P EB,h (t) is the thermal power output of EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of EB operating power; and These are the upper and lower limits of the EC unit’s climbing power.

[0057] (11) General mathematical model of energy storage equipment and operating constraints.

[0058] The operating mechanisms of the three energy storage devices, electric, thermal, and cold, are treated in the same way, so the energy storage devices are modeled uniformly as shown below:

[0059]

[0060] In the formula, n is one of electricity, heat, and cold; and is the charging and discharging power of the energy storage device at time t; and is a binary variable, Indicates that the energy storage device is charged at time t, Indicates the release of energy; SOC ES,n (t) is the final remaining capacity of the energy storage device at time t; and is the charging and discharging efficiency; T is the maximum operating time; is the maximum capacity of the energy storage device; is the initial capacity factor of the energy storage device; in order to reduce the impact of overcharge and overdischarge on the life of the energy storage device, the minimum and maximum capacity factors are set

[0061] (12) Electric power balance constraints.

[0062]

[0063] Where P eLoad (t) is the electric load demand at time t.

[0064] (13) Thermal power balance constraints.

[0065]

[0066] Where P hLoad (t) is the heat load demand at time t.

[0067] (14) Cold power balance constraints.

[0068]

[0069] (15) Hydrogen power balance constraint.

[0070]

[0071] (16) Natural gas power balance constraints.

[0072] P MR,g (t) = P g,GT (t)+P g,GB (t) (24)

[0073] Step 1.2: Establish an optimization objective function for the off-grid zero-carbon park system:

[0074] F=min(C opm )(25)

[0075]

[0076] In the formula, C opm is the total operation and maintenance cost of each device; c opm,i is the unit operation and maintenance cost of the i-th device; P i is the operating power of the ith device.

[0077] Step 2: Describe the off-grid zero-carbon park system optimization scheduling reinforcement learning framework as a Markov decision process:

[0078] Step 2.1: Create state space:

[0079] s t =[v w (t),S a (t),P eLoad (t),P hLoad (t),P cLoad (t),SOC ES,n (t),t](27)

[0080] Step 2.2: Create an action space:

[0081]

[0082] Step 2.2: Create a reward function:

[0083] r t =-C opm (29)

[0084] Step 3: Solve the model based on the improved flexible actor-critic algorithm.

[0085] Specifically, it refers to combining advantage learning with transfer learning and SAC algorithm.

[0086] The objective function expression of the SAC algorithm is as follows:

[0087]

[0088] In the formula, (s q ,a q )~P π is the state action trajectory of strategy π; α is a hyperparameter that determines the influence of entropy on reward; αH(π(·|s t )) is state s t The specific calculation formula is as follows:

[0089]

[0090] The above formula represents the strategy π(·|s t ) corresponds to the randomness of the probability distribution. The more random the strategy, the greater the entropy.

[0091] The SAC algorithm also uses the AC framework, but it has been improved by proposing a soft value function and adding entropy to the state value function and Q value function formulas, as shown below:

[0092]

[0093] Where, T π is the Bellman operator under strategy π, γ is the discount factor of the reward, and the Bellman operator is shown below:

[0094]

[0095] In the formula, is the value function at the kth calculation. The soft value function is iterated through formula (33) and eventually converges to the soft Q value function under the fixed strategy π.

[0096] In order to avoid the situation where the strategy difference is too large and causes the training to be unstable, in the strategy update, the strategy output is Gaussian distribution, and then KL divergence is used to narrow the gap between the new and old strategies. The formula is as follows:

[0097]

[0098] Where D KL is the KL divergence; For the old strategy π old The value function under ; Π is the strategy set; is the allocation function under the old strategy, which can normalize the Q value.

[0099] The parameters of the Critic and Actor neural networks of SAC are updated as follows:

[0100]

[0101] In the formula, θ and φ are the parameters of the Q-value network and the strategy network; and is the updated function, Z θ (S t ) is the allocation function.

[0102] Action entropy is also output in the policy network, where the update of the hyperparameter α is crucial to the entropy, and its update formula is:

[0103]

[0104] Where H0 is the dimension of the action matrix.

[0105] Furthermore, in the estimation of the value function, advantage learning will accelerate the convergence of the algorithm during training by reducing the Q value estimation of non-optimal actions and increasing the probability of the agent selecting the optimal action; the state value function formula of advantage learning is as follows:

[0106]

[0107] In the formula, A * (s,a) is the advantage function, which is defined as follows:

[0108] A * (s,a)=V * (s)-α[V * (s)-Q * (s,a)] (40)

[0109] In the formula, α[V * (s)-Q * (s,a)] is the correction term, Q * When the value of (s,a) is the Q value of the optimal action, the correction term is zero, otherwise it is positive.

[0110] The environmental state s t Input to the policy network, output action a from this network t+1 As the optimal action, a t+1 and status s t Input to the Q value network critic, the output value is regarded as the current optimal state value Q(s t ,a t+1 θ -), then the correction function is rewritten as:

[0111] F(s t ,a t )=α[Q(s t ,a t+1 θ - )-Q(s t ,a t θ - )] (41)

[0112] To prevent the network output Q(s t ,a t θ - ) is greater than Q(s t ,a t θ - ), F(s t ,a t ) is less than zero, then subtract F(s t ,a t ) will increase the Q value and cause overestimation, so add restrictions: when F(s t ,a t )>0, F(s t ,a t ) remains unchanged; F(s t ,a t )<0, F(s t ,a t ) has a value of 0.

[0113] The stability of the neural network also needs to be judged, so the rate of decrease of the network loss function is used as the basis for judging whether to start advantage learning; the formula for the rate of decrease of the network loss function is as follows:

[0114]

[0115] Where, L oss (t) is the neural network loss function, k is the descent rate; if k is less than the threshold set for starting advantage learning, advantage learning will not be started; otherwise, advantage learning will be started.

[0116] The target Q value formula is changed to:

[0117]

[0118] When dealing with different tasks in the same environment, a knowledge transfer strategy is adopted to maximize the use of existing knowledge, thereby improving the efficiency of deep reinforcement learning.

[0119] In actual operation, the neural network weights of the target network are initialized with the neural network weights updated by historical data training, instead of the random weight initialization of the neural network.

[0120] The present invention has the following beneficial technical effects:

[0121] The present invention mainly considers the off-grid integrated energy system in remote areas. In view of the volatility of renewable energy and load and the low-carbon and environmental protection issues, the integrated energy system including multiple energy sources such as electricity, heat, cooling and hydrogen is taken as the research object. By introducing two sets of cogeneration systems, CCHP and HFC, the advantages of multi-energy complementarity are fully utilized to improve the economy and environmental protection of the system; the improved flexible actor-critic algorithm is adopted to effectively avoid the "dimensionality curse" problem of the action space, further reduce the training time and speed up the algorithm iteration speed, so as to realize flexible and efficient scheduling of the integrated energy system. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Figure 1 This is a system topology diagram of the off-grid zero-carbon park of the present invention. DETAILED DESCRIPTION

[0123] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0124] The invention discloses an off-grid zero-carbon park wind, solar, thermal, cold and hydrogen optimization scheduling method, comprising the following steps:

[0125] Step 1: Establish an off-grid zero-carbon park system model. Figure 1 As shown, it includes energy supply equipment, energy conversion equipment, and energy storage equipment. The energy sources are renewable energy such as wind energy and light energy. The load types include electrical load, thermal load, and cooling load.

[0126] Energy supply equipment includes wind turbines (WT) and photovoltaic panels (PV); energy conversion equipment includes combined cooling, heating and power systems (CCHP), gas boilers (GB), electrolyzers (EL), fuel cells (HFC), methane reactors (MR), carbon capture equipment (CCUS), electric refrigerators (EC) and electric boilers (EB); energy storage equipment includes batteries (BAT), hydrogen storage tanks (HST), heat storage devices (HES) and cold storage devices (CES).

[0127] The combined heat, cooling and power system consists of a gas turbine (GT), a waste heat recovery device (WHR), an absorption chiller (AC) and a waste heat boiler (WHB).

[0128] Since the research object of this invention is mainly aimed at remote areas, the situation of buying electricity from the power grid and buying natural gas from the gas grid is not considered.

[0129] Step 1.1: Based on the above off-grid zero-carbon park system model, construct a mathematical model and operating constraints.

[0130] (1) Mathematical model of wind turbine (WT) and operating constraints.

[0131] A wind turbine (WT) is a device that converts wind energy into electrical energy. Its principle is to receive wind energy through the wind rotor blades, causing the wind rotor to rotate, and then drive the generator to rotate through transmission devices such as a speed increase gearbox, thereby converting wind energy into electrical energy. The amount of power generated depends on the local wind speed conditions.

[0132]

[0133] 0≤P WT (t)≤P WT,max (t) (2)

[0134] Where P WT,max (t) is the maximum power output of the wind turbine generator set at time t; N WT is the number of fans; P WT,rated is the rated power of a single fan; ν w (t) is the wind speed of the wind turbine at the time t at the height of the wind turbine; ν rated is the rated wind speed of the wind turbine; ν in is the cut-in wind speed of the wind turbine; out P is the cut-out wind speed of the wind turbine; WT (t) is the actual output power of the wind turbine generator set at time t.

[0135] (2) Mathematical model of photovoltaic generator (PV) and operating constraints.

[0136] Photovoltaic panel generators (PV) are mainly power generation equipment composed of photovoltaic panels. Its principle is based on the photoelectric effect. When sunlight shines on photovoltaic panels, photons interact with electrons in semiconductor materials, causing electrons to jump, thereby generating direct current. Its output power is mainly affected by factors such as light intensity, sunshine duration, temperature, and shadow.

[0137]

[0138] Where P PV,max (t) is the maximum power output of the photovoltaic generator set at time t; f PV is the attenuation coefficient of the photovoltaic panel; P PV,rated is the rated power of a single photovoltaic panel; S a (t) is the actual light intensity at time t; S stcis the light intensity under standard environment; k PV is the power temperature coefficient; T pv (t) is the actual temperature of the photovoltaic panel surface at time t; T stc The temperature of the photovoltaic panel surface under standard conditions.

[0139] The actual temperature T of the photovoltaic panel surface pv The calculation formula of (t) is:

[0140]

[0141] 0≤P PV (t)≤P PV,max (t) (5)

[0142] Where, T amb is the ambient temperature; NOCT is the nominal operating temperature of the photovoltaic module, provided by the photovoltaic manufacturer; P PV (t) is the actual power output of the photovoltaic generator set at time t.

[0143] (3) Mathematical model and operating constraints of the combined cooling, heating and power (CCHP) system.

[0144] Combined cooling, heating and power (CCHP) system is an energy conversion device. Its principle is to use the high-temperature gas generated by the combustion of gas turbines to generate electricity, and at the same time recover the waste heat discharged by the gas turbine. The waste heat is used for heating through the waste heat recovery device and waste heat boiler, and the waste heat is used for cooling using the absorption refrigerator, thereby realizing the combined supply of three energy forms: cold, heat and electricity.

[0145]

[0146] Where P g,GT (t) is the natural gas power input to GT at time t, i.e., the operating power of GT; P GT,e (t) and P GT,h (t) are the electrical power and thermal power output of GT at time t; η GT is the energy conversion efficiency of GT; and They represent the upper and lower limits of the adjustable thermal-to-electric ratio of GT respectively; and The upper and lower limits of the natural gas power input to the GT unit; and These are the upper and lower limits of the GT unit's climbing power.

[0147]

[0148] In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6J; is the calorific value of natural gas (J / Kg), which indicates the heat released by the combustion of unit volume of natural gas; M g,GT (t) is the mass of natural gas consumed by the GT unit at time t (Kg); The mass of CO2 produced by complete combustion of unit mass of natural gas; is the mass of carbon dioxide produced by the GT unit at time t.

[0149]

[0150] Where P h,AC (t) is the thermal power input to AC at time t, and the operating power of AC; P AC,c (t) is the cooling power output by AC at time t; The conversion efficiency of AC from heat to cold; and The upper and lower limits of AC operating power; and These are the upper and lower limits of the AC unit's ramp-up power.

[0151]

[0152] Where P h,WHB (t) is the thermal power input to WHB at time t, and the operating power of WHB; P WHB,h (t) is the thermal power output of WHB at time t; is the WHB heat-to-heat conversion efficiency; and The upper and lower limits of WHB operating power; and These are the upper and lower limits of the WHB unit’s climbing power.

[0153] (4) Mathematical model of gas boiler (GB) and operating constraints.

[0154] Gas boiler (GB) is an energy conversion device. The principle is to convert chemical energy into thermal energy by burning natural gas and other fuels, and use the generated heat to heat water into steam or hot water. Its characteristics are high thermal efficiency, and it can quickly and stably provide a large amount of thermal energy, which can effectively meet the heat load demand.

[0155]

[0156] Where P g,GB (t) is the natural gas power input to GB at time t, and the operating power of GB; P GB,h (t) is the thermal power output by GB at time t; is the conversion efficiency of GB natural gas to heat; and The upper and lower limits of GB operating power; and These are the upper and lower limits of the GB unit’s climbing power.

[0157]

[0158] In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6 J; is the calorific value of natural gas (J / Kg), which indicates the heat released by the combustion of unit volume of natural gas; M g,GB (t) is the volume of natural gas consumed by GB unit at time t; The mass of CO2 produced by complete combustion of unit mass of natural gas (Kg); is the mass of carbon dioxide produced by unit GB at time t.

[0159] (5) Electrolyzer (EL) mathematical model and operating constraints.

[0160] An electrolyzer (EL) uses electrical energy to decompose electrolytes such as water into hydrogen and oxygen, thereby converting electrical energy into chemical energy. Its characteristic is that it can convert excess electricity generated by renewable energy into hydrogen energy storage, effectively balancing energy supply and demand. Its operating efficiency is affected by many factors such as electrolyte concentration, temperature, electrode materials, etc., and the equipment cost and maintenance cost are relatively high.

[0161]

[0162] Where P e,EL (t) is the natural gas power input to EL at time t, and the operating power of EL; P EL,H (t) is the hydrogen power output by EL at time t; is the conversion efficiency of EL to hydrogen; and is the upper and lower limits of EL operating power; and These are the upper and lower limits of the EL unit's climbing power.

[0163] (6) Fuel cell (HFC) mathematical model and operating constraints.

[0164] Fuel cells (HFC) convert the chemical energy of hydrogen directly into electrical energy through an electrochemical reaction between hydrogen and oxygen. Only water and a small amount of heat are produced during the reaction. Its characteristics are high energy conversion efficiency, which has great advantages over traditional power generation methods. It has fast startup, low noise, and a wide power generation range, which can flexibly meet different electrical load requirements.

[0165]

[0166] Where P H,HFC (t) is the hydrogen power input to HFC at time t, i.e., the operating power of HFC; P HFC,e (t) and P HFC,h (t) are the electrical power and thermal power output of HFC respectively; η HFC is the energy conversion efficiency of HFC; and They represent the upper and lower limits of the adjustable heat-to-electricity ratio of HFC respectively; and The upper and lower limits of the hydrogen power input to the HFC unit; and These are the upper and lower limits of the HFC unit's ramp-up power.

[0167] (7) Mathematical model of methane reactor (MR) and operating constraints.

[0168]

[0169] Where P H,MR (t) is the hydrogen power input to MR at time t, and the operating power of MR; P MR,g (t) is the natural gas power output by MR at time t; The conversion efficiency of hydrogen to natural gas for MR production; and is the upper and lower limits of hydrogen power input to the MR unit; and These are the upper and lower limits of the MR unit's climbing power.

[0170]

[0171] In the formula, is the mass of natural gas generated by MR at time t; The mass of carbon dioxide required to produce a unit mass of natural gas; is the mass of carbon dioxide consumed by MR at time t.

[0172] (8) Carbon capture (CCUS) mathematical model and operating constraints.

[0173] Carbon capture equipment is a key device in the energy conversion process. The principle is to use physical or chemical methods to separate and capture carbon dioxide from waste gas generated in industrial production or energy utilization. Its characteristic is that it can effectively reduce the emission of carbon dioxide into the atmosphere and help the park achieve its zero-carbon goal.

[0174]

[0175] Where ηCCUS The amount of electricity required to absorb a unit mass of carbon dioxide; P e,CCUS (t) is the mass of carbon dioxide absorbed by CCUS at time t.

[0176] (9) Electric refrigeration (EC) mathematical model and operating constraints.

[0177] Electric refrigerators use electricity to drive compressors and other components, so that the refrigerant circulates in the system and converts electricity into cold energy. They have high refrigeration efficiency and can quickly and accurately adjust the cooling capacity to meet the demand for cooling load. However, the refrigeration system needs to be maintained regularly, such as replenishing refrigerant and checking components such as compressors to ensure its normal operation.

[0178]

[0179] Where P e,EC (t) is the electric power input to the EC at time t, and the operating power of the EC; P EC,c (t) is the cooling power output by the EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of the EC operating power; and These are the upper and lower limits of the EC unit’s climbing power.

[0180] (10) Electric boiler (EB) mathematical model and operating constraints.

[0181] Electric boiler (EB) converts electrical energy into thermal energy, and heats water through internal heating elements (such as resistance wire) to turn water into hot water or steam. It is pollution-free and does not produce combustion exhaust gas like traditional oil and gas boilers; it has high thermal efficiency, fast heating speed, and can control temperature more accurately; but its operation depends on sufficient power supply, and its energy consumption is relatively large, so the energy cost may be relatively high.

[0182]

[0183] Where P e,EB (t) is the electric power input to EB at time t, and the operating power of EB; P EB,h (t) is the thermal power output of EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of EB operating power; and These are the upper and lower limits of the EC unit’s climbing power.

[0184] (11) General mathematical model of energy storage equipment and operating constraints.

[0185] The operating mechanisms of the three energy storage devices, electric, thermal, and cold, are treated in the same way, so the energy storage devices are modeled uniformly as shown below:

[0186]

[0187] In the formula, n is one of electricity, heat, and cold; and is the charging and discharging power of the energy storage device at time t; and is a binary variable, Indicates that the energy storage device is charged at time t, Indicates the release of energy; SOC ES,n (t) is the final remaining capacity of the energy storage device at time t; and is the charging and discharging efficiency; T is the maximum operating time; is the maximum capacity of the energy storage device; is the initial capacity factor of the energy storage device; in order to reduce the impact of overcharge and overdischarge on the life of the energy storage device, the minimum and maximum capacity factors are set

[0188] (12) Electric power balance constraints.

[0189]

[0190] Where P eLoad (t) is the electric load demand at time t.

[0191] (13) Thermal power balance constraints.

[0192]

[0193] Where P hLoad (t) is the heat load demand at time t.

[0194] (14) Cold power balance constraints.

[0195]

[0196] (15) Hydrogen power balance constraint.

[0197]

[0198] (16) Natural gas power balance constraints.

[0199] P MR,g (t) = P g,GT (t)+Pg,GB (t) (24)

[0200] Step 1.2: Establish an optimization objective function for the off-grid zero-carbon park system:

[0201] F=min(C opm )(25)

[0202]

[0203] In the formula, C opm is the total operation and maintenance cost of each device; c opm,i is the unit operation and maintenance cost of the i-th equipment (yuan / KW); P i is the operating power (KW) of the ith device.

[0204] Step 2: Describe the off-grid zero-carbon park system optimization scheduling reinforcement learning framework as a Markov decision process:

[0205] Step 2.1: Create state space:

[0206] s t =[v w (t),S a (t),P eLoad (t),P hLoad (t),P cLoad (t),SOC ES,n (t),t] (27)

[0207] Step 2.2: Create an action space:

[0208]

[0209] Step 2.2: Create a reward function:

[0210] r t =-C opm (29)

[0211] Step 3: Solve the model based on the improved flexible actor-critic algorithm.

[0212] Specifically, it refers to combining advantage learning and transfer learning with the SAC algorithm. The introduction of advantage learning can improve the convergence speed and improve the algorithm performance; the introduction of transfer learning allows the intelligent agent to obtain the scheduling knowledge learned in historical scenarios before learning new scenarios. The intelligent agent directly learns based on the scheduling knowledge, which reduces the learning time and improves the learning efficiency of the intelligent agent.

[0213] The SAC algorithm introduces the concept of maximum entropy on the basis of maximizing future cumulative rewards. The purpose of adding entropy is to enhance robustness and the exploration ability of the agent. The purpose of the SAC algorithm is to maximize the future cumulative reward value and entropy, making the strategy as random as possible, that is, the probability of each action output is as dispersed as possible, rather than concentrated on one action.

[0214] The objective function expression of the SAC algorithm is as follows:

[0215]

[0216] In the formula, (s q ,a q )~P π is the state action trajectory of strategy π; α is a hyperparameter that determines the influence of entropy on reward; αH(π(·|s t )) is state s t The specific calculation formula is as follows:

[0217]

[0218] The above formula represents the strategy π(·|s t ) corresponds to the randomness of the probability distribution. The more random the strategy, the greater the entropy.

[0219] The SAC algorithm also uses the AC framework, but it has been improved by proposing a soft value function and adding entropy to the state value function and Q value function formulas, as shown below:

[0220]

[0221] Where, T π is the Bellman operator under strategy π, γ is the discount factor of the reward, and the Bellman operator is shown below:

[0222]

[0223] In the formula, is the value function at the kth calculation. The soft value function is iterated through formula (33) and eventually converges to the soft Q value function under the fixed strategy π.

[0224] In order to avoid the situation where the strategy difference is too large and causes the training to be unstable, in the strategy update, the strategy output is Gaussian distribution, and then KL divergence is used to narrow the gap between the new and old strategies. The formula is as follows:

[0225]

[0226] Where D KL is the KL divergence; For the old strategy πold The value function under ; Π is the strategy set; is the allocation function under the old strategy, which can normalize the Q value.

[0227] The parameters of the Critic and Actor neural networks of SAC are updated as follows:

[0228]

[0229] In the formula, θ and φ are the parameters of the Q-value network and the strategy network; and is the updated function, Z θ (S t ) is the allocation function.

[0230] Action entropy is also output in the policy network, where the update of the hyperparameter α is crucial to the entropy, and its update formula is:

[0231]

[0232] Where H0 is the dimension of the action matrix.

[0233] Furthermore, in the estimation of the value function, advantage learning will accelerate the convergence of the algorithm during training by reducing the Q value estimate of non-optimal actions (the smaller the Q value, the smaller the probability of being selected) and increasing the probability of the agent selecting the optimal action; the state value function formula of advantage learning is as follows:

[0234]

[0235] In the formula, A * (s,a) is the advantage function, which is defined as follows:

[0236] A * (s,a)=V * (s)-α[V * (s)-Q * (s,a)] (40)

[0237] In the formula, α[V * (s)-Q * (s,a)] is the correction term, Q * When the value of (s,a) is the Q value of the optimal action, the correction term is zero, otherwise it is positive.

[0238] The environmental state s t Input to the policy network, output action a from this network t+1 As the optimal action, a t+1 and status s tInput to the Q value network (critic), and its output value is regarded as the current optimal state value Q(s t ,a t+1 θ - ), then the correction function is rewritten as:

[0239] F(s t ,a t )=α[Q(s t ,a t+1 θ - )-Q(s t ,a t θ - )] (41)

[0240] To prevent the network output Q(s t ,a t θ - ) is greater than Q(s t ,a t θ - ), F(s t ,a t ) is less than zero, then subtract F(s t ,a t ) will increase the Q value and cause overestimation, so add restrictions: when F(s t ,a t )>0, F(s t ,a t ) remains unchanged; F(s t ,a t )<0, F(s t ,a t ) has a value of 0.

[0241] Although the SAC algorithm can quickly obtain a better strategy, the network parameters are randomly generated during the early iteration of the neural network. At this time, the Q value network estimates the value of the action inaccurately. If advantage learning is started at this time, it will disrupt the parameter iteration of the network, causing interference in the early training. To solve this problem, it is also necessary to judge the stability of the neural network. Therefore, the rate of decrease of the network loss function is used as the basis for judging whether to start advantage learning; the formula for the rate of decrease of the network loss function is as follows:

[0242]

[0243] Where, L oss (t) is the neural network loss function, k is the descent rate; if k is less than the threshold set for starting advantage learning, advantage learning will not be started; otherwise, advantage learning will be started.

[0244] The target Q value formula is changed to:

[0245]

[0246] Many tasks in deep reinforcement learning are usually independent of each other, which means that the experience gained in previous tasks is difficult to apply to new challenges, resulting in a waste of previous training efforts and resources. Learning is restarted every time, which not only consumes a lot of computing resources but is also inefficient. When dealing with different tasks in the same environment, a knowledge transfer strategy can be adopted to maximize the use of existing knowledge, thereby improving the efficiency of deep reinforcement learning.

[0247] In actual operation, the neural network weights of the target network are usually initialized with the neural network weights updated by historical data training, instead of the random weight initialization of the neural network.

Claims

1. A method for optimizing the dispatching of wind, solar, thermal, cooling and hydrogen in an off-grid zero-carbon park, characterized in that: The following steps are involved: Step 1: Establish an off-grid zero-carbon park system model, which includes energy supply equipment, energy conversion equipment, and energy storage equipment. The energy source is renewable energy wind energy and light energy, and the load types include electrical load, thermal load, and cooling load; The energy supply equipment includes a wind turbine WT and a photovoltaic panel generator PV; the energy conversion equipment includes a combined cooling, heating and power system CCHP, a gas boiler GB, an electrolyzer EL, a fuel cell HFC, a methane reactor MR, a carbon capture device CCUS, an electric refrigerator EC and an electric boiler EB; the energy storage equipment includes a battery BAT, a hydrogen storage tank HST, a heat storage device HES and a cold storage device CES; The combined cooling, heating and power system consists of a gas turbine GT, a waste heat recovery device WHR, an absorption chiller AC and a waste heat boiler WHB. Step 1.1: Based on the above off-grid zero-carbon park system model, construct a mathematical model and operating constraints; (1) Mathematical model of wind turbine WT and operating constraints; 0≤P WT (t)≤P WT,max (t) (2) Where P WT,max (t) is the maximum power output of the wind turbine generator set at time t; N WT is the number of fans; P WT,rated is the rated power of a single fan; ν w (t) is the wind speed of the wind turbine at the time t at the height of the wind turbine; ν rated is the rated wind speed of the wind turbine, v in is the cut-in wind speed of the wind turbine; v out P is the cut-out wind speed of the wind turbine; WT (t) is the actual output power of the wind turbine generator set at time t; (2) PV mathematical model and operating constraints of photovoltaic generator sets; Where P PV,max (t) is the maximum power output of the photovoltaic generator set at time t; f PV is the attenuation coefficient of the photovoltaic panel; P PV,rated is the rated power of a single photovoltaic panel; S a (t) is the actual light intensity at time t; S stc is the light intensity under standard environment; k PV is the power temperature coefficient; T pv (t) is the actual temperature of the photovoltaic panel surface at time t; T stc The temperature of the photovoltaic panel surface under standard environment; The actual temperature T of the photovoltaic panel surface pv The calculation formula of (t) is: 0≤P PV (t)≤P PV,max (t) (5) Where, T amb is the ambient temperature; NOCT is the nominal operating temperature of the photovoltaic module, provided by the photovoltaic manufacturer; P PV (t) is the actual power output of the photovoltaic generator set at time t; (3) CCHP mathematical model and operation constraints; Where P g,GT (t) is the natural gas power input to GT at time t, i.e., the operating power of GT; P GT,e (t) and P GT,h (t) are the electrical power and thermal power output of the GT at time t, respectively; η GT is the energy conversion efficiency of GT; and They represent the upper and lower limits of the adjustable thermal-to-electric ratio of GT respectively; and The upper and lower limits of the natural gas power input to the GT unit; and The upper and lower limits of the GT unit climbing power; In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6 J; M is the calorific value of natural gas, which indicates the heat released by the combustion of unit volume of natural gas; g,GT (t) is the mass of natural gas consumed by the GT unit at time t; The mass of CO2 produced by complete combustion of unit mass of natural gas; is the mass of carbon dioxide produced by the GT unit at time t; Where P h,AC (t) is the thermal power input to AC at time t, and the operating power of AC; P AC,c (t) is the cooling power output by AC at time t; The AC heat-to-cool conversion efficiency; and The upper and lower limits of AC operating power; and The upper and lower limits of the AC unit's climbing power; Where P h,WHB (t) is the thermal power input to WHB at time t, and the operating power of WHB; P WHB,h (t) is the thermal power output of WHB at time t; is the WHB heat-to-heat conversion efficiency; and The upper and lower limits of WHB operating power; and The upper and lower limits of the WHB unit climbing power; (4) Mathematical model of gas boiler GB and operating constraints; Where P g,GB (t) is the natural gas power input to GB at time t, and the operating power of GB; P GB,h (t) is the thermal power output by GB at time t; is the conversion efficiency of GB natural gas to heat; and The upper and lower limits of GB operating power; and The upper and lower limits of GB unit climbing power; In the formula, K is the energy corresponding to 1KWh of electricity, that is, 3.6×10 6 J; M is the calorific value of natural gas, which indicates the heat released by the combustion of unit volume of natural gas; g,GB (t) is the volume of natural gas consumed by GB unit at time t; The mass of CO2 produced by complete combustion of unit mass of natural gas; is the mass of carbon dioxide produced by GB unit at time t; (5) Electrolyzer EL mathematical model and operating constraints; Where P e,EL (t) is the natural gas power input to EL at time t, and the operating power of EL; P EL,H (t) is the hydrogen power output by EL at time t; is the conversion efficiency of EL to hydrogen; and is the upper and lower limits of EL operating power; and The upper and lower limits of the EL unit climbing power; (6) Mathematical model of fuel cell HFC and operating constraints; Where P H,HFC (t) is the hydrogen power input to HFC at time t, i.e., the operating power of HFC; P HFC,e (t) and P HFC,h (t) are the electrical power and thermal power output of HFC respectively; η HFC is the energy conversion efficiency of HFC; and They represent the upper and lower limits of the adjustable heat-to-electricity ratio of HFC respectively; and The upper and lower limits of hydrogen power input to the HFC unit; and The upper and lower limits of the HFC unit's ramp-up power; (7) Mathematical model of methane reactor MR and operating constraints; Where P H,MR (t) is the hydrogen power input to MR at time t, and the operating power of MR; P MR,g (t) is the natural gas power output by MR at time t; The conversion efficiency of hydrogen to natural gas for MR production; and is the upper and lower limits of hydrogen power input to the MR unit; and The upper and lower limits of the MR unit climbing power; In the formula, is the mass of natural gas generated by MR at time t; μ CO2 The mass of carbon dioxide required to produce a unit mass of natural gas; is the mass of carbon dioxide consumed by MR at time t; (8) Carbon capture CCUS mathematical model and operation constraints; Where η CCUS The amount of electricity required to absorb a unit mass of carbon dioxide; P e,CCUS (t) is the mass of carbon dioxide absorbed by CCUS at time t; (9) Mathematical model of electric refrigeration EC and operating constraints; Where P e,EC (t) is the electric power input to the EC at time t, and the operating power of the EC; P EC,c (t) is the cooling power output by the EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of the EC operating power; and The upper and lower limits of the EC unit's climbing power; (10) Electric boiler EB mathematical model and operating constraints; Where P e,EB (t) is the electric power input to EB at time t, and the operating power of EB; P EB,h (t) is the thermal power output of EC at time t; The conversion efficiency of EC electricity to cooling; and The upper and lower limits of EB operating power; and The upper and lower limits of the EC unit's climbing power; (11) General mathematical model of energy storage equipment and operating constraints; The operating mechanisms of the three energy storage devices, electric, thermal, and cold, are treated in the same way, so the energy storage devices are modeled uniformly as shown below: In the formula, n is one of electricity, heat, and cold; and is the charging and discharging power of the energy storage device at time t; and is a binary variable, Indicates that the energy storage device is charged at time t, Indicates the release of energy; SOC ES,n (t) is the final remaining capacity of the energy storage device at time t; and is the charging and discharging efficiency; T is the maximum operating time; is the maximum capacity of the energy storage device; is the initial capacity factor of the energy storage device; in order to reduce the impact of overcharge and overdischarge on the life of the energy storage device, the minimum and maximum capacity factors are set (12) Electric power balance constraints; Where P eLoad (t) is the electric load demand at time t; (13) Thermal power balance constraints; Where P hLoad (t) is the heat load demand at time t; (14) Cold power balance constraints; (15) Hydrogen power balance constraints; (16) Natural gas power balance constraints; P MR,g (t)=P g,GT (t)+P g,GB (t) (24) Step 1.2: Establish an optimization objective function for the off-grid zero-carbon park system; F=min(C opm ) (25) In the formula, C opm is the total operation and maintenance cost of each device; c opm,i is the unit operation and maintenance cost of the i-th device; P i is the operating power of the i-th device; Step 2: Describe the off-grid zero-carbon park system optimization scheduling reinforcement learning framework as a Markov decision process: Step 2.1: Create state space: s t =[v w (t),S a (t),P eLoad (t),P hLoad (t),P cLoad (t),SOC ES,n (t),t] (27) Step 2.2: Create an action space: Step 2.2: Create a reward function: r t ＝-C opm (29) Step 3: Solve the model based on the improved flexible actor-critic algorithm; Specifically, it refers to combining advantage learning and transfer learning with the SAC algorithm; The objective function expression of the SAC algorithm is as follows: In the formula, (s q ,a q )~P π is the state action trajectory of strategy π; α is a hyperparameter that determines the influence of entropy on reward; αH(π(·|s t )) is state s t The specific calculation formula is as follows: The above formula represents the strategy π(·|s t ) corresponds to the randomness of the probability distribution. The more random the strategy, the greater the entropy. The SAC algorithm also uses the AC framework, but it has been improved by proposing a soft value function and adding entropy to the state value function and Q value function formulas, as shown below: Where, T π is the Bellman operator under strategy π, γ is the discount factor of the reward, and the Bellman operator is shown below: In the formula, is the value function at the kth calculation. The soft value function is iterated through formula (33) and eventually converges to the soft Q value function under the fixed strategy π; In order to avoid the situation where the strategy difference is too large and causes the training to be unstable, in the strategy update, the strategy output is Gaussian distribution, and then KL divergence is used to narrow the gap between the new and old strategies. The formula is as follows: Where D KL is the KL divergence; For the old strategy π old The value function under ; Π is the strategy set; is the allocation function under the old strategy, which can normalize the Q value; The parameters of the Critic and Actor neural networks of SAC are updated as follows: In the formula, θ and φ are the parameters of the Q-value network and the strategy network; and is the updated function, Z θ (S t ) is the allocation function; Action entropy is also output in the policy network, where the update of the hyperparameter α is crucial to the entropy, and its update formula is: Where H0 is the dimension of the action matrix.

2. The off-grid zero-carbon park wind, solar, thermal, cooling and hydrogen optimization scheduling method according to claim 1 is characterized in that: In step 3, in the estimation of the value function, advantage learning will accelerate the convergence speed of the algorithm during training by reducing the Q value estimation of non-optimal actions and increasing the probability of the agent selecting the optimal action; wherein, the state value function formula of advantage learning is as follows: In the formula, A * (s,a) is the advantage function, which is defined as follows: A * (s,a)=V * (s)-α[V * (s)-Q * (s,a)] (40) In the formula, α[V * (s)-Q * (s,a)] is the correction term, Q * When the value of (s,a) is the Q value of the optimal action, the correction term is zero, otherwise it is positive; The environmental state s t Input to the policy network, output action a from this network t+1 As the optimal action, a t+1 and status s t Input to the Q value network critic, the output value is regarded as the current optimal state value Q(s t ,a t+1 θ - ), then the correction function is rewritten as: F(s t ,a t )=α[Q(s t ,a t+1 ;θ - )-Q(s t ,a t ;θ - )] (41) To prevent the network output Q(s t ,a t θ - ) is greater than Q(s t ,a t θ - ), F(s t ,a t ) is less than zero, then subtract F(s t ,a t ) will increase the Q value and cause overestimation, so add restrictions: when F(s t ,a t )>0, F(s t ,a t ) remains unchanged; F(s t ,a t )<0, F(s t ,a t ) has a value of 0; The stability of the neural network also needs to be judged, so the rate of decrease of the network loss function is used as the basis for judging whether to start advantage learning; the formula for the rate of decrease of the network loss function is as follows: Where, L oss (t) is the neural network loss function, k is the drop rate; if k is less than the threshold set for starting advantage learning, advantage learning will not be started; otherwise, advantage learning will be started; The target Q value formula is changed to: When dealing with different tasks in the same environment, adopt a knowledge transfer strategy to maximize the use of existing knowledge, thereby improving the efficiency of deep reinforcement learning; In actual operation, the neural network weights of the target network are initialized with the neural network weights updated by historical data training, instead of the random weight initialization of the neural network.

Images