Comprehensive energy microgrid multi-objective optimization scheduling method based on efficiency

Abstract

The invention relates to an efficiency-based comprehensive energy micro-grid multi-target optimization scheduling method, which comprises the following steps of: proportionally sharing the power of natural gas generated by a methane reactor to calculate the power of natural gas input into a hydrogen-doped combined heat and power generation and a hydrogen-doped gas boiler by a natural gas network so as to obtain natural gas input into a comprehensive energy micro-grid; a hydrogen production apportionment proportion coefficient is set according to the proportion of the hydrogen power output by photocatalytic hydrogen production to the sum of the hydrogen power output by the alkaline electrolytic cell hydrogen production and the photocatalytic hydrogen production; a hydrogen storage tank output allocation proportion coefficient is set according to the proportion of hydrogen energy of photocatalytic hydrogen production and hydrogen production of the alkaline electrolytic cell stored in the hydrogen storage tank, and hydrogen energy of the photocatalytic hydrogen production input comprehensive energy microgrid is obtained; calculating the overall efficiency of the integrated energy microgrid; and real-time scheduling is carried out by taking improvement of the overall efficiency, economical efficiency and low-carbon property of the comprehensive energy microgrid as an optimization target. According to the method, high-efficiency energy consumption of the system is realized on the premise of economy and low carbon.

Description

Technical Field

[0001] The present invention relates to the technical field of optimal operation of integrated energy microgrids with multiple hydrogen production sources. Specifically, it relates to a multi-objective optimal scheduling method for integrated energy microgrids based on exergy efficiency. Background Technique

[0002] Integrated energy microgrids integrate various distributed energy sources (solar energy, wind energy, energy storage, etc.), combined heat and power generation and other equipment, and can achieve the efficient collaborative operation of equipment, which is crucial for the development of new power systems. Therefore, studying the optimal scheduling method of integrated energy microgrids has important practical value and practical significance.

[0003] At present, there have been a large number of studies on the optimal scheduling method of integrated energy microgrids. The invention patent CN118630735A considers the diversified utilization of hydrogen energy, and constructs a low-carbon economic scheduling model for a hydrogen-containing energy power system with the minimization of carbon trading cost and system operation cost as the optimization objectives. The invention patent CN118863430A constructs and solves a master-slave game model for a hydrogen-containing integrated energy system, realizing the dual optimization of the economy and low carbon of different stakeholders. However, with the rapid development of new power systems, in addition to improving the economy and low carbon of system operation, higher requirements are also put forward for the operation energy efficiency of integrated energy microgrids. However, the above-mentioned invention patents only focus on the "quantity" of energy, while ignoring the "quality" difference between different energies, and their optimal scheduling results do not consider the system operation energy efficiency, which has certain limitations. Summary of the Invention

[0004] Aiming at the deficiencies of the existing technology, the technical problem to be solved by the present invention is to provide a multi-objective optimal scheduling method for integrated energy microgrids based on exergy efficiency to solve the problems existing in the existing technology.

[0005] The present invention adopts the following technical solutions to solve the above technical problems: A multi-objective optimal scheduling method for integrated energy microgrids based on exergy efficiency includes the following contents: Proportionally allocate the natural gas power generated by the methane reactor to calculate the natural gas power input to the hydrogen-blended combined heat and power generation and hydrogen-blended gas boiler in the natural gas network, and then calculate the natural gas exergy input to the integrated energy microgrid; Set the hydrogen production sharing ratio coefficient according to the proportion of the hydrogen power output by photocatalytic hydrogen production in the total hydrogen power output by alkaline electrolysis hydrogen production and photocatalytic hydrogen production; Set the hydrogen storage tank output sharing ratio coefficient according to the proportion of the hydrogen energy of photocatalytic hydrogen production and alkaline electrolysis hydrogen production stored in the hydrogen storage tank; Calculate the hydrogen power allocated to each hydrogen-consuming device by the photocatalytic hydrogen production using the recycling hydrogen production sharing ratio coefficient and the hydrogen storage tank output sharing ratio coefficient, and then calculate the exergy of hydrogen energy input into the integrated energy microgrid by the photocatalytic hydrogen production. Combined with the exergy of the electrical load, the exergy of the thermal load, the exergy of the integrated energy microgrid selling electricity to the main power grid, the exergy of the integrated energy microgrid purchasing electricity, the exergy of electricity generated by the photovoltaic power generation unit, and the exergy of electricity generated by the wind power generation unit, calculate the overall exergy efficiency of the integrated energy microgrid. Taking the improvement of the overall exergy efficiency, economy, and low carbon of the integrated energy microgrid as the optimization goal, use deep reinforcement learning to achieve the real-time scheduling of the integrated energy microgrid.

[0006] The equipment used in the integrated energy microgrid includes photocatalytic hydrogen production, alkaline electrolyzer, new energy power generation units, hydrogen-doped gas boilers, hydrogen-doped combined heat and power, hydrogen fuel cells, carbon capture, energy storage equipment, and methane reactors; among them, the energy storage equipment includes electrical energy storage, thermal energy storage, and hydrogen storage tanks, and the new energy power generation units include photovoltaic power generation units and wind power generation units.

[0007] The calculation formula for the overall exergy efficiency of the integrated energy microgrid is; ; Among them, is the exergy efficiency at time t; , are the exergy of the electrical load and the exergy of the thermal load at time t, respectively; is the exergy of the integrated energy microgrid selling electricity to the main power grid; is the exergy of the integrated energy microgrid purchasing electricity; , are the exergy of electricity generated by the photovoltaic power generation unit and the exergy of electricity generated by the wind power generation unit, respectively; is the exergy of hydrogen energy input into the integrated energy microgrid by the photocatalytic hydrogen production, is the exergy of natural gas input into the integrated energy microgrid.

[0008] Use the optimal solution distance method to integrate the three indicators of exergy efficiency, economy, and low carbon, and construct an objective function for multi-objective optimization; among them, the economic indicator is the operating cost of the integrated energy microgrid, and the low-carbon indicator is the actual carbon emission E of the integrated energy microgrid q ; Set the objective function Target(I t ) of the multi-objective optimization as:

[0009] Among them, I t is the high-dimensional point in the objective value space combining exergy efficiency, economy, and low carbon at time t; I best is the theoretical optimal solution of the scheduling scheme; "|| || 2 " represents the two-norm.

[0010] Based on the constructed objective function, the integrated energy microgrid scheduling process is formulated as a Markov decision process, including a state space, an action space, and a reward function. The deep reinforcement learning adopts the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm; the action space is the action space under over-limit truncation protection , which is defined as:

[0011] where is the lower limit of the hydrogen blending ratio, with a value of 0; is the adjustment variable of the charging / discharging power of the electrical energy storage; is the power change rate of device i; P i,ramping is the ramp-up limit of device i; P i,t-1 is the operating power of device i at time t-1; and are the adjustment variables of the heat-electricity ratio and the hydrogen blending ratio respectively; is the adjustable upper limit of the heat-electricity ratio; is the adjustable lower limit of the heat-electricity ratio; is the mapping coefficient of the hydrogen blending ratio; P i,min is the minimum operating power of device i; P i,max is the maximum operating power of device i.

[0012] By constructing a Markov decision process in a stochastic environment and setting the action space under over-limit truncation protection, the system operation is ensured to be within the safe constraint range, realizing the safe and stable operation of the system; based on physical model knowledge, a reward function with a penalty term including the objective function and operation constraints is set to guide the intelligent agent to explore an efficient, economic, and low-carbon scheduling scheme with safety in an uncertain environment.

[0013] Furthermore, the scheduling method further includes a knowledge rule module, which includes a condition judgment process and a knowledge rule replacement process. The condition judgment process refers to identifying the current state of the integrated energy microgrid and judging whether the current state is within the application range of the knowledge rules; the knowledge rule replacement process is that if the current state is within the application range of the knowledge rules, the actions of the corresponding devices are generated, and the corresponding actions in the action space are randomly replaced according to the set action output probability.

[0014] Furthermore, the action output probability is:

[0015] where is the action selection control probability decay factor; z is the number of episodes of interaction between the Twin Delayed Deep Deterministic Policy Gradient algorithm and the integrated energy microgrid.

[0016] The knowledge rules are the requirements for the operation of corresponding devices set based on the system optimization objectives and the operating status of the integrated energy microgrid.

[0017] Introduce the knowledge rule module into the double-delayed deep deterministic policy gradient algorithm, and execute the double-delayed deep deterministic policy gradient algorithm with the actions processed by the knowledge rule module.

[0018] Considering that the electric energy storage should supply electric energy to the integrated energy microgrid during high electricity price periods to relieve the energy supply pressure and improve the system economy at the same time, the knowledge rule of the electric energy storage is to set the discharging action of the electric energy storage according to the upper and lower limit constraints of the electric energy storage capacity and the upper limit of the discharging power of the electric energy storage; its knowledge rule is expressed as:

[0019] where, is the adjustment variable of the charging / discharging power of the electric energy storage; Clip is the truncation function, and the truncation range is [0,1]; represents the Gaussian distribution function with mean µ and variance ; is the storage capacity of the electric energy storage at time t-1; is the upper limit of the electric energy storage capacity; is the upper limit of the charging / discharging power; , are the start time and end time of the high electricity price period.

[0020] Considering that the electric boiler should minimize electricity consumption during high electricity price periods to avoid increasing the operating cost of the integrated energy microgrid due to the increase in the electricity consumption level, the knowledge rule of the electric boiler is to set the stop / start action of the electric boiler according to the ramp constraint of the electric boiler; its knowledge rule is expressed as:

[0021] where, is the change rate of the electric power input to the electric boiler, is the electric power input to the electric boiler at the previous moment, P EB,ramping is the ramp-up upper limit of the electric power of the electric boiler.

[0022] Considering that the alkaline electrolyzer should also minimize electricity consumption during high electricity price periods to avoid increasing the operating cost of the integrated energy microgrid due to the increase in the electricity consumption level, the knowledge rule of the alkaline electrolyzer is to set the stop / start action of the alkaline electrolyzer according to the ramp constraint of the alkaline electrolyzer; its knowledge rule is expressed as:

[0023] where, is the change rate of the electric power input to the alkaline electrolyzer, P P2G,t-1 is the electric power input to the alkaline electrolyzer at the previous moment; PP2G,ramping is the upper limit of the power ramp of the alkaline electrolyzer.

[0024] Considering the increased power output of the hydrogen-blended cogeneration during high electricity price periods, while meeting the electricity load demand of the integrated energy microgrid and supplying the excess generated electricity to the grid for higher economic efficiency, the knowledge rule of the hydrogen-blended cogeneration is to set the full-power output action of the hydrogen-blended cogeneration according to the upper limit of the power output and the ramp constraint of the hydrogen-blended cogeneration; its knowledge rule is expressed as:

[0025] Wherein, is the change rate of the output electric power of the hydrogen-blended cogeneration, is the output electric power of the hydrogen-blended cogeneration at the previous moment, is the upper limit of the power output of the hydrogen-blended cogeneration, P CHP,ramping is the upper limit of the power ramp of the cogeneration.

[0026] Furthermore, the process of obtaining the natural gas exergy input to the integrated energy microgrid is as follows: Distribute the gas power generated by the methane reactor to the corresponding units as fuel according to the ratio of the gas power required by the hydrogen-blended cogeneration and the hydrogen-blended gas boiler; Subtract the gas power provided by the methane reactor for the corresponding equipment from the gas power required in the dispatching process of the hydrogen-blended cogeneration and the hydrogen-blended gas boiler to obtain the natural gas power provided by the natural gas network to the hydrogen-blended cogeneration and the hydrogen-blended gas boiler; Based on the temperatures during the combustion processes of the hydrogen-blended cogeneration and the hydrogen-blended gas boiler, calculate the natural gas exergy input to the integrated energy microgrid. The specific natural gas exergy input to the integrated energy microgrid is calculated according to the following formula:

[0027]

[0028] Wherein, and are the natural gas exergies input to the hydrogen-blended cogeneration and the hydrogen-blended gas boiler respectively, and are the gas powers input to the hydrogen-blended cogeneration and the hydrogen-blended gas boiler respectively, and are the temperatures during the combustion processes of the hydrogen-blended cogeneration and the hydrogen-blended gas boiler, is the natural gas power generated by the methane reactor, T 1 is the ambient temperature.

[0029] Furthermore, the exergy of the hydrogen energy input into the integrated energy microgrid by photocatalytic hydrogen production is the sum of the exergy of the hydrogen energy produced by photocatalysis flowing into the hydrogen-fired combined heat and power generation and the hydrogen-fired gas boiler, and the exergy of the hydrogen energy produced by photocatalysis flowing into the hydrogen fuel cell and the methane reactor. Specifically, the exergy of the hydrogen energy input into the integrated energy microgrid by photocatalytic hydrogen production is:

[0030] Among them, 、 is the exergy of the hydrogen energy produced by photocatalysis flowing into the hydrogen-fired combined heat and power generation and the hydrogen-fired gas boiler, 、 is the exergy of the hydrogen energy produced by photocatalysis flowing into the hydrogen fuel cell and the methane reactor;

[0031] Among them, 、 is the hydrogen power of the hydrogen energy produced by photocatalysis input into the hydrogen-fired combined heat and power generation and the hydrogen-fired gas boiler; T 1 represents the environmental temperature, with the unit of K; 、 are the temperatures during the combustion processes of the hydrogen-fired combined heat and power generation and the hydrogen-fired gas boiler.

[0032] Furthermore, the hydrogen power of the hydrogen energy produced by photocatalysis input into the hydrogen-fired combined heat and power generation and the hydrogen-fired gas boiler 、 is obtained according to the following formula;

[0033] Among them, is the hydrogen production sharing ratio coefficient at time t; is the hydrogen storage tank output sharing ratio coefficient at time t; is the hydrogen storage tank output sharing ratio coefficient at time t - 1; is the hydrogen production sharing ratio coefficient at time t - 1; is a 0-1 variable, which is 1 when the hydrogen storage tank stores hydrogen and 0 when it releases hydrogen; 、 are the charging / discharging efficiencies respectively; is the storage capacity of the hydrogen storage tank at the scheduling period T, and T takes 24; is the initial output sharing ratio coefficient of the hydrogen storage tank; is the initial capacity of the hydrogen storage tank; is the hydrogen power of the photocatalytic hydrogen production input device j; is the hydrogen power of the input device j; j takes values in [1, 2, 3, 4], representing the hydrogen-fired combined heat and power generation, the hydrogen-fired gas boiler, the methane reactor, and the hydrogen fuel cell respectively; P EL,t is the hydrogen power output by the alkaline electrolyzer at time t; PPHP,t The hydrogen power output by photocatalytic hydrogen production at time t; The hydrogen storage / discharge power at time t; The hydrogen storage / discharge power at t - 1, which is hydrogen discharge when greater than 0 and hydrogen storage otherwise.

[0034] Furthermore, the index values of exergy efficiency, economy, and low - carbon are transformed using the Sigmoid function to map their numerical values to the same dimension: where x k is the k - th index to be transformed, is the numerical value of the k - th index after transformation; k takes values of 1, 2, and 3, representing the exergy efficiency index, economy index, and low - carbon index respectively; θ k is the weight parameter of the k - th index; After the above transformation, I t ={ , , }, , and are the exergy efficiency, system operation cost, and actual carbon emissions of the system after Sigmoid transformation respectively.

[0035] A prioritized experience replay mechanism can also be introduced in deep reinforcement learning.

[0036] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention fills the gap in the calculation of exergy efficiency of the existing hydrogen - containing integrated energy system, constructs a "black - box" exergy efficiency analysis model for the hydrogen - containing integrated energy microgrid, expands the calculation scenario of exergy efficiency of the hydrogen - containing integrated energy microgrid. For a system with multiple hydrogen production sources, by setting the sharing ratio coefficients of different energy flows, such as the hydrogen production sharing ratio coefficient and the hydrogen storage tank output sharing ratio coefficient, the photocatalytic hydrogen production and alkaline electrolytic hydrogen production are calculated separately, realizing the real - time "tracking" of the hydrogen energy flow direction of different hydrogen production units and the calculation of exergy under multiple hydrogen energy flow directions.

[0037] 2. The present invention comprehensively considers multiple indicators of exergy efficiency, economy, and low - carbon, integrates the three optimization indicators of exergy efficiency, economy, and low - carbon, constructs an objective function for multi - objective optimization, effectively balances the contradiction between the energy efficiency, economy, and carbon emission reduction in system operation, and ensures the high - efficiency energy utilization of the system on the premise of economy and low - carbon.

[0038] 3. The present invention also proposes a deep reinforcement learning framework that integrates knowledge rules with an improved Twin Delayed Deep Deterministic Policy Gradient algorithm (TD3 algorithm + prioritized experience replay pool). In the initial stage of training, high-quality training samples generated by optimizing knowledge rules are used to guide the agent to explore better solutions. At the same time, overrun truncation is set to ensure that the output actions of the agent are within the range of constraint conditions, realizing the safe operation of the integrated energy microgrid and ensuring the safety of deep reinforcement learning in actual application scenarios. At the same time, the overestimation problem in the update process of the Twin Delayed Policy Gradient algorithm (DDPG algorithm) and the Deep Neural Network algorithm (DQN algorithm) is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 FIG. is a schematic structural diagram of an electric-gas-thermal-hydrogen integrated energy microgrid for an embodiment; Figure 2 FIG. is a schematic diagram of the exergy efficiency analysis model of the "black box" of the integrated energy microgrid in the present invention; Figure 3 FIG. is a schematic diagram of the network architecture of deep reinforcement learning for an embodiment of the present invention; Figure 4 FIG. is a reward curve graph of the training process of Embodiment 3 of the present invention; Figure 5 FIG. is a graph of the electric power scheduling result output by the scheduling method in Embodiment 3 of the present invention; Figure 6 FIG. is a graph of the thermal power scheduling result output by the scheduling method in Embodiment 3 of the present invention; Figure 7 FIG. is a graph of the hydrogen power scheduling result output by the scheduling method in Embodiment 3 of the present invention; Figure 8 FIG. is a graph of the carbon emission and exergy efficiency scheduling results output by the scheduling method in Embodiment 3 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0040] The present invention will be further explained below in conjunction with the drawings and embodiments, but this is not intended to limit the protection scope of the present application.

[0041] Embodiment 1: The multi-objective optimal scheduling method for an integrated energy microgrid based on exergy efficiency in this embodiment, the integrated energy microgrid includes multiple hydrogen production sources, and the equipment used includes photocatalytic hydrogen production, alkaline electrolyzer, new energy generator sets, hydrogen-doped gas boilers, hydrogen-doped combined heat and power, hydrogen fuel cells, carbon capture, energy storage equipment, methane reactors; among them, the energy storage equipment includes electrical energy storage, thermal energy storage and hydrogen storage tanks (hydrogen energy storage), and the new energy generator sets include photovoltaic generator sets and wind turbine generator sets. The specific steps are as follows: Proportionally allocate the natural gas power generated by the methane reactor to calculate the natural gas power input to the hydrogen - blended combined heat and power generation and hydrogen - blended gas boiler in the natural gas network, and then calculate the exergy of the natural gas input to the integrated energy micro - grid; Set the hydrogen production sharing ratio coefficient according to the proportion of the hydrogen power output by photocatalytic hydrogen production in the total hydrogen power output by both alkaline electrolysis hydrogen production and photocatalytic hydrogen production; Set the hydrogen storage tank output sharing ratio coefficient according to the proportion of the hydrogen energy of photocatalytic hydrogen production and alkaline electrolysis hydrogen production stored in the hydrogen storage tank; Then use the hydrogen production sharing ratio coefficient and the hydrogen storage tank output sharing ratio coefficient to calculate the hydrogen power allocated to each hydrogen - using device by photocatalytic hydrogen production, and then calculate the exergy of the hydrogen energy input to the integrated energy micro - grid by photocatalytic hydrogen production; Then, combined with the exergy of the electrical load, the exergy of the thermal load, the exergy of the integrated energy micro - grid selling electricity to the main power grid, the exergy of the integrated energy micro - grid purchasing electricity, the exergy of the photovoltaic power generation unit, and the exergy of the wind power generation unit, calculate the overall exergy efficiency of the integrated energy micro - grid; Taking improving the overall exergy efficiency, economy, and low - carbon performance of the integrated energy micro - grid as the optimization goal, use deep reinforcement learning to achieve the real - time scheduling of the integrated energy micro - grid.

[0042] The overall exergy efficiency of the integrated energy micro - grid at time t is:

[0043] Among them, is the exergy efficiency; and are the exergy of the electrical load and the exergy of the thermal load at time t, respectively; is the exergy of the integrated energy micro - grid selling electricity to the main power grid; is the exergy of the integrated energy micro - grid purchasing electricity; and are the exergy of the photovoltaic power generation unit and the exergy of the wind power generation unit, respectively; is the exergy of the hydrogen energy input to the integrated energy micro - grid by photocatalytic hydrogen production, is the exergy of the natural gas input to the integrated energy micro - grid.

[0044] Example 2: The scheduling method of this example also includes a knowledge rule module. The knowledge rule module includes a condition judgment process and a knowledge rule replacement process. The condition judgment process refers to judging whether the current state of the integrated energy micro - grid is within the application range of the knowledge rules by identifying the current state of the integrated energy micro - grid; The knowledge rule replacement process means that if the current state is within the application range of the knowledge rules, generate the actions of the corresponding devices, and randomly replace the corresponding actions in the action space according to the set action output probability.

[0045] The action output probability is:

[0046] Wherein, is the action selection control probability reduction factor; z is the number of rounds of interaction between the double-delay deep deterministic policy gradient algorithm and the integrated energy microgrid.

[0047] The knowledge rules are the knowledge rule requirements for the operation of corresponding devices set based on the system optimization objectives and the operation status of the integrated energy microgrid; including the knowledge rules of electric energy storage, electric boilers, alkaline electrolyzers, and hydrogen-blended combined heat and power.

[0048] Mainly based on the autonomous exploration of the double-delay deep deterministic policy gradient algorithm, the knowledge rule module is introduced into the double-delay deep deterministic policy gradient algorithm, and the action processed by the knowledge rule module is used to execute the double-delay deep deterministic policy gradient algorithm.

[0049] Based on the optimal solution distance method, three indicators of exergy efficiency, economy, and low carbon are integrated to construct an objective function for multi-objective optimization; among them, the economic indicator is the operating cost of the integrated energy microgrid, and the low-carbon indicator is the actual carbon emission E of the integrated energy microgrid q ; Set the objective function Target(I t ) of the multi-objective optimization as:

[0050] Wherein, I t is the high-dimensional point in the objective value space combining exergy efficiency, economy, and low carbon at time t, I best is the theoretical optimal solution of the scheduling scheme, I best = {1, 0, 0}; "|| || 2 " represents the two-norm.

[0051] The historical data in the present invention includes electric / thermal load demand, power generation of new energy generating units, time-of-use electricity price, environmental temperature, etc. The network architecture (i.e., the intelligent agent) of deep reinforcement learning is trained using historical data such as electric / thermal load demand, power generation of new energy generating units, time-of-use electricity price, environmental temperature, etc., and the real-time state of the integrated energy microgrid is input into the trained network architecture of deep reinforcement learning to realize the real-time scheduling of the integrated energy microgrid under uncertain environments.

[0052] Embodiment 3: In this embodiment, the electric-gas-thermal-hydrogen integrated energy microgrid, as Figure 1As shown in the figure, it includes photocatalytic hydrogen production, alkaline electrolyzers, dynamic hydrogen blending units (hydrogen-blended gas boilers, hydrogen-blended combined heat and power), methane reactors, hydrogen fuel cells, carbon capture, electric boilers, energy storage devices (electrical energy storage, thermal energy storage, hydrogen storage tanks), and new energy generating units. Among them, the hydrogen storage tank realizes the function of hydrogen energy storage, and photocatalytic hydrogen production, alkaline electrolyzers, methane reactors, dynamic hydrogen blending units (hydrogen-blended gas boilers, hydrogen-blended combined heat and power), hydrogen fuel cells, and hydrogen storage tanks constitute a diversified hydrogen energy utilization structure. The hydrogen demand of the integrated energy microgrid is jointly met by hydrogen production through electrolyzing water by alkaline electrolyzers and photocatalytic hydrogen production. The electricity demand is jointly supplied by new energy generating units, hydrogen-blended combined heat and power, hydrogen fuel cells, electrical energy storage, and the superior power grid. The heat demand is jointly met by dynamic hydrogen blending units, electric boilers, and thermal energy storage.

[0053] The process of the multi-objective optimal scheduling method for the integrated energy microgrid based on exergy efficiency is as follows: S1. Calculate the exergy efficiency; Construct an exergy efficiency analysis model for the integrated energy microgrid with coupled electricity-gas-heat-hydrogen multi-energy flows as follows: For the exergy efficiency analysis of the integrated energy microgrid, first, the integrated energy microgrid is equivalently processed from the perspective of external characteristics, and an exergy flow "black box" model is constructed that can transform the complex energy flow and coupling relationship inside the system into an input-output relationship. The "black box" exergy efficiency analysis model of the integrated energy microgrid is as Figure 2 shown. The supply end inputs energy E x,in into the integrated energy microgrid, which is used as the payment exergy. After a series of processes such as transmission, conversion, storage, and distribution inside the integrated energy microgrid, energy E x,out is output to the demand end, which is used as the revenue exergy to meet the demand for all forms of energy in the integrated energy microgrid. According to the exergy reduction principle of the second law of thermodynamics, during the process of energy conversion inside the integrated energy microgrid, inevitably, a part of the quantity will be dissipated into the natural environment, which is the exergy loss E x,loss .

[0054] The exergy loss will lead to a reduction in exergy efficiency. The mathematical model of exergy efficiency is: (1) In the integrated energy microgrid, the payment exergy input into the "black box" model consists of the exergy of purchasing electricity from the superior power grid, the exergy of electricity production by new energy generating units (wind power, photovoltaic), the exergy of hydrogen energy input into the system by photocatalytic hydrogen production, and the exergy of purchasing natural gas by the system. The revenue exergy output from the "black box" model consists of the exergy of the electrical load, the exergy of the heat load, and the exergy of selling electricity by the integrated energy microgrid. Through the "black box" model, the sharing ratio coefficients of different energy flows are introduced, and based on the second law of thermodynamics, the exergy calculation of the energy flow with multiple hydrogen production sources is realized.

[0055] The exergy efficiency of the integrated energy microgrid at time t can be expressed as: (2) Where, is the exergy efficiency at time t; , are the exergy of the electrical load and the exergy of the thermal load at time t, respectively; is the exergy of the integrated energy microgrid selling electrical energy to the main power grid; is the exergy of the integrated energy microgrid purchasing electrical energy; , are the exergy of electricity generated by the photovoltaic power generation unit and the exergy of electricity generated by the wind power generation unit, respectively; is the exergy of hydrogen energy input into the integrated energy microgrid by photocatalytic hydrogen production, is the exergy of natural gas input into the integrated energy microgrid.

[0056] According to the first and second laws of thermodynamics, due to the high conversion efficiency of electrical energy, it can be completely converted into other forms of energy. Therefore, it is regarded as pure exergy in exergy analysis, and its energy quality coefficient is defined as 1. Therefore, the exergy of the integrated energy microgrid selling electrical energy to the main power grid and the exergy (3) Where, is the energy quality coefficient of electrical energy; is a 0-1 variable, which is 1 when the integrated energy microgrid sells electricity to the power grid and 0 when it purchases electricity; P grid,t is the interactive electricity quantity between the integrated energy microgrid and the power grid. A value greater than 0 indicates buying electricity, and a value less than 0 indicates selling electricity; is the electrical load in the integrated energy microgrid.

[0057] The exergy of the thermal load (4) Where, is the energy quality coefficient of thermal energy; T 2 represents the heat source temperature, with the unit of K; T 1 represents the ambient temperature, with the unit of K; is the thermal load of the integrated energy microgrid.

[0058] The exergy In theory, it can be directly solved from the chemical reaction formula. However, in practical applications, it is limited by the heat resistance level of existing equipment, and the theoretical combustion temperature of the fuel often cannot be reached. Therefore, the exergy of natural gas is equivalent to the heat exergy generated at its combustion temperature during the combustion process. The natural gas purchased from the natural gas network and the natural gas produced by the methane reactor jointly flow into the hydrogen-blended combined heat and power generation and the hydrogen-blended gas boiler as raw materials for energy production. According to the "black box" model of exergy efficiency, the natural gas produced by the methane reactor is converted from the energy of the remaining energy flow in the system, rather than the input exergy. Therefore, when calculating the input natural gas exergy through the gas power of the input gas equipment, the exergy of the natural gas generated by the methane reaction should be subtracted. According to the "proportion sharing" principle, the gas power generated by the methane reactor is proportionally distributed to the corresponding units as fuel according to the gas power input to the hydrogen-blended combined heat and power generation and the gas power input to the hydrogen-blended gas boiler. The specific exergy of the natural gas input to the integrated energy microgrid can be expressed as: (5) (6) where 、 are the exergies of natural gas input to the hydrogen-blended combined heat and power generation and the hydrogen-blended gas boiler respectively, 、 are the gas powers input to the hydrogen-blended combined heat and power generation and the hydrogen-blended gas boiler respectively, 、 are the temperatures during the combustion processes of the hydrogen-blended combined heat and power generation and the hydrogen-blended gas boiler, is the gas power of the natural gas produced by the methane reactor.

[0059] 、 respectively represent the gas powers of natural gas provided by the natural gas network to the hydrogen-blended combined heat and power generation and the hydrogen-blended gas boiler.

[0060] The hydrogen power generated by photocatalytic hydrogen production has multiple flow directions, flowing into the hydrogen storage tank for storage, flowing into the hydrogen fuel cell and the dynamic hydrogen-blended units (hydrogen-blended gas boiler, hydrogen-blended combined heat and power generation). To accurately calculate the payment exergy of the input hydrogen energy and avoid confusion with the hydrogen produced by the alkaline electrolyzer inside the "black box", the following measures are taken: The sources of hydrogen energy flowing into the target energy production equipment are hydrogen production by alkaline electrolysis, photocatalytic hydrogen production, and the hydrogen storage tank, as shown in the following formula: (7) where P EL,t is the hydrogen power output by the alkaline electrolyzer; P PHP,t is the hydrogen power output by photocatalytic hydrogen production; is the hydrogen storage / discharge power at time t, which is hydrogen discharge when it is greater than 0, and hydrogen storage otherwise; is the hydrogen power input to the methane reactor at time t; The hydrogen power input to the hydrogen fuel cell at time t; The hydrogen power input to the hydrogen co - generation; The hydrogen power input to the hydrogen - doped gas boiler.

[0061] To achieve real - time "tracking" of the hydrogen energy flow produced by photocatalysis, according to the "proportional sharing" principle, set the hydrogen production sharing ratio coefficient And the output sharing ratio coefficient of the hydrogen storage tank : (8) Among them, The hydrogen production sharing ratio coefficient at time t; The output sharing ratio coefficient of the hydrogen storage tank at time t; The output sharing ratio coefficient of the hydrogen storage tank at t - 1; The hydrogen production sharing ratio coefficient at t - 1; Is a 0 - 1 variable, which is 1 when the hydrogen storage tank stores hydrogen and 0 when it releases hydrogen; 、 Are the charging / discharging efficiencies respectively; Is the storage capacity of the hydrogen storage tank at the scheduling period T, and T takes 24; Is the initial output sharing ratio coefficient of the hydrogen storage tank; Is the initial capacity of the hydrogen storage tank; Is the hydrogen power of the photocatalytic hydrogen production input device j; Is the hydrogen power of the input device j; j takes values in [1, 2, 3, 4], representing hydrogen co - generation, hydrogen - doped gas boiler, methane reactor, and hydrogen fuel cell respectively; t - 1 is the previous period of t; P EL,t Is the hydrogen power output by the alkaline electrolyzer at time t; P PHP,t Is the hydrogen power output by photocatalytic hydrogen production at time t; Is the hydrogen storage / release power at time t, Is the hydrogen storage / release power at t - 1, when it is greater than 0, it is hydrogen release, otherwise it is hydrogen storage.

[0062] According to the hydrogen storage / release action, the exergy payment of the hydrogen energy produced by photocatalysis is: (9) Among them, 、 Are the exergies of the hydrogen energy produced by photocatalytic hydrogen production flowing into the hydrogen co - generation and the hydrogen - doped gas boiler, 、 Are the hydrogen powers of the hydrogen energy produced by photocatalytic hydrogen production input into the hydrogen co - generation and the hydrogen - doped gas boiler.

[0063] Since the hydrogen energy input into the hydrogen fuel cell and the methane reactor is different from that produced through the combustion process, the hydrogen fuel cell directly converts chemical energy into electrical energy through an electrochemical reaction without intermediate changes in thermal energy and mechanical energy. The methane reactor converts hydrogen energy into methane, and its exergy calculation model is as follows: (10) (11) Among them, 、 are the exergies of the hydrogen energy produced by photocatalytic hydrogen production flowing into the hydrogen fuel cell and the methane reactor, 、 are the hydrogen powers of the hydrogen energy produced by photocatalytic hydrogen production input into the hydrogen fuel cell and the methane reactor, are the energy quality coefficients of the hydrogen energy input into the hydrogen fuel cell and the methane reactor respectively, and E HFC is the maximum electric work that can be generated during the chemical reaction process of the hydrogen fuel cell; G is the change in Gibbs free energy, is the Gibbs free energy of hydrogen, is the Gibbs free energy of oxygen, is the Gibbs free energy of water; H is the enthalpy change of the substance, representing the maximum energy that the hydrogen energy can release; is the enthalpy value of hydrogen, is the enthalpy value of oxygen, is the enthalpy value of water. The derivation form of the quality factor of the methane reactor is similar to (11), so it will not be elaborated here.

[0064] The exergy of the hydrogen energy input into the integrated energy microgrid by photocatalytic hydrogen production is equal to the sum of the above-mentioned exergies, which is specifically expressed as: (12) S2. Based on the optimal solution distance method, integrate the three indicators of exergy efficiency, economy, and low carbon, and construct an objective function for multi-objective optimization; Among them, the economic indicator is the operating cost of the integrated energy microgrid, and the low-carbon indicator is the actual carbon emission E q ; The exergy efficiency indicator represents the operating energy efficiency of the integrated energy microgrid and is determined by formula (2) in step S1.

[0065] The mathematical model of the economic indicator is expressed as follows: (13) Among them, F total,t is the operating cost of the integrated energy microgrid, C YS,t 、C MG,t 、C NG,t are CO 2Sequestration and transportation costs, electricity trading fees, and natural gas trading fees.

[0066] CO 2 The expression for sequestration and transportation costs is: (14) Where, is the unit cost of CO 2 sequestration and transportation, E CC,t is the carbon emissions during the operation of gas equipment (hydrogen-blended combined heat and power and hydrogen-blended gas boilers), is the amount of CO 2 required for the reaction of the methane reactor at time t, is the capture rate of carbon capture.

[0067] The electricity trading fee C MG,t expression is: (15) Where, , are the unit prices of electricity purchased and sold from the power grid respectively; P grid,t is the electricity volume exchanged between the integrated energy microgrid and the power grid.

[0068] The natural gas cost C NG,t expression is: (16) Where, is the natural gas price at time t, V CHP,t , V GB,t are the volumes of natural gas required for the operation of hydrogen-blended combined heat and power and hydrogen-blended gas boilers, V CH 4 ,t is the volume of methane generated by the methane reactor.

[0069] The mathematical model of the low-carbon index is expressed as follows:

[0070] Where, E q,t is the actual carbon emissions of the integrated energy microgrid at time t, is the carbon emission factor per unit of electricity purchased; is a 0-1 variable, 1 when purchasing electricity and 0 when selling electricity; M r represents the ratio of the relative molecular masses of CO 2 to carbon, E CHP,t , E GB,t are the carbon emissions of hydrogen-blended combined heat and power and hydrogen-blended gas boilers at time t, , are the carbon content per unit calorific value and the carbon oxidation rate of natural gas respectively, is the thermoelectric conversion coefficient, is the gas power input to the hydrogen - doped combined heat and power generation, is the gas power input to the hydrogen - doped gas boiler.

[0071] If the traditional TOPSIS method is used, it is necessary to calculate the optimal evaluation value and the worst evaluation value for the scheduling result, and the performance of the scheme needs to be sorted according to the distance from the optimal and the worst solutions. However, the setting of the distance from the worst solution differentiates the solutions with the same performance, further restricting the exploration range of the intelligent agent and leading to difficulties in the training of the intelligent agent. Therefore, in this embodiment, the traditional TOPSIS method is improved, and the optimal - solution - distance method is used to construct the objective function for multi - objective optimization: (19) where I t is the high - dimensional point in the objective - value space considering exergy efficiency, economy, and low - carbon at time t, and I best is the theoretical optimal solution of the scheduling scheme, and || || 2 represents the two - norm; At the same time, considering that the inconsistent index dimensions of the above three indicators will affect the calculation of the space distance, the index values of exergy efficiency, economy, and low - carbon are transformed using the Sigmoid function to map their numerical values to the same dimension: (20) where x k is the k - th index to be transformed, is the numerical value of the k - th index after transformation; k takes values of 1, 2, and 3, representing the exergy - efficiency index, economy index, and low - carbon index respectively; θ k is the weight parameter of the k - th index; after the above transformation, I t = { , , }, , and are the exergy efficiency, system operation cost, and system actual carbon emissions after Sigmoid transformation respectively. During the system operation process, the higher the exergy efficiency, the closer it is to a high - energy - efficiency system, while the smaller the operation cost and system carbon emissions, the more economical and low - carbon the system operation is. Therefore, the theoretical optimal solution of the scheduling scheme is set as I best = {1, 0, 0}.

[0072] S3. Based on the system optimization objectives and system operation status, set the knowledge - rule requirements for the operation of each device in the integrated energy micro - grid: This embodiment mainly focuses on the intelligent agent to explore the optimal solution in the environment, supplemented by expert knowledge, and establishes the knowledge rules for the operation of each device in the integrated energy microgrid during high electricity price periods. The main devices involved in the knowledge rules are electric energy storage, electric boiler, alkaline electrolyzer, and hydrogen-blended cogeneration.

[0073] During high electricity price periods, the electric energy storage is subject to the upper and lower limit constraints of its capacity, and its knowledge rule is expressed as follows: (21) Where, is the adjustment variable of the charge / discharge power of the electric energy storage; Clip is the truncation function, and the truncation range is [0,1]; represents the Gaussian distribution function with mean µ and variance ; is the storage capacity of the electric energy storage at time t-1; is the upper limit of the electric energy storage capacity; is the upper limit of the charge / discharge power; , are the start time and end time of the high electricity price period.

[0074] Using electricity in the electric boiler during high electricity price periods will not only reduce the operating economy of the system, but also lead to a decline in the energy quality of the system due to the conversion of high-quality electrical energy into heat energy. Its knowledge rule is expressed as follows: (22) Where, is the change rate of the electric power input to the electric boiler, is the electric power input to the electric boiler at the previous moment, P EB,ramping is the ramp-up limit of the electric power of the electric boiler.

[0075] Generating electricity in the alkaline electrolyzer during high electricity price periods will cause the electricity consumption level to increase during the peak electricity consumption period of the system, resulting in a decrease in the electricity sales volume of the integrated energy microgrid, thus affecting the operating economy of the system. Its knowledge rule is expressed as follows: (23) Where, is the change rate of the electric power input to the alkaline electrolyzer, P P2G,t-1 is the input electric power of the alkaline electrolyzer at the previous moment; P P2G,ramping is the ramp-up limit of the electric power of the alkaline electrolyzer.

[0076] Increasing the power generation of the hydrogen-blended cogeneration during high electricity price periods can not only provide electric energy for the peak load demand period, but also sell the excess electric energy to the power grid during high electricity price periods, improving the economy of the system. Its knowledge rule is expressed as follows: (24) Among them, is the change rate of the output electric power of the hydrogen-doped combined heat and power generation; is the output electric power of the hydrogen-doped combined heat and power generation at time t-1; is the upper limit of the output of the electric power of the hydrogen-doped combined heat and power generation, P CHP,ramping is the ramp-up limit of the electric power of the combined heat and power generation.

[0077] S4. Construct a Markov decision process under a random environment, and determine the state space of the data-driven reinforcement learning agent, the action space under overrun truncation guarantee, and the reward function considering the optimization objective and constraint conditions: The state of the integrated energy microgrid includes the storage capacity of the energy storage device at time t, the demand for electric and heat loads, the output of new energy generating units, the unit price of power purchase / sale from / to the power grid, the photocatalytic hydrogen production power, the environmental temperature, the action at the previous moment, and the scheduling moment; since the carbon trading base price and the natural gas price are fixed values, they are not used as state variables, and the state space is expressed as: (25) Among them, S m,t is the storage capacity of the energy storage device at time t, including hydrogen storage tanks, electrical energy storage, and thermal energy storage; P new,t is the power generation power of new energy generating units, including wind power and photovoltaic power generation; T 1,t is the environmental temperature at time t; a t-1 is the action at the previous moment; P PHP,t is the hydrogen power output by the photocatalytic hydrogen production device; , are the unit prices of power purchase / sale from / to the power grid respectively; is the electric load of the integrated energy microgrid; is the heat load of the integrated energy microgrid.

[0078] The action space consists of the input electric power P of the alkaline electrolyzer P2G,t , the output electric power of the hydrogen-doped combined heat and power generation , the output heat power of the gas boiler , the heat-to-electricity ratio of the heat-hydrogen-doped combined heat and power generation , the hydrogen doping ratio of the dynamic hydrogen-doped unit , the hydrogen power input to the methane reactor , the hydrogen power input to the hydrogen fuel cell , the charge / discharge power of the electrical energy storage , and the output heat power of the electric boiler . When the above variables are determined, the output of other parts can be obtained according to the corresponding mathematical model and equality constraints. The action space is defined as follows: (26) Among them, is the traditional state space; P P2G,t is the input electric power of the alkaline electrolyzer; is the output electric power of the hydrogen - blended combined heat and power generation; is the output thermal power of the gas boiler; is the heat - to - power ratio of the hydrogen - blended combined heat and power generation; is the hydrogen blending ratio of the dynamic hydrogen - blending unit, including the hydrogen blending ratios of the hydrogen - blended combined heat and power generation and the hydrogen - blended gas boiler; is the charge / discharge power of the electrical energy storage; is the output thermal power of the electric boiler.

[0079] To meet the ramp constraints of the equipment and normalize the action space, the above - mentioned action space is decomposed as follows: (27) (28) (29) where, is the power change rate of equipment i; P i,ramping is the ramp - up limit of equipment i; P i,t is the operating power of equipment i at time t; P i,t-1 is the operating power of equipment i at time t - 1; is the charge / discharge power upper limit of the electrical energy storage; , , are the adjustment variables of the charge / discharge power of the electrical energy storage, the heat - to - power ratio, and the hydrogen blending ratio respectively; is the adjustable upper limit of the heat - to - power ratio; is the mapping coefficient of the hydrogen blending ratio, with a value of 0.2; is the decomposed action space, which is output after passing through the Tanh layer, so that the action variable is in the range of [- 1,1].

[0080] By the above steps, the ramp constraints of the equipment are met, but the upper and lower limits of the operating power, the upper and lower limits of the heat - to - power ratio, and the upper and lower limits of the hydrogen blending ratio are not fully met. By truncating the over - limit actions on the action space, the output actions can strictly meet the upper and lower limits of the operation: (30) where, is the action space under the over - limit truncation guarantee, is the lower limit of the hydrogen blending ratio, with a value of 0; is the adjustment variable of the charge / discharge power of the electrical energy storage; is the power change rate of equipment i; P i,ramping is the ramp - up limit of equipment i; P i,t-1is the operating power of device i at t-1; and are the adjustment variables of the thermoelectric ratio and the hydrogen blending ratio respectively; is the upper limit of the adjustable thermoelectric ratio; is the lower limit of the adjustable thermoelectric ratio; is the mapping coefficient of the hydrogen blending ratio; P i,min is the minimum operating power of device i; P i,max is the maximum operating power of device i.

[0081] In this embodiment, the operation objective function and constraint conditions of the integrated energy microgrid are transformed into a reward function with penalty terms, and the reward r obtained by the agent at time t t is expressed as: (31) where, F c (s t , a t ) is the constraint penalty function, and F u (s t , a t ) is the power imbalance penalty function; are the cost scaling coefficient, the constraint penalty scaling coefficient, and the power imbalance penalty scaling coefficient respectively, and , and F total (s t , a t ) is the operation cost of the integrated energy microgrid; s t is the state space; a t is the action space under the over-limit truncation guarantee.

[0082] The calculation model of the constraint penalty function F c (s t , a t ) is: (32) (33) where, M i,t is the over-limit value of the operating power of device i; M MG,t is the over-limit value of the power exchanged with the superior power grid; is the over-limit value of the storage capacity, including electricity, heat, and hydrogen storage tanks; is the over-limit value of the hydrogen blending ratio, including hydrogen blending cogeneration and hydrogen blending gas boilers; K CHP,t is the over-limit value of the thermoelectric ratio of cogeneration; is the specific gravity coefficient, which is used to increase the penalty proportion of the over-limit hydrogen blending ratio; P i,t is the operating power of device i at t; M MG,t , , , K CHP,t The calculation process of CHP,t is similar to the calculation of the upper limit value of the device operating power, so it will not be elaborated here.

[0083] Power imbalance penalty function F u (s t , a t ) is expressed as: (34) Among them, P CCS,t is the electric energy consumed by carbon capture, is the electric energy consumed by the electric boiler, is the output electric power of the hydrogen fuel cell at time t, is the output thermal power of the hydrogen-doped combined heat and power generation at time t, is the output thermal power of the hydrogen-doped gas boiler, is the output thermal power of the electric boiler, is the charging / discharging power of the thermal energy storage at time t.

[0084] S5. Integrate the knowledge rules with the double-delay deep deterministic policy gradient algorithm to obtain the network architecture of the deep reinforcement learning of the integrated energy microgrid: In this embodiment, knowledge rules are set for the electric energy storage, electric boiler, alkaline electrolyzer, and hydrogen-doped combined heat and power generation. Based on the autonomous exploration of the double-delay deep deterministic policy gradient algorithm, a knowledge rule module is introduced. The knowledge rule module includes a condition judgment process and a knowledge rule replacement process. The condition discrimination process judges whether the current state of the intelligent agent is within the application scope of the knowledge rules by identifying the current state of the intelligent agent; the knowledge rule replacement process means that if the current state is within the application scope of the knowledge rules, the actions of the corresponding devices are generated, and the corresponding actions in the action space are randomly replaced according to the set action probability, as Figure 3 shown. The action output probability of the knowledge rules is: (35) Among them, is the action selection control probability decay factor; z is the number of rounds of interaction between the algorithm and the integrated energy microgrid; the action output probability of the knowledge rules decreases with the increase of the number of learning times. That is, at the initial stage of training, the action output probability of the knowledge rules is high, enabling the intelligent agent to quickly master the knowledge rules; as the number of learning times increases, the action output probability of the knowledge rules is gradually reduced, while increasing the autonomous exploration opportunity of the intelligent agent, so that the intelligent agent can quickly autonomously explore the optimization strategy beyond the knowledge rules.

[0085] The network architecture of the deep reinforcement learning in this embodiment is as Figure 3As shown, it includes a policy network, a value network 1, a value network 2, a knowledge rule module, and a prioritized experience replay pool. The policy network includes a main policy network and a target policy network. The value network 1 includes a main value network 1 and a target value network 1. The value network 2 includes a main value network 2 and a target value network 2, denoted as the improved TD3 algorithm. The specific construction and update process are as follows: To reduce the overestimation problem of the Q value in the value network, the smaller Q value in the two target value networks is selected to construct the target value y of the temporal difference: (36) where r is the reward value at training sample t, γ is the discount factor, 、 are the target Q values of the target value network 1 and the target value network 2 respectively, is the target policy of the target policy network, s t+1 is the state space at t + 1.

[0086] Adding a random Gaussian noise to the target policy network can alleviate the overfitting problem of the valuation function in the policy network: (37) where ε represents the Gaussian noise, c represents the truncation boundary value of the policy smoothing noise, the clip function represents the truncation function, represents the Gaussian distribution function with a mean of 0 and a variance of .

[0087] Finally, the gradient descent algorithm is used to minimize the error between the valuation and the target value , thereby updating the parameters in the two value networks: (38) where E( ) is the expectation function, are the network parameters of the main value network u, is the learning rate of the main value network u, represents the gradient calculation function of the parameters of the main value network u, is the action-value function of the main value network u.

[0088] By using the sampled policy gradient for the main policy network parameters update according to formula (39): (39) (40) where, is the learning rate of the main policy network, is the gradient information of the value network, is the gradient information of the main policy network, and E represents the expectation function. is the policy of the main policy network. are the parameters of the main policy network.

[0089] In the double-delayed deep deterministic policy gradient algorithm, a prioritized experience replay mechanism is introduced to enable the agent to more efficiently utilize important experience tuples and improve the training efficiency. In the prioritized experience replay mechanism, the temporal difference error δ n can be used to measure the priority of the sampled experience tuple n.

[0090] The priority is determined using a sorting-based priority method: p n = 1 / rank n In, the rank rank of the nth sample n can be obtained from the sorting of the absolute temporal difference error |δ n |, and |δ n | is expressed as follows: (41) where S n+1 is the state space of the (n + 1)th sample, S n is the state space of the nth sample, and a n is the action space of the nth sample.

[0091] During the training process, the agent calculates the sampling probability of each experience tuple according to the priority p n of the experience tuples in the buffer, and then performs importance sampling on the experience tuples according to the sampling probability. The sampling probability P n is expressed as follows: (42) where determines the number of priorities used. When = 0, it means random sampling, and m is the total number of samples.

[0092] However, since experiences with high temporal difference errors will be replayed more frequently, it may cause the neural network to oscillate or diverge during training. To solve the above problem, importance sampling weights can be used when calculating the weight change:

[0093] where N b is the size of the prioritized experience buffer, and the parameter β determines the amount of correction used; is the importance sampling weight of the sample l ; l ranges from 1 ~m to an integer.

[0094] After introducing the prioritized experience replay mechanism, the loss function and sampling policy gradient of the policy network are reformulated as follows: (44) (45) Updating the parameters of the target value network and the target policy network by using the soft update method of the following formula can improve the stability during the learning process: (46) where, represents the soft update rate, is the main policy network parameter, is the target policy network parameter, is the main value network parameter, is the target value network parameter, and u takes 1, 2.

[0095] S6. Use historical data such as electric heating load demand, power generation of new energy generating units, time-of-use electricity price, and ambient temperature to train the agent in step S5, and input the real-time state of the integrated energy microgrid into the trained agent to realize the real-time scheduling of the integrated energy microgrid under uncertain environments.

[0096] The system scheduling duration of this embodiment is 24h, the time interval is 1h, the electricity price participating in the electricity market transaction adopts the time-of-use electricity price, the peak electricity price period is 12:00 - 18:00, the natural gas price is 4.0 yuan / m³, and the carbon trading base price is 140 yuan / ton. The training process is completed on the AIStation artificial intelligence platform, using the Pytorch deep learning framework, and a total of 5000 rounds are trained. The initial learning rates of the policy network and the value network are both set to 0.0001, the reward discount factor is taken as 0.99, and the soft update rate of the target value network and the target policy network is taken as 0.001. The exploration noise is set to 0.25, and the policy noise is set to 0.15. Both the policy network and the value network have 4 hidden layers, with 64 neurons in each layer. The maximum capacity of the prioritized experience replay pool is 50000, and the batch size for training is 1000.

[0097] The training process includes experience replay, sampling, calculating the target value, calculating the loss, updating the value network parameters, calculating the policy gradient, updating the policy network parameters, and updating the target value network and the target policy network. Through these steps, the agent can continuously optimize the value network and the policy network, learn the optimal policy, and the training results are as Figure 4 shown (the reward value converges close to 0, and the training reaches the optimal). After training is completed, input the real-time state of the integrated energy microgrid and output the cooperative control action to realize the real-time scheduling of the integrated energy microgrid.

[0098] Input the environmental state of winter days into the trained deep reinforcement learning network architecture, and the system operation results are as Figure 5 shown. It can be seen from Figure 5 that during the peak electricity price period, the electricity demand is jointly borne by new energy generating units, hydrogen-doped cogeneration, and hydrogen fuel cells. Considering the exergy efficiency index and economic index, the system sells the excess electricity to the power grid, obtaining higher economic benefits while maintaining the "quality" of energy. Figure 6 shows that during the peak electricity price period, hydrogen-doped cogeneration and hydrogen-doped gas boilers are the main heating equipment, while during the valley electricity price period, the heat load demand is mainly met through the coordinated operation of hydrogen-doped cogeneration, hydrogen-doped gas boilers, and electric boilers. It can be obtained from Figure 7 that the hydrogen production power of photocatalytic hydrogen production is the largest during the peak period of photovoltaic power generation, and the electrolyzer does not produce hydrogen during the peak electricity price period. It can be obtained from Figure 8 that the exergy efficiency is at a relatively low level during the peak heating period, i.e., 00:00 - 6:00, compared with other periods, because part of the high-quality electrical energy is converted into heat energy, but the exergy efficiency remains above 0.5. The average exergy efficiency for 24 hours is 0.701, showing a relatively high level. The total daily operating cost is 0.538 ten thousand yuan, and the daily carbon emission is 20.38 tons. Through analysis, it shows that the method of the present invention can achieve the coordinated operation of economy, low carbon, and high energy efficiency of the integrated energy microgrid.

[0099] To verify the superiority of the improved TD3 algorithm in the present invention in the integrated energy microgrid scheduling, its scheduling results are compared and analyzed with those of the integrated energy microgrid scheduling based on the deep Q-network algorithm (DQN algorithm) and the deep deterministic policy gradient algorithm (DDPG algorithm). Fifteen days are randomly selected from the March data as test samples to evaluate its scheduling results. The scheduling results are the average values of the total cost, carbon emissions, and exergy efficiency of the 15-day test data, as shown in Table 1. It can be seen from Table 1 that the integrated energy microgrid scheduling strategy output by the improved TD3 algorithm of the present invention is significantly better than the DQN algorithm and the DDPG algorithm.

[0100]

[0101] The prior art only focuses on the "quantity" of energy and ignores the "quality" difference between different energies. Its optimized scheduling results do not consider the system operation energy efficiency and have certain limitations. The present invention models the exergy efficiency and uses it in multi-objective optimization. Taking the exergy efficiency as the energy efficiency index, it fully considers the difference in the energy use quality between different energies. The present invention uses the optimal solution distance method to construct the objective function of multi-objective optimization including exergy efficiency, economy, and low carbon, balancing the contradiction between carbon emission reduction, economy, and energy efficiency operation, so that the optimized scheduling scheme takes into account economy, low carbon, and high-quality energy use.

[0102] There is no clear theoretical guidance in the prior art to calculate the exergy efficiency of the system under multiple hydrogen production sources. The present invention expands the exergy efficiency calculation scenario of the hydrogen-containing integrated energy microgrid. By constructing the sharing ratio coefficient, the real-time "tracking" of the hydrogen energy flow direction of different hydrogen production units is realized, and the calculation of the exergy under multiple hydrogen energy flow directions is realized. In addition, the present invention adopts a data-driven deep reinforcement learning method to realize the multi-objective optimal scheduling of the integrated energy microgrid, and can realize the real-time calculation of the exergy under different environmental changes and the dynamic formulation of the scheduling scheme according to the external environmental changes, effectively coping with the multiple uncertainties in the system.

[0103] The present invention introduces knowledge rules into the TD3 algorithm, dynamically optimizes the knowledge rules at the initial stage of training to generate excellent training samples, accelerates the convergence speed of the training process of the intelligent agent, guides the intelligent agent to "jump out" of the local optimal solution, and improves the convergence performance of the algorithm.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: without departing from the basic ideas and essential features of the present invention, the technical solutions of the present invention can be modified or equivalently replaced. These modifications or equivalent replacements are all considered to be included within the protection scope defined by the claims of the present invention.

[0105] Matters not described in the present invention are applicable to the prior art.

Claims

1. A multi-objective optimization scheduling method for integrated energy microgrid based on exergy efficiency, characterized in that: The scheduling method Includes the following: The natural gas power generated by the methane reactor is apportioned to calculate the natural gas power input to the hydrogen-blended cogeneration and hydrogen-blended gas boilers, and then the natural gas exergy input to the integrated energy microgrid is calculated; The hydrogen production apportionment ratio coefficient is set according to the ratio of the hydrogen power output by photocatalytic hydrogen production to the total hydrogen power output by the alkaline electrolyzer hydrogen production and the photocatalytic hydrogen production; The output apportionment ratio coefficient of the hydrogen storage tank is set according to the ratio of hydrogen energy stored in the hydrogen storage tank by photocatalytic hydrogen production and hydrogen produced by alkaline electrolysis cell; Then, the hydrogen production apportionment ratio coefficient and the hydrogen storage tank output apportionment ratio coefficient are used to calculate the hydrogen power allocated to each hydrogen-using device by photocatalytic hydrogen production, and then the hydrogen energy exergy input into the comprehensive energy microgrid by photocatalytic hydrogen production is calculated; Combined with the exergy of the electrical load and the heat load, the exergy of the integrated energy microgrid selling electricity to the main grid, the exergy of the integrated energy microgrid purchasing electricity, the exergy of the photovoltaic generator set and the exergy of the wind turbine set, the overall exergy efficiency of the integrated energy microgrid is calculated. With the optimization goal of improving the overall exergy efficiency, economy and low carbon of the integrated energy microgrid, deep reinforcement learning is used to achieve real-time scheduling of the integrated energy microgrid.

2. The scheduling method according to claim 1, characterized in that: The equipment used in the integrated energy microgrid includes photocatalytic hydrogen production, alkaline electrolyzers, new energy generator sets, hydrogen-blended gas boilers, hydrogen-blended cogeneration, hydrogen fuel cells, carbon capture, energy storage equipment and methane reactors; among which the energy storage equipment includes electrical energy storage, thermal energy storage and hydrogen storage tanks, and the new energy generator sets include photovoltaic generator sets and wind turbine generator sets.

3. The scheduling method according to claim 1, characterized in that: The process of obtaining natural gas exergy input into the integrated energy microgrid is: The gas power generated by the methane reactor is distributed to the corresponding units as fuel according to the ratio of the gas power required by the hydrogen-blended cogeneration and hydrogen-blended gas boiler; The gas power required for the hydrogen-doped cogeneration and hydrogen-doped gas boiler scheduling process is subtracted from the gas power provided by the methane reactor for the corresponding equipment to obtain the natural gas power provided by the natural gas grid to the hydrogen-doped cogeneration and hydrogen-doped gas boiler; Based on the temperature of the hydrogen-doped cogeneration and hydrogen-doped gas boiler combustion process, the natural gas exergy input into the integrated energy microgrid is calculated.

4. The scheduling method according to claim 1, characterized in that: The hydrogen energy exergy of the photocatalytic hydrogen production input into the comprehensive energy microgrid is the sum of the exergy of the photocatalytic hydrogen production flowing into the hydrogen-blended cogeneration and hydrogen-blended gas boiler, and the exergy of the photocatalytic hydrogen production flowing into the hydrogen fuel cell and the methane reactor.

5. The scheduling method according to claim 1, characterized in that: The scheduling method further includes a knowledge rule module, which includes a condition judgment process and a knowledge rule replacement process. The condition judgment process determines whether the current state of the integrated energy microgrid is within the application scope of the knowledge rule by identifying the current state of the integrated energy microgrid; Knowledge The rule replacement process is that if the current state is within the application range of the knowledge rule, the action of the corresponding device is generated, and the corresponding action in the action space is randomly replaced according to the set action output probability.

6. The scheduling method according to claim 5, characterized in that: The deep reinforcement learning adopts a double-delay deep deterministic policy gradient algorithm, and introduces a knowledge rule module into the double-delay deep deterministic policy gradient algorithm, and executes the double-delay deep deterministic policy gradient algorithm with actions processed by the knowledge rule module.

7. The scheduling method according to claim 5, characterized in that: The knowledge rules are the knowledge rule requirements for the operation of corresponding equipment set based on the system optimization objectives and the operating status of the integrated energy microgrid.

8. The scheduling method according to claim 7, characterized in that: During periods of high electricity prices, the knowledge rules for electric energy storage are to set the electric energy storage discharge action based on the upper and lower limit constraints of the electric energy storage capacity and the upper limit of the electric energy storage discharge power; the knowledge rules for electric boilers are to set the electric boiler shutdown action based on the electric boiler climbing constraints; the knowledge rules for alkaline electrolyzers are to set the alkaline electrolyzer shutdown action based on the alkaline electrolyzer climbing constraints; the knowledge rules for hydrogen-doped cogeneration are to set the full power output action of hydrogen-doped cogeneration based on the electric power output upper limit and climbing constraints of hydrogen-doped cogeneration.

9. The scheduling method according to claim 1, characterized in that: In deep reinforcement learning, the over-limit actions are truncated and the action space is protected by over-limit truncation.

Images