Integrated energy system scheduling method based on optimized reinforcement learning DDPG

Abstract

The invention discloses a comprehensive energy system scheduling method based on optimized reinforcement learning (DDPG), and aims to improve the scheduling efficiency and adaptability of the system by establishing a comprehensive energy system model containing four energy sources of electricity, heat, cold and hydrogen and performing optimized scheduling based on an optimized depth deterministic policy gradient algorithm (DDPG). A simulation result shows that under a typical daily working condition, the convergence of the DDPG algorithm is optimized, the efficient scheduling of the integrated energy system is realized, and the four kinds of energy storage equipment operate in a reasonable working condition. According to the method, the optimization performance and the stability in a complex continuous decision problem are remarkably improved by processing a continuous action space, efficient strategy learning, experience playback, a target network and other mechanisms. The research of the invention provides theoretical support for the application of the deep reinforcement learning technology in the multi-energy coupled integrated energy system.

Description

Technical Field

[0001] The present invention belongs to the field of energy system engineering, and in particular to a comprehensive energy system scheduling method based on optimized reinforcement learning DDPG. Background Art

[0002] Energy is the material basis for the development of human society. With the continuous maturity of the technology of renewable energy utilization, such as light energy, wind energy, hydrogen energy and other energy sources, people's demand for energy diversity has been met. Integrated energy utilization can achieve coordinated scheduling of various forms of energy such as electricity, heat, and cold, thereby achieving energy complementarity at the system level and improving energy utilization efficiency. The accurate prediction and precise control of the electricity / heat / cold / hydrogen integrated energy system in the integrated energy system are difficult, which brings great risks to the operation of the integrated energy system. In order to solve this problem, the precise control of the electricity / heat / cold / hydrogen integrated energy system has become one of the current hot spots. At present, most studies focus on using mathematical model-related knowledge to model and optimize the system economy. There is insufficient research on rapid response, stability, oscillation, etc., and it is difficult to solve the problem of unstable operation of the integrated energy system. On the other hand, algorithms based on deep learning can achieve efficient energy scheduling of integrated energy systems, promoting the intelligent and efficient development of energy systems, but there are deficiencies in data acquisition, model interpretability, and training response speed. In addition, existing studies have used reinforcement learning DDPG to solve simple energy scheduling problems in electric and thermal coordinated systems. For more complex integrated energy systems, traditional DDPG algorithms cannot perform energy scheduling. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide an electricity / cold / heat / hydrogen integrated energy system scheduling method based on optimized reinforcement learning DDPG to solve the defects of the prior art.

[0004] Technical solution: A comprehensive energy system scheduling method based on optimized reinforcement learning DDPG of the present invention comprises the following steps:

[0005] Step 1: Construct the mathematical model, objective function and operation constraints of the electricity / cold / heat / hydrogen integrated energy system;

[0006] Step 2: Introduce Ornstein-Uhlenbeck OU noise and Gaussian noise suitable for inertial systems, write the optimized reinforcement learning DDPG algorithm based on the Markov decision process MDP using Matlab solver and input the mathematical model;

[0007] Step 3: Use randomly generated working conditions for DDPG training;

[0008] Step 4: Analyze the DDPG training results and obtain the optimized reinforcement learning DDPG to achieve the coordinated scheduling of multiple energies such as electricity, heat, cold, and hydrogen in the integrated energy system.

[0009] Furthermore, in step 1, the electricity / cold / heat / hydrogen integrated energy system includes photovoltaic modules, wind turbines, power grids, proton exchange membrane electrolyzers, proton exchange membrane fuel cells, batteries, hydrogen storage tanks, air source heat pumps, electric heating boilers, steam heat accumulators, chillers and ice storage equipment.

[0010] Further, photovoltaic modules: As a partial power source, the output power of photovoltaic modules can be simplified as:

[0011]

[0012] Among them, η PV and η inv are photovoltaic efficiency and inverter efficiency respectively, A PV is the effective illumination area, I0 is the solar irradiance, the unit is watt per square meter;

[0013] Wind turbines:

[0014] When the wind turbine is running, the equation for power generation is as follows:

[0015]

[0016] in, is the wind power, ρ and R are respectively the air density and the radius of the wind turbine impeller, C p is the wind energy utilization coefficient. According to Betz theory, the maximum wind energy utilization rate C pmax ≈0.593, λ, β, and V are the tip speed ratio, pitch angle, and wind speed, respectively;

[0017] The power generation efficiency of wind turbines is as follows:

[0018]

[0019] Among them, V in is the starting wind speed, V out is the shutdown wind speed;

[0020] Proton exchange membrane electrolyzer:

[0021] The proton exchange membrane electrolyzer is used as a hydrogen production device, and its hydrogen production rate is as follows:

[0022]

[0023] Among them, H ET is the hydrogen production rate, η F is the Faraday efficiency, n ETis the number of electrolytic cells, I ET is the working current, F is the Faraday constant;

[0024] Proton exchange membrane fuel cell:

[0025] The hydrogen consumption of proton exchange membrane fuel cells is calculated using Faraday's law:

[0026]

[0027] Among them, H FC is the hydrogen consumption rate, I FC is the operating current, n FC is the number of batteries;

[0028] The fuel cell reaction rate can be expressed by the current density j:

[0029]

[0030] Among them, A cell is the reaction area of ​​the battery unit;

[0031] Thermal power output of fuel cells It is expressed as:

[0032]

[0033] Among them, η T is the fuel cell thermal efficiency, η E is the fuel cell electrical efficiency;

[0034] Battery:

[0035] The battery operates in battery pack mode, and its total power is expressed as:

[0036]

[0037] in, is the total power output of the battery pack, is the battery unit power, n1 and n2 are the number of battery rows and columns respectively;

[0038] The current of the battery cell is as follows:

[0039]

[0040] Among them, the open-loop voltage V OC and the battery cell internal resistance R b All can be regarded as fixed values;

[0041] The dynamic characteristic SOC of the battery cell can be expressed as:

[0042]

[0043] Among them, t (0≤k≤T) is the scheduling time, T is the scheduling period, Q B is the battery cell capacity, I b,k is the current, charging is negative and discharging is positive;

[0044] Hydrogen storage tank:

[0045] A high-pressure hydrogen storage tank model HST is used to store hydrogen produced by the electrolyzer and provide a source of hydrogen for fuel cell power generation. The pressure inside the hydrogen storage tank changes dynamically with the storage and release of hydrogen, expressed as:

[0046]

[0047] Among them, TP t H and are the internal pressure and hydrogen flow rate of the hydrogen storage tank at time t, in mol / s, V H is the volume of the hydrogen storage tank, T H is the operating temperature, R is the universal gas constant, z t is the compressibility factor of hydrogen, which can be calculated based on NIST data and the Lemmon equation:

[0048]

[0049] Among them, a i 、b i 、c i All are constants, T0 = 100K, R = 8.3145 J / mol·K;

[0050] Air source heat pump

[0051] The air source heat pump HP provides peak thermal power under high load. The thermal performance coefficient COP of the heat pump is related to the temperature difference, and its empirical relationship is fitted as follows:

[0052]

[0053] Where, ΔT HP is the cooling / heating target temperature T D / T H With the current ambient temperature T a The difference, h1, h2 and h3 are the fitting coefficients;

[0054] The heating power of the heat pump is expressed by the formula:

[0055]

[0056] in, is the heat pump thermal power, For the corresponding electrical power, COP H is the heating performance coefficient;

[0057] The cooling power of the heat pump is expressed as:

[0058]

[0059] in, is the cooling power of the heat pump, For the corresponding electrical power, COP C is the cooling performance coefficient;

[0060] Electric heating boiler:

[0061] As a thermal equipment that converts electrical energy into thermal energy, the heating power of the electric boiler EB can be expressed as:

[0062]

[0063] Among them, η EB is the conversion efficiency of the electric boiler;

[0064] Steam accumulator:

[0065] The thermal storage tank TES component model embeds the PCM module on the top of the hot water tank and ensures that the TES outlet temperature is constant according to the steam supply standard of a specific industrial park. To describe the dynamic characteristics of TES, a heat storage state HSD similar to the battery SOC is introduced and defined as:

[0066]

[0067] in, is the charging or releasing power of the steam accumulator at time t, H c The maximum heat storage value of the steam accumulator; HSD is limited between 0 and 1;

[0068] Refrigeration unit:

[0069] The refrigeration unit RAC converts electrical energy into cooling capacity, and its cooling power can be expressed as:

[0070]

[0071] Among them, COP RAC is the energy efficiency ratio of the refrigeration unit;

[0072] Ice storage device:

[0073] In order to describe the dynamic characteristics of the ice storage device CES, the cold storage state CSD similar to the steam accumulator is introduced and defined as:

[0074]

[0075] in, is the charging or discharging power of the ice storage device at time t, C c It is the maximum cold storage capacity of the ice storage device. CSD is limited between 0 and 1.

[0076] Furthermore, in step 1, the objective function is specifically: the objective function mainly includes the operating cost of the system Cost of purchasing electricity from the grid and the status of the energy storage device

[0077]

[0078] Among them, c i and are the operating cost per kWh of the equipment in the system and the power at time t, N is the number of equipment, p t is the power purchase cost from the power grid, α, k1, k2, k3 and k4 are all weight coefficients.

[0079] Furthermore, in step 1, the operation constraints are specifically:

[0080] Electrical balance constraints:

[0081] Thermal balance constraints:

[0082] Cold balance constraints:

[0083] Hydrogen balance constraints:

[0084] Refrigeration unit power upper and lower limit constraints:

[0085] The upper and lower limits of proton exchange membrane fuel cell power:

[0086] Air source heat pump power upper and lower limits:

[0087] Electric heating boiler power upper and lower limit constraints:

[0088] The upper and lower limits of the power of the proton exchange membrane electrolyzer are:

[0089] Hydrogen flow rate upper and lower limits:

[0090] Battery state upper and lower limit constraints: SOC min ≤SOC t ≤SOCmax

[0091] Hydrogen storage tank pressure upper and lower limit constraints: TP min ≤TP t ≤TP max

[0092] Steam accumulator heat storage state upper and lower limit constraints: HSD min ≤HSD t ≤HSD max

[0093] Ice storage device cold storage state upper and lower limit constraints: CSD min ≤CSD t ≤CSD max

[0094] Corresponding device initialization state constraints: .

[0096] Furthermore, step 2 specifically includes the following steps:

[0097] Step 2.1: First, add the Ornstein-Uhlenbeck OU noise applicable to the inertial system Due to the complexity of the system, Gaussian noise G is introduced And set the noise attenuation to achieve a balance between exploration and utilization. The specific mathematical expression is as follows:

[0098]

[0099]

[0100] Where t is the time step, is the OU noise value, θ is the mean reversion rate, μ is the mean, σ is the volatility, W t For Brownian motion, is the G noise value, z is a random variable sampled from a standard normal distribution, with a mean of 0 and a variance of 1; a t is the action, P(s t |θ P ) is the actor network in DDPG, θ p is the network parameter, s t is the system state. The temporal correlation of the OU process enables the system to perform multiple steps of exploration in the same direction to accumulate training experience, thereby effectively improving the exploration efficiency of the inertial system. The G process introduces a certain degree of randomness in the strategy when selecting actions, thereby encouraging exploration;

[0101] Step 2.2, reinforcement learning tasks are usually described as a Markov decision process MDP, defined as a five-tuple: S, A, P, R, γ, which are state space, action space, transition matrix, reward function and discount factor respectively.

[0102] Furthermore, in step 2.2, the state space is as follows:

[0103]

[0104] Where t is the time step, and are the photovoltaic power generation and wind power generation at time t, and They are the electrical load, thermal load, cooling load and hydrogen load at time t, SOC t TP t , HSD t and CSD t They are the battery storage state, hydrogen storage tank pressure, heat storage state of the heat storage tank and cold storage state of the ice storage device at time t respectively;

[0105] The action space is as follows:

[0106]

[0107] in, The flow rate of hydrogen to generate electricity for the fuel cell, and They are the power of refrigeration unit, heat pump, electric boiler and electrolyzer hydrogen production;

[0108] The reward function is as follows:

[0109]

[0110] in, and H out,t They represent the corresponding power surplus or shortage after the four energy storage devices participate in the dispatch, and l1~l6 are the weights adjusted according to the training experience and effect. and They respectively mean that when electricity prices are low, it is encouraged to purchase electricity from the power grid to electrolyze water to produce hydrogen to make up for the hydrogen consumption in hydrogen storage tanks, and to use refrigeration units to make up for the cold consumption of ice storage equipment.

[0111] Furthermore, step 3 specifically includes the following steps:

[0112] Step 3.1, neural network initialization:

[0113] According to the state space and action space, the strategy network P and evaluation network Q are established respectively. Except for the input layer, the number of layers of the P network and the Q network are 11 and 10 respectively. The P network takes the input state and outputs the action. The activation function of the output layer selects Sigmoid mapping to the interval 0-1 to facilitate the scaling and restoration of the action. The Q network takes the input state and action and outputs the reward.

[0114] Step 3.2, algorithm parameter setting:

[0115] Training parameter setting: Set the learning rate α of the network P and α Q , target network update coefficient τ, scheduling period T, number of training rounds e max , the discount factor γ in step 2.2, the experience recycling area Memory, the training batch size Batchsize;

[0116] OU and G noise parameter settings: value regression rate θ, mean μ, volatility σ, maximum volatility σ max , minimum fluctuation σ min , noise attenuation period T Nd ;

[0117] Reward function parameter settings: coefficients k1, k2, k3, k4, l1, l2, l3, l4, l5 and l6;

[0118] Step 3.3, training scene settings:

[0119] Before each training round, within the typical day scenario and the feasible domain of equipment safe operation, a normal random fluctuation of no more than 30% is added to the mean of the load source data, and new typical day training scenarios are continuously generated until the end of the training;

[0120] Step 3.4, MDP training loop:

[0121] Before each training, the initial state S0 of the system is obtained; for each time step state s t , the policy network is based on the transfer matrix P(s t |θ P ) Output action a t , the system gets reward r after executing the action t , and get the next state s t+1 ; training experience (s t ,a t ,r t ,s t+1 ) is saved to the experience replay pool, the current state is updated, and the MDP is repeated until the end to complete one round of training; the report calculation is performed through multi-step returns to better estimate the Q value. The multi-step return formula is:

[0122]

[0123] in, represents the multi-step return value, γ represents the discount factor, r t+n (n=0,1,...,n-1) represents the reward value of each step.

[0124] Step 3.5: Network parameter update:

[0125] When the experience parameters of the experience replay pool meet the batch size, the network parameters are updated at each time step, the policy network and the evaluation network are updated, and the target network is soft-updated.

[0126] The present invention also discloses a computer device, comprising a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method of the present invention.

[0127] The present invention also discloses a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method of the present invention are implemented.

[0128] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0129] The integrated energy system model established by the present invention fully considers the system operation status and continuous equipment scheduling, and can fully reflect the dynamic operation process of the equipment in the integrated energy system;

[0130] The present invention adds G noise on the basis of traditional DDPG which only adds OU noise, and weights the noise to increase the diversity of strategy exploration. At the same time, multi-step returns are used when updating returns to improve the stability of the strategy. The optimized reinforcement learning DDPG algorithm has the ability to solve the complex electricity / cold / heat / hydrogen integrated energy system scheduling problem.

[0131] By dealing with complex continuous control problems, the optimized DDPG algorithm in the present invention can effectively realize the coordinated scheduling of multiple energies such as electricity, heat, cold, hydrogen, etc. in the integrated energy system, thereby improving the overall performance and adaptability of the system.

[0132] Under different uncertainty conditions, the optimized DDPG algorithm of the present invention demonstrates higher real-time scheduling capability and system adaptability, indicating that it has advantages in dealing with multi-energy flows and strongly coupled energy systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 This is a schematic diagram of the electricity / heat / cold / hydrogen integrated energy system;

[0134] Figure 2Schematic diagram of the DDPG strategy network and evaluation network structure;

[0135] Figure 3 are examples of mean and range values ​​for random conditions;

[0136] Figure 4 Cumulative reward training curve for DDPG;

[0137] Figure 5 The dispatch results of SOC and HSD;

[0138] Figure 6 The scheduling results of TP and CSD;

[0139] Figure 7 For the optimized reinforcement learning DDPG process. DETAILED DESCRIPTION

[0140] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0141] The present invention provides a method for scheduling an electricity / cold / heat / hydrogen integrated energy system based on optimized reinforcement learning DDPG, comprising the following steps:

[0142] Step S1: Establishing a mathematical model, objective function and constraints of an electricity / cold / heat / hydrogen integrated energy system;

[0143] Step S2 uses Matlab solver to write optimized reinforcement learning DDPG code and input the mathematical model;

[0144] Step S3 uses randomly generated working conditions to perform DDPG training;

[0145] Step S4 analyzes the DDPG training results.

[0146] The electricity / cold / heat / hydrogen integrated energy system of step S1 is as follows Figure 1 As shown, it includes photovoltaic modules, wind turbines, power grids, PEM electrolyzers, PEM fuel cells, batteries, hydrogen storage tanks, air source heat pumps, electric heating boilers, steam heat accumulators, chillers, and ice storage equipment.

[0147] Photovoltaic modules:

[0148] As a partial power source, the output power of photovoltaic modules can be simplified as:

[0149]

[0150] Among them, η PV and η inv are photovoltaic efficiency and inverter efficiency respectively, A PVis the effective illumination area, I0 is the solar irradiance, the unit is watt per square meter (W / m 2 ).

[0151] Wind turbines:

[0152] When the wind turbine is running, the equation for power generation is as follows:

[0153]

[0154] in, is the wind power, ρ and R are respectively the air density and the radius of the wind turbine impeller, C p is the wind energy utilization coefficient. According to Betz theory, the maximum wind energy utilization rate C pmax ≈0.593, λ, β, and V are the tip speed ratio, pitch angle, and wind speed, respectively.

[0155] The power generation efficiency of wind turbines is as follows:

[0156]

[0157] Among them, V in is the starting wind speed, V out The shutdown wind speed.

[0158] Proton exchange membrane electrolyzer

[0159] As a hydrogen production equipment, the proton exchange membrane electrolyzer has the advantages of high electrolysis efficiency, fast response and compact design. The hydrogen production rate is as follows:

[0160]

[0161] Among them, H ET is the hydrogen production rate, η F is the Faraday efficiency, n ET is the number of electrolytic cells, I ET is the working current, and F is the Faraday constant.

[0162] Proton Exchange Membrane Fuel Cell

[0163] The hydrogen consumption of proton exchange membrane fuel cells is calculated using Faraday's law:

[0164]

[0165] Among them, H FC is the hydrogen consumption rate, I FC is the operating current, n FC is the number of batteries.

[0166] The fuel cell reaction rate can be expressed by the current density j:

[0167]

[0168] Among them, A cell is the reaction area of ​​the battery cell.

[0169] Thermal power output of fuel cells It is expressed as:

[0170]

[0171] Among them, η T is the fuel cell thermal efficiency, η E is the electrical efficiency of the fuel cell.

[0172] Battery

[0173] The battery is usually operated in battery pack mode, and its total power can be expressed as:

[0174]

[0175] in, is the total power output of the battery pack, is the battery unit power, n1 and n2 are the number of battery rows and columns respectively.

[0176] The current of the battery cell is as follows:

[0177]

[0178] Among them, the open-loop voltage V OC and the battery cell internal resistance R b can all be considered as fixed values.

[0179] The dynamic characteristic SOC of the battery cell can be expressed as:

[0180]

[0181] Among them, t (0≤k≤T) is the scheduling time, T is the scheduling period, Q B is the battery cell capacity. b,k is the current, charging is negative and discharging is positive.

[0182] Hydrogen storage tank

[0183] This system uses a high-pressure hydrogen storage tank (HST) model to store hydrogen produced by the electrolyzer and provide a source of hydrogen for fuel cell power generation. The pressure inside the hydrogen storage tank changes dynamically with the storage and release of hydrogen, which can be expressed as:

[0184]

[0185] Among them, TPt H and are the internal pressure and hydrogen flow rate (mol / s) of the hydrogen storage tank at time t, V H is the volume of the hydrogen storage tank, T H is the operating temperature, and R is the universal gas constant. t is the compressibility factor of hydrogen, which can be calculated based on NIST data and the Lemmon equation:

[0186]

[0187] Among them, a i 、b i 、c i They are all constants, and their specific values ​​are given in the table. T0 = 100K, R = 8.3145 J / mol·K.

[0188]

[0189] Air source heat pump

[0190] Air source heat pumps (HP) can provide peak thermal power under high load. The coefficient of performance (COP) of the heat pump is related to the temperature difference, and its empirical relationship can be fitted as:

[0191]

[0192] Where, ΔT HP is the cooling / heating target temperature T D / T H With the current ambient temperature T a The difference, h1, h2 and h3 are fitting coefficients.

[0193] The heating power of the heat pump can be expressed by the formula:

[0194]

[0195] Among them, P H HP is the heat pump thermal power, P E HP For the corresponding electrical power, COP H is the heating performance coefficient.

[0196] The cooling power of the heat pump can be expressed as:

[0197]

[0198] in, is the cooling power of the heat pump, For the corresponding electrical power, COP C is the cooling performance coefficient.

[0199] Electric heating boiler

[0200] This system is equipped with an electric-heated boiler (EB) as a thermal device that can convert electrical energy into thermal energy. In addition, the electric boiler does not produce pollution and noise and occupies a small area. Its thermal efficiency is as high as 98% and has good energy conversion performance.

[0201] The heating power of an electric boiler can be expressed as:

[0202]

[0203] Among them, η EB is the conversion efficiency of the electric boiler.

[0204] Steam accumulator

[0205] The thermal energy storage (TES) component model of this system embeds the PCM module on the top of the hot water tank and ensures a constant TES outlet temperature according to the steam supply standard of a specific industrial park. To describe the dynamic characteristics of TES, a heat storage state (heat storage degree, HSD) similar to the battery SOC is introduced and defined as:

[0206]

[0207] in, is the charging (negative) or releasing (positive) power of the steam accumulator at time t, H c The maximum heat storage capacity of the steam accumulator. HSD is limited to between 0 and 1.

[0208] Refrigeration Unit

[0209] The refrigeration unit (refrigeration / air condition / chiller, RAC) selected in this system can convert electrical energy into cooling capacity. The refrigeration unit has a high energy efficiency ratio and can achieve efficient energy conversion.

[0210] The cooling power of the refrigeration unit can be expressed as:

[0211]

[0212] Among them, COP RAC is the energy efficiency ratio of the refrigeration unit.

[0213] Ice storage device

[0214] In order to describe the dynamic characteristics of the ice energy storage (CES), the cold storage degree (CSD) similar to the steam accumulator is introduced and defined as:

[0215]

[0216] in, is the charging (negative) or discharging (positive) power of the ice storage device at time t, C c It is the maximum cold storage capacity of the ice storage device. CSD is limited to 0~1.

[0217] Objective Function

[0218] The objective function mainly includes the operating cost of the system Cost of purchasing electricity from the grid and the status of the energy storage device

[0219]

[0220] Among them, c i and are the operating cost per kWh of the equipment in the system and the power at time t, N is the number of equipment, p t is the power purchase cost of the power grid. α, k1, k2, k3 and k4 are all weight coefficients.

[0221] Operational constraints

[0222] Electrical balance constraints:

[0223] Thermal balance constraints:

[0224] Cold balance constraints:

[0225] Hydrogen balance constraints:

[0226] Refrigeration unit power upper and lower limit constraints:

[0227] The upper and lower limits of proton exchange membrane fuel cell power:

[0228] Air source heat pump power upper and lower limits:

[0229] Electric heating boiler power upper and lower limit constraints:

[0230] The upper and lower limits of the power of the proton exchange membrane electrolyzer are:

[0231] Hydrogen flow rate upper and lower limits:

[0232] Battery state upper and lower limit constraints: SOC min ≤SOC t ≤SOC max

[0233] Hydrogen storage tank pressure upper and lower limit constraints: TP min ≤TP t ≤TP max

[0234] Steam accumulator heat storage state upper and lower limit constraints: HSD min ≤HSD t ≤HSD max

[0235] Ice storage device cold storage state upper and lower limit constraints: CSD min ≤CSD t ≤CSD max

[0236] Corresponding device initialization state constraints:

[0237] Step S2, use Matlab solver to write reinforcement learning DDPG code and input the mathematical model.

[0238] DDPG algorithm training process

[0239] Exploring Noise and Markov Decision Processes

[0240] DDPG is a classic deep reinforcement learning algorithm that can be used to solve the continuous control problem of integrated energy systems. In order to better explore the unknown environment in the decision-making process, the Ornstein-Uhlenbeck (OU) noise suitable for inertial systems is first added. Since this system is more complex than the traditional electric-heat synergistic system, Gaussian noise (Gaussian, G) is introduced And set the noise attenuation to achieve a balance between exploration and utilization. The specific mathematical expression is as follows:

[0241]

[0242] Where t is the time step, is the OU noise value, θ is the mean reversion rate, μ is the mean, σ is the volatility, W t For Brownian motion, is the G noise value, and z is a random variable sampled from a standard normal distribution (mean 0, variance 1).t is the action, P(s t |θ P ) is the actor network in DDPG, θ p is the network parameter, s t is the system state. The temporal correlation of the OU process enables the system to perform multiple steps of exploration in the same direction to accumulate training experience, thereby effectively improving the exploration efficiency of the inertial system. The G process introduces a certain degree of randomness in the strategy when selecting actions, thereby encouraging exploration.

[0243] Reinforcement learning tasks are usually described as Markov decision processes (MDPs), which are defined as five tuples: (S, A, P, R, γ), which are state, action, transition matrix, reward, and discount factor. Since this paper studies the model-free DDPG algorithm, there is no transition matrix. Based on the characteristics of the electric-heat synergistic system, the state space, action space, and reward function are defined as follows:

[0244] State Space

[0245]

[0246] Where t is the time step, and are the photovoltaic power generation and wind power generation at time t, and They are the electrical load, thermal load, cooling load and hydrogen load at time t, SOC t TP t , HSD t and CSD t They are the battery storage state, hydrogen storage tank pressure, heat storage state of the heat storage tank and cold storage state of the ice storage device at time t respectively.

[0247] Action Space

[0248]

[0249] in, The flow rate of hydrogen to generate electricity for the fuel cell, and They are refrigeration unit power, heat pump power, electric boiler power, and electrolyzer hydrogen production power.

[0250] Reward Function

[0251]

[0252] in, and H out,tThey represent the corresponding power surplus (greater than 0) or shortage (less than 0) after the four energy storage devices participate in the dispatch. l1~l6 are weights adjusted according to training experience and results. and They respectively mean that when electricity prices are low, it is encouraged to purchase electricity from the power grid to electrolyze water to produce hydrogen to make up for the hydrogen consumption in hydrogen storage tanks, and to use refrigeration units to make up for the cold consumption of ice storage equipment.

[0253] DDPG algorithm training process

[0254] (1) Neural network initialization

[0255] According to the state space and action space, the policy network (P network) and evaluation network (Q network) are established respectively, as shown in the figure. Except for the input layer, the number of layers of the P network and Q network are 11 and 10 respectively. The P network has 12 inputs (states) and 5 outputs (actions). The activation function of the output layer selects Sigmoid mapping to the interval 0-1 to facilitate the scaling and restoration of actions; the Q network has 17 inputs (states + actions) and 1 output (reward).

[0256] (2) Algorithm parameter setting

[0257] Training parameter setting: Set the learning rate α of the network P and α Q , target network update coefficient τ, scheduling period T, number of training rounds e max , discount factor γ, experience recycling area Memory, training batch size Batchsize.

[0258] OU and G noise parameter settings: value regression rate θ, mean μ, volatility σ, maximum volatility σ max , minimum fluctuation σ min , noise attenuation period T Nd .

[0259] Reward function parameter settings: coefficients k1, k2, k3, k4, l1, l2, l3, l4, l5 and l6.

[0260] (3) Training scenario setting

[0261] Before the start of each training round, within the typical day scenario and the feasible domain of safe operation of the equipment, a normal random fluctuation of no more than 30% is added to the mean of the load source data, and new typical day training scenarios are continuously generated until the end of the training.

[0262] (4) MDP training loop

[0263] Before each training, the initial state S0 of the system is obtained; for each time step state s t , the policy network is based on P(s t|θ P ) Output action at, the system gets reward rt after executing the action, and gets the next state s t+1 ; training experience (s t ,a t ,r t ,s t+1 ) is saved to the experience replay pool, the current state is updated, and the MDP is repeated until the end to complete one round of training. Traditional DDPG uses single-step returns for report calculation. Through multi-step returns, the Q value is better estimated and the stability is improved. Multi-step return formula:

[0264] (5) Network parameter update

[0265] When the experience replay pool experience parameters meet the batch size, the network parameters will be updated at each time step. The policy network and evaluation network are updated, and the target network is soft-updated.

[0266] Step S3, using randomly generated working conditions to perform DDPG training.

[0267] The number of training rounds is set to 200, and 200 sets of training scenarios need to be randomly generated. Examples of the mean and fluctuation range of electric load, thermal load, cooling load, hydrogen load, photovoltaic power generation, and wind power generation are as follows: Figure 3 The solid line is the load mean, and the shaded line is the range in which the data can fluctuate.

[0268] Step S4, analyzing the DDPG training results.

[0269] Embodiment 1:

[0270] Training scenarios and system settings

[0271] The number of training rounds in this embodiment is 200, and 200 sets of training scenarios need to be randomly generated. The mean and fluctuation range of electric load, heat load, cooling load, hydrogen load, photovoltaic power generation and wind power generation are as follows: Figure 3 The solid line is the load mean, and the shaded line is the range in which the data can fluctuate.

[0272] Based on typical daily load data, the specifications and safety constraints of each device in the integrated energy system are given in Table 1.

[0273] In Table 1, the effective area of ​​the photovoltaic power generation module is 10m 2 , the converter efficiency is 0.95. The radius of the wind turbine impeller is 30m, and the air density is 1.29kg / m 3The Faraday efficiency of the electrolyzer is 0.95, and the maximum and minimum powers are 10kW and 1kW. The battery cell capacity is 100Ah, the battery internal resistance is 20mΩ, the open circuit voltage is 4.8V, the upper and lower limits of the state of charge SOC are 0.9 and 0.2 respectively, and the number of rows and columns of the battery array are both 13; the fuel cell rated power is 5kW, the maximum and minimum outputs are 5kW and 0.5kW, the low calorific value of hydrogen is 119.96kJ / g, the rated electrical efficiency is 37%, and the rated thermal efficiency is 53%. The hydrogen storage tank capacity is 5m 3 , the upper and lower limits of TP in hydrogen storage state are 7 and 3.5 respectively. The rated power of the heat pump is 10kW, the maximum heating power is 10kW, and the minimum heating power is 0kW. The thermal efficiency of the electric heating boiler is 0.98, and the maximum and minimum heating powers are 5kW and 0kW. The capacity of the steam accumulator is 5kWh, and the upper and lower limits of HSD in heat storage state are 1 and 0 respectively. The rated power of the refrigeration unit is 5kW, the capacity is 4kWh, and the upper and lower limits of CSD in cold storage state are 0.9 and 0.2 respectively.

[0274] Table 1 Main parameters of the equipment

[0275]

[0276] Algorithm implementation and parameter adjustment

[0277] Before each round of training, a set of system operating conditions is randomly generated based on typical daily operating conditions and the safe feasible domain of equipment operation, including ambient temperature, solar irradiance, electrical load and thermal load. The random conditions correspond to the multiple uncertainties in system operation. In addition, the state of the energy storage device is initialized.

[0278] In each training round, according to the current state S t and action output a t After interacting with the environment, you get a reward r t and the next state S t+1 , and the data group (S t ,a t ,r t ,S t+1 ) is saved to the experience replay memory. When updating the network parameters, a batch of training data is randomly selected from the experience replay memory and shuffled to avoid network overfitting.

[0279] Table 2 Main parameter settings of DDPG algorithm

[0280]

[0281] In this problem, the power of the fuel cell and heat pump is scaled by the 0-1 output value of the Sigmoid function, while the energy storage state of the battery and heat storage tank is updated by the electric and thermal power balance calculation at the current moment. If the power balance cannot be guaranteed, the current excess or shortage of electric and thermal power is calculated. In terms of parameter adjustment, the parameters of the energy storage reward item can be adjusted appropriately according to the scheduling results, but should not be too large to avoid affecting the power balance at the previous moment; the reward values ​​for energy surplus and shortage should be greater than the fuel cost reward value to ensure energy efficiency; due to the high efficiency of the heat pump, the coefficient of the heat surplus reward item should be slightly larger than the coefficient of the heat shortage reward item, while the opposite is true for electric energy, so as to avoid the algorithm from failing to converge.

[0282] The DDPG cumulative training curve is shown in the figure, where “mean” represents the mean of the cumulative rewards for every 5 training sessions. Figure 4 ,The cumulative reward converges quickly in the first 25 rounds, and converges in the 173 rounds as the noise continues to decrease.

[0283] Depend on Figure 5 and Figure 6 As shown in the figure, the scheduling results of the four energy storage devices are all within reasonable operating conditions, indicating that the scheduling strategy can ensure the effective storage and release of energy and absorb the excess energy of the system.

Claims

1. A comprehensive energy system scheduling method based on optimized reinforcement learning DDPG, characterized in that: The steps include: Step 1: Construct the mathematical model, objective function and operation constraints of the electricity / cold / heat / hydrogen integrated energy system; Step 2: Introduce Ornstein-Uhlenbeck OU noise and Gaussian noise suitable for inertial systems, write the optimized reinforcement learning DDPG algorithm based on the Markov decision process MDP using Matlab solver and input the mathematical model; Step 3: Use randomly generated working conditions for DDPG training; Step 4: Analyze the DDPG training results and obtain the optimized reinforcement learning DDPG to achieve the coordinated scheduling of multiple energies such as electricity, heat, cold, and hydrogen in the integrated energy system.

2. According to claim 1, a comprehensive energy system scheduling method based on optimized reinforcement learning DDPG is characterized in that: In step 1, the electricity / cold / heat / hydrogen integrated energy system includes photovoltaic modules, wind turbines, power grids, proton exchange membrane electrolyzers, proton exchange membrane fuel cells, batteries, hydrogen storage tanks, air source heat pumps, electric heating boilers, steam heat accumulators, chillers and ice storage equipment.

3. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 2 is characterized in that: Photovoltaic modules: As a partial power source, the output power of photovoltaic modules can be simplified as: Among them, η PV and η inv are photovoltaic efficiency and inverter efficiency, A PV is the effective illumination area, I0 is the solar irradiance, the unit is watt per square meter; Wind turbines: When the wind turbine is running, the equation for power generation is as follows: in, is the wind power, ρ and R are respectively the air density and the radius of the wind turbine impeller, C p is the wind energy utilization coefficient. According to Betz theory, the maximum wind energy utilization rate C pmax ≈0.593, λ, β, and V are the tip speed ratio, pitch angle, and wind speed, respectively; The power generation efficiency of wind turbines is as follows: Among them, V in is the starting wind speed, V out is the shutdown wind speed; Proton exchange membrane electrolyzer: The proton exchange membrane electrolyzer is used as a hydrogen production device, and its hydrogen production rate is as follows: Among them, H ET is the hydrogen production rate, η F is the Faraday efficiency, n ET is the number of electrolytic cells, I ET is the working current, F is the Faraday constant; Proton exchange membrane fuel cell: The hydrogen consumption of proton exchange membrane fuel cells is calculated using Faraday's law: Among them, H FC is the hydrogen consumption rate, I FC is the operating current, n FC is the number of batteries; The fuel cell reaction rate can be expressed by the current density j: Among them, A cell is the reaction area of ​​the battery cell; Thermal power output of fuel cells It is expressed as: Among them, η T is the thermal efficiency of the fuel cell, η E is the fuel cell electrical efficiency; Battery: The battery operates in battery pack mode, and its total power is expressed as: in, is the total power output of the battery pack, is the battery unit power, n1 and n2 are the number of battery rows and columns respectively; The current of the battery cell is as follows: Among them, the open-loop voltage V OC and the battery cell internal resistance R b All can be regarded as fixed values; The dynamic characteristic SOC of the battery cell can be expressed as: Among them, t is the scheduling time, T is the scheduling period, Q B is the battery cell capacity, I b,k is the current, charging is negative and discharging is positive; Hydrogen storage tank: A high-pressure hydrogen storage tank model HST is used to store hydrogen produced by the electrolyzer and provide a source of hydrogen for fuel cell power generation. The pressure inside the hydrogen storage tank changes dynamically with the storage and release of hydrogen, expressed as: Among them, TP t H and are the internal pressure and hydrogen flow rate of the hydrogen storage tank at time t, in mol / s, V H is the volume of the hydrogen storage tank, T H is the operating temperature, R is the universal gas constant, z t is the compressibility factor of hydrogen, which can be calculated based on NIST data and the Lemmon equation: Among them, a i , b i 、c i All are constants, T0 = 100K, R = 8.3145 J / mol·K; Air source heat pump: The air source heat pump HP provides peak thermal power under high load. The thermal performance coefficient COP of the heat pump is related to the temperature difference, and its empirical relationship is fitted as follows: Where, ΔT HP is the cooling / heating target temperature T D / T H With the current ambient temperature T a The difference, h1, h2 and h3 are the fitting coefficients; The heating power of the heat pump is expressed by: in, is the heat pump thermal power, For the corresponding electrical power, COP H is the heating performance coefficient; The cooling power of the heat pump is expressed as: in, is the cooling power of the heat pump, For the corresponding electrical power, COP C is the cooling performance coefficient; Electric heating boiler: As a thermal equipment that converts electrical energy into thermal energy, the heating power of the electric boiler EB can be expressed as: Among them, η EB is the conversion efficiency of the electric boiler; Steam accumulator: The thermal storage tank TES component model embeds the PCM module on the top of the hot water tank and ensures that the TES outlet temperature is constant according to the steam supply standard of a specific industrial park. To describe the dynamic characteristics of TES, a heat storage state HSD similar to the battery SOC is introduced and defined as: in, is the charging or releasing power of the steam accumulator at time t, H c The maximum heat storage value of the steam accumulator; HSD is limited between 0 and 1; Refrigeration unit: The refrigeration unit RAC converts electrical energy into cooling capacity, and its cooling power can be expressed as: Among them, COP RAC is the energy efficiency ratio of the refrigeration unit; Ice storage device: In order to describe the dynamic characteristics of the ice storage device CES, the cold storage state CSD similar to the steam accumulator is introduced and defined as: in, is the charging or discharging power of the ice storage device at time t, C c It is the maximum cold storage capacity of the ice storage device. CSD is limited between 0 and 1.

4. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 3 is characterized in that: In step 1, the objective function is specifically: The objective function mainly includes the operating cost of the system Cost of purchasing electricity from the grid and the status of the energy storage device Among them, c i and are the operating cost per kWh of the equipment in the system and the power at time t, N is the number of equipment, p t is the power purchase cost from the power grid, α, k1, k2, k3 and k4 are all weight coefficients.

5. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 3 is characterized in that: In step 1, the operation constraints are specifically: Electrical balance constraints: Thermal balance constraints: Cold balance constraints: Hydrogen balance constraints: Refrigeration unit power upper and lower limit constraints: The upper and lower limits of proton exchange membrane fuel cell power: Air source heat pump power upper and lower limits: Electric heating boiler power upper and lower limits: The upper and lower limits of the power of the proton exchange membrane electrolyzer are: Hydrogen flow rate upper and lower limits: Battery state upper and lower limit constraints: SOC min ≤SOC t ≤SOC max Hydrogen storage tank pressure upper and lower limit constraints: TP min ≤TP t ≤TP max Steam accumulator heat storage state upper and lower limit constraints: HSD min ≤HSD t ≤HSD max Ice storage device cold storage state upper and lower limit constraints: CSD min ≤CSD t ≤CSD max Corresponding device initialization state constraints:

6. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 1 is characterized in that: Step 2 specifically includes the following steps: Step 2.1: First, add the Ornstein-Uhlenbeck OU noise applicable to the inertial system Due to the complexity of the system, Gaussian noise G is introduced And set the noise attenuation to achieve a balance between exploration and utilization. The specific mathematical expression is as follows: Where t is the time step, is the OU noise value, θ is the mean reversion rate, μ is the mean, σ is the volatility, W t For Brownian motion, is the G noise value, z is a random variable sampled from a standard normal distribution, with a mean of 0 and a variance of 1; a t is the action, P(s t |θ P ) is the actor network in DDPG, θ p is the network parameter, s t is the system status; Step 2.2, reinforcement learning tasks are usually described as a Markov decision process MDP, defined as a five-tuple: S, A, P, R, γ, which are state space, action space, transition matrix, reward function and discount factor respectively.

7. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 6 is characterized in that: In step 2.2, the state space is as follows: Where t is the time step, and are the photovoltaic power generation and wind power generation at time t, and They are the electrical load, thermal load, cooling load and hydrogen load at time t, SOC t TP t , HSD t and CSD t They are the battery storage state, hydrogen storage tank pressure, heat storage state of the heat storage tank and cold storage state of the ice storage device at time t respectively; The action space is as follows: in, The flow rate of hydrogen to generate electricity for the fuel cell, and They are the power of refrigeration unit, heat pump, electric boiler and electrolyzer hydrogen production; The reward function is as follows: in, and H out,t They represent the corresponding power surplus or shortage after the four energy storage devices participate in the dispatch, and l1~l6 are the weights adjusted according to the training experience and effect. and They respectively mean that when electricity prices are low, it is encouraged to purchase electricity from the power grid to electrolyze water to produce hydrogen to make up for the hydrogen consumption in hydrogen storage tanks, and to use refrigeration units to make up for the cold consumption of ice storage equipment.

8. The integrated energy system scheduling method based on optimized reinforcement learning DDPG according to claim 1 is characterized in that: Step 3 specifically includes the following steps: Step 3.1, neural network initialization: According to the state space and action space, the strategy network P and evaluation network Q are established respectively. Except for the input layer, the number of layers of the P network and the Q network are 11 and 10 respectively. The P network takes the input state and outputs the action. The activation function of the output layer selects Sigmoid mapping to the interval 0-1 to facilitate the scaling and restoration of the action. The Q network takes the input state and action and outputs the reward. Step 3.2, algorithm parameter setting: Training parameter setting: Set the learning rate α of the network P and α Q , target network update coefficient τ, scheduling period T, number of training rounds e max , the discount factor γ in step 2.2, the experience recycling area Memory, the training batch size Batchsize; OU and G noise parameter settings: value regression rate θ, mean μ, volatility σ, maximum volatility σ max , minimum fluctuation σ min , noise attenuation period T Nd ; Reward function parameter settings: coefficients k1, k2, k3, k4, l1, l2, l3, l4, l5 and l6; Step 3.3, training scene settings: Before each training round, within the typical day scenario and the feasible domain of equipment safe operation, a normal random fluctuation of no more than 30% is added to the mean of the load source data, and new typical day training scenarios are continuously generated until the end of the training; Step 3.4, MDP training loop: Before each training, the initial state S0 of the system is obtained; for each time step state s t , the policy network is based on the transfer matrix P(s t |θ P ) Output action a t , the system gets reward r after executing the action t , and get the next state s t+1 ; training experience (s t ,a t ,r t ,s t+1 ) is saved to the experience replay pool, the current state is updated, and the MDP is repeated until the end to complete one round of training; report calculation is performed through multi-step returns to better estimate the Q value. The multi-step return formula is: in, represents the multi-step return value, γ represents the discount factor, r t+n (n=0,1,...,n-1) represents the reward value of each step; Step 3.5: Network parameter update: When the experience parameters of the experience replay pool meet the batch size, the network parameters are updated at each time step, the policy network and the evaluation network are updated, and the target network is soft-updated.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method of claim 1.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to claim 1 are implemented.

Images