Comprehensive energy system model prediction control method based on deep reinforcement learning

Abstract

The invention discloses an integrated energy system model prediction control method based on deep reinforcement learning, which is used in the field of energy prediction control, and comprises the following steps: constructing a state prediction model according to the state, output increment and disturbance quantity of each unit in integrated energy equipment in a park; predicting the output state of each device of the system at each moment in the prediction time domain within the day, establishing a rolling optimization objective function based on a prediction value and a day-ahead plan value, and calculating an intra-day scheduling optimal control quantity; and constructing an intelligent agent, and learning weight parameters in the rolling optimization objective function by using a double-delay depth deterministic strategy gradient algorithm to realize optimization of model prediction control of the integrated energy system. According to the invention, through learning of the neural network on different scenes, the optimization efficiency and quality of the algorithm on the integrated energy system are improved.

Description

Technical Field

[0001] The present invention relates to the field of energy prediction and control, and in particular, to a comprehensive energy system model prediction and control method based on deep reinforcement learning. Background Art

[0002] Traditional energy systems usually adopt decentralized dispatching methods, and lack coordination and optimization between various energy subsystems, resulting in low system efficiency and serious energy waste. To solve this problem, integrated energy system optimization dispatching technology has emerged, using advanced information technology, mathematical modeling and optimization algorithms to achieve comprehensive monitoring, prediction and dispatching of energy systems. At present, integrated energy optimization still faces some challenges, especially when dealing with the complexity and uncertainty of energy systems, which may show unstable phenomena. For example, in the park-level integrated energy system, taking photovoltaic power generation as an example, it has the advantages of clean and renewable, but due to the instability of factors such as weather, photovoltaic power generation has temporal fluctuations, so there will be insufficient utilization of photovoltaic power.

[0003] Model Predictive Control (MPC) is favored for its efficient optimization capabilities and handling of complex constraints in integrated energy systems. It uses system models to predict future states and make real-time control decisions to achieve optimal energy allocation and stable system operation. However, the shortcoming of the MPC algorithm is its dependence on model accuracy. Fixed model parameters may not be able to fully adapt to the dynamic changes and fluctuations of the external environment, resulting in a decrease in control performance. This deficiency is particularly evident when facing rapidly changing energy demand and supply conditions. In order to overcome this challenge, it is necessary to develop more flexible and adaptive control strategies that can update model parameters in real time to better reflect the actual operating status of the system, thereby improving the robustness and adaptability of the control system.

[0004] By taking advantage of its data-driven and model-free characteristics, reinforcement learning algorithms have the potential to handle the increased uncertainty caused by the introduction of renewable energy generation. Reinforcement learning can be trained and learned through the temporal difference (TD) method, without the need for a dynamic model of the environment, and can be learned simply through interaction with the environment. The main advantage of reinforcement learning-based algorithms is that, if properly designed and trained, they can capture the uncertainty of the energy system by leveraging historical data, provide high-quality solutions in real time, and avoid the computational burden during operation; however, the most critical challenge for the deployment of reinforcement learning algorithms in the actual energy system scheduling framework is that they lack constraint enforcement guarantees; although some safe reinforcement learning algorithms have addressed this issue, these methods cannot meet the required safety level for energy system operation. In general, model-based optimization methods, such as model predictive control, can ensure the feasibility of the defined integrated energy system scheduling by setting hard constraints in mathematical formulas, which is not possible in current reinforcement learning algorithms. Therefore, combining reinforcement learning and model predictive control algorithms to achieve scheduling is a promising direction.

[0005] For example, Chinese Patent Publication No. CN116245333A discloses a method for optimizing the scheduling of a campus integrated energy system based on model predictive control, which is used to reduce the system operating cost and achieve complementary advantages in the performance of equipment within the system; Chinese Patent Publication No. CN116488150A introduces the Q-learning algorithm into model predictive control to track and evaluate the prediction errors of uncertain variables such as microgrid power generation and power load, and dynamically optimizes and adjusts the control parameters of MPC accordingly.

[0006] The technical disadvantages of existing model predictive control algorithm solutions are as follows: the control performance of the model predictive control algorithm depends on the accuracy of the model and the quality of the parameters. However, with the increasing proportion of renewable energy power generation connected to the grid and the uncertainty of user load, fixed models and parameters cannot accurately capture the system status at all times, which will lead to deviations in prediction and control.

[0007] Technical disadvantages of existing reinforcement learning schemes: The most critical challenge in deploying reinforcement learning algorithms in the actual energy system scheduling framework is that they lack constraint enforcement guarantees; although some safe reinforcement learning algorithms have solved this problem, these methods cannot meet the safety level required for energy system operation; in a new energy microgrid optimization operation method based on model predictive control, the reinforcement learning Q-learning algorithm can be used to learn the weight parameters of the MPC rolling optimization objective function and obtain the optimal value of the scheduling control at each moment. However, this method still faces the following problems:

[0008] (1) Limitations of state space and action space: In high-dimensional state-action space, the tabular method of the Q-learning algorithm is difficult to handle, resulting in low learning efficiency or even failure to learn.

[0009] (2) It is impossible to output continuous action values. The Q-learning algorithm must discretize the action values ​​in order to learn.

[0010] (3) Learning parameters are limited. It is difficult for the Q-learning algorithm to learn multiple parameters at the same time. When the system scale increases, it is necessary to adjust the relationship between multiple parameters at the same time to achieve better results.

[0011] (4) The Q-table learned by the Q-learning algorithm is specific to the training environment and has very limited ability to migrate between different scenarios, which greatly limits its application in complex and dynamic environments.

[0012] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention

[0013] In order to overcome the above problems, the present invention aims to propose a model predictive control method for an integrated energy system based on deep reinforcement learning, with the aim of solving the problem that the Q table learned by Q-learning is specific to the training environment and has very limited migration ability between different scenarios, and Q-learning can only generate discrete actions, which greatly limits its application in complex and dynamic environments.

[0014] To this end, the specific technical solution adopted by the present invention is as follows:

[0015] A comprehensive energy system model predictive control method based on deep reinforcement learning, the comprehensive energy system model predictive control method comprising the following steps:

[0016] S 1. Establish a comprehensive energy system model based on the energy supply and demand balance of the load in the park's comprehensive energy equipment;

[0017] S2. Based on the minimum full-day energy cost objective function, the distributed robust optimization algorithm is used to calculate the day-ahead output plan value of the integrated energy system model; a state prediction model is constructed according to the state, output increment and disturbance of each unit in the park's integrated energy equipment;

[0018] S3. Use the state prediction model to predict the output state of each device in the system at each time in the prediction time domain within the day, and establish a rolling optimization objective function based on the predicted value and the day-ahead planned value, calculate the optimal control quantity for intraday scheduling, and optimize the state prediction model according to the calculation results;

[0019] S4. Build a deep reinforcement learning agent based on the state prediction model. Based on the framework, use the double-delay deep deterministic policy gradient algorithm to learn the weight parameters in the rolling optimization objective function to achieve the optimization of the predictive control of the integrated energy system model.

[0020] Optionally, based on the minimum all-day energy cost objective function, a distributed robust optimization algorithm is used to calculate the output day-ahead planned value of the integrated energy system model; establishing a state prediction model according to the state, output increment and disturbance of each unit in the park integrated energy equipment includes the following steps:

[0021] S21. Define minimization of the energy cost throughout the day as the objective function and establish a constraint function;

[0022] S22, using KL divergence as a measure of the distance between the actual distribution and the reference distribution, defining the distance between the actual distribution and the reference distribution, and establishing an uncertainty set based on the distance between the actual distribution and the reference distribution;

[0023] S23. Construct a chance-constrained optimization model for uncertain sets using a distributed robust optimization algorithm;

[0024] S24, solving the opportunity constraint optimization model using a mathematical solver, and obtaining the day-ahead planned output value of the integrated energy system equipment;

[0025] S25. Construct a state prediction model based on the state, output increment and disturbance of each unit in the park's comprehensive energy equipment.

[0026] Optionally, using KL divergence as a measure of the distance between the actual distribution and the reference distribution, and defining the distance between the actual distribution and the reference distribution, establishing the uncertainty set based on the distance between the actual distribution and the reference distribution includes the following steps:

[0027] S221, setting a limit value γ of the KL divergence, and using the KL divergence as a measure of the distance between the actual distribution and the reference distribution to define the distance between the actual distribution and the reference distribution;

[0028] S222. Establish an uncertainty set of distribution functions based on the distance between the actual distribution and the reference distribution.

[0029] Optionally, the expression defining the distance between the actual distribution and the reference distribution is:

[0030]

[0031] Where D KL (P||P0) represents the distance between the distribution function P and the reference distribution P0 measured using KL divergence;

[0032] f(ζ) represents the probability density function under the distribution function P;

[0033] f0(ζ) represents the probability density function under the reference distribution P0;

[0034] d ζ represents the integration of ξ.

[0035] Optionally, the expression of the chance-constrained optimization model for uncertain sets constructed using the distributed robust optimization algorithm is:

[0036]

[0037] Ω p ={P|D KL (P||P0)≤γ}

[0038] In the formula, h(x, ζ) represents the optimization objective function;

[0039] x represents the decision variable;

[0040] X represents the feasible domain where the decision variable x is located;

[0041] st represents the constraint condition;

[0042] Pr ～p represents the probability under the uncertainty distribution p;

[0043] H(x,ζ) represents the constraint function;

[0044] Ωp represents the uncertain set of distribution functions;

[0045] γ represents the limit of KL divergence;

[0046] α represents the new confidence level considering the limit γ of KL divergence and the initial confidence α;

[0047] p represents the uncertainty distribution.

[0048] Optionally, constructing a chance-constrained optimization model for an uncertain set using a distributed robust optimization algorithm includes the following steps:

[0049] S231, obtaining an uncertain set, and constructing a robust optimization framework based on the uncertain set by using a distributed robust optimization algorithm;

[0050] S232. Through the robust optimization framework, the objective function is reconstructed, while ensuring that the control quantity satisfies the constraint function under uncertain conditions;

[0051] S232. Decompose the robust optimization framework using a distributed optimization algorithm, and achieve the global optimization goal by collaboratively solving the decomposed robust optimization framework.

[0052] Optionally, the expression for constructing the state prediction model according to the state, output increment and disturbance of each unit in the park comprehensive energy equipment is:

[0053] x t+1 =Ax t +Bu t +Cr t ;

[0054] In the formula, x t Indicates the status of each unit in the park's comprehensive energy equipment at time t;

[0055] x t+1 Table t+1 time each unit in the park comprehensive energy equipment status;

[0056] u t It represents the output increment of each unit at time t;

[0057] r t It represents the disturbance amount of each unit at time t;

[0058] A represents x t Coefficient matrix, B represents u t The coefficient matrix of t The coefficient matrix of , which can be derived from the system energy balance.

[0059] Optionally, the state prediction model is used to predict the output state of each device in the system at each time in the prediction time domain within the day, and a rolling optimization objective function is established based on the predicted value and the day-ahead planned value. The calculation of the optimal control quantity for intraday scheduling includes the following steps:

[0060] S31, using the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain, and establishing a rolling optimization objective function based on the deviation between the predicted value and the day-ahead planned value;

[0061] S32, setting the rolling optimization objective function to include physical limitations on equipment output, smoothness constraints on changes in control quantities, and constraints on the amount of electricity and gas purchased;

[0062] S33, predicting the state of each unit and solving the optimal control quantity within the constraints of each rolling time domain;

[0063] S34. Adjust the state prediction model based on the optimal control quantity and the actual output of the system, and build a feedback mechanism.

[0064] Optionally, using the state prediction model to predict the output state of each device of the system at each moment in the prediction time domain within a day, and establishing a rolling optimization objective function based on the predicted value and the day-ahead planned value includes the following steps:

[0065] S311. At each sampling moment, detect the actual state of each unit in the park's comprehensive energy equipment, and obtain the actual output of each unit at the current moment;

[0066] S312, using the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain;

[0067] S313, establishing a rolling optimization objective function based on the predicted value and the deviation in the day-ahead plan, calculating the optimal control amount and issuing the control instruction at the first moment;

[0068] S314, calculating the error between the model prediction state and the actual output of the system, and correcting the model prediction control model through the feedback correction coefficient according to the calculated error, and entering the next moment rolling optimization.

[0069] Optionally, the expression for establishing the rolling optimization objective function is:

[0070] min f=(X forecast -X ref ) T Q(X forecast -X ref )+u T Hu+YR;

[0071] Where, X forecast Represents the prediction output of MPC in the prediction time domain;

[0072] X ref represents the reference state calculated by the day-ahead optimization;

[0073] H represents the control amount weight matrix;

[0074] Q represents the weight coefficient;

[0075] u represents the system control quantity in the prediction time domain;

[0076] R represents the price of electricity and gas purchased at the corresponding moment;

[0077] Y represents the amount of electricity and gas purchased at each moment in the forecast time domain;

[0078] minf represents the minimization of the rolling optimization objective function.

[0079] Optionally, a deep reinforcement learning agent is constructed based on the state prediction model, and based on the constructed agent, a double-delayed deep deterministic policy gradient algorithm is used to learn the weight parameters in the rolling optimization objective function to achieve the optimization of the integrated energy system model predictive control, including the following steps:

[0080] S41. Determine the dimensions of the state space and the action space using the optimized state prediction model, and construct the critic and actor neural network of the intelligent agent based on the dimensions of the state space and the action space, wherein the state space determines the number of neurons in the input layer, and the action space determines the number of neurons in the output layer;

[0081] S42. Use the double-delayed deep deterministic policy gradient algorithm to initialize the agent's critic and actor neural networks, and design the reward function and penalty mechanism;

[0082] S43, initializing the experience replay unit, storing sample data interacting with the environment, and randomly sampling sample data therefrom for learning and training;

[0083] S44, generating actions with exploration noise in the actor network, executing the actions and storing the experience data, generating new actions with policy noise through the target actor network, calculating the target value using the dual critic network, taking the minimum value of the two networks, thereby reducing network overestimation;

[0084] S45. The gradient of the actor network is calculated and gradually updated through the critic network, and soft update and double delay update strategies are adopted to ensure the stability of learning and achieve optimal control of the integrated energy system model.

[0085] Optionally, determining the dimensions of the state space and the action space using the optimized state prediction model, and constructing the critic and actor neural network of the agent based on the dimensions of the state space and the action space comprises the following steps:

[0086] S411. Use historical data and model evaluation indicators to train the state prediction model, and optimize the state prediction model rolling optimization objective function weight parameters;

[0087] S412, identifying all key state variables in the state prediction model and determining all control actions;

[0088] S413. Design an actor-critic neural network of an intelligent agent with corresponding input layer and output layer; wherein the input layer receives state variables and the output layer generates control actions.

[0089] Optionally, using a double-delayed deep deterministic policy gradient algorithm to initialize the agent's critic and actor neural networks and designing a reward function and a penalty mechanism includes the following steps:

[0090] S421, initialize the actor network and critic network in the dual-delay deep deterministic policy gradient algorithm; wherein the actor network is responsible for generating continuous control actions based on the current environment state and outputting the optimal control strategy; the critic network is used to evaluate the actions generated by the actor network, and the evaluation of the same action by the dual critic networks is used to improve stability and prevent the estimated value from being too high;

[0091] S422. After the actor network generates an action, a reward function is defined. The design of the reward function is centered on minimizing the value of the objective function, encouraging the agent to take actions that help optimize the goal; at the same time, a penalty mechanism is designed;

[0092] S423. The agent maximizes the cumulative reward value by executing the optimal control strategy.

[0093] Optionally, the gradient of the actor network is calculated and gradually updated through the critic network, and the soft update and double-delay update strategies are adopted to ensure the stability of learning and realize the optimal control of the integrated energy system model, including the following steps:

[0094] S451, calculate the gradient of the actor network through the critic network and update the parameters of the actor network;

[0095] S452, train the critic network by minimizing the mean square error;

[0096] S453, based on minimizing the mean square error, a soft update mechanism is used to gradually update the target critic network;

[0097] S454, adopting a double-delay update strategy to update the target actor network at a preset step interval; S454, repeating the steps of S451-S54 until the training is completed, ensuring the stability of the actor network learning and training, and realizing the optimal control of the integrated energy system model.

[0098] Optionally, the expression for calculating the gradient of the actor network is:

[0099]

[0100] In the formula, represents the gradient of the loss function with respect to the actor network parameter φ;

[0101] represents the gradient of the critic network output with respect to action a, and calculates the gradient of the value function Q given the action a generated by the actor network in the current state;

[0102] represents the gradient of the action a output by the actor network with respect to the actor network parameter φ;

[0103] N-1 It means one-N, where N is the number of samples taken at one time.

[0104] Compared with the prior art, the present application has the following beneficial effects: the present invention can improve the ability of the model predictive control algorithm to cope with the uncertainty of the external environment and improve the accuracy and economy of scheduling by dynamically adjusting the control parameters of the model predictive control algorithm; by adopting a policy-based reinforcement learning method, continuous action output can be achieved, and the powerful characterization ability of deep learning can be used to more accurately describe the system state characteristics, and more parameters can be tracked and learned at the same time; the double-delay deep deterministic policy gradient algorithm has better generalization, and the generalization of the intelligent agent scheduling is enhanced through the learning of different scenarios by the neural network, which can be better applied in complex and dynamic environments, thereby improving the optimization efficiency and quality of the algorithm for the integrated energy system. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] The above characteristics, features and advantages of the present invention and their implementation methods and methods will become more clearly understood in conjunction with the following description of the embodiments, which are described in detail in conjunction with the accompanying drawings. Herein, a schematic diagram is shown:

[0106] Figure 1 is a flow chart of a comprehensive energy system model predictive control method based on deep reinforcement learning according to an embodiment of the present invention;

[0107] Figure 2 It is a framework diagram of optimized dispatching control of a comprehensive energy system in a comprehensive energy system model predictive control method based on deep reinforcement learning according to an embodiment of the present invention;

[0108] Figure 3 It is a specific implementation diagram of a comprehensive energy system model predictive control method based on deep reinforcement learning according to an embodiment of the present invention. DETAILED DESCRIPTION

[0109] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0110] According to an embodiment of the present invention, a model predictive control method for an integrated energy system based on deep reinforcement learning is provided.

[0111] The present invention is further described with reference to the accompanying drawings and specific embodiments. Figure 1-Figure 3As shown, according to the comprehensive energy system model predictive control method based on deep reinforcement learning according to an embodiment of the present invention, the comprehensive energy system model predictive control method includes the following steps:

[0112] S 1. Establish a comprehensive energy system model based on the energy supply and demand balance of the load in the park's comprehensive energy equipment.

[0113] Preferably, the comprehensive energy system model includes an equipment model and an energy consumption model.

[0114] It should be explained that the equipment model includes photovoltaic power generation equipment, external power grid, boiler equipment, CHP unit, refrigeration equipment and energy storage equipment;

[0115] The energy consumption model includes the energy supply and demand balance of cooling load, heating load and electrical load.

[0116] Establish equipment models, including photovoltaic power generation equipment, external power grid, boiler equipment, CHP units, refrigeration equipment and energy storage equipment.

[0117] (1) Photovoltaic power generation equipment

[0118] Photovoltaic power calculation formula:

[0119]

[0120] Where Y PV represents the photovoltaic array capacity, f PV Indicates the derating factor of the photovoltaic array, represents the irradiance reaching the photovoltaic array, Indicates irradiance scalar (1kW / m 2 ), P PV Represents the photovoltaic power. The model assumes that the photovoltaic arrays are equipped with maximum power point tracking.

[0121] (2) External power grid

[0122] The external power grid mainly plays the role of power balance. When the power generated in the integrated energy system is insufficient, the load can be met by purchasing power from the power grid; when there is surplus power, it can also be sold to the power grid. The power interaction constraints between the system and the large power grid are as follows:

[0123]

[0124] in, Indicates the upper limit of power transmission of the power grid, P grid，t It represents the power transmission of the power grid at time t.

[0125] (3) Boiler equipment

[0126] Boiler equipment can be divided into electric boilers and gas boilers. The model is expressed as follows:

[0127] H EB，t =η EB P EB，t ;

[0128] H GB，t =η GB G GB，t ;

[0129] Where P EB，t Indicates the power consumption of electric boiler, G GB，t Indicates the gas consumption of gas boiler, H EB，t , H GB，t They represent the heat output of the electric boiler and the gas boiler at time t, respectively, EB , η GB They represent the electric-to-heat conversion efficiency and the gas-to-heat conversion efficiency respectively.

[0130] (4) CHP unit

[0131] H CHP，t =η H，CHP G CHP，t ;

[0132] P CHP，t =η P，CHP G CHP，t ;

[0133] In the formula, G CHP，t Indicates the gas consumption of the CHP unit at time t, H CHP，t , P CHP，t They represent the heat production and power generation of the CHP unit at time t, respectively, H，CHP , η P，CHP They represent the gas-to-heat conversion efficiency and gas-to-electricity conversion efficiency of the CHP unit respectively.

[0134] Natural gas is supplied to CHP units and gas boilers. Considering the reliability and safety of energy network operation, it is necessary to set power constraints:

[0135]

[0136] in, Indicates the upper limit of gas grid transmission power, P gass,t Represents the gas grid transmission power at time t.

[0137] (5) Refrigeration equipment: divided into electric refrigeration equipment and absorption refrigeration equipment, the models are as follows:

[0138] C AC，t =η AC P AC，t ;

[0139] C LBR，t =η LBR H LBR，t ;

[0140] Where η AC , η LBR They are the electric cooling conversion coefficient and the thermal coefficient; P AC，t H is the power consumption of the refrigerator during period t; LBR，t C is the heat consumption of the absorption refrigerator during period t; AC，t , C LBR,t The columns represent the cooling capacity of the electric refrigerator and the absorption refrigerator during period t respectively.

[0141] (6) Energy storage equipment

[0142] The energy storage device model models the energy charging and discharging process of the electric energy storage device, and its general state equation is as follows:

[0143]

[0144] In the formula, i represents the type of energy storage device, E i,t represents the storage capacity of the energy storage device at time t, They represent charging efficiency and discharging efficiency respectively. Respectively represent the charging power and discharging power of the energy storage device, ε i Represents the self-loss coefficient; E i，t+1 Represents the storage capacity of the energy storage device at time t+1.

[0145] The battery energy storage device constraints are:

[0146]

[0147] in, and Respectively represent the battery energy storage charging and discharging status, and Respectively represent the minimum and maximum storage capacity of the battery. The battery can only work in one state: charging or discharging.

[0148] An energy consumption model is established, which includes cooling load, heating load and electric load. The load data is loaded into the system as known external data for operation optimization. The electric load, heating load and cooling load must meet the following power balance constraints:

[0149] P PV，t +P CHP，t +P grid，t +P Ess，t =P Load，t +P EB，t +P AC，t;

[0150] Where P PV，t , P CHP,t , P grid,t , P Ess，t They represent the photovoltaic power generation, CHP power generation, power purchase from the grid, and battery discharge at time t respectively; P Load，t , P EB，t , P AC，t They represent the power consumption of the park, electric boiler and electric refrigerator at time t respectively.

[0151] H CHP，t +H EB，t +H GB，t =H Heat，t +H LBR,t ;

[0152] In the formula, H CHP，t , H EB，t , H GB,t They represent the heat generated by CHP, electric boiler, and gas boiler at time t respectively; H Heat,t , H LBR，t They represent the heat consumption of the park and the heat consumption of absorption cooling at time t respectively.

[0153] C AC，t +C LBR，t =C Cold，t ;

[0154] Among them, C AC，t , C LBR，t They represent the cooling capacity of the electric refrigeration machine and the absorption refrigeration machine at time t respectively; C Cold,t Indicates the cooling consumption of the park at time t.

[0155] S2. Based on the minimum full-day energy cost objective function, the distributed robust optimization algorithm is used to calculate the day-ahead output plan value of the integrated energy system model; a state prediction model is constructed according to the state, output increment and disturbance of each unit in the park's integrated energy equipment.

[0156] Preferably, based on the minimum all-day energy cost objective function, a distributed robust optimization algorithm is used to calculate the output day-ahead planning value of the integrated energy system model; establishing a state prediction model according to the state, output increment and disturbance of each unit in the park integrated energy equipment includes the following steps:

[0157] S21. Define minimization of the energy cost throughout the day as the objective function and establish a constraint function;

[0158] S22, using KL divergence as a measure of the distance between the actual distribution and the reference distribution, defining the distance between the actual distribution and the reference distribution, and establishing an uncertainty set based on the distance between the actual distribution and the reference distribution;

[0159] S23. Construct a chance-constrained optimization model for uncertain sets using a distributed robust optimization algorithm;

[0160] S24, using a mathematical solver to solve the opportunity constraint optimization model and obtain the day-ahead planned output value of the integrated energy system equipment; wherein the mathematical solver includes gurobi (high-efficiency solver) or other mathematical solvers;

[0161] S25. Construct a state prediction model based on the state, output increment and disturbance of each unit in the park's comprehensive energy equipment.

[0162] Preferably, KL divergence is used as a measure of the distance between the actual distribution and the reference distribution, and the distance between the actual distribution and the reference distribution is defined, and establishing the uncertainty set based on the distance between the actual distribution and the reference distribution includes the following steps:

[0163] S221, setting a limit value γ of the KL divergence, and using the KL divergence as a measure of the distance between the actual distribution and the reference distribution to define the distance between the actual distribution and the reference distribution;

[0164] S222. Establish an uncertainty set of distribution functions based on the distance between the actual distribution and the reference distribution.

[0165] Preferably, the expression defining the distance between the actual distribution and the reference distribution is:

[0166]

[0167] Where D KL (P||P0) represents the distance between the distribution function P and the reference distribution P0 measured using KL divergence;

[0168] f(ζ) represents the probability density function under the distribution function P;

[0169] f0(ζ) represents the probability density function under the reference distribution P0;

[0170] d ζ represents the integration of ξ.

[0171] Preferably, the expression of the chance-constrained optimization model for uncertain sets constructed using the distributed robust optimization algorithm is:

[0172]

[0173] Q p ={P|D KL (P||P0)≤γ};

[0174] In the formula, h(x, ζ) represents the optimization objective function;

[0175] x represents the decision variable;

[0176] X represents the feasible domain where the decision variable x is located;

[0177] st represents the constraint condition;

[0178] Pr ～p represents the probability under the uncertainty distribution p;

[0179] H(x,ζ) represents the constraint function;

[0180] Ω p represents an uncertain set of distribution functions;

[0181] represents the new confidence level considering the limit γ of KL divergence and the initial confidence α;

[0182] p represents the uncertainty distribution;

[0183] γ represents the limit of KL divergence, which is used to define Ω p The maximum allowed distance between the distribution function and the reference distribution.

[0184] Preferably, constructing a chance-constrained optimization model for an uncertain set using a distributed robust optimization algorithm comprises the following steps:

[0185] S231, obtaining an uncertain set, and constructing a robust optimization framework based on the uncertain set by using a distributed robust optimization algorithm;

[0186] S232. Through the robust optimization framework, the objective function is reconstructed, while ensuring that the control quantity satisfies the constraint function under uncertain conditions;

[0187] S232. Decompose the robust optimization framework using a distributed optimization algorithm, and achieve the global optimization goal by collaboratively solving the decomposed robust optimization framework.

[0188] Preferably, the expression for constructing the state prediction model according to the state, output increment and disturbance of each unit in the park comprehensive energy equipment is:

[0189] x t+1 =Ax t +Bu t +Cr t ;

[0190] In the formula, x t Indicates the status of each unit in the park's comprehensive energy equipment at time t;

[0191] x t+1Table t+1 time each unit in the park comprehensive energy equipment status;

[0192] u t It represents the output increment of each unit at time t;

[0193] r t It represents the disturbance amount of each unit at time t;

[0194] A represents x t Coefficient matrix, B represents u t The coefficient matrix of t The coefficient matrix of , which can be derived from the system energy balance.

[0195] It should be explained that in the above comprehensive energy scenario, the optimization goal is to optimize the decision variables such as power purchased from the power grid and natural gas supply power based on the multiple uncertainties of the fluctuations of parameters such as photovoltaic power generation, power load, thermal load, and day-ahead electricity price. The operating cost of the comprehensive energy system is:

[0196]

[0197] In the formula, Cost dayahead represents the objective function for solving the output of each device on the previous day; P grid，t represents the power purchased by the power grid in period t; Costgrid,t represents the day-ahead predicted power purchase price per unit time in period t; ζ cg，t represents the electricity price forecast error during period t; P bat，t Cost represents the charging and discharging power of the energy storage in the time period (t-1, t], where a positive value indicates discharging and a negative value indicates charging. bat P represents the cost per unit power of energy storage; gas，t Cost represents the energy consumption power of natural gas during period t; gas Represents the unit consumption cost of natural gas.

[0198] Cost of energy used throughout the day dayahead As the objective function, the above-mentioned distributed robust optimization algorithm is used to solve the day-ahead planned value X of the system equipment output power ref .

[0199] In this embodiment, the vector consisting of the electric energy storage charging and discharging power, the charge state of the electric energy storage system, the electric refrigeration power, the electric boiler heating power, the total power generated by the CHP unit gas consumption, the gas boiler power, the absorption refrigeration power, and the power exchanged between the microgrid and the external grid tie line is selected as the state variable:

[0200] x(k)=[P batt (k), SOC(k), P EC (k), P EB(k)

[0201] G CHP (k), P GB (k), P AC (k), P grid (k)] T ;

[0202] Where x(k) represents the system state vector at time k; P batt (k), SOC(k), P EC (k), P EB (k), G CHP (k), P GB (k), P AC (k), P grid (k) represents the charging and discharging power of the energy storage system at time k, the charge state of the energy storage system, the electric refrigeration power, the heating power of the electric boiler, the total power generated by the gas consumption of the CHP unit, the gas boiler power, the absorption refrigeration power, and the exchange power of the microgrid and the external grid connection line.

[0203] The output increments of electric energy storage, electric refrigeration, electric boiler, CHP unit, gas boiler, and absorption refrigeration are used as control variables:

[0204] u(k)=[ΔP batt (k), ΔP EC (k), ΔP EB (k)

[0205] ΔP CHP (k), ΔP GB (k), ΔP AC (k)] T ;

[0206] u(k) represents the control vector of the system at time k;

[0207] ΔP batt (k), ΔP EC (k), ΔP EB (k), ΔP CHP (k), ΔP GB (k), ΔP AC (k) represents the output increment of electric energy storage, electric refrigeration, electric boiler, CHP unit, gas boiler and absorption refrigeration equipment at time k respectively.

[0208] The vector consisting of the daily predicted power increment of photovoltaic, cooling and heating power loads is used as the disturbance variable:

[0209] r(k)=[ΔP PV (k), ΔP load_elec (k), ΔP load_heat(k), ΔP load_cold (k)| T ;

[0210] r(k) represents the disturbance variable of the system at time k, ΔP PV (k) represents the predicted photovoltaic power increment at time k; ΔP load_elec (k) represents the predicted power increment of the electric load during the day at time k; ΔP load_heat (k) represents the predicted power increment of the heat load during the day at time k; ΔP load_cold (k) represents the predicted power increment of cooling load during the day at time k.

[0211] Using power balance to build an MPC prediction model:

[0212] x(k+Δt)=Ax(k)+Bu(k)+Cr(k).

[0213] In the formula, A represents x t Coefficient matrix, B represents u t The coefficient matrix of t The coefficient matrix can be derived from the constraint-related formula; x(k) represents the system state at time k, and x(k+Δt) represents the system state at the next moment predicted based on the control action u(k) and the disturbance r(k) at time k.

[0214] S3. Use the state prediction model to predict the output state of each device in the system at each moment in the prediction time domain within the day, and establish a rolling optimization objective function based on the predicted value and the day-ahead planned value, calculate the optimal control quantity for intraday scheduling, and optimize the state prediction model based on the calculation results.

[0215] It should be explained that the vector consisting of the total power generated by CHP gas consumption, the power generated by gas boiler gas consumption, and the exchange power between the microgrid and the external grid is used as the variable for calculating the daily gas and electricity purchase costs:

[0216] y(k)=[P CHP (k), P GB (k), P grid (k)];

[0217] Where y(k) represents the power generated by natural gas consumption at time k and the power exchanged with the power grid, P CHP (k), P GB (k), P grid (k) represents the total power generated by CHP gas consumption, the power generated by gas boiler gas consumption, and the exchange power between the microgrid and the external grid at time k.

[0218] By repeatedly iterating the state space prediction model, the system state variable x(k) can be obtained in the prediction time N. p The prediction vector x forecast And the prediction vector Y of y(k) in the prediction domain:

[0219] X forecast =[P batt (k+i|k), SOC(k+i|k), P EC (k+i|k),

[0220] P EB (k+i|k),…,P grid (k+i|k)] T , i=1,2,…N p ;

[0221] Y=[P CHP (k+i|k), P GB (k+i|k),

[0222] P grid (k+i|k)] T , i=1,2,…N p ;

[0223] Where R is the price matrix of gas and electricity purchases in each period within the forecast domain:

[0224] R=[S CH4 (k+Δt), S CH4 (k+Δt), S grid (k+Δt)…,

[0225] S CH4 (k+N p Δt), S CH4 (k+N p Δt), S grid (k+N p Δt)];

[0226] In the formula, S CH4 (k+Δt), S grid (k+Δt) represents the gas purchase price and the power grid purchase price at time k+Δt; S CH4 (k+N p Δt), S grid (k+N p Δt) represents k+N p The price of purchasing gas and electricity at time Δt.

[0227] q represents the weight coefficient matrix of single-moment error tracking in the rolling optimization objective function:

[0228] q=diag[wESS , w SOC , w EC , w EB , w CHP , w GB , w AC , w grid ];

[0229] In the formula, w ESS 、w SOC 、w EC 、w EB 、w CHP 、w GB 、w AC 、w grid They are respectively the charging and discharging power of electric energy storage, the state of charge of the electric energy storage system, the electric refrigeration power, the heating power of the electric boiler, the total power generated by the gas consumption of the CHP unit, the power of the gas boiler, the absorption refrigeration power, and the weight coefficient of the power exchange between the microgrid and the external grid connection line. Iteratively extending it to the MPC prediction domain yields:

[0230] Q = diag[w ESS (k+i|k), w SOC (k+i|k), w EC (k+i|k),

[0231] w EB (k+i|k),...,w grid (k+i|k)], i=1, 2, ... N p ;

[0232] Where Q is the weight coefficient matrix in the prediction time domain, which is expanded by q iterations, where w ESS (k+i|k), w SOC (k+i|k), w EC (k+i|k), w EB (k+i|k), w grid (k+i|k) represents the weight coefficients of the energy storage charging and discharging power, the energy storage system charge state, the electric cooling power, the electric boiler heating power and the power exchanged between the microgrid and the external grid at time k+i. The total power generated by the CHP unit gas consumption, the gas boiler power and the absorption cooling power weight coefficient are omitted in the middle. i=1, 2, ...N p , N p Indicates the prediction time domain degree.

[0233] Since the system model is usually simplified and there may be modeling errors, external interference and other factors, the actual system output state may be different from the value predicted by the model. Therefore, in order to suppress the control deviation caused by model mismatch or environmental interference, at the new sampling time, the actual output of the object is first detected, and the prediction results based on the model are corrected using this real-time information, and the actual value of the system state is used as the starting state of the next round of rolling optimization, and then a new optimization is performed.

[0234] S4. Build a deep reinforcement learning agent based on the state prediction model. Based on the framework, use the double-delay deep deterministic policy gradient algorithm to learn the weight parameters in the rolling optimization objective function to achieve the optimization of the predictive control of the integrated energy system model.

[0235] Preferably, the state prediction model is used to predict the output state of each device in the system at each moment in the prediction time domain within the day, and a rolling optimization objective function is established based on the predicted value and the day-ahead planned value. The calculation of the optimal control quantity for intraday scheduling includes the following steps:

[0236] S31. Use the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain, and establish a rolling optimization objective function based on the deviation between the predicted value and the day-ahead planned value; S32. Set the rolling optimization objective function to include physical limitations on equipment output, smoothness constraints on control quantity changes, and constraints on the amount of electricity and gas purchased;

[0237] S33, predicting the state of each unit and solving the optimal control quantity within the constraints of each rolling time domain;

[0238] S34. Adjust the state prediction model based on the optimal control quantity and the actual output of the system, and build a feedback mechanism.

[0239] Preferably, the state prediction model is used to predict the output state of each device in the system at each moment in the prediction time domain within the day, and a rolling optimization objective function is established based on the predicted value and the day-ahead planned value. The calculation of the optimal control quantity for intraday scheduling includes the following steps:

[0240] S311. At each sampling moment, detect the actual state of each unit in the park's comprehensive energy equipment, and obtain the actual output of each unit at the current moment;

[0241] S312, using the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain;

[0242] S313, establishing a rolling optimization objective function based on the predicted value and the deviation in the day-ahead plan, calculating the optimal control amount and issuing the control instruction at the first moment;

[0243] S314, calculating the error between the model prediction state and the actual output of the system, and correcting the model prediction control model through the feedback correction coefficient according to the calculated error, and entering the next moment rolling optimization.

[0244] Preferably, the expression for establishing the rolling optimization objective function is:

[0245] minf=(W forecast -X ref ) T Q(X forecast -X ref )+u T Hu+YR;

[0246] Where, X forecast Represents the prediction output of MPC in the prediction time domain;

[0247] X ref represents the reference state calculated by the day-ahead optimization;

[0248] H represents the control amount weight matrix;

[0249] Q represents the weight coefficient;

[0250] u represents the system control quantity in the prediction time domain;

[0251] R represents the price of electricity and gas purchased at the corresponding moment;

[0252] Y represents the amount of electricity and gas purchased at each moment in the forecast time domain;

[0253] minf represents the minimization of the rolling optimization objective function.

[0254] In addition, it should be noted that the objective function optimizes three indicators. The first item makes the state vector in the prediction time domain as close to the day-ahead reference value as possible under the constraint of the weight coefficient Q. The second item is to ensure the stability and security of the power grid, and the control action should not change too much, so the optimization objective function adds constraints on the control quantity. The third item is the constraint on the cost of purchasing electricity and gas in the prediction domain. Solve the objective function to obtain the system control action u.

[0255] Preferably, a deep reinforcement learning agent is constructed based on the state prediction model, and based on the agent of the framework, a double-delayed deep deterministic policy gradient algorithm is used to learn the weight parameters in the rolling optimization objective function, and the optimization of the predictive control of the integrated energy system model is realized, which includes the following steps:

[0256] S41. Determine the dimensions of the state space and the action space using the optimized state prediction model, and construct the critic and actor neural network of the intelligent agent based on the dimensions of the state space and the action space, wherein the state space determines the number of neurons in the input layer, and the action space determines the number of neurons in the output layer;

[0257] S42. Use the double-delayed deep deterministic policy gradient algorithm to initialize the agent's critic and actor neural networks, and design the reward function and penalty mechanism;

[0258] S43, initializing the experience replay unit, storing sample data interacting with the environment, and randomly sampling sample data therefrom for training;

[0259] S44, generating actions with exploration noise in the actor network, executing the actions and storing the experience data, generating new actions with policy noise through the target actor network, calculating the target value using the dual critic network, taking the minimum value of the two networks, thereby reducing network overestimation;

[0260] S45. The gradient of the actor network is calculated and gradually updated through the critic network, and soft update and double delay update strategies are adopted to ensure the stability of learning and achieve optimal control of the integrated energy system model.

[0261] Preferably, determining the dimensions of the state space and the action space using the optimized state prediction model, and constructing the critic and actor neural network of the intelligent agent based on the dimensions of the state space and the action space comprises the following steps:

[0262] S411, using historical data and model evaluation indicators to learn and train weight parameters of the rolling optimization objective function of the state prediction model;

[0263] S412, identifying all key state variables in the state prediction model and determining all control actions;

[0264] S413. Design a critic and actor neural network of an intelligent agent with corresponding input layer and output layer; wherein the input layer receives state variables and the output layer generates control actions.

[0265] Preferably, using a double-delayed deep deterministic policy gradient algorithm to initialize the agent's critic and actor neural networks, and designing a reward function and a penalty mechanism includes the following steps:

[0266] S421, initialize the actor network and critic network in the dual-delay deep deterministic policy gradient algorithm; wherein the actor network is responsible for generating continuous control actions based on the current environment state and outputting the optimal control strategy; the critic network is used to evaluate the actions generated by the actor network, and the evaluation of the same action by the dual critic networks is used to improve stability and prevent the estimated value from being too high;

[0267] S422. After the actor network generates an action, a reward function is defined. The design of the reward function is centered on minimizing the value of the objective function, encouraging the agent to take actions that help optimize the goal; at the same time, a penalty mechanism is designed;

[0268] S423. The agent maximizes the cumulative reward value through the actor's optimal control strategy.

[0269] Preferably, the gradient of the actor network is calculated and gradually updated through the critic network, and the soft update and double delay update strategies are adopted to ensure the stability of learning, and to achieve the optimal control of the integrated energy system model, including the following steps:

[0270] S451, calculate the gradient of the actor network through the critic network and update the parameters of the actor network;

[0271] S452, train the critic network by minimizing the mean square error;

[0272] S453, based on minimizing the mean square error, a soft update mechanism is used to gradually update the target critic network;

[0273] S454, adopting a double-delay update strategy to update the target actor network at a preset step interval; S454, repeating the steps of S451-S54 until the training is completed, ensuring the stability of the actor network learning and training, and realizing the optimal control of the integrated energy system model.

[0274] Preferably, the expression for calculating the gradient of the actor network is:

[0275]

[0276] In the formula, represents the gradient of the loss function with respect to the actor network parameter φ;

[0277] represents the gradient of the critic network output with respect to action a, and calculates the gradient of the value function Q given the action a generated by the actor network in the current state;

[0278] represents the gradient of the action a output by the actor network with respect to the actor network parameter φ;

[0279] N - 1 represents one - Nth, and N represents the number of samples in one sampling.

[0280] It should be noted that in this embodiment, the dimensions of the state space and the action space are determined by feature extraction. Since the battery charge - discharge power and the state of charge (SOC) change synchronously, only the battery charge - discharge power needs to be selected. The neural network input is:

[0281] [P batt (k), P EC (k), P EB (k), G CHP (k), P GB (k), P AC (k), P grid (k)];

[0282] According to the control action of MPC, the neural network output is:

[0283] [w ESS (k + i|k), w SOC (k + i|k), w EC (k + i|k),

[0284] w EB (k + i|k),..., w grid (k + i|k)], i = 1, 2,..., N p ;

[0285] Considering that the situations of curtailment of photovoltaic power and insufficient power supply should be reduced, the reward function is designed as:

[0286]

[0287] In the formula, r(s t , a t ) represents the reward feedback by the environment at time t, f t is the objective function value at time t, σ scale represents the scaling factor, which scales the reward value for easy calculation and processing; the goal of the agent is to minimize the objective value and maximize the cumulative reward value; represents the curtailment of photovoltaic power penalty term, represents the penalty term for the grid interaction power exceeding the upper limit of the grid transmission power. If there is no curtailment of photovoltaic power and the grid exchange power does not exceed the upper limit, both are 0; then, according to the training method in the technical solution, TD3 is learned and trained by random sampling to achieve closed - loop optimal control; f_t represents the objective function value at time t.

[0288] The dimensions of the state space and action space are determined by feature extraction, the number of neurons in the input layer of the neural network is determined by the state space, and the number of neurons in the output layer of the neural network is determined by the action space. The present invention proposes to establish a reinforcement learning agent using the TD3 algorithm. The TD3 algorithm is selected because of its Actor-Critic reinforcement learning network architecture, in which the policy-based actor network can achieve continuous action output, and the value-based dual-Critic network structure design can prevent the network from overestimating.

[0289] The replay memory unit D is used to store the data (s, a, r, s') obtained from each interaction with the environment for sampling learning of the network, where s and a represent the current state and action taken, respectively, r represents the reward value obtained by taking the current action, and s' represents the next state entered after taking the action.

[0290] Initialize the neural network, initialize the experience replay buffer D, randomly initialize the noise ε for strategy smoothing, delay the update frequency d, and the soft update coefficient τ. The learning and training of TD3 is repeated according to the following steps:

[0291] 1) In Generate action a=π in the network φ (S)+ε1, ε1 represents exploration noise, which is used to increase exploration, calculate reward r and observe the new state s', and store the experience (s, a, r, s') in the buffer D;

[0292] 2) When the experience pool D has accumulated enough data on the interaction between the agent and the environment, N sets of experience data (s, a, r, s')∈D are randomly sampled from the replay memory unit for training;

[0293] 3) For each sample data (s, a, r, s') taken out, use the target actor network π' φ Select action a′=π′ φ (s′)+ε2, ε2 represents the policy noise, which is used for policy smoothing, and then the target critic network is used to calculate the target value

[0294] 4) The critic network is trained by minimizing the mean square error between the two Q values Perform training updates. The loss function for minimizing the mean squared error (MSE) is as follows:

[0295]

[0296] L(θ i ) is the loss function, y i is the target value of the sample calculated above, is a sample (sj , a j ) in Q value;

[0297] 5) Become a critic network After a certain number of updates, the double-delayed update actor network π φ and the target actor network π′ φ , and the target critic network

[0298] Calculate the current step gradient to update the actor network π φ The expression is:

[0299]

[0300] In the formula, Represents the gradient of the loss function with respect to the actor network φ, which is used to guide how to update the parameters of the actor network. Represents the gradient of the critic network output with respect to action a, and calculates the gradient of the value function Q given the action a generated by the actor network in the current state. represents the gradient of the action a output by the actor network with respect to the actor network parameters φ.

[0301] Soft update target critic network and target actor network:

[0302]

[0303] π′ φ =τπ φ +(1-τ)π′ φ ;

[0304] Soft update is a gradual update, τ is a small number, and the target network only updates a small number of parameters each time to ensure the stability of the execution network learning. This cycle continues until the training is completed.

[0305] In addition, the design of the integrated energy system optimization dispatch framework is as follows Figure 2 The model predictive control algorithm module receives the system initial state x, starts rolling optimization and calculates the optimal control parameter u, which is applied to the integrated energy system for scheduling. After the action is executed, the system enters the next state x', and x' is then transmitted back to the model predictive control module to enter the next round of rolling optimization. In this process, (x, a, r, x') will be stored in the deep reinforcement learning agent experience pool D for sampling learning, and the optimal weight coefficient parameter will be learned in the process of interaction with the environment. It is passed to the model predictive control model for scheduling control to achieve the best optimization effect.

[0306] The reinforcement learning module uses the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. It is a policy-based actor network that generates action strategies in a given state. is the target actor network, which is used to generate actions to calculate the target Q value. The network is The network's delayed update of the network replica is mainly to stabilize the learning process of the actor network. ε represents Noise is added to the actions generated by the network to smooth the strategy and encourage exploration. ε is sampled from a normal distribution. In order to prevent sampling of very large or very small values, the noise size is clipped to limit it to a certain range:

[0307] ε~clip(N(0,σ),-n,n);

[0308] Where N(0, σ) represents the normal distribution, clip means limiting the noise size to between -n and n, and ε represents the added strategy noise.

[0309] and It is a value-based critic network that evaluates the value of actions generated by the actor network and updates Target and Target is the target critic network, used to evaluate Target The network generates the value of the action and calculates the target value for updating and Target yes The delayed update network copy of Target yes The delayed update copy of the dual network structure is to stabilize the learning process and prevent overfitting. When calculating the Q value, the minimum value calculated by the two target critic networks is selected to suppress the problem of network overestimation. Moreover, when the critic network is updated multiple times, the actor network is updated to ensure that the training of the actor network is more stable.

[0310] Based on the deep reinforcement learning algorithm, a closed-loop control mechanism that can dynamically and adaptively search for the optimal MPC exogenous control parameters is established. First, an MPC state prediction model is established based on the integrated energy system model; then, a neural network model of a deep reinforcement learning agent is established based on the MPC state model to determine the dimensions of the state space and action space, where the state space is the input layer of the neural network, and the action space is the output layer of the neural network; then, the agent's reward function is designed based on the objective function; then, MPC performs intraday rolling optimization based on the reference value optimized the day before, and adjusts the equipment output; in this process, the agent's neural network will learn based on the MPC execution results, the objective function, and the reward. By learning and training a large amount of historical data, the agent can have the ability to autonomously assist in MPC scheduling optimization, and can achieve better economic efficiency. The scope of application of the present invention is not only limited to the optimized operation of integrated energy systems, but can also be further extended to the rolling optimization problems of time domain systems.

[0311] The MPC (Model Predictive Control) algorithm is an advanced control strategy that calculates the control input by solving an optimization problem at each control step. The optimization problem predicts the future behavior based on a system model and finds a control sequence that makes the system output track the desired trajectory while satisfying a series of constraints.

[0312] TD3 (Twin Delayed Deep Deterministic Policy Gradient) is a deep reinforcement learning algorithm based on the actor-critic framework, which is used to solve reinforcement learning problems with continuous action space. It improves on the DDPG (Deep Deterministic Policy Gradient) algorithm by introducing three key mechanisms to improve learning stability and performance: dual network mechanism, delayed update and target policy smoothing. The TD3 algorithm uses a dual critic network and takes the smaller value of the two when calculating the target value, thereby suppressing the network over-estimation problem; a delay strategy is used in the update of the target network, so that the updates of the policy network and the target network are not synchronized, improving the stability of network learning. Target policy smoothing regularization adds perturbations to the action of the next state when calculating the target value, so that the value evaluation is more accurate. These improvements enable the TD3 algorithm to perform superior performance in multiple continuous control tasks, especially in those environments with sparse rewards.

[0313] The explanations of relevant technical terms are as follows:

[0314] MSE: Mean Squared Error, minimize the mean square error loss function.

[0315] KL divergence (Kullback-Leibler Divergence), also known as relative entropy, is a measure of the difference between two probability distributions. It has a wide range of applications in information theory, machine learning, and deep learning.

[0316] To sum up, with the help of the above-mentioned technical scheme of the present invention, the present invention can improve the ability of the model predictive control algorithm to deal with the uncertainty of the external environment and improve the accuracy and economy of scheduling by dynamically adjusting the control parameters of the model predictive control algorithm (MPC); by adopting a policy-based reinforcement learning method, continuous action output can be achieved, and the powerful characterization ability of deep learning can be used to more accurately describe the system state characteristics, and more parameters can be tracked and learned at the same time; the double-delay deep deterministic policy gradient algorithm has better generalization, and the generalization of the intelligent agent scheduling is enhanced through the learning of different scenarios by neural networks, which can be better applied in complex and dynamic environments, thereby improving the optimization efficiency and quality of the algorithm for the integrated energy system.

[0317] Although the present invention has been disclosed as above with preferred embodiments, the embodiments are merely examples for the convenience of description and are not intended to limit the present invention. Those skilled in the art may make several changes and modifications without departing from the spirit and scope of the present invention. The scope of protection claimed by the present invention shall be based on the claims.

Claims

1. A model predictive control method for an integrated energy system based on deep reinforcement learning, the integrated energy system model predictive control method comprising the following steps: S1. Establish a comprehensive energy system model based on the energy supply and demand balance of the load in the park's comprehensive energy equipment; S2. Based on the minimum full-day energy cost objective function, the distributed robust optimization algorithm is used to calculate the day-ahead output plan value of the integrated energy system model; a state prediction model is constructed according to the state, output increment and disturbance of each unit in the park's integrated energy equipment; S3. Use the state prediction model to predict the output state of each device in the system at each time in the prediction time domain within the day, and establish a rolling optimization objective function based on the predicted value and the day-ahead planned value to calculate the optimal control quantity for intraday scheduling; S4. Build a deep reinforcement learning agent based on the state prediction model. Based on the framework, use the double-delay deep deterministic policy gradient algorithm to learn the weight parameters in the rolling optimization objective function to achieve the optimization of the predictive control of the integrated energy system model.

2. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 1 is characterized in that: The method of calculating the output day-ahead plan value of the integrated energy system model based on the minimum all-day energy cost objective function by using a distributed robust optimization algorithm; and establishing a state prediction model according to the state, output increment and disturbance of each unit in the park integrated energy equipment includes the following steps: S21. Define minimization of the energy cost throughout the day as the objective function and establish a constraint function; S22, using KL divergence as a measure of the distance between the actual distribution and the reference distribution, defining the distance between the actual distribution and the reference distribution, and establishing an uncertainty set based on the distance between the actual distribution and the reference distribution; S23. Construct a chance-constrained optimization model for uncertain sets using a distributed robust optimization algorithm; S24, solving the opportunity constraint optimization model using a mathematical solver, and obtaining the day-ahead planned output value of the integrated energy system equipment; S25. Construct a state prediction model based on the state, output increment and disturbance of each unit in the park's comprehensive energy equipment.

3. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 2 is characterized in that: The method of using KL divergence as a measure of the distance between the actual distribution and the reference distribution, defining the distance between the actual distribution and the reference distribution, and establishing an uncertain set based on the distance between the actual distribution and the reference distribution includes the following steps: S221, setting a limit value of KL divergence, and using KL divergence as a measure of the distance between the actual distribution and the reference distribution to define the distance between the actual distribution and the reference distribution; S222. Establish an uncertainty set of distribution functions based on the distance between the actual distribution and the reference distribution.

4. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 3 is characterized in that: The expression defining the distance between the actual distribution and the reference distribution is: Where D KL (P||P0) represents the distance between the actual distribution P and the reference distribution P0 measured using KL divergence; f(ζ) represents the probability density function under the distribution function P; f0(ζ) represents the probability density function under the reference distribution P0; d ζ represents the integration of ξ.

5. The method for predictive control of an integrated energy system model based on deep reinforcement learning according to claim 4, characterized in that: The expression of the chance-constrained optimization model for uncertain sets constructed by using the distributed robust optimization algorithm is: Oh p ={P|D KL (P||P0)≤γ} In the formula, h(x, ζ) represents the optimization objective function x represents the decision variable; X represents the feasible domain where the decision variable x is located; st represents the constraint condition; Pr ～p represents the probability under the uncertainty distribution p; H(x,ζ) represents the constraint function; Ωp represents the uncertain set of distribution functions; γ represents the limit of KL divergence; represents the new confidence level considering the limit γ of KL divergence and the initial confidence α; p represents the uncertainty distribution.

6. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 5, characterized in that: The method of constructing a chance-constrained optimization model for an uncertain set using a distributed robust optimization algorithm comprises the following steps: S231, obtaining an uncertain set, and constructing a robust optimization framework based on the uncertain set by using a distributed robust optimization algorithm; S232. Through the robust optimization framework, the objective function is reconstructed, while ensuring that the control quantity satisfies the constraint function under uncertain conditions; S232. Decompose the robust optimization framework using a distributed optimization algorithm, and achieve the global optimization goal by collaboratively solving the decomposed robust optimization framework.

7. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 6 is characterized in that: The expression for constructing the state prediction model according to the state, output increment and disturbance of each unit in the park comprehensive energy equipment is: x t+1 =Ax t +Bu t +Cr t ; In the formula, x t Indicates the status of each unit in the park's comprehensive energy equipment at time t; x t+1 Table t+1 time each unit in the park comprehensive energy equipment status; u t It represents the output increment of each unit at time t; r t It represents the disturbance amount of each unit at time t; A represents x t coefficient matrix; B stands for u t The coefficient matrix of C stands for r t The coefficient matrix of .

8. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 7 is characterized in that: The state prediction model is used to predict the output state of each device in the system at each time in the prediction time domain within the day, and a rolling optimization objective function is established based on the predicted value and the day-ahead planned value. The calculation of the optimal control quantity for the intraday scheduling includes the following steps: S31. Use the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain, and establish a rolling optimization objective function based on the deviation between the predicted value and the day-ahead planned value; S32. Set the rolling optimization objective function to include physical limitations on equipment output, smoothness constraints on control quantity changes, and constraints on the amount of electricity and gas purchased; S33, predicting the state of each unit and solving the optimal control quantity within the constraints of each rolling time domain; S34. Perform scheduling control based on the optimal control quantity, build a feedback mechanism, and adjust the state prediction model.

9. The comprehensive energy system model predictive control method based on deep reinforcement learning according to claim 8, characterized in that: The method of using the state prediction model to predict the output state of each unit in the park comprehensive energy equipment within the prediction time domain and establishing a rolling optimization objective function based on the deviation between the predicted value and the day-ahead planned value includes the following steps: S311. At each sampling moment, detect the actual state of each unit in the park's comprehensive energy equipment, and obtain the actual output of each unit at the current moment; S312, using the state prediction model to predict the output state of each unit in the park's comprehensive energy equipment within the prediction time domain; S313, establishing a rolling optimization objective function based on the predicted value and the deviation in the day-ahead plan, calculating the optimal control amount and issuing the control instruction at the first moment; S314, calculating the error between the model prediction state and the actual output of the system, and correcting the model prediction control model through the feedback correction coefficient according to the calculated error, and entering the next moment rolling optimization.

10. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 9, characterized in that: The expression for establishing the rolling optimization objective function is: min f=(X forecast -X ref ) T Q(X forecast -X ref )+u T Hu+YR; Where, X forecast Represents the prediction output of MPC in the prediction time domain; X ref represents the reference state calculated by the day-ahead optimization; H represents the control amount weight matrix; Q represents the weight coefficient; u represents the output increment of each unit in the system within the prediction time domain; R represents the price of electricity and gas purchased at the corresponding moment; Y represents the amount of electricity and gas purchased at each moment in the forecast time domain; minf represents the minimization of the rolling optimization objective function.

11. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 1, characterized in that: The deep reinforcement learning agent is constructed based on the state prediction model. Based on the agent of the framework, the weight parameters in the rolling optimization objective function are learned by using the double-delayed deep deterministic policy gradient algorithm to realize the optimization of the predictive control of the integrated energy system model, which includes the following steps: S41. Determine the dimensions of the state space and the action space using the optimized state prediction model, and construct the critic and actor neural network of the intelligent agent based on the dimensions of the state space and the action space, wherein the state space determines the number of neurons in the input layer, and the action space determines the number of neurons in the output layer; S42. Use the double-delayed deep deterministic policy gradient algorithm to initialize the agent's critic and actor neural networks, and design the reward function and penalty mechanism; S43, initializing the experience replay unit, storing sample data interacting with the environment, and randomly sampling sample data therefrom for training; S44, generating actions with exploration noise in the actor network, executing the actions and storing the experience data, generating new actions with policy noise through the target actor network, calculating the target value using the dual critic network, taking the minimum value of the two networks, thereby reducing network overestimation; S45. The gradient of the actor network is calculated and gradually updated through the critic network, and soft update and double delay update strategies are adopted to ensure the stability of learning and achieve optimal control of the integrated energy system model.

12. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 11, characterized in that: The method of determining the dimensions of the state space and the action space by using the optimized state prediction model and constructing the critic and actor neural network of the intelligent agent based on the dimensions of the state space and the action space comprises the following steps: S411. Use historical data and model evaluation indicators to train the state prediction model and optimize the state prediction model parameters; S412, identifying all key state variables in the state prediction model and determining all control actions; S413. Design a critic and actor neural network of an intelligent agent with corresponding input layer and output layer; wherein the input layer receives state variables and the output layer generates control actions.

13. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 12, characterized in that: The method of initializing the critic and actor neural networks of the agent using the double-delayed deep deterministic policy gradient algorithm and designing the reward function and penalty mechanism includes the following steps: S421, initialize the actor network and critic network in the dual-delay deep deterministic policy gradient algorithm; wherein the actor network is responsible for generating continuous control actions based on the current environment state and outputting the optimal control strategy; the critic network is used to evaluate the value of the actions generated by the actor network, and the evaluation of the same action by the dual critic networks is used to improve stability and prevent the estimated value from being too high; S422. After the actor network generates an action, a reward function is defined. The design of the reward function is centered on minimizing the value of the objective function, encouraging the agent to take actions that help optimize the goal; at the same time, a penalty mechanism v is designed. S423. The agent maximizes the cumulative reward value by executing the optimal control strategy.

14. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 13, characterized in that: The method of calculating the gradient of the actor network through the critic network and gradually updating it, adopting the soft update and double delay update strategies, thereby ensuring the stability of learning and realizing the optimal control of the integrated energy system model includes the following steps: S451, calculate the gradient of the actor network through the critic network and update the parameters of the actor network; S452, train the critic network by minimizing the mean square error; S453, based on minimizing the mean square error, a soft update mechanism is used to gradually update the target critic network; S454, adopting a double-delay update strategy to update the target actor network at a preset step interval; S454, repeating the steps of S451-S54 until the training is completed, ensuring the stability of the actor network learning and training, and realizing the optimal control of the integrated energy system model.

15. The method for predictive control of a comprehensive energy system model based on deep reinforcement learning according to claim 14, characterized in that: The expression for calculating the gradient of the actor network is: In the formula, represents the gradient v of the loss function with respect to the actor network parameter φ Represents the calculation of the gradient of the value function Q when given the action a generated by the actor network in the current state; represents the gradient of the action a output by the actor network with respect to the actor network parameter φ; N-1 means one-N, and N means the number of samples taken at one time.

Images