Comprehensive energy system planning optimization method and system based on deep reinforcement learning

Abstract

The present invention belongs to the technical field of integrated energy system planning and optimization. In order to solve the problems of difficulty in solving and long time consumption in integrated energy system planning, an integrated energy system planning and optimization method and system based on deep reinforcement learning are proposed. A mathematical model of the integrated energy system is established, and the objective function and constraint conditions of the integrated energy system are determined to form an initial dynamic programming optimization problem; the initial dynamic programming optimization problem is converted into a Markov decision process; the historical operation data of the integrated energy system is offline learned using the DDPG algorithm and the prediction network; the optimization parameters of offline learning are applied to the Online-DDPG algorithm to solve the initial dynamic programming optimization problem and obtain the optimal planning scheme of the integrated energy system. The solution model problem caused by the dimensionality disaster and information loss is effectively solved, and the anti-interference ability to load and renewable energy fluctuations is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of integrated energy system planning and optimization, and in particular relates to an integrated energy system planning and optimization method and system based on deep reinforcement learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Various heterogeneous energy sources have been interconnected, integrated and complemented through integrated energy systems, providing an important solution for improving energy efficiency and alleviating global environmental pressure. Integrated energy systems can efficiently integrate and utilize various energy units, such as renewable energy generation units, cogeneration units, energy storage units, etc. The coexistence and interaction of multiple energy units increase the difficulty of designing energy scheduling strategies for integrated energy systems. Renewable energy generation and energy loads in integrated energy systems have significant volatility and randomness. Therefore, this dynamic uncertainty poses severe challenges to the safe and economic operation of integrated energy systems.

[0004] In the integrated energy system, the instability of clean energy such as solar energy and wind energy and the coupling of multiple energy sources make the physical model very complex. The access of various intelligent communication devices will generate a large amount of high-dimensional data, which increases the difficulty of data processing. The commonly used traditional optimization algorithms are difficult to apply due to the increase in computational costs, resulting in a slow solution rate. In recent years, the development of deep reinforcement learning has been impressive. It is a machine learning-based technology that simulates the human learning process and has an efficient solution capability for problems with large amounts of data and complex models, providing a more efficient decision-making method for artificial intelligence systems. Deep reinforcement learning has been used in various scenarios, such as autonomous driving, board games, motion detection, etc. It has also been studied in the safe and stable control of power systems, power dispatching, optimal power flow control, voltage and reactive power optimization control, etc. Through continuous interaction with the environment, the intelligent agent of deep reinforcement learning can obtain a lot of information and experience about the environment, and use it in subsequent actions to maximize the reward value.

[0005] The continuous development of reinforcement learning theory provides new ideas for solving optimization decision-making problems with uncertain factors. Reinforcement learning is a data-driven method that continuously improves strategies through continuous interaction between the agent and the environment. Deep reinforcement learning combines the nonlinear fitting ability of deep learning with the decision-making ability of reinforcement learning. It is an effective means to solve high-dimensional complex problems and avoid over-reliance on sophisticated physical models. In addition, since the application of deep reinforcement learning in integrated energy systems has been less explored, it is necessary to further explore the potential of deep reinforcement learning in energy scheduling of integrated energy systems. Summary of the invention

[0006] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides an integrated energy system planning optimization method and system based on deep reinforcement learning, which converts the dynamic planning problem of the integrated energy system into a reinforcement learning problem of continuous state and action space. The optimization method based on reinforcement learning can adapt to randomness, ensure real-time requirements, and has a strong ability to adapt to various uncertain interferences.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a comprehensive energy system planning optimization method based on deep reinforcement learning, comprising:

[0009] Establish a mathematical model of the integrated energy system, determine the objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem;

[0010] Convert the initial dynamic programming optimization problem into a Markov decision process;

[0011] Use DDPG algorithm and prediction network to conduct offline learning on historical operation data of integrated energy system;

[0012] The optimization parameters learned offline are applied in the Online-DDPG algorithm to solve the initial dynamic programming optimization problem and obtain the optimal planning scheme for the integrated energy system.

[0013] In a second aspect, the present invention provides a comprehensive energy system planning and optimization system based on deep reinforcement learning, comprising:

[0014] A building module is configured to: establish a mathematical model of the integrated energy system, determine an objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem;

[0015] A conversion module configured to: convert an initial dynamic programming optimization problem into a Markov decision process;

[0016] An offline learning module, which is configured to: perform offline learning on historical operation data pairs of the integrated energy system using a DDPG algorithm and a prediction network;

[0017] The optimization planning module is configured to apply the optimization parameters learned offline to the Online-DDPG algorithm, solve the initial dynamic programming optimization problem, and obtain the optimal planning solution for the integrated energy system.

[0018] In a third aspect, the present invention provides an electronic device comprising a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in the first aspect is performed.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium for storing computer instructions, wherein when the computer instructions are executed by a processor, the method described in the first aspect is performed.

[0020] One or more of the above technical solutions have the following beneficial effects:

[0021] In the present invention, the dynamic programming problem of the integrated energy system is transformed into a reinforcement learning problem of continuous state and action space, and does not need to rely on a sophisticated mathematical model, and can use the prediction information to update the scheduling algorithm in real time, so that the scheduling algorithm always maintains the ability to perceive system changes, effectively solves the model problem caused by dimensionality disaster and information loss, and improves the ability to resist interference with load and renewable energy fluctuations. In addition, the traditional DDPG algorithm is an offline learning algorithm, which relies too much on historical data and requires enough historical data, which not only requires too much data storage capacity for the integrated energy system, but also causes the consumption of reinforcement learning algorithm training time and training resources. The online-DDPG algorithm proposed in the present invention combines offline learning with online learning, and only requires a small part of historical data for offline learning before online learning and optimized scheduling can be performed, which greatly saves training costs and improves algorithm efficiency.

[0022] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0024] Figure 1 This is a structural block diagram of the integrated energy system in Embodiment 1 of the present invention;

[0025] Figure 2 This is a framework diagram of the online-DDPG algorithm in Embodiment 1 of the present invention;

[0026] FIG3 (a) is a performer network structure diagram of the DDPG algorithm in the first embodiment of the present invention;

[0027] FIG3( b ) is a diagram showing the critic network structure of the DDPG algorithm in the first embodiment of the present invention;

[0028] Figure 4 This is a flow chart of the integrated energy system planning optimization method based on deep reinforcement learning in Example 1 of the present invention. DETAILED DESCRIPTION

[0029] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0030] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0031] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0032] Embodiment 1

[0033] This embodiment discloses a comprehensive energy system planning optimization method based on deep reinforcement learning, including:

[0034] Establish a mathematical model of the integrated energy system, determine the objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem;

[0035] Convert the initial dynamic programming optimization problem into a Markov decision process;

[0036] Use DDPG algorithm and prediction network to conduct offline learning on historical operation data of integrated energy system;

[0037] The optimization parameters learned offline are applied in the Online-DDPG algorithm to solve the initial dynamic programming optimization problem and obtain the optimal planning scheme for the integrated energy system.

[0038] The dynamic energy scheduling of integrated energy systems is a random sequential decision-making problem. Reinforcement learning provides a model-free method to solve random decision-making problems without prior environmental knowledge. It can guarantee real-time requirements and has a strong ability to adapt to various uncertain interferences. Therefore, the scheduling method based on reinforcement learning can adapt to randomness and does not need to model the source load in advance as in traditional methods. In this sense, the unreliability and large deviation of the scheduling strategy caused by inaccurate modeling can be successfully avoided.

[0039] This embodiment combines the deep reinforcement learning algorithm based on policy gradient and the structure of the integrated energy system of solar-wind combined energy supply, and establishes an optimization scheduling model of the integrated energy system of solar-wind combined energy supply based on deep reinforcement learning of policy gradient. Then, by comparing several different types of deep reinforcement learning models based on policy gradient, it is determined to use the Deep Deterministic Policy Gradient (DDPG) framework for reinforcement learning, and finds its own advantages in combination with the characteristics of the algorithm, and improves it into a real-time deep deterministic policy gradient algorithm (online-DDPG) suitable for specific integrated energy systems for planning optimization, thereby achieving a more generalized, more robust and more efficient optimized scheduling of integrated energy systems.

[0040] The structure and mathematical modeling of the integrated energy system include energy coupling units, energy storage units and distributed power supply units; then the balance constraints of thermal energy, electrical energy and gas energy are established respectively. Based on this, the optimization objective function of the integrated energy system is proposed, including economic benefits and environmental benefits; then the dynamic programming problem of the traditional integrated energy system is transformed into a reinforcement learning framework based on the Markov decision process. In reinforcement learning, the observation space includes renewable energy generation, user thermal load, battery pack status, and grid price. The action space of the intelligent agent includes the scheduling of cogeneration units, electric boilers and battery packs. The reward function consists of economic benefits and environmental benefits, and a penalty term for unreasonable use of battery packs and electric boilers is added. Through such modeling, the optimization problem of the integrated energy system can be solved by reinforcement learning.

[0041] The power system and heating system in the integrated energy system are connected through coupling devices (such as cogeneration units and electric boilers), allowing energy flow and interaction between the power system and the heating system. Cogeneration units consume fossil fuels (such as coal and natural gas) to generate electricity and heat at the same time, while electric boilers and gas boilers convert electricity and gas energy into heat energy respectively. This coupling relationship between the power system and the heating system increases the flexibility of the integrated energy system and is conducive to the grid-connected intermittent renewable energy.

[0042] In terms of structure, the integrated energy system can be regarded as an energy unit with multiple input, output, conversion and storage devices. Based on the common form of the integrated energy system, this embodiment constructs an integrated energy system including an energy coupling unit, a distributed power source and an energy storage device. The structure of the integrated energy system is as follows: Figure 1As shown, the energy coupling unit includes a gas turbine cogeneration unit, a gas boiler, and an electric boiler; the distributed power source includes a distributed wind turbine generator set and a distributed photovoltaic generator set; and the energy storage device is an electric energy storage. The integrated energy system purchases online energy and natural gas from the upstream external power grid and gas supplier. Energy is reasonably distributed through conversion and storage equipment to meet the various needs of users.

[0043] 1. Mathematical modeling of each component unit.

[0044] 1-1. Energy coupling unit.

[0045] Gas turbine cogeneration unit is an important part of the comprehensive energy system. It is a device that uses natural gas or other fuel as fuel to generate electricity and heat energy at the same time. It usually consists of a gas turbine, a waste heat boiler and other auxiliary equipment.

[0046] The planning optimization task of this embodiment can be satisfied by only considering its characteristic of utilizing natural gas to generate electricity and heat. The electric power and heat power output by the cogeneration turbine set at time t are shown in the following formula.

[0047]

[0048]

[0049] In the formula, They represent the electrical power and thermal power output of the cogeneration turbine unit at time t, represents the natural gas consumption of the CHP turbine unit in period t, is the dispatch factor of the CHP turbine unit at time t, The low calorific value of natural gas. is the power generation efficiency of the gas turbine. The constraints of its thermal power and electrical power are shown in the following formula:

[0050]

[0051]

[0052] In the formula, The tables show the minimum and maximum power generation of the combined heat and power turbine unit, respectively; They represent the minimum and maximum heat production power of the combined heat and power turbine unit respectively.

[0053] When the heat energy recovered by the waste heat recovery device of the cogeneration turbine unit cannot meet the heat load demand, the gas boiler can burn natural gas to directly supply part of the heat load. The heat power output and constraints of the gas boiler can be expressed as follows:

[0054]

[0055]

[0056] In the formula, represents the thermal power output of the gas boiler at time t, represents the natural gas consumption of the electric boiler in period t, The low calorific value of natural gas. The power generation efficiency of the gas turbine. They represent the minimum and maximum heating power of the gas boiler respectively.

[0057] The thermal power output and constraints of the electric boiler are shown in the following equations:

[0058]

[0059]

[0060] In the formula, represents the thermal power output of the electric boiler at time t, represents the electric power input to the gas boiler at time t, is the electric-to-heat conversion efficiency of the electric boiler, They represent the minimum and maximum heating power of the gas boiler respectively.

[0061] 1-2. Energy storage unit.

[0062] Energy storage units mainly store excess electrical energy in the form of batteries. As one of the most commonly used energy storage devices, batteries can store energy generated by generators and renewable energy generators and release it when needed.

[0063] Therefore, the battery pack has two operating states: charging and discharging, which can be represented by the change of its state of charge and can be expressed as follows.

[0064]

[0065]

[0066] In the formula, δ represents the discharge rate of the battery, They represent the battery charge state of the battery pack at time t and time t+1 respectively, represents the charge-discharge power of the battery pack at time t, assuming that the charging , when discharging , They represent the charging efficiency and discharging efficiency respectively, Indicates the battery pack capacity.

[0067] The operation of a battery pack has expenses associated with acquisition and maintenance costs, which can be expressed as follows:

[0068]

[0069] In the formula, is the cost factor of the battery pack, The cost of the battery pack.

[0070] The charge and discharge constraints of the battery pack are shown in the following formula:

[0071]

[0072] In the formula, They are the minimum power generation and maximum power generation of the battery pack respectively.

[0073] 1-3. Distributed power supply unit.

[0074] The distributed power supply unit is composed of wind turbines and photovoltaic generators, which are mainly used to convert wind energy and light energy into electrical energy and transmit it to the integrated energy system. Usually it depends on natural conditions such as wind speed, temperature, weather and sunshine time, etc. Since renewable energy power generation does not consume any fossil fuels, the cost of its power generation is not considered in this embodiment.

[0075] 2. Constraints.

[0076] The thermal energy balance constraint is shown in the following formula:

[0077]

[0078] In the formula, Indicates the heat load power, is the efficiency of the heat exchanger.

[0079] The power balance constraint condition is as follows:

[0080]

[0081] In the formula, It represents the electric power purchased by the integrated energy system from the large power grid. Indicates the electrical load power.

[0082] The gas energy balance condition is shown in the following formula:

[0083]

[0084] In the formula, represents the user's natural gas load, Represents the amount of natural gas purchased by the integrated energy system.

[0085] Since the price fluctuation of natural gas is much smaller than that of power grid, the economic effect of the integrated energy system providing gas energy to users is basically unchanged. Therefore, this embodiment does not consider the economic benefits brought by the integrated energy system providing gas energy to users.

[0086] 3. Optimize goals.

[0087] The optimization objectives of the integrated energy system include economic benefits and environmental benefits, as shown in the following formula:

[0088]

[0089] In the formula, Represents a scheduling cycle, usually N is 24, t is 1h, They represent the economic benefits and environmental benefits in the kth time period, respectively, as shown in the following formula:

[0090]

[0091]

[0092] In the formula, They respectively represent the environmental cost and heating power borne by each unit that consumes fossil fuels to generate heat.

[0093] Economic benefits are the benefits gained from the integrated energy system providing electricity and heat to users. , the cost of purchasing electricity and gas from upstream large power grids and natural gas suppliers for the integrated energy system and battery pack operating costs Composition, see the following formula:

[0094]

[0095]

[0096] In the formula, , Respectively expressed in The unit price of electricity and heat sold during the period; Respectively expressed in The power of the user's electrical and thermal loads during the period; They represent the economic expenditure of purchasing electricity from the large power grid and the economic expenditure of purchasing gas from the natural gas supplier, as shown in the following formulas:

[0097]

[0098]

[0099] In the formula, Representatives in The electricity price of the large power grid during this period, Represents the natural gas selling price.

[0100] 4. Reinforcement learning framework based on Markov decision process.

[0101] The integrated energy system includes multiple dynamic elements such as power generation equipment, energy storage equipment, and loads. There are complex coupling relationships between these elements. The future state of the system depends on the current state and the real-time control strategy of these dynamic elements. There are some dynamic processes in the integrated energy system with time delays, such as battery charging and discharging processes. The current control decision will affect the system state in the future. The system involves factors with random fluctuations such as renewable energy generation such as wind power and photovoltaic power generation and user load. The future state of the integrated energy system is also affected by these random factors. The integrated energy system is restricted by multiple constraints such as equipment capacity, high efficiency and economy. The current control decision must meet these constraints to ensure that the system evolves to the desired future state. Therefore, the optimal scheduling problem of the integrated energy system needs to consider the dynamic characteristics, random uncertainty and various constraints of the system at the same time. This problem characteristic allows it to be modeled as a Markov decision process, and then use deep reinforcement learning methods to seek the optimal real-time scheduling strategy to make the system develop to the desired future state.

[0102] According to the various unit mathematical expressions established above, the mathematical model of the traditional integrated energy system dynamic programming problem can be transformed into a Markov decision process model. The Markov decision process consists of five tuples:<S, A, P, R, γ> Definition: In the deep reinforcement learning method used in this embodiment, the integrated energy system is the environment of the agent, and the agent makes the optimal scheduling decision by adjusting the output of each unit device in the system. At time t, the environment provides the agent with the observed system state , the agent follows the strategy Generate Action Therefore, the basic components of the deep reinforcement learning model for dynamic programming of integrated energy systems include the state space describing the environment , the action space describing the agent’s behavior , and the reward obtained by the agent .

[0103] 4-1. State Space

[0104] The observed state of the solar-wind combined energy supply integrated energy system is the renewable energy power provided by photovoltaic power generation and wind power generation. , User power load , User heat load , Battery charge status , large power grid time-of-use electricity price , integrated energy system time-of-use electricity price , which can be expressed as the following formula.

[0105]

[0106] 4-2. Action Space

[0107] At the moment t , the action of the intelligent agent can be represented by the output of each unit of the integrated energy system. When determining the natural gas consumption of the gas turbine cogeneration unit When the electric power produced is and thermal power , and determine the electric boiler input power The amount of natural gas consumed by the gas boiler can be determined Similarly, in addition to determining the charge-discharge power of the battery pack When the power purchased from the large power grid is determined .

[0108] So the action of the intelligent agent is the natural gas consumption of the gas turbine cogeneration unit , Electric boiler input power , Charge-discharge power of the battery pack , which can be expressed as follows:

[0109]

[0110] 4-3. Reward Function

[0111] The correct setting of the reward function can not only guide the agent towards the optimization goal, but also reduce the probability of making wrong choices when exploring the environment, thereby improving the optimization efficiency. In addition to the above-mentioned integrated energy system planning objective function value, it is also necessary to add a penalty coefficient for unreasonable charging and discharging of the battery pack. , and the penalty coefficient for unreasonable use of electric boilers , the agent's reward value expression is as follows:

[0112]

[0113] In the formula, The purpose is to reduce the reward value and control the reward value within a reasonable range, which will help the convergence of the algorithm and improve the training effect.

[0114] The penalty constant can constrain the unreasonable strategies explored by the agent, reduce its use of unreasonable strategies, and accelerate the convergence of the algorithm. Less than or equal to 0 or greater than or equal to 1, the penalty coefficient A large negative number indicates a penalty, and the penalty coefficients for other charge states are When the heat supply of the gas turbine cogeneration unit has met the heat load, the electric boiler input power When it is still a positive number, the penalty coefficient A large negative number is used to penalize unreasonable strategies. In other cases, the penalty coefficient is 0.

[0115] 5. In the Online-DDPG algorithm, the neural network parameters are first learned offline using historical data, and then online learning is performed after the parameters are optimized to a certain value. Offline learning consists of two parts: agent learning and prediction network learning. Online learning and application can continue to optimize and memorize parameters according to special changes in the external environment while optimizing the scheduling of the integrated energy system. See the Online-DDPG algorithm framework diagram. Figure 2 .

[0116] 5-1. Offline learning.

[0117] 5-11 Agent Learning: The offline DDPG algorithm uses an actor-critic architecture and learns deterministic strategies. First, two neural networks, Actor and Critic, are introduced to fit the action strategy function respectively. and the state-action-value function , the Actor network receives input current state To output actions to maximize the long-term accumulated reward, its neural network parameters Updates are made under the guidance of the Critic network; the Critic network estimates the value of each state-action pair (s, a) and updates the Actor network parameters by backpropagating the gradients .

[0118] In addition, the target network is used to update parameters to stabilize the training process, so the entire offline DDPG algorithm contains four neural networks, namely the Actor network mentioned above and Critic Network And the target Actor network and target critic network ,in, , Represent the neural network parameters in the Actor network and the Critic network respectively, , Represent the neural network parameters in the target Actor network and the target Critic network respectively, Represents the output of the target Actor network.

[0119] The Critic network minimizes the loss function To learn the state action value function, the loss function is:

[0120]

[0121] Among them, the objective function , r is the current environment reward, γ is the discount factor. The Actor network uses deterministic policy gradient To update the network parameters .

[0122] The policy gradient is as follows:

[0123]

[0124] Where N is the sampling batch size of the experience replay pool.

[0125] The algorithm performs soft updates (conservative strategy iteration) on the target actor network and target critic network parameters. The update formula is as follows:

[0126]

[0127] In the formula, , represents the target network parameters. In this way, the value of the target network is constrained to change slowly.

[0128] Use the Replay Buffer (RB) to store previous experience Each time the training parameters are updated, a small batch of experience is randomly sampled from the experience revisit pool to enable the target critic network and the target actor network to predict and infer the optimization target of the critic network at the current moment. This is also the reason why the offline DDPG algorithm needs to know the environmental history data in advance and learn offline.

[0129] 5-12 Prediction network learning: The data from offline learning will also be used for training the prediction network. By learning historical data such as user thermal power load, renewable energy power generation of the integrated energy system, electricity prices of the large power grid and the integrated energy system, and the charge state of the battery pack, the future state can be predicted.

[0130] The prediction network is composed of a Long Short-Term Memory network with an attention mechanism added, which can provide the next-moment state required for optimization for online-DDPG in the future.

[0131] LSTM is a special recurrent neural network structure that controls the flow of information through input gates, forget gates, and output gates. The input gate determines how much new information can enter the memory unit, the forget gate controls how much historical information to discard, and the output gate determines how much information to output at the current moment. This design enables LSTM to store important information for a long time while selectively forgetting irrelevant information. The attention mechanism is a mechanism that allows the model to selectively focus on important parts of the input information. It calculates the similarity between the query and the key to obtain the attention weights, and then applies these weights to the value to obtain a weighted context representation. This mechanism mimics the way humans allocate attention, allowing the model to dynamically decide to focus on different parts of the input at different times.

[0132] LSTM combined with the attention mechanism has unique advantages in prediction tasks: LSTM can handle the temporal dependencies of sequences well through its gating mechanism, and can update short-term information in a timely manner while maintaining long-term memory; while the attention mechanism calculates the similarity between the query and the key, assigns different weights to information at different time steps, so that the model can dynamically focus on important parts of historical information according to current prediction needs. This combination not only ensures the continuity of temporal information, but also highlights the important features of key time points. At the same time, the visualization of attention weights also provides interpretability of prediction results. The complementary effect of the two mechanisms not only improves the accuracy and robustness of predictions, but also achieves a good balance in computational efficiency.

[0133] This embodiment adds an attention mechanism to the last layer of the LSTM prediction neural network, uses the last time step as query information, the LSTM layer is used to process temporal dependencies, the attention layer processes key information, and finally outputs prediction information.

[0134] For the two tasks of solar-wind renewable energy power generation and user thermal power load demand forecasting, LSTM combined with attention mechanism can handle the periodic and sudden characteristics of data well: for renewable energy power generation, LSTM can learn the seasonal laws and continuous characteristics of weather changes through its memory mechanism, while the attention mechanism can timely capture key factors that may cause sudden weather changes such as cold air and typhoons, and improve the accuracy of extreme weather forecasts; for user behavior forecasting, LSTM is good at capturing users' daily behavior patterns such as work and rest patterns, consumption habits, etc., while the attention mechanism can identify special events that affect user behavior such as holidays and promotional activities. This combination can accurately predict users' regular behavior and respond to behavioral changes in special circumstances in a timely manner. The electricity price of large power grids and integrated energy systems is related to user electricity loads, and the state of charge of battery packs is related to user electricity loads and solar-wind renewable energy power generation, so this prediction model can make good predictions for them.

[0135] 5-2. Online learning and application.

[0136] Online-DDPG is based on DDPG after offline learning and LSTM with added self-attention mechanism. It removes the experience replay pool of offline DDPG and uses the prediction network for online learning.

[0137] Actor network receives current state Output value Add noise N to get the action Acting on the integrated energy system, that is, the scheduling strategy at the current moment.

[0138] Current state of integrated energy systems Input the prediction network to get the predicted value of the next state ; The target Actor network receives the state prediction value of the prediction network And the output result Send them to the target Critic network together and get , providing optimization objectives for the Critic network so that the Actor network and the Critic network can update parameters and complete online learning.

[0139] The architectures of the actor network and the critic network are shown in Figure 3(a)-Figure 3(b). The target actor network architecture is consistent with the actor network architecture, and the target critic network architecture is consistent with the actor network architecture.

[0140] This embodiment uses the historical data of the integrated energy system as the environmental state to train the neural networks in DDPG and LSTM+AM offline. The environmental state includes electrical load, thermal load, renewable energy generation, battery energy storage status, integrated energy system electricity purchase price, and integrated energy system electricity sales price. After offline training, the neural network parameters in DDPG and LSTM+AM will be transplanted for the continued update of the online-DDPG algorithm and the scheduling of the integrated energy system.

[0141] In the integrated energy system optimization scheduling method proposed in this embodiment, the actor network is based on the current integrated energy system state. Select an action , including natural gas consumption of gas turbine combined heat and power units , Charge-discharge power of the battery pack Input power of electric boiler , and then execute the action, the integrated energy system environment transfers to the next state, and the agent receives a reward The agent collects the state information of the integrated energy system at the next moment as the new state and makes the scheduling decision at that moment.

[0142] The original DDPG algorithm is an off-policy algorithm, which relies too much on historical data and requires a sufficient amount of historical data. This not only places too much demand on the data storage capacity of the integrated energy system, but also wastes the training time and training resources of the reinforcement learning algorithm. The online-DDPG algorithm proposed in this embodiment combines offline learning with online learning, and only requires a small part of historical data for offline learning before online learning and optimization scheduling can be performed, which greatly saves training costs and improves algorithm efficiency.

[0143] This embodiment proposes a solar-wind combined energy supply integrated energy system planning optimization method based on deep reinforcement learning - online-DDPG, which transforms the dynamic planning problem of the integrated energy system into a reinforcement learning problem of continuous state and action space, without relying on sophisticated mathematical models, and can use prediction information to update the scheduling algorithm in real time, so that the scheduling algorithm always maintains the ability to perceive system changes, effectively solving the model problem caused by dimensionality disaster and information loss, and improving the ability to resist interference with load and renewable energy fluctuations. The specific flow chart of the solar-wind combined energy supply integrated energy system planning optimization method based on deep reinforcement learning is as follows: Figure 4 shown.

[0144] The innovative point of the integrated energy planning method proposed in this embodiment is that it transforms the complex dynamic planning optimization problem of the integrated energy system into a reinforcement learning framework, and uses deep learning technology to autonomously learn action strategies, thereby improving the optimization performance of the system and simplifying the mathematical modeling process of the integrated energy system.

[0145] Embodiment 2

[0146] The purpose of this embodiment is to provide a comprehensive energy system planning and optimization system based on deep reinforcement learning, including:

[0147] A building module is configured to: establish a mathematical model of the integrated energy system, determine an objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem;

[0148] A conversion module configured to: convert an initial dynamic programming optimization problem into a Markov decision process;

[0149] An offline learning module, which is configured to: perform offline learning on historical operation data pairs of the integrated energy system using a DDPG algorithm and a prediction network;

[0150] The optimization planning module is configured to apply the optimization parameters learned offline to the Online-DDPG algorithm, solve the initial dynamic programming optimization problem, and obtain the optimal planning solution for the integrated energy system.

[0151] In further embodiments, there is also provided:

[0152] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method described in Embodiment 1 is performed. For the sake of brevity, no further description is given here.

[0153] A computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method described in embodiment 1 is completed.

[0154] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A comprehensive energy system planning optimization method based on deep reinforcement learning, characterized in that: include: Establish a mathematical model of the integrated energy system, determine the objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem; The mathematical model of the comprehensive energy system is a planning optimization model with the minimum economic benefit and environmental benefit as the optimization target; the constraint conditions include thermal energy balance constraint, electrical energy balance constraint and gas energy balance constraint; Convert the initial dynamic programming optimization problem into a Markov decision process; Use DDPG algorithm and prediction network to conduct offline learning on historical operation data of integrated energy system; The optimization parameters learned offline are applied to the Online-DDPG algorithm to solve the initial dynamic programming optimization problem and obtain the optimal planning scheme for the integrated energy system. The prediction network is composed of a long short-term memory network with an attention mechanism, which provides the next state required for optimization for online-DDPG. The Online-DDPG algorithm performs online learning, specifically: Input the current state of the integrated energy system into the prediction network to obtain the predicted value of the state of the integrated energy system at the next moment; The target Actor network receives the state prediction value of the prediction network and inputs the output action value function into the target Critic network to obtain the optimization target of the Critic network. The Critic network updates the parameters of the Actor network and the Critic network according to the optimization target to complete online learning.

2. The integrated energy system planning and optimization method based on deep reinforcement learning according to claim 1, characterized in that: The initial dynamic programming optimization problem is transformed into a Markov decision process, specifically: The Markov decision process consists of five tuples<S, A, P, R, γ> definition, It is used to describe the state space of the environment, A is used to describe the action space of the agent's behavior, R is the reward obtained by the agent, P is used to represent the state transition, and γ is used to represent the discount factor; Among them, the status Renewable energy power provided for photovoltaic power generation and wind power generation, user electrical load, user thermal load, battery pack charge state, large power grid time-of-use electricity price and integrated energy system time-of-use electricity price; dynamic The natural gas consumption of the gas turbine cogeneration unit, the electric power input to the electric boiler, and the charge-discharge power of the battery pack; The reward function includes the objective function value, the unreasonable charging and discharging penalty coefficient of the battery pack, and the unreasonable utilization penalty coefficient of the electric boiler.

3. The integrated energy system planning optimization method based on deep reinforcement learning according to claim 2, characterized in that: When the state of charge of the battery pack at the next moment is ≤0 or ≥1, the unreasonable charging and discharging penalty coefficient of the battery pack is set to a large negative number to indicate the penalty, and the unreasonable charging and discharging penalty coefficients under other charge states are 0; when the heating of the gas turbine cogeneration unit has met the thermal load and the electric boiler input power is still a positive number, the unreasonable utilization penalty coefficient of the electric boiler is set to a large negative number to punish the unreasonable strategy, and the unreasonable utilization penalty coefficient of the electric boiler is 0 in other cases.

4. The integrated energy system planning and optimization method based on deep reinforcement learning according to claim 1, characterized in that: The DDPG algorithm and prediction network are used to conduct offline learning on the historical operation data of the integrated energy system, specifically: The Actor network selects actions based on the current state to maximize the long-term accumulated rewards, and the Actor network parameters are updated under the guidance of the Critic network; The Critic network estimates the value of each state-action pair and updates the Actor network parameters through gradient backpropagation; Using the network parameters of the Actor network and the Critic network, the network parameters of the target critic network and the target actor network are updated; The experience recycling pool is used to store previous experience. A small batch of experience is randomly sampled from the experience recycling pool each time the parameters are updated during training. This is used to enable the target critic network and the target actor network to predict and infer the optimization target of the critic network at the current moment.

5. The integrated energy system planning optimization method based on deep reinforcement learning according to claim 1, characterized in that: The mathematical model of the integrated energy system includes energy coupling, energy storage and distributed power sources; wherein the energy coupling includes the electric power and thermal power output of the cogeneration turbine set and the corresponding constraints, the thermal power output of the gas boiler and the corresponding constraints, and the thermal power output of the electric boiler and the corresponding constraints; the energy storage includes: battery pack charging and discharging, the cost of the battery pack, and the charging and discharging constraints of the battery pack; the distributed power source includes the power generation power of the wind turbine set and the photovoltaic generator set.

6. Comprehensive energy system planning and optimization system based on deep reinforcement learning, characterized by: include: A building module is configured to: establish a mathematical model of the integrated energy system, determine an objective function and constraints of the integrated energy system, and form an initial dynamic programming optimization problem; The mathematical model of the comprehensive energy system is a planning optimization model with the minimum economic benefit and environmental benefit as the optimization target; the constraint conditions include thermal energy balance constraint, electrical energy balance constraint and gas energy balance constraint; A conversion module configured to: convert an initial dynamic programming optimization problem into a Markov decision process; An offline learning module, which is configured to: perform offline learning on historical operation data of the integrated energy system using a DDPG algorithm and a prediction network; The optimization planning module is configured to: apply the optimization parameters learned offline to the Online-DDPG algorithm to solve the initial dynamic programming optimization problem and obtain the optimal planning scheme for the integrated energy system; The prediction network is composed of a long short-term memory network with an attention mechanism, which provides the next state required for optimization for online-DDPG. The Online-DDPG algorithm performs online learning, specifically: Input the current state of the integrated energy system into the prediction network to obtain the predicted value of the state of the integrated energy system at the next moment; The target Actor network receives the state prediction value of the prediction network and inputs the output action value function into the target Critic network to obtain the optimization target of the Critic network. The Critic network updates the parameters of the Actor network and the Critic network according to the optimization target to complete online learning.

7. An electronic device, characterized in that: The method comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor, wherein when the computer instructions are executed by the processor, the method according to any one of claims 1 to 5 is completed.

8. A computer-readable storage medium, characterized in that: Used to store computer instructions, which, when executed by a processor, complete the method described in any one of claims 1 to 5.

Images