A comprehensive energy system optimization scheduling method, system, equipment and medium

Abstract

The present invention discloses a method, system, device and medium for optimizing and dispatching an integrated energy system. The method comprises the following steps: upon receiving a dispatching task, obtaining the state S at the current time t based on a pre-built and trained digital twin model of the integrated energy system; t ; Use the pre-trained DDPG algorithm in the digital twin model of the integrated energy system to select action a t ; Execute action a t And get the state S at time t+1 t+1 ; Based on the state S at time t+1 obtained t+1 And the pre-trained DDPG algorithm, by selecting the action a at time t+1 t+1 The optimization scheduling method provided by the present invention is specifically a comprehensive energy optimization scheduling method based on digital twins and deep reinforcement learning, which can improve the accuracy and timeliness of the optimized output of the units in the comprehensive energy system, thereby reducing the operating cost of the comprehensive energy system.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power dispatching decision-making, and in particular relates to a comprehensive energy system optimization dispatching method, system, equipment and medium. Background Art

[0002] Under the pressure of expanding energy demand and reducing carbon emissions, building a sound integrated energy system is an important way to promote energy transformation; by building a sound integrated energy system, we can absorb new energy, optimize the energy supply structure and improve energy utilization efficiency.

[0003] The integrated energy system needs to consider multiple mutually coupled heterogeneous energy networks and equipment at the same time to meet the energy needs of electricity, heat, cooling, etc. on the user side. In addition, the high-penetration injection of new energy with random output makes the system structure more complex and it is difficult to collect system operating parameters. The integrated energy system has higher requirements on the accuracy and real-time performance of the unit output. Inaccurate system parameters and rapidly changing real-time data can easily lead to untimely updates, which will affect the output optimization accuracy of the unit and reduce the reliability of the results.

[0004] Based on the above statements, it can be seen that how to optimize unit output to reduce system operating costs has become an important issue in the study of integrated energy systems; the accuracy and timeliness of the optimized output of units in existing integrated energy systems are insufficient, and a new integrated energy optimization scheduling method is urgently needed. Summary of the invention

[0005] The purpose of the present invention is to provide a comprehensive energy system optimization scheduling method, system, device and medium to solve one or more of the above-mentioned technical problems. The optimization scheduling method provided by the present invention is specifically a comprehensive energy optimization scheduling method based on digital twins and deep reinforcement learning, which can improve the accuracy and timeliness of the optimized output of the units in the comprehensive energy system, thereby reducing the operating cost of the comprehensive energy system.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A first aspect of the present invention provides a method for optimizing and scheduling an integrated energy system, comprising the following steps:

[0008] When receiving a scheduling task, the state S of the integrated energy system at the current time t is obtained based on the pre-built and trained integrated energy system digital twin model. t ; The integrated energy system digital twin model includes a decision-making layer;

[0009] The pre-trained DDPG algorithm deployed in the decision layer is used based on the state S at the current time t. t , determine action a t ;

[0010] Based on the digital twin model of the integrated energy system, perform action a t And get the state S at time t+1 t+1 ;

[0011] Based on the state S at time t+1 obtained t+1 And the pre-trained DDPG algorithm, by selecting the action a at time t+1 t+1 A dynamic dispatching action is obtained, and the output of the units included in the integrated energy system is adjusted based on the dynamic dispatching action.

[0012] A further improvement of the present invention is that the step of obtaining the pre-built integrated energy system digital twin model includes:

[0013] Build the physical entity of the integrated energy system and construct the physical layer;

[0014] Installing multi-dimensional sensors at preset positions of the physical entities of the integrated energy system to construct a perception layer;

[0015] Constructing a data transmission layer; wherein the data transmission layer is used to store or transmit the data acquired by the perception layer;

[0016] Constructing a data processing layer; wherein the data processing layer is used to receive and pre-process the data transmitted by the data transmission layer, and construct a digital twin environment of the integrated energy system based on the pre-processed data; used to treat each type of unit in the integrated energy system as an intelligent agent to achieve continuous interaction between the intelligent agent and the digital twin environment; used to use historical data to correct the prediction results generated by the decision-making layer;

[0017] Build a virtual entity of the integrated energy system and construct a decision-making layer; wherein the decision-making layer is used to find the optimal strategy for the economic operation output of the unit based on the interaction data between the intelligent agent and the digital twin environment, using the pre-trained DDPG algorithm, and generate a unit output scheduling plan;

[0018] Based on the physical layer, perception layer, data transmission layer, data processing layer and decision-making layer, a digital twin model of the comprehensive energy system is obtained.

[0019] A further improvement of the present invention is that the digital twin environment includes power balance constraints within the integrated energy system and operating range constraints of each unit;

[0020] The internal power balance constraint includes the electrical power and thermal power balance constraints, and the expression is:

[0021] P grid (t)+P RG (t)+PBES (t)+P CHP (t)-P EB (t) = P load (t);

[0022] H CHP (t)+H GB (t)+H EB (t) = H load (t);

[0023] Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging power or discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the combined heat and power unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load;

[0024] The operating range constraints of each unit are expressed as:

[0025]

[0026] In the formula, are the lower and upper limits of the interaction power between the integrated energy system and the main grid side, respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; They are the lower limit and upper limit of the output thermal power of the gas boiler respectively; are the lower and upper limits of the output thermal power of the electric boiler respectively; They are the lower and upper limits of the energy storage state of charge, respectively.

[0027] A further improvement of the present invention is that the step of obtaining the pre-trained DDPG algorithm comprises:

[0028] Constructing the state space, action space, and reward function in a deep reinforcement learning framework for integrated energy systems;

[0029] The expression of the state space is, S t = {P load (t), Hload (t), P RG (t), C SOC (t-1), t}; where C SOC (t-1) is the charge state of the electric energy storage;

[0030] The action space expression is: t = {P CHP (t), P BES (t), H GB (t)};

[0031] The reward function expression is:

[0032] Where, energy purchase cost C E , depreciation cost C BES ;

[0033]

[0034] In the formula, ε e (t) is the electricity price in time period t, ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, Δt is the time slot length, ρ BES is the depreciation cost coefficient of electric energy storage;

[0035] When the system state S in the integrated energy system t OK, schedule action a t The degree of goodness or badness adopts the action value function Q π (S, a) evaluation until the optimal strategy π is found * To maximize Q π (S, a); where the action value function Q π The expression of (S, a) is, In the formula, E π (.) is the expectation under strategy π, γ is the discount factor;

[0036] Use the value network θ Q and the policy network θ π To approximate the critic and actor functions in the DDPG framework, the value network θ Q and the policy network θ π are two independent networks, each with its own target network θ Q ' and θ π '; In the iterative calculation at time t, the policy network θ Q The state S in the input state spacet , output action a in the action space t ; The agent transfers state according to the control strategy at time t and obtains the state S at the next moment t+1 , and gives the reward r at time t t (S t , a t ) is fed back to the agent, which records the experience e t =(S t , a t , r t (S t , a t ), S t+1 ) and put it into the experience pool; value network θ π The state S in the input state space t and action a in the action space t , output action-value function Q π (S t , a t );

[0037] During training, we repeat the iterations and randomly extract small batches of samples from the experience pool each time. Q and θ π The network updates the network parameters by the gradient descent method until it is stable, and the pre-trained DDPG algorithm is obtained.

[0038] A second aspect of the present invention provides an integrated energy system optimization and scheduling system, comprising:

[0039] The first state acquisition module is used to obtain the state S of the integrated energy system at the current time t based on the pre-built and trained integrated energy system digital twin model when receiving the scheduling task. t ; The integrated energy system digital twin model includes a decision-making layer;

[0040] The first action acquisition module is used to adopt the pre-trained DDPG algorithm deployed in the decision layer, based on the state S at the current time t t , determine action a t ;

[0041] The second state acquisition module is used to perform action a based on the digital twin model of the integrated energy system. t And get the state S at time t+1 t+1 ;

[0042] The second action acquisition module is used to obtain the state S at time t+1 based on the state S at time t+1. t+1 And the DDPG algorithm, by selecting action a at time t+1 t+1A dynamic dispatching action is obtained, and the output of the units included in the integrated energy system is adjusted based on the dynamic dispatching action.

[0043] A further improvement of the present invention is that the step of obtaining the pre-built integrated energy system digital twin model includes:

[0044] Build the physical entity of the integrated energy system and construct the physical layer;

[0045] Installing multi-dimensional sensors at preset positions of the physical entities of the integrated energy system to construct a perception layer;

[0046] Constructing a data transmission layer; wherein the data transmission layer is used to store or transmit the data acquired by the perception layer;

[0047] Constructing a data processing layer; wherein the data processing layer is used to receive and pre-process the data transmitted by the data transmission layer, and construct a digital twin environment of the integrated energy system based on the pre-processed data; used to treat each type of unit in the integrated energy system as an intelligent agent to achieve continuous interaction between the intelligent agent and the digital twin environment; used to use historical data to correct the prediction results generated by the decision-making layer;

[0048] Build a virtual entity of the integrated energy system and construct a decision-making layer; wherein the decision-making layer is used to find the optimal strategy for the economic operation output of the unit based on the interaction data between the intelligent agent and the digital twin environment, using the pre-trained DDPG algorithm, and generate a unit output scheduling plan;

[0049] Based on the physical layer, perception layer, data transmission layer, data processing layer and decision-making layer, a digital twin model of the comprehensive energy system is obtained.

[0050] A further improvement of the present invention is that the digital twin environment includes power balance constraints within the integrated energy system and operating range constraints of each unit;

[0051] The internal power balance constraint includes the electrical power and thermal power balance constraints, and the expression is:

[0052] P grid (t)+P RG (t)+P BES (t)+P CHP (t)-P EB (t) = P load (t);

[0053] H CHP (t)+H GB (t)+H EB (t) = H load (t);

[0054] Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging power or discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the combined heat and power unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load;

[0055] The operating range constraints of each unit are expressed as:

[0056]

[0057] In the formula, are the lower and upper limits of the interaction power between the integrated energy system and the main grid side, respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; They are the lower limit and upper limit of the output thermal power of the gas boiler respectively; are the lower and upper limits of the output thermal power of the electric boiler respectively; They are the lower and upper limits of the energy storage state of charge, respectively.

[0058] A further improvement of the present invention is that the step of obtaining the pre-trained DDPG algorithm comprises:

[0059] Constructing the state space, action space, and reward function in a deep reinforcement learning framework for integrated energy systems;

[0060] The expression of the state space is, S t = {P load (t), H load (t), P RG (t), C SOC (t-1), t}; where C SOC (t-1) is the charge state of the electric energy storage;

[0061] The action space expression is: t = {P CHP (t), P BES (t), HGB (t)};

[0062] The reward function expression is:

[0063] Where, energy purchase cost C E , depreciation cost C BES ;

[0064]

[0065] In the formula, ε e (t) is the electricity price in time period t, ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, Δt is the time slot length, ρ BES is the depreciation cost coefficient of electric energy storage;

[0066] When the system state S in the integrated energy system t OK, schedule action a t The degree of goodness or badness adopts the action value function Q π (S, a) evaluation until the optimal strategy π is found * To maximize Q π (S, a); where the action value function Q π The expression of (S, a) is, In the formula, E π (.) is the expectation under strategy π, γ is the discount factor;

[0067] Use the value network θ Q and the policy network θ π To approximate the critic and actor functions in the DDPG framework, the value network θ Q and the policy network θ π are two independent networks, each with its own target network θ Q ' and θ π '; In the iterative calculation at time t, the policy network θ Q The state S in the input state space t , output action a in the action space t ; The agent transfers state according to the control strategy at time t and obtains the state S at the next moment t+1 , and gives the reward r at time t t (S t , a t ) is fed back to the agent, which records the experience e t =(S t , at , r t (S t , a t ), S t+1 ) and put it into the experience pool; value network θ π The state S in the input state space t and action a in the action space t , output action-value function Q π (S t , a t );

[0068] During training, we repeat the iterations and randomly extract small batches of samples from the experience pool each time. Q and θ π The network updates the network parameters by the gradient descent method until it is stable, and the pre-trained DDPG algorithm is obtained.

[0069] A third aspect of the present invention provides an electronic device, comprising:

[0070] at least one processor; and,

[0071] a memory communicatively connected to the at least one processor; wherein,

[0072] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any of the above-mentioned integrated energy system optimization and scheduling methods of the present invention.

[0073] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned integrated energy system optimization scheduling methods of the present invention.

[0074] Compared with the prior art, the present invention has the following beneficial effects:

[0075] The comprehensive energy optimization dispatching method provided by the present invention can comprehensively improve the control level of power grid enterprises in the field of power dispatching decision-making from the following two aspects:

[0076] (1) Accurate acquisition of data parameters of the integrated energy system can be achieved; specifically, the present invention obtains accurate system operation data by establishing a digital twin virtual environment, and simulates the unit output in real time through the digital twin virtual environment, which can provide relevant data basis for subsequent deep reinforcement learning decision-making.

[0077] (2) It is possible to obtain the optimal strategy for the output of the integrated energy system units; specifically, the present invention uses a deep reinforcement learning algorithm to search for the best decision through the interaction between the unit and the environment, and generates a corresponding scheduling plan to guide the optimal output of the integrated energy system units.

[0078] To sum up, the optimization scheduling method provided by the present invention is specifically a comprehensive energy optimization scheduling method based on digital twins and deep reinforcement learning, which can improve the accuracy and timeliness of the optimized output of the units in the comprehensive energy system. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art; obviously, the drawings described below are some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0080] Figure 1 It is a flowchart of a comprehensive energy optimization scheduling method according to an embodiment of the present invention;

[0081] Figure 2 It is a schematic diagram of the structural workflow of the comprehensive energy digital twin model in an embodiment of the present invention;

[0082] Figure 3 It is a schematic diagram of the decision-making layer workflow based on the comprehensive energy digital twin model in an embodiment of the present invention;

[0083] Figure 4 It is a schematic diagram of a comprehensive energy optimization scheduling system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0085] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0086] See also Figure 1 , a comprehensive energy system optimization scheduling method according to an embodiment of the present invention comprises the following steps:

[0087] When the integrated energy system receives a scheduling task, it obtains the state S at the current time t based on the pre-built integrated energy system digital twin model. t ;

[0088] Based on the current state S at time t t , using the pre-trained DDPG algorithm strategy network in the integrated energy system digital twin model to select action a t ;

[0089] Based on the digital twin model of the integrated energy system, perform action a t And get the state S at time t+1 t+1 ;

[0090] Based on the state S at time t+1 obtained t+1 , using the pre-trained DDPG algorithm strategy network in the integrated energy system digital twin model to select action a t+1 , dynamic scheduling actions are obtained to achieve optimal scheduling of the integrated energy system.

[0091] In an embodiment of the present invention, the digital twin model includes: a physical layer, a perception layer, a data transmission layer, a data processing layer and a decision layer; the decision layer uses a deep reinforcement learning algorithm (DDPG algorithm) to find the optimal strategy for the economic operation output of the unit and generate a unit output scheduling plan; based on the digital twin model, the optimized scheduling of the integrated energy system can improve the accuracy and timeliness of the optimized output of the units in the integrated energy system.

[0092] See also Figure 2The embodiment of the present invention provides a comprehensive energy dispatching strategy based on digital twins and deep reinforcement learning. At the same time, a comprehensive energy system unit output optimization digital twin structure system consisting of five parts: physical layer, sensor layer, data transmission layer, data processing layer and decision layer is established to provide technical support for improving the power dispatching level of power grid enterprises; wherein, the construction process of the comprehensive energy system unit output optimization digital twin structure system is as follows:

[0093] Step 1: Establish the physical layer and build the physical entity of the integrated energy system; specifically, establish the data carrier of the entire digital twin, including various types of units in the integrated energy system, such as renewable energy units (photovoltaic, wind turbine units), electric energy storage, cogeneration units, gas boilers, electric boilers, etc., and provide parameters such as unit installed capacity, operating data, and operating environment of each type of equipment.

[0094] Step 2: Establish a perception layer to collect physical data. Specifically, install multi-dimensional sensors in the integrated energy system to collect weather data, operating data of renewable energy units, cogeneration units, gas boilers and other equipment, and user electrical and thermal loads, thereby driving the normal operation of the digital twin system.

[0095] Step 3: Establish a data transmission layer to realize data transmission and storage; for example, data transmission is centered on switches and Ethernet, and a wireless network transmission system is built to achieve efficient transmission of meteorological data, equipment operation data, etc. Data storage uses a combination of distributed local storage and centralized cloud storage, which can achieve dynamic response and mutual call of data according to system requirements.

[0096] Step 4: Establish a data processing layer. Specifically, on the one hand, the data processing layer receives data from the perception layer and the data transmission layer, and uses big data processing technology to preprocess the data, including data loading, data cleaning, data conversion, data extraction, etc., and establishes a digital twin environment for the integrated energy system. On the other hand, it regards each type of unit as an intelligent agent, and realizes continuous interaction between the intelligent agent and the environment. It uses historical data to correct the prediction results generated by the decision-making layer, and provides a basis for the decision-making layer to generate the final scheduling plan.

[0097] In an embodiment of the present invention, the established digital twin environment of the integrated energy system needs to comprehensively consider the power balance constraints within the system, the external energy supply constraints and the equipment operation constraints.

[0098] The power balance constraints in the integrated energy system mainly consider the following electric power and thermal power balance constraints:

[0099] P grid (t)+P RG (t)+PBES (t)+P CHP (t)-P EB (t) = P load (t) (1)

[0100] H CHP (t)+H GB (t)+H EB (t) = H load (t) (2)

[0101] Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging / discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the cogeneration unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load.

[0102] Based on the consideration of grid-side operation stability, the main grid constrains the interactive power of the integrated energy system. At the same time, in order to ensure the continuous and stable operation of the equipment in the system, the operating range of each unit is constrained as follows:

[0103]

[0104] In the formula, The lower and upper limits of the interaction power between the integrated energy system and the main grid side are respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; The lower and upper limits of the thermal output power of the gas boiler; are the lower and upper limits of the output thermal power of the electric boiler respectively; The lower and upper limits of the energy storage state of charge respectively.

[0105] Step 5: Establish a decision-making layer and build a virtual entity of the integrated energy system; specifically, based on the interaction data between the agent and the environment, the decision-making layer uses a deep reinforcement learning algorithm to find the best strategy for the economic operation output of the unit and generate a unit output scheduling plan. The plan is fed back to the terminal device to guide the output of each unit in the integrated energy system.

[0106] See also Figure 3 In the embodiment of the present invention, the step of obtaining the pre-trained DDPG algorithm includes:

[0107] When the economic dispatch of the integrated energy system is transformed into a reinforcement learning framework, the user's electric load, thermal load, output power of renewable energy units, charge state of the electric energy storage and dispatch time period t are used as the state information in the deep reinforcement learning framework of the integrated energy system. The state space can be described as:

[0108] S t = {P load (t), H load (t), P RG (t), C SOC (t-1), t} (4)

[0109] For a combined heat and power unit, when P CHP After (t) is determined, H can be obtained from the "electrical-thermal characteristics" between the output electrical power and thermal power. CHP (t); H GB After (t) is determined, H can be obtained by formula (2) EB (t), which can be further converted into P through the output efficiency of the electric boiler EB (t). Therefore, the action space in the deep reinforcement learning framework of the integrated energy system can be described as:

[0110] a t = {P CHP (t), P BES (t), H GB (t)} (5)

[0111] The problem of minimizing the total cost of the integrated energy system takes into account the energy purchase cost C E Hedian Energy Storage Charge / Discharge Depreciation Cost C BES , then the reward function can be taken as:

[0112]

[0113] in,

[0114]

[0115] In the formula, ε e (t) is the electricity price in time period t, ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, Δt is the time slot length, ρ BESis the depreciation cost coefficient of electric energy storage;

[0116] When the state S in the integrated energy system t OK, action a t The quality of the action can be expressed as the action-value function Q π (S, a) evaluation until the optimal strategy π is found * To maximize Q π (S, a).

[0117]

[0118] In the formula, E π (.) is the expectation under strategy π, and γ is the discount factor.

[0119] In this paper, the DDPG algorithm based on the actor-critic framework is used as the core of the decision layer, and the optimal policy function is estimated through a deep neural network, that is, two independent network value networks θ are used respectively. Q and the policy network θ π To approximate the critic and actor functions; each network has its own target network θ Q ' and θ π '. The main process includes:

[0120] 1) In the iterative calculation at time t, the policy network θ Q Input state S as shown in formula (4) t , output action a as shown in formula (5) t ;

[0121] 2) The agent transfers state according to the control strategy at this time and reaches the state S at the next moment t+1 , and gives the reward r at time t t (S t , a t ), feedback to the agent, the agent records the experience e t =(S t , a t , r t (S t , a t ), S t+1 ) and put it into the experience pool;

[0122] 3) Value network θ π The input state is shown in equation (4) and the action is shown in equation (5), and the output action-value function Q is shown in equation (8) π (S t , a t );

[0123] 4) Repeat steps 1) to 3) during training, randomly extracting small batches of samples from the experience pool each time, θ Q and θ π The network parameters are updated by the gradient descent method until they are stable, and the network training is completed.

[0124] When the integrated energy system receives a new scheduling task, it can t Use the trained DDPG algorithm strategy network to select scheduling action a t , perform action a t and the next environment state S t+1 , and then collect the system status information at time t+1, and perform S t+1 The decision at each moment can be used to obtain dynamic scheduling actions.

[0125] In summary, the embodiment of the present invention specifically discloses a comprehensive energy dispatching framework and strategy based on digital twin and deep reinforcement learning, especially for the technical field of power system dispatching decision-making. The principle of the present invention using the model for optimization and dispatching is as follows: first, the actual operation data of the comprehensive energy system is collected by sensors to build a virtual environment of the digital twin of the comprehensive energy system; second, through the continuous interaction between various types of unit agents and the digital twin environment, the relevant system parameter basis is provided for the decision of the deep reinforcement learning model; finally, the deep reinforcement learning model is used to find the optimal output strategy of the unit, and the information is fed back to the terminal controller in real time through the digital twin, and the corresponding control instructions are generated to guide the output dispatching of the unit of the comprehensive energy system. Digital twin technology uses interactive simulation between physical entities, sensors and historical databases to realize high-fidelity mapping between digital virtual space and actual physical equipment. Through deep reinforcement learning as the decision-making center, the synchronous evolution of the output of the physical system unit throughout the life cycle can be realized. The combination of digital twin and artificial intelligence technology provided by the embodiment of the present invention provides a new idea for solving the accuracy and timeliness of optimizing the output of the unit of the comprehensive energy system.

[0126] The following are device embodiments of the present invention, which can be used to perform method embodiments of the present invention. For details not disclosed in the device embodiments, please refer to the method embodiments of the present invention.

[0127] See also Figure 4 In another embodiment of the present invention, a comprehensive energy system optimization and scheduling system is provided, comprising:

[0128] The first state acquisition module is used to obtain the state S at the current time t based on the pre-built digital twin model of the integrated energy system when the integrated energy system receives the scheduling task. t ;

[0129] The first action acquisition module is used to obtain the state S at the current time t based on the state S at the current time t. t , using the pre-trained DDPG algorithm strategy network in the integrated energy system digital twin model to select action a t ;

[0130] The second state acquisition module is used to perform action a based on the digital twin model of the integrated energy system. t And get the state S at time t+1 t+1 ;

[0131] The second action acquisition module is used to obtain the state S at time t+1 based on the state S at time t+1. t+1 And the pre-trained DDPG algorithm, by selecting the action a at time t+1 t+1 Get a dynamically dispatched action.

[0132] The first state acquisition module acquires the state S at the current time t based on the pre-built integrated energy system digital twin model t In the process, the steps of obtaining the pre-built integrated energy system digital twin model include:

[0133] Build the physical entity of the integrated energy system and construct the physical layer;

[0134] Installing multi-dimensional sensors at preset positions of the physical entities of the integrated energy system to construct a perception layer;

[0135] Constructing a data transmission layer; wherein the data transmission layer is used to store or transmit the data acquired by the perception layer;

[0136] Constructing a data processing layer; wherein the data processing layer is used to receive and pre-process the data transmitted by the data transmission layer, and construct a digital twin environment of the integrated energy system based on the pre-processed data; used to treat each type of unit in the integrated energy system as an intelligent agent to achieve continuous interaction between the intelligent agent and the digital twin environment; used to use historical data to correct the prediction results generated by the decision-making layer;

[0137] Build a virtual entity of the integrated energy system and construct a decision-making layer; wherein the decision-making layer is used to find the optimal strategy for the economic operation output of the unit based on the interaction data between the intelligent agent and the digital twin environment, using the pre-trained DDPG algorithm, and generate a unit output scheduling plan;

[0138] Based on the physical layer, perception layer, data transmission layer, data processing layer and decision-making layer, a digital twin model of the comprehensive energy system is obtained.

[0139] The digital twin environment includes power balance constraints within the integrated energy system and operating range constraints for each unit;

[0140] The internal power balance constraint includes the electrical power and thermal power balance constraints, and the expression is:

[0141] P grid (t)+P RG (t)+P BES (t)+P CHP (t)-P EB (t) = P load (t);

[0142] H CHP (t)+H GB (t)+H EB (t) = H load (t);

[0143] Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging power or discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the combined heat and power unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load;

[0144] The operating range constraints of each unit are expressed as:

[0145]

[0146] In the formula, are the lower and upper limits of the interaction power between the integrated energy system and the main grid side, respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; They are the lower limit and upper limit of the output thermal power of the gas boiler respectively; are the lower and upper limits of the output thermal power of the electric boiler respectively; They are the lower and upper limits of the energy storage state of charge, respectively.

[0147] The first action acquisition module selects action a using a pre-trained DDPG algorithm strategy network in the digital twin model of the integrated energy system.t In the process, the steps of obtaining the pre-trained DDPG algorithm include:

[0148] Constructing the state space, action space, and reward function in a deep reinforcement learning framework for integrated energy systems;

[0149] The expression of the state space is, S t = {P load (t), H load (t), P RG (t), C SOC (t-1), t}; where C SOC (t-1) is the charge state of the electric energy storage;

[0150] The action space expression is: t = {P CHP (t), P BES (t), H GB (t)};

[0151] The reward function expression is:

[0152] Where, energy purchase cost C E , depreciation cost C BES ;

[0153]

[0154] In the formula, ε e (t) is the electricity price in time period t, ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, Δt is the time slot length, ρ BES is the depreciation cost coefficient of electric energy storage;

[0155] When the system state S in the integrated energy system t OK, schedule action a t The degree of goodness or badness adopts the action value function Q π (S, a) Evaluate until the optimal policy π* is found to maximize Q π (S, a); where the action value function Q π The expression of (S, a) is, In the formula, E π (.) is the expectation under strategy π, γ is the discount factor;

[0156] Use the value network θ Q and the policy network θ πTo approximate the critic and actor functions in the DDPG algorithm framework, the value network θ Q and the policy network θ π are two independent networks, each with its own target network θ Q ' and θ π '; In the iterative calculation at time t, the policy network θ Q The state S in the input state space t , output action a in the action space t ; The agent transfers state according to the control strategy at time t and obtains the state S at the next moment t+1 , and gives the reward r at time t t (S t , a t ) is fed back to the agent, which records the experience e t =(S t , a t , r t (S t , a t ), S t+1 ) and put it into the experience pool; value network θ π The state S in the input state space t and action a in the action space t , output action-value function Q π (S t , a t );

[0157] During training, we repeat the iterations and randomly extract small batches of samples from the experience pool each time. Q and θ π The network updates the network parameters by the gradient descent method until it is stable, and the pre-trained DDPG algorithm is obtained.

[0158] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of the integrated energy system optimization scheduling method.

[0159] In another embodiment of the present invention, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It can be understood that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the method for optimizing and scheduling the integrated energy system in the above embodiment.

[0160] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0161] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0162] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0163] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A comprehensive energy system optimization scheduling method, characterized in that: The following steps are involved: When receiving a scheduling task, the state S of the integrated energy system at the current time t is obtained based on the pre-built and trained integrated energy system digital twin model. t ; The integrated energy system digital twin model includes a decision-making layer; The pre-trained DDPG algorithm deployed in the decision layer is used based on the state S at the current time t. t , determine action a t ; Based on the digital twin model of the integrated energy system, perform action a t And get the state S at time t+1 t+1 ; Based on the state S at time t+1 obtained t+1 And the DDPG algorithm, by selecting action a at time t+1 t+1 Obtaining a dynamic dispatching action, and adjusting the output of the units included in the integrated energy system based on the dynamic dispatching action; The step of obtaining the pre-trained DDPG algorithm includes: Constructing the state space, action space, and reward function in a deep reinforcement learning framework for integrated energy systems; The expression of the state space is, S t = {P load (t), H load (t), P RG (t), C SOC (t-1), t}; where S t represents the state space; P load (t) represents the user's electricity load in the time period t; H load (t) represents the user heat load in time period t; P RG (t) represents the output power of the renewable energy unit in the time period t; C SOC (t-1) is the charge state of the electric energy storage; The action space expression is: t = {P CHP (t), P BES (t), H GB (t)}; where a t represents the action space; P CHP (t) represents the output power of the combined heat and power unit in the time period t; P BES (t) represents the charging power or discharging power of the energy storage in the time period t; H GB (t) represents the output thermal power of the gas boiler in the time period t; The reward function expression is: In the formula, C E represents the energy purchase cost, C BES represents the depreciation cost; In the formula, ε e (t) is the electricity price in time period t; P grid (t) represents the interaction power with the main grid side in the time period t; ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, △t is the time slot length, ρ BES is the depreciation cost coefficient of electric energy storage; When the system state S in the integrated energy system t OK, schedule action a t The degree of goodness or badness adopts the action value function Q π (S, a) evaluation until the optimal policy π* is found to maximize Q π (S, a); where the action value function Q π The expression of (S,a) is, In the formula, E π (.) is the expectation under strategy π, γ is the discount factor; Use the value network θ Q and the policy network θ π To approximate the critic and actor functions in the DDPG algorithm framework, the value network θ Q and the policy network θ π are two independent networks, each with its own target network θ Q‘ and θ π’ ; Among them, in the iterative calculation at time t, the policy network θ Q The state S in the input state space t , output action a in the action space t ; The agent transfers state according to the control strategy at time t and obtains the state S at the next moment t+1 , and gives the reward r at time t t (S t ,a t ) is fed back to the agent, which records the experience e t =(S t ,a t ,r t (S t ,a t ),S t+1 ) and put it into the experience pool; value network θ π The state S in the input state space t and action a in the action space t , output action-value function Q π (S t ,a t ); During training, we repeat the iterations and randomly extract small batches of samples from the experience pool each time. Q and θ π The network updates the network parameters by the gradient descent method until it is stable, and the pre-trained DDPG algorithm is obtained.

2. A comprehensive energy system optimization scheduling method according to claim 1, characterized in that: The steps to obtain a pre-built integrated energy system digital twin model include: Build the physical entity of the integrated energy system and construct the physical layer; Installing multi-dimensional sensors at preset positions of the physical entities of the integrated energy system to construct a perception layer; Constructing a data transmission layer; wherein the data transmission layer is used to store or transmit the data acquired by the perception layer; Constructing a data processing layer; wherein the data processing layer is used to receive and pre-process the data transmitted by the data transmission layer, and construct a digital twin environment of the integrated energy system based on the pre-processed data; used to treat each type of unit in the integrated energy system as an intelligent agent to achieve continuous interaction between the intelligent agent and the digital twin environment; used to use historical data to correct the prediction results generated by the decision-making layer; Build a virtual entity of the integrated energy system and construct a decision-making layer; wherein the decision-making layer is used to find the optimal strategy for the economic operation output of the unit based on the interaction data between the intelligent agent and the digital twin environment, using the pre-trained DDPG algorithm, and generate a unit output scheduling plan; Based on the physical layer, perception layer, data transmission layer, data processing layer and decision-making layer, a digital twin model of the comprehensive energy system is obtained.

3. The method for optimizing and dispatching a comprehensive energy system according to claim 2, characterized in that: The digital twin environment includes power balance constraints within the integrated energy system and operating range constraints for each unit; The internal power balance constraint includes the electrical power and thermal power balance constraints, and the expression is: P grid (t)+P RG (t)+P BES (t)+P CHP (t)-P EB (t)=P load (t); H CHP (t)+H GB (t)+H EB (t)=H load (t); Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging power or discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the cogeneration unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load in the time period t; The operating range constraints of each unit are expressed as: In the formula, are the lower and upper limits of the interaction power between the integrated energy system and the main grid side, respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; They are the lower limit and upper limit of the output thermal power of the gas boiler respectively; are the lower and upper limits of the output thermal power of the electric boiler respectively; They are the lower and upper limits of the energy storage state of charge, respectively.

4. A comprehensive energy system optimization and dispatching system, characterized in that: include: The first state acquisition module is used to obtain the state S of the integrated energy system at the current time t based on the pre-built and trained integrated energy system digital twin model when receiving the scheduling task. t ; The integrated energy system digital twin model includes a decision-making layer; The first action acquisition module is used to adopt the pre-trained DDPG algorithm deployed in the decision layer, based on the state S at the current time t t , determine action a t ; The second state acquisition module is used to perform action a based on the digital twin model of the integrated energy system. t And get the state S at time t+1 t+1 ; The second action acquisition module is used to obtain the state S at time t+1 based on the state S at time t+1. t+1 And the DDPG algorithm, by selecting action a at time t+1 t+1 Obtaining a dynamic dispatching action, and adjusting the output of the units included in the integrated energy system based on the dynamic dispatching action; The step of obtaining the pre-trained DDPG algorithm includes: Constructing the state space, action space, and reward function in a deep reinforcement learning framework for integrated energy systems; The expression of the state space is, S t = {P load (t),H load (t),P RG (t),C SOC (t-1),t}; where S t represents the state space; P load (t) represents the user's electricity load in the time period t; H load (t) represents the user heat load in time period t; P RG (t) represents the output power of the renewable energy unit in the time period t; C SOC (t-1) is the charge state of the electric energy storage; The action space expression is: t = {P CHP (t),P BES (t),H GB (t)}; where a t represents the action space; P CHP (t) represents the output power of the combined heat and power unit in the time period t; P BES (t) represents the charging power or discharging power of the energy storage in the time period t; H GB (t) represents the output thermal power of the gas boiler in the time period t; The reward function expression is: In the formula, C E represents the energy purchase cost, C BES represents the depreciation cost; In the formula, ε e (t) is the electricity price in time period t; P grid (t) represents the interaction power with the main grid side in the time period t; ε gas (t) is the unit calorific value price of natural gas purchased in period t, is the electrical efficiency of the combined heat and power unit, η GB is the efficiency of the gas boiler, T is the total scheduling time, △t is the time slot length, ρ BES is the depreciation cost coefficient of electric energy storage; When the system state S in the integrated energy system t OK, schedule action a t The degree of goodness or badness adopts the action value function Q π (S, a) evaluation until the optimal policy π* is found to maximize Q π (S, a); where the action value function Q π The expression of (S,a) is, In the formula, E π (.) is the expectation under strategy π, γ is the discount factor; Use the value network θ Q and the policy network θ π To approximate the critic and actor functions in the DDPG algorithm framework, the value network θ Q and the policy network θ π are two independent networks, each with its own target network θ Q‘ and θ π’ ; Among them, in the iterative calculation at time t, the policy network θ Q The state S in the input state space t , output action a in the action space t ; The agent transfers state according to the control strategy at time t and obtains the state S at the next moment t+1 , and gives the reward r at time t t (S t ,a t ) is fed back to the agent, which records the experience e t =(S t ,a t ,r t (S t ,a t ),S t+1 ) and put it into the experience pool; value network θ π The state S in the input state space t and action a in the action space t , output action-value function Q π (S t ,a t ); During training, we repeat the iterations and randomly extract small batches of samples from the experience pool each time. Q and θ π The network updates the network parameters by the gradient descent method until it is stable, and the pre-trained DDPG algorithm is obtained.

5. The integrated energy system optimization and dispatching system according to claim 4, characterized in that: The steps of obtaining the pre-built integrated energy system digital twin model include: Build the physical entity of the integrated energy system and construct the physical layer; Installing multi-dimensional sensors at preset positions of the physical entities of the integrated energy system to construct a perception layer; Constructing a data transmission layer; wherein the data transmission layer is used to store or transmit the data acquired by the perception layer; Constructing a data processing layer; wherein the data processing layer is used to receive and pre-process the data transmitted by the data transmission layer, and construct a digital twin environment of the integrated energy system based on the pre-processed data; used to treat each type of unit in the integrated energy system as an intelligent agent to achieve continuous interaction between the intelligent agent and the digital twin environment; used to use historical data to correct the prediction results generated by the decision-making layer; Build a virtual entity of the integrated energy system and construct a decision-making layer; wherein the decision-making layer is used to find the optimal strategy for the economic operation output of the unit based on the interaction data between the intelligent agent and the digital twin environment, using the pre-trained DDPG algorithm, and generate a unit output scheduling plan; Based on the physical layer, perception layer, data transmission layer, data processing layer and decision-making layer, a digital twin model of the comprehensive energy system is obtained.

6. The integrated energy system optimization and dispatching system according to claim 5, characterized in that: The digital twin environment includes power balance constraints within the integrated energy system and operating range constraints for each unit; The internal power balance constraint includes the electrical power and thermal power balance constraints, and the expression is: P grid (t)+P RG (t)+P BES (t)+P CHP (t)-P EB (t)=P load (t); H CHP (t)+H GB (t)+H EB (t)=H load (t); Where P grid (t), P RG (t), P BES (t), P CHP (t), P EB (t), P load (t) are the interaction power with the main grid, the output power of renewable energy units, the charging power or discharging power of electric energy storage, the output power of combined heat and power units, the input power of electric boilers and the user's electric load in the time period t; H CHP (t), H GB (t), H EB (t), H load (t) are the output thermal power of the combined heat and power unit, the output thermal power of the gas boiler, the output thermal power of the electric boiler and the user's thermal load; The operating range constraints of each unit are expressed as: In the formula, are the lower and upper limits of the interaction power between the integrated energy system and the main grid side, respectively; are the lower limit and upper limit of the output electric power of the combined heat and power unit respectively; are the lower and upper limits of the charge / discharge power of the electric energy storage, respectively; They are the lower limit and upper limit of the output thermal power of the gas boiler respectively; are the lower and upper limits of the output thermal power of the electric boiler respectively; They are the lower and upper limits of the energy storage state of charge, respectively.

7. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the integrated energy system optimization scheduling method as described in any one of claims 1 to 3.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the integrated energy system optimization scheduling method described in any one of claims 1 to 3 is implemented.

Images