Integrated energy system anti-fragility optimization operation method and system based on safety reinforcement learning, electronic equipment and medium

Abstract

The invention provides an integrated energy system anti-fragility optimization operation method and system based on safety reinforcement learning, electronic equipment and a medium, and belongs to the technical field of integrated energy system operation analysis, and the method comprises the following steps: S1, building a mathematical model of an integrated energy system anti-fragility optimization problem, setting an objective function and constraint conditions of the mathematical model of the anti-fragile optimization problem of the integrated energy system; s2, converting an anti-fragile optimization problem of the integrated energy system into an optimization model based on a constrained Markov decision process; and S3, designing a security reinforcement learning solution algorithm, and updating and converging each network to obtain an optimal solution of the anti-fragility optimization problem of the integrated energy system. According to the integrated energy system anti-fragility optimization operation method and system based on safety reinforcement learning, the electronic equipment and the medium, the anti-fragility ability of the system to deal with uncertainty can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated energy system operation analysis, and in particular to an anti-fragile optimization operation method, system, electronic equipment and medium for an integrated energy system based on secure reinforcement learning. Background Art

[0002] Energy is vital to modern society. Integrated energy systems integrate multiple energy forms and control technologies, and are an important way to improve energy security, economic benefits and the environment. With the development of clean energy and renewable energy technologies, and the introduction of concepts such as energy Internet and smart grid, the concept and theory of integrated energy systems have gradually been deepened and expanded. In recent years, the high penetration of new energy and the large-scale access of power electronic equipment have increased the uncertainty of the optimization and scheduling of integrated energy systems, and the difficulty of scheduling and regulation has increased. Therefore, how to deal with the uncertainty of integrated energy systems and enhance their anti-fragility capabilities is a key issue that needs to be solved.

[0003] In recent years, traditional optimization scheduling methods for integrated energy systems include analytical methods and heuristic algorithms. The analytical method accurately solves problems based on strict mathematical theories. However, for large-scale, nonlinear, multi-variable, and multi-constrained optimization problems, the amount of calculation may be very large or even impossible to solve. In addition, the actual problem usually needs to be simplified and assumed during the solution process, resulting in a decrease in the quality of the solution and a large deviation from the actual situation. Heuristic algorithms do not require complex mathematical models and are relatively simple to implement, but they can usually only find approximate solutions or local optimal solutions, and cannot guarantee global optimality. In addition, the algorithm is too dependent on parameters and is not suitable for practical applications of large-scale problems. With the development of intelligent and digital energy scheduling, artificial intelligence methods, led by reinforcement learning, have been widely used in the field of energy scheduling by virtue of their "de-modeling" advantages. Although reinforcement learning can reduce the impact of model errors and uncertainty factors on scheduling strategies, in the face of increasingly complex dynamic integrated energy systems, it is necessary to enhance its anti-fragility capabilities to achieve safe and economical scheduling. Summary of the invention

[0004] The purpose of the present invention is to provide an anti-fragile optimization operation method, system, electronic device and medium for an integrated energy system based on secure reinforcement learning, which can enhance the anti-fragile ability of the system to cope with uncertainty.

[0005] To achieve the above object, the present invention provides an anti-fragile optimization operation method for an integrated energy system based on security reinforcement learning, comprising the following steps:

[0006] Step S1, establishing a mathematical model for the anti-fragile optimization problem of the integrated energy system, and setting the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system under five constraints: equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint;

[0007] Step S2, transforming the anti-fragile optimization problem of the integrated energy system into an optimization model based on a constrained Markov decision process;

[0008] Step S3, design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network and a target strategy network, and update each network to obtain the optimal solution to the anti-fragile optimization problem of the integrated energy system after convergence.

[0009] Preferably, in step S1, the objective function is as follows:

[0010]

[0011] F BES (t) = α BES |P BES (t)| (4);

[0012] Among them, F is the system operation cost; T is the number of time periods in the operation cycle; F NG (t), F grid (t), F BES (t) are the gas purchase cost, electricity purchase cost and electric energy storage depreciation cost at time t; are the unit prices of gas and electricity at time t; M GB (t), M CHP (t) are the total amount of natural gas consumed by gas boilers and cogeneration units at time t; P grid (t) is the total amount of electricity purchased from the superior power grid at time t; α BES is the depreciation cost coefficient of the electric energy storage device; P BES (t) is the charging and discharging power of the energy storage device at time t, with positive values ​​representing discharge and negative values ​​representing charge.

[0013] Preferably, in step S1, the equipment operation constraints are as follows:

[0014]

[0015] in, are the minimum and maximum output power of the cogeneration unit respectively; P CHP (t) is the electric power output of the cogeneration unit at time t; P CHP (t-1) is the electric power output of the cogeneration unit at time t-1; are the minimum and maximum values ​​of thermal power output of electric boiler respectively; H EB (t) is the thermal power output of the electric boiler at time t; H EB (t-1) is the thermal power output of the electric boiler at time t-1; are the minimum and maximum values ​​of thermal power output of gas boiler respectively; H GB (t) is the thermal power output of the gas boiler at time t; H GB (t-1) is the thermal power output of the gas boiler at time t-1; are the minimum and maximum output power of the electric energy storage device respectively; P BES (t) is the electric power output by the electric energy storage device at time t; They are the upper and lower limits of the ramp constraints for the cogeneration unit, respectively; They are the upper and lower limits of the climbing constraints of the electric boiler respectively; They are the upper and lower limits of the climbing constraints of the gas boiler respectively.

[0016] Preferably, in step S1, the external energy supply constraints are as follows:

[0017]

[0018] in, are the minimum and maximum values ​​of the exchange power between the integrated energy system and the upper power grid; P grid (t) is the exchange power between the integrated energy system and the upper-level power grid at time t.

[0019] Preferably, in step S1, the electric energy storage device is constrained as follows:

[0020]

[0021] Among them, C SOC (t) is the charge state of the electric energy storage device at time t; C SOC (t-1) is the state of charge of the electric energy storage device at time t-1; η BES is the charge and discharge coefficient; Q BES is the capacity of the electric energy storage device; Δt is the time interval between adjacent scheduling periods; η ch is the charging coefficient; η dis is the discharge coefficient; are the minimum and maximum charge values ​​respectively.

[0022] Preferably, in step S1, the power balance constraint is as follows:

[0023] P grid (t)+P CHP (t)+P PV (t)+P BES(t)-P EB (t) = P load (t)(16);

[0024] Among them, P PV (t) is the output power of the photovoltaic unit at time t; P load (t) is the electric load demand that the comprehensive energy system needs to meet at time t; P EB (t) is the electric power consumed by the electric boiler at time t; P grid (t) is the electric power provided by the upper power grid at time t.

[0025] Preferably, in step S1, the thermal energy balance constraint is as follows:

[0026] H CHP (t)+H GB (t)+H EB (t) = H load (t)(17);

[0027] Among them, H CHP (t) is the output thermal power of the cogeneration unit at time t; H load (t) is the heat load demand that the comprehensive energy system needs to meet at time t.

[0028] Preferably, in step S2, the optimization model based on the constrained Markov decision process is expressed as:

[0029]

[0030] Where Q(π) is the discounted cost of future rewards; E π is the expected function under strategy π; γ is the discount factor; r(s t ,a t ) is an immediate reward; t is the state at time t; a t is the action at time t; C(π) is the future discounted security cost; d is the constraint tolerance.

[0031] Preferably, in step S2, the constrained Markov decision process includes a state space, an action space, a reward function, a state transition design, a discount factor and a safety constraint function.

[0032] Preferably, the state space includes the electric load demand, heat load demand, output power of the photovoltaic unit and the charge state of the electric energy storage device at time t that the comprehensive energy system needs to meet, and the random disturbance simulation uncertainty is superimposed on the source and load historical data to enhance the anti-fragility capability of the system, which is specifically expressed as:

[0033] s t =[P load(t),H load (t),P PV (t),C SOC (t-1)] (19).

[0034] Preferably, the action space includes the electric power output by the cogeneration unit, the thermal power output by the gas boiler and the electric power output by the electric energy storage, which is specifically expressed as:

[0035] a t =[P CHP (t),H GB (t),P BES (t)] (20).

[0036] Preferably, the inverse of the operating cost of the integrated energy system at time t is taken as the reward function of reinforcement learning, which is specifically expressed as:

[0037] r(s t ,a t )=-ξ(F NG (t)+F grid (t)+F BES (t)) (21);

[0038] Where ξ is the scaling factor of the immediate reward;

[0039] Define the future reward discount cost Q(π), expressed as:

[0040]

[0041] Preferably, the safety limit function is as follows:

[0042]

[0043] Where ξ1 is the safety limit scaling factor; operator |a + =max(0,a);P x (t) is the active power output of unit x at time t; They are the minimum and maximum output of unit x at any moment respectively.

[0044] Preferably, the updating process of the economic value network, the security value network, the policy network, the economic target network, the security target network and the target policy network is as follows:

[0045] The economic value network update process is:

[0046]

[0047] Among them, L(φ Q ) is the economic value network loss function; y tis the target economic value at time t; Q(s t ,α t ∣φ Q ) is the action value function output by the economic value network; φ Q The economic value network parameters before this round of update; is the economic value network parameter after this round of update; α φ is the learning rate of the economic value network; ▽ is the gradient calculation function;

[0048] The security value network update process is:

[0049]

[0050] Among them, L(ψ C ) is the security value network loss function; is the target safety value at time t; C(s t ,a t ∣ψ C ) is the safety value function corresponding to different constraints; ψ C The security value network parameters before this round of update; is the security value network parameter after this round of update; α ψ is the learning rate of the security value network;

[0051] The policy network update process is:

[0052]

[0053] Among them, L(s t ,a t |λ) is the policy network loss function; λ is the Lagrange multiplier; θ π The policy network parameters before this round of update; is the policy network parameter after this round of update; l π is the learning rate of the policy network;

[0054] The updating process of the economic target network, security target network and target strategy network is as follows:

[0055]

[0056] in, is the economic target network parameter after this round of update; τ is the soft update coefficient; The economic target network parameters before this round of update; The security target network parameters after this round of update; The security target network parameters before this round of update; The target strategy network parameters after this round of update; The target strategy network parameters before this round of update;

[0057] After the updates of each network converge, the optimal solution to the anti-fragile optimization problem of the integrated energy system is obtained.

[0058] Preferably, the integrated energy system includes an energy transfer unit, an energy conversion unit, an energy storage unit and an energy output unit. The energy transfer unit includes a superior power grid, an external gas grid and a photovoltaic power generation device. The energy conversion equipment includes a cogeneration unit, an electric boiler and a gas boiler. The energy storage unit includes an electric energy storage device. The energy output unit includes an electric load and a thermal load.

[0059] The present invention also provides an anti-fragile optimization operation system for an integrated energy system based on security reinforcement learning, comprising:

[0060] The modeling module is used to establish a mathematical model for the anti-fragile optimization problem of the integrated energy system. Under the five constraints of equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint, the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system is set;

[0061] The conversion module converts the anti-fragile optimization problem of the integrated energy system into an optimization model based on the constrained Markov decision process;

[0062] The solution module is used to design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network, and a target strategy network. Each network is updated and the optimal solution to the anti-fragile optimization problem of the integrated energy system is obtained after convergence.

[0063] The present invention also provides a computer device, comprising: a memory and a processor; the memory stores a computer program, and when the processor executes the computer program, the steps of the above-mentioned anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning are implemented.

[0064] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning are implemented.

[0065] Therefore, the present invention adopts the above-mentioned anti-fragile optimization operation method, system, electronic device and medium of the comprehensive energy system based on security reinforcement learning, and the beneficial technical effects are as follows:

[0066] (1) The present invention designs a safety reinforcement learning method for solving the optimization scheduling problem of the integrated energy system, which can improve the economic efficiency of the system while ensuring safety;

[0067] (2) In the process of establishing a mathematical model, the present invention superimposes randomness on historical data, so that it can flexibly respond to the uncertainty fluctuations of the system and has good anti-fragile optimization capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a diagram of the energy supply architecture of the test system of Example 1;

[0069] Figure 2 This is a graph showing the convergence of the reward function. DETAILED DESCRIPTION

[0070] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0071] Unless otherwise defined, technical or scientific terms used in the present invention shall have the common meanings understood by one having ordinary skills in the field to which the present invention belongs.

[0072] Embodiment 1

[0073] A method for anti-fragile optimization operation of an integrated energy system based on security reinforcement learning, comprising the following steps:

[0074] Step S1, establish a mathematical model for the anti-fragile optimization problem of the integrated energy system, and set the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system under five constraints: equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint.

[0075] The objective function is as follows:

[0076]

[0077] F BES (t) = α BES |P BES (t)| (4);

[0078] Among them, F is the system operation cost; T is the number of time periods in the operation cycle; F NG (t), F grid (t), F BES (t) are the gas purchase cost, electricity purchase cost and electric energy storage depreciation cost at time t; are the unit prices of gas and electricity at time t; M GB (t), M CHP (t) are the total amount of natural gas consumed by gas boilers and cogeneration units at time t; P grid (t) is the total amount of electricity purchased from the superior power grid at time t; α BES is the depreciation cost coefficient of the electric energy storage device; P BES(t) is the charging and discharging power of the energy storage device at time t, with positive values ​​representing discharge and negative values ​​representing charge.

[0079] The equipment operation constraints are as follows:

[0080]

[0081]

[0082] in, are the minimum and maximum output power of the cogeneration unit respectively; P CHP (t) is the electric power output of the cogeneration unit at time t; P CHP (t-1) is the electric power output of the cogeneration unit at time t-1; are the minimum and maximum values ​​of thermal power output of electric boiler respectively; H EB (t) is the thermal power output of the electric boiler at time t; H EB (t-1) is the thermal power output of the electric boiler at time t-1; are the minimum and maximum values ​​of thermal power output of gas boiler respectively; H GB (t) is the thermal power output of the gas boiler at time t; H GB (t-1) is the thermal power output of the gas boiler at time t-1; are the minimum and maximum output power of the electric energy storage device respectively; P BES (t) is the electric power output by the electric energy storage device at time t; They are the upper and lower limits of the ramp constraints for the cogeneration unit, respectively; They are the upper and lower limits of the climbing constraints of the electric boiler respectively; They are the upper and lower limits of the climbing constraints of the gas boiler respectively.

[0083] The external energy supply constraints are as follows:

[0084]

[0085] in, are the minimum and maximum values ​​of the exchange power between the integrated energy system and the upper power grid; P grid (t) is the exchange power between the integrated energy system and the upper-level power grid at time t.

[0086] The constraints on electrical energy storage devices are as follows:

[0087]

[0088] Among them, C SOC (t) is the charge state of the electric energy storage device at time t; C SOC(t-1) is the state of charge of the electric energy storage device at time t-1; η BES is the charge and discharge coefficient; Q BES is the capacity of the electric energy storage device; Δt is the time interval between adjacent scheduling periods; η ch is the charging coefficient; η dis is the discharge coefficient; are the minimum and maximum charge values ​​respectively.

[0089] The power balance constraints are as follows:

[0090] P grid (t)+P CHP (t)+P PV (t)+P BES (t)-P EB (t) = P load (t)(16);

[0091] Among them, P PV (t) is the output power of the photovoltaic unit at time t; P load (t) is the electric load demand that the comprehensive energy system needs to meet at time t; P EB (t) is the electric power consumed by the electric boiler at time t; P grid (t) is the electric power provided by the upper power grid at time t.

[0092] In step S1, the thermal energy balance constraint is as follows:

[0093] H CHP (t)+H GB (t)+H EB (t) = H load (t)(17);

[0094] Among them, H CHP (t) is the output thermal power of the cogeneration unit at time t; H load (t) is the heat load demand that the comprehensive energy system needs to meet at time t.

[0095] Step S2: transform the anti-fragile optimization problem of the integrated energy system into an optimization model based on a constrained Markov decision process.

[0096] The optimization model based on constrained Markov decision process is expressed as:

[0097]

[0098] Where Q(π) is the discounted cost of future rewards; E π is the expected function under strategy π; γ is the discount factor; r(s t ,a t ) is an immediate reward;t is the state at time t; a t is the action at time t; C(π) is the future discounted security cost; d is the constraint tolerance.

[0099] The constrained Markov decision process includes state space, action space, reward function, state transition design, discount factor and safety limit function.

[0100] The state space contains the state information needed by the agent to decide the next action. The state space includes the electric load demand, thermal load demand, output power of the photovoltaic unit, and the charge state of the energy storage device at time t that the integrated energy system needs to meet. On the basis of the historical data of source and load, random disturbance simulation uncertainty is superimposed to enhance the anti-fragility ability of the system, which is specifically expressed as:

[0101] s t =[P load (t),H load (t),P PV (t),C SOC (t-1)](19).

[0102] The action space is the set of all possible actions that the agent can perform in the current state, which will directly affect the training process of the agent. The action space includes the electric power output of the cogeneration unit, the thermal power output of the gas boiler, and the electric power output of the electric energy storage, which is specifically expressed as:

[0103] a t =[P CHP (t),H GB (t),P BES (t)](20).

[0104] The goal of reinforcement learning is to maximize the reward function. Therefore, the inverse of the system operation cost at time t is taken as the reward function of reinforcement learning, which is specifically expressed as:

[0105] r(s t ,a t )=-ξ(F NG (t)+F grid (t)+F BES (t))(21);

[0106] Where ξ is the scaling factor of the immediate reward;

[0107] On this basis, the future reward discount cost Q(π) is defined as:

[0108]

[0109] State transfer design:

[0110] The state transition function describes the probability that the agent will transition to the next state after taking an action, which is usually related to the uncertainty of the environment.

[0111] Discount Factor:

[0112] The discount factor is used to calculate the present value of future rewards, allowing the agent to balance short-term and long-term benefits.

[0113] The security restriction function is as follows:

[0114]

[0115] Where ξ1 is the safety limit scaling factor; the operator |a| + =max(0,a);P x (t) is the active power output of unit x at time t; They are the minimum and maximum output of unit x at any moment respectively.

[0116] Step S3, design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network and a target strategy network, and update each network to obtain the optimal solution to the anti-fragile optimization problem of the integrated energy system after convergence.

[0117] The economic value network update process is:

[0118]

[0119] Among them, L(φ Q ) is the economic value network loss function; y t is the target economic value at time t; Q(s t ,α t ∣φ Q ) is the action value function output by the economic value network; φ Q The economic value network parameters before this round of update; is the economic value network parameter after this round of update; α φ is the learning rate of the economic value network; ▽ is the gradient calculation function.

[0120] The security value network update process is:

[0121]

[0122] Among them, L(ψ C ) is the security value network loss function; is the target safety value at time t; C(s t ,a t ∣ψ C) is the safety value function corresponding to different constraints; ψ C The security value network parameters before this round of update; is the security value network parameter after this round of update; α ψ is the learning rate of the security value network.

[0123] The policy network update process is:

[0124]

[0125] Among them, L(s t ,a t |λ) is the policy network loss function; λ is the Lagrange multiplier; θ π The policy network parameters before this round of update; is the policy network parameter after this round of update; l π is the learning rate of the policy network.

[0126] The updating process of the economic target network, security target network and target strategy network is as follows:

[0127]

[0128] in, is the economic target network parameter after this round of update; τ is the soft update coefficient; The economic target network parameters before this round of update; The security target network parameters after this round of update; The security target network parameters before this round of update; The target strategy network parameters after this round of update; It is the target strategy network parameters before this round of update.

[0129] After the updates of each network converge, the optimal solution to the anti-fragile optimization problem of the integrated energy system is obtained.

[0130] The anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning is applied to Figure 1 The method is further illustrated in the test system shown. Figure 1The test system shown is an integrated energy system with electric and thermal coupling. The test system includes an energy transfer unit, an energy conversion unit, an energy storage unit and an energy output unit. The energy transfer unit includes an upper power grid, an external gas grid and a photovoltaic power generation device, wherein the upper power grid provides electric energy, the external gas grid provides gas, and the photovoltaic power generation device provides electric energy converted from solar energy; the energy conversion equipment includes a cogeneration unit, an electric boiler and a gas boiler, wherein gas is converted into electric energy and thermal energy (cogeneration unit), electric energy is converted into thermal energy (electric boiler) and gas is converted into thermal energy (gas boiler); the energy storage unit includes an electric energy storage device, and the energy output unit includes electric load and thermal load. Its source load data is the actual historical data of a domestic industrial park.

[0131] Based on historical data and the internal structure of the system, a constrained Markov decision process is constructed.

[0132] Based on the constrained Markov decision process, the model of the present invention is solved by the secure reinforcement learning algorithm to obtain the convergence of the algorithm reward function, such as Figure 2 shown.

[0133] Embodiment 2

[0134] An anti-fragile optimization operation system for an integrated energy system based on secure reinforcement learning, comprising:

[0135] The modeling module is used to establish a mathematical model for the anti-fragile optimization problem of the integrated energy system. Under the five constraints of equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint, the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system is set;

[0136] The conversion module converts the anti-fragile optimization problem of the integrated energy system into an optimization model based on the constrained Markov decision process;

[0137] The solution module is used to design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network, and a target strategy network. Each network is updated and the optimal solution to the anti-fragile optimization problem of the integrated energy system is obtained after convergence.

[0138] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0139] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0140] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0141] It is worth noting that the contents not elaborated in detail in the present invention are all prior art and are well known to those skilled in the art.

[0142] Therefore, the present invention adopts the above-mentioned anti-fragile optimization operation method, system, electronic device and medium of the comprehensive energy system based on security reinforcement learning, which can enhance the system's anti-fragile ability to deal with uncertainty.

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

Claims

1. A method for anti-fragile optimization operation of an integrated energy system based on security reinforcement learning, characterized in that: The following steps are involved: Step S1, establishing a mathematical model for the anti-fragile optimization problem of the integrated energy system, and setting the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system under five constraints: equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint; Step S2, transforming the anti-fragile optimization problem of the integrated energy system into an optimization model based on a constrained Markov decision process; Step S3, design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network and a target strategy network, and update each network to obtain the optimal solution to the anti-fragile optimization problem of the integrated energy system after convergence.

2. According to claim 1, the anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning is characterized in that: In step S1, the objective function is as follows: F BES (t)=α BES |P BES (t)| (4); Among them, F is the system operation cost; T is the number of time periods in the operation cycle; F NG (t), F grid (t), F BES (t) are the gas purchase cost, electricity purchase cost and electric energy storage depreciation cost at time t; are the unit prices of gas and electricity at time t; M GB (t), M CHP (t) are the total amount of natural gas consumed by gas boilers and cogeneration units at time t; P grid (t) is the total amount of electricity purchased from the superior power grid at time t; α BES is the depreciation cost coefficient of the electric energy storage device; P BES (t) is the charging and discharging power of the energy storage device at time t, with positive values ​​representing discharge and negative values ​​representing charge.

3. According to claim 2, the anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning is characterized in that: In step S1, the equipment operation constraints are as follows: in, are the minimum and maximum output power of the cogeneration unit respectively; P CHP (t) is the electric power output of the cogeneration unit at time t; P CHP (t-1) is the electric power output of the cogeneration unit at time t-1; are the minimum and maximum values ​​of thermal power output of electric boiler respectively; H EB (t) is the thermal power output of the electric boiler at time t; H EB (t-1) is the thermal power output of the electric boiler at time t-1; are the minimum and maximum values ​​of thermal power output of gas boiler respectively; H GB (t) is the thermal power output of the gas boiler at time t; H GB (t-1) is the thermal power output of the gas boiler at time t-1; are the minimum and maximum output power of the electric energy storage device respectively; P BES (t) is the electric power output by the electric energy storage device at time t; They are the upper and lower limits of the ramp constraints for the cogeneration unit, respectively; They are the upper and lower limits of the climbing constraints of the electric boiler respectively; They are the upper and lower limits of the climbing constraints of the gas boiler respectively.

4. According to claim 3, the anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning is characterized in that: In step S1, the external energy supply constraints are as follows: in, are the minimum and maximum values ​​of the exchange power between the integrated energy system and the upper power grid; P grid (t) is the exchange power between the integrated energy system and the upper-level power grid at time t.

5. According to claim 4, the anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning is characterized in that: In step S1, the electric energy storage device constraints are as follows: Among them, C SOC (t) is the charge state of the electric energy storage device at time t; C SOC (t-1) is the state of charge of the electric energy storage device at time t-1; η BES is the charge and discharge coefficient; Q BES is the capacity of the electric energy storage device; Δt is the time interval between adjacent scheduling periods; η ch is the charging coefficient; η dis is the discharge coefficient; are the minimum and maximum charge values ​​respectively.

6. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 5 is characterized in that: In step S1, the power balance constraints are as follows: P grid (t)+P CHP (t)+P PV (t)+P BES (t)-P EB (t)=P load (t)(16); Among them, P PV (t) is the output power of the photovoltaic unit at time t; P load (t) is the electric load demand that the comprehensive energy system needs to meet at time t; P EB (t) is the electric power consumed by the electric boiler at time t; P grid (t) is the electric power provided by the upper power grid at time t.

7. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 6 is characterized in that: In step S1, the thermal energy balance constraint is as follows: H CHP (t)+H GB (t)+H EB (t)=H load (t)(17); Among them, H CHP (t) is the output thermal power of the cogeneration unit at time t; H load (t) is the heat load demand that the comprehensive energy system needs to meet at time t.

8. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 7 is characterized in that: In step S2, the optimization model based on the constrained Markov decision process is expressed as: Where Q(π) is the discounted cost of future rewards; E π is the expected function under strategy π; γ is the discount factor; r(s t ,a t ) is an immediate reward; t is the state at time t; a t is the action at time t; C(π) is the future discounted security cost; d is the constraint tolerance.

9. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 8 is characterized in that: In step S2, the constrained Markov decision process includes a state space, an action space, a reward function, a state transition design, a discount factor, and a safety constraint function.

10. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 9 is characterized in that: The state space includes the electric load demand, thermal load demand, output power of the photovoltaic unit and the charge state of the electric energy storage device at time t that the integrated energy system needs to meet. On the basis of the historical data of source and load, random disturbance simulation uncertainty is superimposed to enhance the anti-fragility capability of the system. It is specifically expressed as: s t =[P load (t),H load (t),P PV (t),C SOC (t-1)](19)。 11. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 10 is characterized in that: The action space includes the electric power output by the cogeneration unit, the thermal power output by the gas boiler and the electric power output by the electric energy storage, which can be specifically expressed as: a t =[P CHP (t),H GB (t),P BES (t)](20)。 12. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 11, characterized in that: The inverse of the operating cost of the integrated energy system at time t is taken as the reward function of reinforcement learning, which is specifically expressed as: r(s t ,a t )=-ξ(F NG (t)+F grid (t)+F BES (t))(21); Among them, ξ is the scaling factor of the immediate reward; Define the future reward discount cost Q(π), expressed as:

13. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 12 is characterized in that: The safety restriction function is as follows: Where ξ1 is the safety limit scaling factor; the operator |a| + =max(0,a);P x (t) is the active power output of unit x at time t; They are the minimum and maximum output of unit x at any moment respectively.

14. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 13, characterized in that: The update process of the economic value network, security value network, policy network, economic target network, security target network and target policy network is as follows: The economic value network update process is: L(φ Q )=[y t -Q(s t ,a t ∣φ Q )] 2 (24); Among them, L(φ Q ) is the economic value network loss function; y t is the target economic value at time t; Q(s t ,α t ∣φ Q ) is the action value function output by the economic value network; φ Q The economic value network parameters before this round of update; is the economic value network parameter after this round of update; α φ is the economic value network learning rate; is the gradient calculation function; The security value network update process is: Among them, L(ψ C ) is the security value network loss function; is the target safety value at time t; C(s t ,a t ∣ψ C ) is the safety value function corresponding to different constraints; ψ C The security value network parameters before this round of update; is the security value network parameter after this round of update; α ψ is the learning rate of the security value network; The policy network update process is: L(s t ,a t ∣λ)=Q(s t ,a t ∣φ Q )-λ[C(s t ,a t ∣ψ C )-d] (28); Among them, L(s t ,a t |λ) is the policy network loss function; λ is the Lagrange multiplier; θ π The policy network parameters before this round of update; is the policy network parameter after this round of update; l π is the learning rate of the policy network; The updating process of the economic target network, security target network and target strategy network is as follows: in, is the economic target network parameter after this round of update; τ is the soft update coefficient; The economic target network parameters before this round of update; The security target network parameters after this round of update; The security target network parameters before this round of update; The target strategy network parameters after this round of update; It is the target strategy network parameter before this round of update.

15. The anti-fragile optimization operation method of a comprehensive energy system based on security reinforcement learning according to claim 14, characterized in that: The comprehensive energy system includes energy transfer units, energy conversion units, energy storage units and energy output units. The energy transfer unit includes the upper-level power grid, external gas grid and photovoltaic power generation equipment. The energy conversion equipment includes cogeneration units, electric boilers and gas boilers. The energy storage unit includes electric energy storage equipment. The energy output unit includes electric load and thermal load. 16.An anti-fragile optimization operation system for an integrated energy system based on secure reinforcement learning, characterized in that: include: The modeling module is used to establish a mathematical model for the anti-fragile optimization problem of the integrated energy system. Under the five constraints of equipment operation constraint, external energy supply constraint, electric energy storage device constraint, electric balance constraint, and thermal energy balance constraint, the objective function of the mathematical model for the anti-fragile optimization problem of the integrated energy system is set; The conversion module converts the anti-fragile optimization problem of the integrated energy system into an optimization model based on the constrained Markov decision process; The solution module is used to design a security reinforcement learning solution algorithm, including an economic value network, an economic target network, a security value network, a security target network, a strategy network, and a target strategy network. Each network is updated and the optimal solution to the anti-fragile optimization problem of the integrated energy system is obtained after convergence.

17. A computer device comprising: Memory and processor; The memory stores a computer program, characterized in that when the processor executes the computer program, the steps of the anti-fragile optimization operation method of the integrated energy system based on security reinforcement learning described in any one of claims 1-15 are implemented.

18. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the anti-fragile optimization operation method of an integrated energy system based on security reinforcement learning described in any one of claims 1 to 15 are implemented.

Images