A research method for load transfer of an integrated electricity-heat-gas energy system based on reinforcement learning

Abstract

The application provides a load transfer method for an electric-thermal-gas integrated energy system based on reinforcement learning, and also belongs to the field of artificial intelligence. The method obtains relevant data from the actual operation of the integrated energy system, and collects the data as training sample data; an agent-environment model is constructed to simulate and constrain the operation of the system; a transfer trigger condition mechanism for different uncertain events of the integrated energy system is established to clearly define various uncertain events of the integrated energy system; a DDQN model is constructed to generate an optimal load transfer strategy of the electric-thermal-gas integrated energy system in different uncertain events through the DDQN algorithm after multiple iterations; and the trained model is used to realize the optimal load transfer strategy of the electric-thermal-gas integrated energy system containing different load types. The application overcomes the limitations of the traditional load transfer method for the electric-thermal-gas integrated energy system containing multiple different types of loads, such as long calculation time and low strategy fault tolerance, accurately describes the complex dynamic behavior of the electric-thermal-gas integrated energy system, and realizes the real-time and accurate load transfer strategy of the electric-thermal-gas integrated energy system containing different types of loads.

Description

TECHNICAL FIELD

[0001] The application of the present application belongs to the field of comprehensive energy, and the technical means of the present application belongs to the field of machine learning and the field of artificial intelligence, in particular, the design of reinforcement learning of comprehensive energy system load transfer path analysis. BACKGROUND

[0002] Under the driving of the "double carbon target" and the construction of a new power system, in order to solve the problem of supply and demand contradiction of traditional power system single energy peak load and other problems, it has become an inevitable trend to build a new comprehensive energy system of multi-energy coupling of power system, heat system and natural gas system. With the increasing of the scale of comprehensive energy system and the grid connection of photovoltaic and wind power, the comprehensive energy system is facing a series of uncertain events such as photovoltaic and wind power consumption, sudden increase and decrease of different regional loads, line fault, etc. There are great challenges to ensure the stability of different regional loads of different comprehensive energy systems. Accurate and real-time formulation of comprehensive energy system load transfer scheme can provide reliable decision-making information for the stable operation of different regional loads of comprehensive energy system, so as to ensure the safety, stability, economy and operation of comprehensive energy system.

[0003] How to realize the load transfer of different regional loads of comprehensive energy system has become a focal problem. The traditional power system, heat system and natural gas system realize the load transfer method by heuristic algorithm (genetic algorithm, particle swarm optimization algorithm, etc.) network interconnection switch, sectionalizing switch or interconnection valve switch device to realize the load transfer, and realize the safety of system. This transfer method has the following limitations: 1. The energy mode of traditional power system, heat system and natural gas system is single, which cannot meet the needs of new multi-type load transfer; 2. There are different types of loads and different flexibility in different regions of comprehensive energy system, and the current research method lacks research on different types of load transfer scheme of comprehensive energy system; 3. Since the realization of comprehensive energy system load transfer involves the action of each switch valve, and the system topology structure is complex, the traditional heuristic algorithm such as genetic algorithm and particle swarm optimization algorithm is difficult to provide real-time and accurate load transfer route strategy; 4. The current load transfer solution method usually only considers the safety of load transfer after system failure, lacks economic load transfer scheme under non-fault condition of system, and is difficult to realize the consideration of safety and economy.

[0004] In view of the difficulty of realizing real-time and accurate strategy of load transfer strategy of integrated energy system, the deep reinforcement learning technology is increasingly mature, which provides necessary conditions for developing real-time and accurate load transfer strategy of integrated energy system in the future, but the technology has not been widely used in integrated energy system, therefore, it is necessary to fully utilize the topology structure of integrated energy system and a large number of load transfer strategies, deeply mine the relationship between different uncertain events and load transfer strategies, and effectively reduce the time of formulating load transfer strategy. At the same time, due to the high sensitivity of integrated energy system to topology operation, incorrect operation may cause serious consequences, so it is also necessary to establish load transfer strategies corresponding to different uncertain events, and improve the accuracy of load transfer strategy. SUMMARY

[0005] The purpose of the present application is to provide a load transfer method of an electricity-heat-gas integrated energy system based on reinforcement learning. Since the load types and flexibility of different regions of the integrated energy system are different, and it is difficult to formulate accurate and real-time load transfer strategies when uncertain events occur in the integrated energy system, the method mainly learns a large number of load transfer strategies of the integrated energy system through a neural network.

[0006] The purpose of the present application is to provide a load transfer method of an electricity-heat-gas integrated energy system based on reinforcement learning, to make up for the shortcomings of the prior art, and the goal of the present application is to minimize the system operation cost on the basis of ensuring the stable operation of the system load when uncertain events occur in the integrated energy system. A model and data driven method is adopted, a model of the integrated energy system containing different load types is established, the limitations of traditional integrated energy system modeling are overcome, and the complex operation state of the integrated energy system can be accurately described; the data of the integrated energy system is learned and trained through reinforcement learning, so that fast and accurate load transfer strategies can be made in highly complex scenarios. The specific steps of the method are as follows:

[0007] S1, an electricity-heat-gas integrated energy system model containing different load types is established from an actual integrated energy system, and data related to system load transfer after different uncertain events are collected, and the collected data is used as training samples;

[0008] S2, an agent-environment model is constructed, the operation environment of the integrated energy system is simulated, in the environment, the actions of the agent are constrained based on the operation constraints of the integrated energy system, and the physical law and actual conditions of the operation load of the system are ensured;

[0009] S3, a reinforcement learning model is constructed, the state function, action function and state transition variable in the algorithm agent are defined, a reporting function is formulated by designing a reward and punishment mechanism, so as to optimize the agent strategy;

[0010] S4, establish a transfer trigger condition mechanism of the comprehensive energy system under different uncertain events, and clearly define various uncertain events of the comprehensive energy system, so that the intelligent agent can make more efficient and accurate load transfer strategy for different uncertain events;

[0011] S5, train the intelligent agent, update the intelligent agent strategy through a large amount of load transfer data in the comprehensive energy system, and train it by using the Dueling Deep-Q-Learning (DDQN) algorithm, and generate the optimal load transfer strategy of the comprehensive energy system under different uncertain events after multiple iterations of the intelligent agent;

[0012] S6, use the trained reinforcement learning model to develop real-time and accurate load transfer strategy of the comprehensive energy system.

[0013] Further, in S1,

[0014] The comprehensive energy system modeling part includes: power system line model, heat network, natural gas network pipe network model, compressor model; electric-gas, electric-heat, heat and power cogeneration coupling device model between each system,

[0015] Pipe model:

[0016] Natural gas pipeline model:

[0017]

[0018] In the formula, f rs,t is the steady-state flow of natural gas pipeline r, s; K rs is the pipeline constant, which depends on the pipe diameter, length, friction coefficient and temperature and other factors; p r,t , p s,t are the pressures of nodes r and s, respectively; s rs represents the direction of natural gas flow, and takes +1 when p r,t >p s,t , otherwise -1.

[0019] Heat pipe model:

[0020]

[0021] In the formula, Q loss (t) is the total heat loss of the pipe at time t, Q loss,q (t) is the heat loss of pipe q section, and N is the total number of pipe sections in the heat network.

[0022]

[0023] In the formula, ΔP is the pressure loss of the pipe section; f is the friction factor, which is usually related to the flow velocity, pipe roughness and fluid properties; L is the length of the pipe section; D is the diameter of the pipe; ρ is the density of the fluid; and v is the flow velocity of the fluid.

[0024] Compressor model:

[0025]

[0026] R com,m,n =ρ n,t / ρ m,t (5)

[0027]

[0028] In the formula: W com,mn,t The power consumed by the compressor; f com,n,t B is the air flow rate through the compressor. com,mn,t z cm m,t are compressor parameters, which depend on compressor temperature, efficiency, and compression factor, etc.; R com,m,n τ is the compression ratio; com,mn The gas flow rate consumed by the gas turbine; α com β com γ com This is the efficiency constant for energy conversion.

[0029] Electro-gas conversion model:

[0030] f P2G,itt =η P2G P P2G,i,t / G HV (7)

[0031] In the formula η P2G For the electro-gas conversion efficiency; G HV The high calorific value of natural gas is taken as 39 MJ / m³. 3 .

[0032] Electric boiler model:

[0033] H EB (t)=μ EB ·η EB ·P EB (t) (8)

[0034] μ in the formula EB ∈{0,1} represents the binary variable representing the start-up and shutdown state of the electric boiler at time t, as supplemented in the comments; η EB This refers to the electrothermal conversion efficiency.

[0035] Gas-fired boilers:

[0036] HGB (t) = η GB · G GB (t) · LHV (9)

[0037] η in the formula GB is the gas heat conversion efficiency coefficient of the gas boiler.

[0038] Modeling of different types of loads:

[0039] P σ (t) = P σ,0 · f(t) (10)

[0040] P in the formula σ,0 is the reference power of the device; f(t) is a function describing the load change, which is usually related to the running period of the device, load change and process requirements. For example, f(t) can be a sine function based on time to simulate periodic fluctuations, or a function driven by process requirements (such as production volume, production time, etc.).

[0041] The relevant data collected include: the topology of the integrated energy system, the initial node voltage, pressure and node temperature of the network after the occurrence of uncertain events such as load surge reduction, line fault, line maintenance, etc., photovoltaic and wind power generation, user electricity, heat and gas load, real-time electricity price, hourly information, and energy storage device information.

[0042] Further, as described in S2:

[0043] S2-1: First, it is clear that the goal is to achieve economic operation of the integrated energy system on the basis of ensuring stable operation of the system load when different uncertain events such as load surge reduction, line fault, line maintenance occur in the integrated energy system of electricity, heat and gas, and the load transfer strategy developed focuses on the amount of load recovery and the minimum operating cost of the integrated energy system; the optimization goal can be described by the following formula:

[0044]

[0045] S2-2: Build the operating constraints of the integrated energy system of electricity, heat and gas. The integrated energy system is coupled with the power system, the heat system and the natural gas system, and the pipe network energy flow balance constraints of each system and the constraints of each energy coupling device are constructed.

[0046] Power system power flow constraints:

[0047]

[0048] P G,i,t , Q G,i,t are the active and reactive power of the generator set at node i at time t; P L,j,t , QL,i,t Pi(t) is the active power of node i at time t; P STO,i,t P is the active power of random energy source such as wind power and photovoltaic power; P P2G,i,t P is the active power of the electric-to-gas unit; P SOC,i,t U is the charging (or discharging) power of the electric energy storage; U i,t V is the voltage modulus of node i; θ ij,t G is the phase angle difference between two nodes; G ij , B ij are the conductance and susceptance between nodes i and j.

[0049] Natural gas system:

[0050]

[0051] In the formula, f N,r,t , f L,r,t , f P2G,r,t , f GT,r,t are the natural gas source injection amount, natural gas load, P2G device injection amount and gas turbine gas consumption of node r at time t; f s,in,r,t , f s,out,r,t are the storage tank natural gas injection and output flow; when node r is the compressor inlet, s com,s,r,t takes +1, otherwise -1.

[0052] Thermal system:

[0053] H buy (t) + H GB (t) + H EB (t) + H dis (t) = H cha (t) + H load (t) (14)

[0054] In the formula, H buy (t) is the heat purchased by the park from the outside; H GB (t), H EB (t) are the output heat powers of the gas boiler and the electric boiler respectively; H cha (t), H dis (t) are the heat storage power and heat release power of the heat storage device respectively; H load (t) is the heat load demand, all in kW.

[0055] Electric-to-gas equipment:

[0056]

[0057] In the formula, f N,i,t represents the actual gas production of the nth equipment at time t respectively represent the minimum and maximum gas flow allowed by the P2G device when operating stably; the minimum and maximum gas flow allowed by the P2G device when operating stably;

[0058] energy storage device:

[0059]

[0060] in the formula respectively represent the maximum charging and discharging rate of the δth energy storage device; respectively represent the minimum and maximum load state of the energy storage of the δth energy storage device; C δ is the rated storage capacity of the energy storage device; E δ (0), E δ (T) respectively represent the energy storage amount at the beginning and the end of the dispatching period.

[0061] Further, the S3 specifically comprises:

[0062] The neural network is built. An agent-environment training architecture is adopted, an input-output model of the system is established by a DDQN algorithm, an agent policy network and a value network are established, an environment variable is input as a parameter, and a comprehensive energy system network connection switch connection valve control strategy is output as a parameter.

[0063] Further, the neural network model policy network is configured, a three-layer neural network is adopted, an input layer, a hidden layer, and an output layer, the input layer includes 128 nodes, the hidden layer includes 256 nodes, and the output layer includes 128 nodes. The input layer input variable is the node voltage, the node gas pressure, the set node temperature, the electric heat gas load demand, the photovoltaic and wind power generation capacity, and the electricity price information of the comprehensive energy system after the fault, a ReLU function is selected as an activation layer, and the output layer is the next time action of each connection switch, sectional switch, and connection valve switch in the comprehensive energy system. The loss function of the policy network is optimized based on a stochastic gradient descent method, and an Adam optimizer is used to find the optimal solution of gradient descent.

[0064] Further, the DDQN model specifically models the control strategy problem as a Markov decision process:

[0065] (1) The observation state S is set, which describes the state information features of the agent at time t. The features are the node voltage, gas pressure, temperature, electric heat gas load demand, photovoltaic and wind power generation capacity, and electricity price information of the comprehensive energy system at the current time.

[0066] (2) The action space A is set. The action space of the agent is the switch action of the connection switch, sectional switch, and connection valve in the natural gas and heat system of the power system of the comprehensive energy system. Each switch action is (0, 1), 0 represents that the switch is open, and 1 represents that the switch is closed.

[0067] (3) The setting of the reward function R, in order to simultaneously depict the safety and economy of the integrated energy system, an integrated reward mechanism is constructed, the reward function is composed of two types of core components, 1) load recovery benefit item, 2) operation cost item, the reward function is defined as follows:

[0068]

[0069] In the formula, N op is the total number of actions; P res,k is the recovered load amount of the kth action; C C / E (t), C M / E (t), C S / E (t), C B / E (t) are the initial construction cost conversion value, operation and maintenance cost, unit start-stop cost and external main network purchase power cost of the power system respectively; C C / H (t), C M / H (t), C S / H (t), C B / H (t) are the initial construction cost depreciation, operation and maintenance cost, unit start-stop cost and external purchase heat cost of the heat system respectively, wherein the initial construction cost depreciation, operation and maintenance cost and unit start-stop cost calculation formula can be analogous to the power system. C / G (t), C M / G (t), C S / G (t), C B / G (t) are the initial construction cost depreciation, operation and maintenance cost, unit start-stop cost and external purchase gas cost of the gas supply system respectively, wherein the initial construction cost, operation and maintenance cost and unit start-stop cost calculation formula can be analogous to the power system.

[0070] Further, the transfer supply trigger condition mechanism mentioned in the S4: in each time step, the agent simulates the integrated energy system result without operation, and uses the trigger condition W defined by formula (16) to judge whether the agent should act.

[0071] W=W1+W2+W3 (18)

[0072] In the formula, W is a load surge and sudden drop flag, W2 is a fault flag, and W3 is a maintenance flag, which correspond to different event drives of load transfer supply respectively.

[0073]

[0074] In the formula, LF is the power flow of the line L, TL LThe limit value of the line L, L is the line number in the system, and Z is the total number of lines in the system. If the load level of the line in the power grid is higher than the predetermined threshold λ, the power flow out-of-limit flag is set to 1.

[0075] The calculation formula of W2 and W3 is as follows:

[0076]

[0077] By adding the transfer trigger condition mechanism, the agent will generate the corresponding load transfer strategy in any case of different types of uncertain events, and the corresponding operation will be performed on the integrated energy system, thereby improving the accuracy of the load transfer strategy.

[0078] Further, the DDQN training model described in S3 and S4: the DDQN training model agent updates the policy network and value network weights according to the environment, and the agent and the environment interact continuously, continuously train the strategy, and thus formulate the optimal strategy. In the training process, the cumulative reward curve is used to reflect the training process, and the agent training situation and convergence are intuitively reflected.

[0079] Select the training number as Ψ, the sample time series length in data collection as T, the state space as S, and the action space as A:

[0080] Step 1: Initialize the weights of the policy network and the value network respectively;

[0081] Step 2: The agent obtains the sampling (time, load, etc.) generated by each step operation;

[0082] Step 3: The agent executes the current operation at (device output) and obtains the reward of the current operation;

[0083] Step 4: The agent executes the transfer trigger condition

[0084] Step 5: The agent executes the switch action;

[0085] Step 6: Randomly extract experience replay area data;

[0086] Step 7: Input parameters in the policy network to generate;

[0087] Step 8: Update the network by using the stochastic gradient descent method;

[0088] Step 9: Input parameters in the value network to generate the target;

[0089] Step 10: Update the value network.

[0090] Compared with the prior art, the present application has the following advantages and benefits:

[0091] The application provides a load transfer strategy method for an electric-thermal-gas integrated energy system based on reinforcement learning.

[0092] The application provides a load transfer strategy method for an electric-thermal-gas integrated energy system based on reinforcement learning, overcomes the limitations of a traditional load transfer method for an electric-thermal-gas integrated energy system containing multiple different types of loads, such as long calculation time and low fault tolerance rate of the strategy, accurately describes the complex dynamic behavior of the electric-thermal-gas integrated energy system, and realizes real-time and accurate load transfer strategy for the electric-thermal-gas integrated energy system containing different types of loads. BRIEF DESCRIPTION OF DRAWINGS

[0093] Figure 1 It is a basic principle diagram of reinforcement learning.

[0094] Figure 2 It is a framework structure diagram of the application.

[0095] Figure 3 It is a structure diagram of an electric-thermal-gas integrated energy system.

[0096] Figure 4 It is a topology structure of an integrated energy system adopted by the application.

[0097] Figure 5 It is a flowchart of a DDQN algorithm.

[0098] Figure 6 It is a DDQN model training curve. DETAILED DESCRIPTION

[0099] The technical solutions described in the application will be further described and illustrated below in combination with embodiments. The method expresses the load transfer strategy process of the integrated energy system as a Markov decision process, which can effectively handle the complex switching problem of the load transfer process of the electric-thermal-gas integrated energy system containing different types of loads. Reinforcement learning interacts with the environment through an intelligent agent, and constantly tries and corrects, so as to learn the optimal load transfer strategy from the complex strategy. Meanwhile, the trained reinforcement learning model has good generalization ability, can adapt to different environments through a large amount of data-driven learning, and does not need a detailed physical model, which greatly simplifies the modeling complexity. The load transfer path method for the electric-thermal-gas integrated energy system based on reinforcement learning is shown in the figure. The method comprises the following steps:

[0100] (1) The data in this example is derived from a California, USA, integrated energy system, including network pipe network model, electric-gas conversion equipment, electric boiler, compressor, gas turbine; Collecting data includes integrated energy system topology, network initial node voltage, pressure and node temperature after uncertain events such as load surge increase and decrease, line fault, line maintenance, etc. Photovoltaic, wind power generation, user electricity, heat, gas load, real-time electricity price, hourly information, energy storage device information.

[0101] (2) Construct an agent-environment model to simulate the operating environment of the integrated energy system. In this environment, the agent's actions are constrained based on the operating constraints of the integrated energy system to ensure the physical laws and actual conditions of the system's operating load:

[0102] Power system flow constraint conditions:

[0103]

[0104] Natural gas system:

[0105]

[0106] Heat system:

[0107] H buy (t)+H GB (t)+H EB (t)+H dis (t)=H cha (t)+H load (t)

[0108] Electric-gas conversion equipment:

[0109] f N,i,min ≤f N,i,t ≤f N,i,max (23)

[0110] Energy storage equipment:

[0111]

[0112] (3) Construct a reinforcement learning model to clearly define the state function, action function, and state transition variable in the algorithm agent. Design a reward and punishment mechanism to formulate a report function, thereby optimizing the agent's strategy;

[0113] The neural network is built. An agent-environment training architecture is adopted, an input-output model of the system is established through a DDQN algorithm, an agent strategy network and a value network are established, input parameters are environment variables, and output parameters are comprehensive energy system network tie-in switch tie-in valve control strategies. A three-layer neural network is adopted, including an input layer, a hidden layer and an output layer, the input layer includes 128 nodes, the hidden layer includes 256 nodes, and the output layer includes 128 nodes. The input layer input variables are node voltage, node gas pressure, set node temperature, electric heat gas load demand, photovoltaic and wind power generation capacity and electricity price information of the comprehensive energy system at the current time after a fault occurs, a ReLU function is selected as an activation layer, and the output layer is the action of each tie-in switch, sectionalizing switch and tie-in valve of the comprehensive energy system at the next time. The loss function of the strategy network is optimized based on a stochastic gradient descent method, and an Adam optimizer is adopted to find the optimal solution of gradient descent.

[0114] The DDQN model specifically models the control strategy problem as a Markov decision process:

[0115] (1) The observation state S is set, which describes the state information features of the agent at t time, and the features are node voltage, gas pressure, temperature, electric heat gas load demand, photovoltaic and wind power generation capacity and electricity price information of the comprehensive energy system at the current time.

[0116] (2) The action space A is set, the action space of the agent is the switch action of the tie-in switch, sectionalizing switch of the power system and the tie-in valve of the natural gas and heat system in the comprehensive energy system, and each switch action is (0, 1), 0 represents that the switch is opened, and 1 represents that the switch is closed.

[0117] (3) The reward function R is set, the present application adopts a reward function design containing safety and economy of the comprehensive energy system, that is, the safety and economy of the agent action for the comprehensive energy system are considered in the process of training the agent, the reward function is composed of two parts, 1) the load recovery amount of the comprehensive energy system, and 2) the minimum operation cost of the comprehensive energy system, and the reward function is defined as

[0118] as follows:

[0119]

[0120] C E (t)=C C / E (t)+C M / E (t)+C S / E (t)+C B / E (t)

[0121] C H (t)=C C / H (t)+C M / H (t)+CS / H (t)+C B / H (t)

[0122] C G (t)=C C / G (t)+C M / G (t)+C S / G (t)+C B / G (t)

[0123] (4) Establish a comprehensive energy system to trigger the transfer of different uncertain events, and clearly define the various uncertain events that occur in the comprehensive energy system, so as to realize the more efficient and accurate load transfer strategy of the intelligent agent for different uncertain events;

[0124] Finally, the DDQN algorithm is trained by the stochastic gradient descent method. The agent updates the policy network and value network weights according to the environment, and the agent and the environment interact continuously, and the strategy is continuously trained to develop the optimal strategy. During the training process, the cumulative reward curve is used to reflect the training process, and the agent training and convergence are intuitively reflected. After the model is trained, the test environment is used to evaluate the model. The evaluation process includes multiple test rounds to measure the stability of the model. It is observed whether the agent can make quick and accurate decisions in an uncertain environment that has not occurred before. If the model performs poorly during the evaluation process, the model can be fine-tuned by adjusting the hyperparameters. For example, the learning rate, the number of layers of the neural network, the number of neurons in the hidden layer, the exploration rate, etc. can be adjusted to further improve the performance of the model.

Claims

1. A load transfer method for an integrated electricity-heat-gas energy system based on reinforcement learning, characterized in that, The method comprises the following steps: S1, establishing an electric-thermal-gas comprehensive energy system model containing different load types from an actual comprehensive energy system, and collecting system load transfer related data after different uncertain events occur, the collected data being used as training samples; S2, constructing an agent-environment model, simulating the operation environment of the comprehensive energy system, and in the environment, based on the operation constraints of the comprehensive energy system, the actions of the agent are subjected to constraints, so as to ensure the physical law and actual conditions of the operation load of the system; S3, constructing a reinforcement learning model, defining the state function, action function and state transition variable in the algorithm agent, and formulating the reward function by designing a reward and punishment mechanism, so as to optimize the agent strategy; S4, establishing a transfer trigger condition mechanism of the comprehensive energy system under different uncertain events, and defining various uncertain events of the comprehensive energy system, so as to realize more efficient and accurate load transfer strategy of the agent for different uncertain events; S5, based on the measurable comprehensive energy system, constructing an observation state S including node voltage, gas pressure, temperature, electric / thermal / gas load, photovoltaic and wind power output and electricity price, defining the action space A as the binary on-off instruction of the electric power connection / sectional switch and the natural gas and heat connection valve, and setting a composite reward R considering safety and economy; establishing a policy network and a target network, training by using experience replay and target network soft update, and optimizing parameters by using stochastic gradient descent, and obtaining the optimal load transfer strategy for different uncertain events; S6, using the trained reinforcement learning model to formulate a real-time and accurate load transfer strategy of the comprehensive energy system.

2. The method of claim 1, wherein: The comprehensive energy system modeling part comprises: a power system line model, a heat network and a natural gas network pipe network model, and a compressor model; and electric-to-gas, electric-to-heat and heat-to-power coupling device models between the systems; The natural gas pipeline model is as follows: where: f rs,t is the steady state flow of the natural gas pipeline r, s; K rs is the pipeline constant, which depends on the internal diameter, length, friction coefficient and temperature of the pipeline; p r,t , p s,t are the pressures at nodes r, s, respectively; s rs characterizes the direction of the natural gas flow, taking +1 when p r,t > p s,t , and -1 otherwise. The heat pipe model is as follows: where Q loss (t) is the total heat loss of the pipe at time t, Q loss,q (t) is the heat loss of pipe segment q, and N is the total number of pipe segments in the heat network. In the formula, ΔP is the pressure loss of the pipeline section; f is the friction factor, which is usually related to the flow rate, pipe roughness and fluid properties; L is the length of the pipeline section; D is the diameter of the pipeline; ρ is the density of the fluid; and v is the flow rate of the fluid; The compressor model is as follows: R com,m,n = p n,t / p m,t (5) where: W com,mn,t is the power consumed by the compressor; f com,n,t is the flow rate of gas through the compressor; B com,mn,t , z cm , m, t are compressor parameters dependent on compressor temperature, efficiency and compression factor; R com,m,n is the compression ratio; τ com,mn is the flow rate of gas consumed by the gas turbine; a com , β com , γ com is the capacity conversion efficiency constant; The electric-to-gas model is as follows: f P2G,i,t = η P2G P P2G,i,t / G HV (7) η in the formula P2G G is the conversion efficiency of the power-to-gas conversion; G HV G is the conversion efficiency of the power-to-gas conversion; G 3 G is the conversion efficiency of the power-to-gas conversion; G The electric boiler model is as follows: H EB (t) = μ EB (t) · η EB · P EB (t) (8) μ EB (t)∈{0,1} is the binary variable of the on-off state of the electric boiler at time t. η EB η is the electric-thermal conversion efficiency; The gas boiler is as follows: H GB (t) = η GB ·G GB (t) · LHV (9) η GB is the coefficient of gas heat conversion efficiency of the gas boiler; Different types of load modeling: P σ (t) = P σ,0 • f(t) (10) P = P0+ f(t) in the equation σ,0 is the reference power of the device; f(t) is a function describing the load variation, which is a sinusoidal function based on time to simulate periodic fluctuations; The collected related data comprises: the topology structure of the comprehensive energy system, the initial node voltage, pressure and node temperature after the occurrence of uncertain events such as load surge, line fault and line maintenance, photovoltaic and wind power generation, user power, heat and gas load, real-time electricity price, hourly information and energy storage device information.

3. The method of claim 1, wherein: In step S2, the following sub-steps are included: S2-1 optimization target; and S2-2 operation constraint; wherein S2-1 to S2-3 are all from step S2; S2-1 first defines the target as realizing the economic operation of the electric-thermal-gas comprehensive energy system when different uncertainties occur in the system, on the basis of ensuring the stable operation of the system load, and the formulated load transfer strategy focuses on the load recovery amount and the minimum operation cost of the comprehensive energy system, and the optimization target is as follows: Based on S2-1, S2-2 constructs the operation constraints of the integrated energy system, which couples the power system, the heat system and the natural gas system, and constructs the energy flow balance constraints of each system pipe network and the constraints of each energy coupling device: Power system flow constraints: In the formula: P G,i,t , Q G,i,t is the active and reactive power of the generator set of node i at time t; P L,j,t , Q L,i,t is the active and reactive load of node i at time t; P STO,i,t is the active power of the new energy source; P P2G,i,t is the active power of the electric-gas conversion unit; P SOC,i,t is the charging power of the electric energy storage; U i,t is the voltage module value of node i; θ ij,t is the phase angle difference between two nodes; G ij , B ij is the conductance and susceptance between nodes i and j; Natural gas system: f N,r,t , f L,r,t , f P2G,r,t , f GT,r,t are the natural gas source injection amount, the natural gas load, the P2G device injection amount and the gas turbine gas consumption amount of node r at time t, respectively; f s,in,r,t , f s,out,r,t are the storage tank natural gas injection and output flow; s com,s,r,t takes +1 when the node r is the compressor inlet, and otherwise takes -1. Heat system: H buy (t)+H GB (t)+H EB (t)+H dis (t)=H cha (t)+H load (t) (14) wherein: H buy (t) is the heat purchased from outside; H GB (t), H EB (t) is the output heat power of the gas boiler and the electric boiler, respectively; H cha (t), H dis (t) is the heat storage power and the heat release power of the heat storage device, respectively; H load (t) is the heat load demand, and the unit is kW; Electricity-to-gas equipment: wherein: f N,i,t represents the time instant t actual gas production of the first respectively represent the time instant t minimum and maximum gas production flow allowed by the first P2G plant when operating in steady state Energy storage equipment: wherein respectively the maximum charging and discharging rate of the δth energy storage device; respectively the minimum and maximum state of charge of the δth energy storage device; C δ is the rated storage capacity of the energy storage device; E δ (0), E δ (T) are the energy storage amount at the beginning and the end of the dispatching period, respectively.

4. The method of claim 1, wherein: The optimal strategy is solved by using the agent-environment training architecture and DDQN, which includes: building a neural network, using the agent-environment training architecture, establishing the input-output model of the system by DDQN algorithm, establishing the agent policy network and value network, the input parameters are environmental variables, and the output parameters are the control strategies of the integrated energy system network tie-in switch and tie-in valve; a three-layer neural network is used, including 128 nodes in the input layer, 256 nodes in the hidden layer, and 128 nodes in the output layer; the input variables of the input layer are the node voltage, node gas pressure, node temperature, load demand, photovoltaic and wind power generation, and electricity price information at the current time after the integrated energy system fails, the ReLU function is selected as the activation layer, and the output layer is the action of each tie-in switch, sectionalizing switch and tie-in valve in the integrated energy system at the next time; the loss function of the policy network is optimized based on the stochastic gradient descent method, and the Adam optimizer is used to find the optimal solution of gradient descent; The DDQN model models the control strategy problem as a Markov decision process: (1) The observation state S is set, which describes the state information characteristics of the agent at time t, which includes the node voltage, gas pressure, temperature, load demand, photovoltaic and wind power generation, and electricity price information at the current time of the integrated energy system; (2) The action space A is set, the action space of the agent is the switch action of the tie-in switch, sectionalizing switch in the power system and the tie-in valve in the natural gas and heat system, each switch action is (0, 1), 0 represents the switch is open, and 1 represents the switch is closed; (3) The reward function R is set, which describes the safety and economy of the integrated energy system at the same time, and a comprehensive reward mechanism is constructed, which is composed of two core components: 1) load recovery benefit term, 2) operation cost term.

5. The method of claim 1, wherein: In each time step, the agent simulates the result of the integrated energy system without operation, and uses the trigger condition W defined by formula (15) to judge whether the agent should act: W = W1 + W2 + W3 (17) In the formula, W is the load surge and sudden drop flag, W2 is the fault flag, and W3 is the maintenance flag, which correspond to different event-driven load transfer; Formula LF L TL is the flow of line L L L is the line number in the system, Z is the total number of lines in the system, and if the load level of a line in the power grid is higher than a predetermined threshold λ, the flow out-of-limit flag is set to 1; The calculation formulas of W2 and W3 are as follows: By adding the transfer trigger condition mechanism, the agent will generate the corresponding load transfer strategy when any one of the different types of uncertain events occurs in the system, and will perform the corresponding operation on the integrated energy system, which improves the accuracy of the load transfer strategy.

6. The method of claim 1, wherein: The DDQN training model updates the policy network and value network weights according to the environment. The agent and the environment interact continuously, and the agent continuously trains the policy to develop the optimal strategy. During the training process, the cumulative reward curve is used to reflect the training progress, and the agent training situation and convergence are intuitively reflected: Select the training frequency as Ψ, the sample time series length in data collection as T, the state space as S, and the action space as A: Step 1: Initialize the weights of the policy network and the value network respectively; Step 2: The agent obtains the sample generated by each step operation; Step 3: The agent executes the current operation at and obtains the reward of the current operation; Step 4: The agent executes the transfer trigger condition; Step 5: The agent executes the switch action; Step 6: Randomly extract data from the experience replay area; Step 7: Input parameters in the policy network to generate; Step 8: Calculate the network update using the stochastic gradient descent method; Step 9: Input parameters in the value network to generate the target; Step 10: Update the value network.

Images